I'm probably doing something wrong...but I have 3 new, 2TB SSDs in a RaidZ1 configured as a ZFS Pool. Even with only a single VM running, the perf is incredibly slow from within the VM guest. From the Unraid host, the perf work as expected. I've attached a perf report from within the guest VM as well as a diagnostics file. Do you see anything misconfigured? What could be causing the perf to be SOOOoooo bad?

 

Thanks

 

Guest Name: Raichu

ZFS Pool: ZVM at path /mnt/zcache/ZVM

Vdisk location: /mnt/zcache/ZVM/Raichu

fio_perf_raichu.log pumbaa-diagnostics-20241029-1436.zip

Also, on the dashboard page I see the following for system stats. I'm not sure why ZFS shows 100%...could this be the root cause? If so how can I fix it?

 

UPDATE -- I switched the PRIMARYCACHE from all to metadata for all zfs pools and restarted the array, now I'm down to only 1% utilization of the arc size (8GB). After a disk io test it's already up to over 25% again. I'll keep an eye on it and see how it goes. The perf test after the reboot isn't much better but the guest VM seems responsive at least...for now.

 

UPDATE -- after being on for just an hour or two ZFS usage is all the way back to 100%

Edited October 29, 20241 yr by geeksheikh
update

12 hours ago, geeksheikh said:

ZFS usage is all the way back to 100%

This is normal and a good thing, it means it's using all allocated ARC RAM.

 

RAIDZ1 is not for VMs since the IOPS are the same as a single device, mirrors would be better, even better for performance would be using a zvol on a mirror, instead of a vdisk.

 

 

Thanks @JorgeB -- the Unraid GUI only has options for qcow and raw. Is there any guide for enabling an Unraid VM on a ZVOL? ChatGPT tells me to adjust the XML directly but I wasn't sure I could trust it.

 

What I understand is convert the vdisk to zvol with qemu command but not sure how to point the vm to it.

3 minutes ago, geeksheikh said:

ChatGPT tells me to adjust the XML directly but I wasn't sure I could trust it.

Currently that's the only way, like this:

 

 

Where "cacheb" is the pool name, "Win2022" is the zvol name

@JorgeB I've been really struggling with pool perf for a long time now and am looking for some help. (Tests ran on Unraid 6.12.13)

 

Forget about VM perf, I boiled it down to just working with raw files, both VDisk files and other large files. I even did some screen records to prove what I'm seeing but either I'm misunderstanding something about pools or the pools are extremely inefficient.

 

I have two SSDs same model, same size. The Single tests below are for both. The files were between 4GB and 100GB.

Single SSD mounted in Unraid via unassigned devices as btrfs and as zfs-encrypted works well

Same Single SSD mounted in Windows NTFS works well (300MB/s+) for the same files tested in Unraid.

300MB/s was the average sustained write speed for the individual disks for the entire duration of the write in Windows. In Unraid sustained averages were around 100MB/s (still ok)

 

I added the same two disks to a btrfs pool (mirrored) and a zfs-encrypted pool (zraid1) and repeated the tests.

Copy starts off between 100-300 MB/s, great.

After a few GBs is copied though, perf goes down to 0-3MB/s. iotop shows me disk utilization is maxed out. I will splice together the video and share on YT if you need it.

 

This means, it's not the disks, it's not the unraid setup, it's not hardware between mobo and ssds.

 

The only thing left I can think of is that mirrored pool operations is extremely inefficient. Am I missing something? What else would help you diagnose?

 

Thank you

Strange if it happens with both btrfs and zfs, which SSD devices are you sing for that test?

Crucial BX500 2TB 3D NAND SATA 2.5-Inch Internal SSD, up to 540MB/s - CT2000BX500SSD1, Solid State Hard Drive

Not the fastest devices, do you have a different you could use? Crucial MX500 for example.

I am testing with these now...but...even "not the fastest" doesn't seem right. Single drive can transfer over 300MB/s sustained for >100GB so long as not using pool. I'm about to upload the video for you. Meanwhile I am also testing the drives below and will let you know.

Samsung Electronics 870 EVO 2TB 2.5 Inch SATA III Internal SSD (MZ-77E2T0B/AM)

@JorgeB -- Here's the video that show the issues. I've added chapters so you can move around quickly.

 

0:00 NTFS Successful (Windows -- Fast Transfer)

2:02 BTRFS Single SSD Success (Unraid Single Drive -- Fast Transfer)

6:36 ZFS Mirror Setup

10:01 ZFS Mirror Slow (ZFS transfer crawling along)

13:18 BTRFS Mirror Setup 

16:42 BTFS Mirror Stuck Plus iostat (transfers just get stuck. Overlaid iostat for debug / review)

I'll take a look tomorrow with more time.

Thanks @JorgeB. FYI -- I did complete my tests with the Samsung EVOs and could not replicate the perf issue. I would still very much appreciate if you could take a look at that video and offer any thoughts you have on what is so different about those SSDs that cause them write at <2MB/s after large transfers get going. Perhaps there something UnRaid can do too to improve perf on SSDs like that or at the very least identify what to look out for when choosing SSDs for high-perf pools.

I took a look at the video, to me, it looked like there may be an issue with the 2nd SSD, did you try the single pool config wit both devices, to confirm they both perfume OK alone?

Wow, I had three disks, I did test them all...but somewhere I must have gotten confused and tested the same one twice and never done appropriate tests on the "bad" drive. I went ahead and ran through a perf test on all ssds again and attached is what I found. Sheesh...whatta waste of life.

 

In the attached look at Serial ending in 9DB4...arg. Will submit RMA for the drive. thank you and sorry for taking up your time on something I should have found.crucial_ssd_perf_reports.pdf