Background: I have 5x 2TB NVMe SSDs in a RaidZ1 pool (non encrypted, some datasets have compression and some do not), used as my cache.  I'm sitting between 80% and 90% utilization at any given time (8TB usable, ~1TB free). 

 

Moving large files within the pool is very slow compared to when I was running the same drives in a BTRFS Raid1 pool (5TB usable).  Moving a 5GB file from folder A to folder B within the cache gives me consistently less than 500MB/s transfer speed (usually much less), while I was seeing around 1.5GB/s on BTRFS.  

 

 

Not sure what I can check here.  Could this be due to the higher pool utilization?  I have a 10th gen i7 and 48GB of RAM, and the system isn't taxed.  The PCIe bus is 3.0 and all NVMe's have DRAM.  

 

Thanks for your suggestions!

Edited February 2, 20242 yr by Andiroo2
Reasons

20 minutes ago, Andiroo2 said:

Not sure what I can check here.  Could this be due to the higher pool utilization?

What is the current pool utilization? 

 

I've moved from a btrfs raid5 pool to zfs raidz1 pool and see similar speeds, even a little faster with zfs, I used to get around 1GB/s, now get 1.5GB/s when copying data inside the pool.

Sitting at 88% right this minute:

 

Mine is not much below that:

 

 

 

Please note that I have a large ARC and I mostly copy data that is on the ARC, for data not on the ARC I usually see around 1GB/s

 

 

OK I increased my ARC to 1/4 of RAM instead of the default 1/8.  Will see what happens.

Did changing ARC size help? I am having speed problems with an nvme cache pool. 4 512gb nvme  raidz1 on asus hyper pcie3 x16 slot. Copying from nvme on lan system with 10g connection fastest transfer is 500-550mb/s. Copying to my sas3 ssd cache of 3 500gb raidz1 will saturate a 10g conection. How can 12g sas be twice as fast as nvme. Could the asus card be the problem, is bifurcation slowing the nvmes down. What about switching to 2 mirrors or just setup 4 cache pools with individual nvmes? Any ideas welcome.

ARC size didn’t help. Saw the same speeds afterwards. 
 

That being said, my speeds jumped to around 800MB/s when I switched to exclusive shares. 

I manually switched my appdata to bypass FUSE a while ago, thats what exclusive shares is mainly for, right? I don't see how exclusive shares change transfer speeds onto unraid, it definitely seems to help dockers. My main problem is still the question of why my nvmes are so slow.

6 hours ago, roberth58 said:

I don't see how exclusive shares change transfer speeds onto unraid

It can make a large difference, but if you are already bypassing FUSE it will be the same.

I have been testing/judging NVMe speed based on network transfers in my Windows VM to Unraid shares. Is there a better way to just test this within Unraid core?  The disk speed docker doesn’t work for pooled cache drives…what else could I try here?  Can I run rsync with some stats?

Rsync is not made for speed, pv can be a good test for actual file speed copy (not actual max device speed, but what you will get in real use):

 

pv /path/to/a/large/file > /path/to/dest

 

You will want to stop and re-start the array between tests to avoid RAM cache.