[6.10.3] Slow Cache Drive Performance

[berta123]

I have 2x MX500 SATA SSDs in a mirror for my cache drive. From a Windows 11 VM running on my unraid server I am only able to get ~160MB/s write speeds and 260 MB/s read. The DiskSpeed docker reports each drive is capable of 500MB/s+. I expect some loss to SMB, but not this severe. 

 

While testing I did notice 2 odd things:

1. A share set to cache only is writing directly to the array even though there is >50% space left on the cache drive 

2. When using the cache disk share directly it does seem to write to the cache BUT it does so sporadically when watching the disk speeds on the "Main" page. So what I'll observe is writing begins in my test (using the AJA system test) and I see no activity change for several seconds on the drives. Then maybe 10 seconds in I'll start seeing the drives show 300-500MB/s writes. Then it will die down, and then pop back up. It's almost like there is another cache before the cache array that is very slow and so the drives just read out of it when there is enough data. But the transfer speed remains constant despite the fluctuations in actual disk write. Adding to the bizarre readings when doing the read test in AJA the disks show no read occurring with just very minimal background noise (a few KB/s). 

 

Anyone have any ideas? @Support?

 

System Specs:

AMD Ryzen Threadripper 3960X 24-Core

192GB Ram

10GBe 

System is basically idle during these tests (~10% load) 

 

Edited August 13, 2022 by berta123

[itimpi]

3 minutes ago, berta123 said:

A share set to cache only is writing directly to the array even though there is >50% space left on the cache drive

The only way I can think of this happening is if you are overwriting a file that already exists there as in this case it is done in situ.    It should NEVER happen for new file if the Use Cache=Only setting is set.

[berta123]

1 minute ago, itimpi said:

The only way I can think of this happening is if you are overwriting a file that already exists there as in this case it is done in situ.    It should NEVER happen for new file if the Use Cache=Only setting is set.

I can create a test word document in the share and it shows only on the array device. The cache only share doesn't even have a folder on the cache drive!

[itimpi]

Just now, berta123 said:

I can create a test word document in the share and it shows only on the array device. The cache only share doesn't even have a folder on the cache drive!

That does not make sense - something has to be set wrong for that to happen.   I would suggest you post your system’s diagnostics zip file and mention which share is showing this behaviour. 

[berta123]

28 minutes ago, itimpi said:

That does not make sense - something has to be set wrong for that to happen.   I would suggest you post your system’s diagnostics zip file and mention which share is showing this behaviour. 

Generating this now... How long should it take? It's been going for almost 30 mins now running commands like:
sed -i 's/\/mnt\/disk9\/os\/docker\/gitlab\/data\/postgresql\/data\/base\/16386\/29604/\/\/..4\/.../g' '/beta-diagnostics-20220813-1504/logs/syslog.1.txt' 2>/dev/null

 

What's weird is that file doesn't even exist on disk 9. In fact that docker isn't even in use anymore.

[itimpi]

It should only take a minute or two - definitely something going wrong.

[berta123]

Finally finished. Share I was using as cache only was called "fast"

 

 

beta-diagnostics-20220813-1520.zip

[JorgeB]

Are you using a rootshare? IIRC someone else had shares issues recently because of that.

[berta123]

No rootshare but I did somewhat recently replace my cache (set all shares to cache yes, run mover, disable VMs docker etc, deleted cache, swapped a drive, then re-setup cache, reset shares to proper settings and ran mover again)

[berta123]

Bumping this up. Anyone have ideas?

[JorgeB]

Create a new test share, set it to cache yes, leave all remaining settings as default, then create a file:

 

touch /mnt/user/test_share/file

 

Then post new diags.

[berta123]

On 8/16/2022 at 1:12 AM, JorgeB said:

Create a new test share, set it to cache yes, leave all remaining settings as default, then create a file:

 

touch /mnt/user/test_share/file

 

Then post new diags.

Ok test file was created on cache drive. But I am still seeing only HDD speeds on the share. I see identical speeds writing/reading to the share set to cache yes and cache no from a VM. Test share was called fast2

beta-diagnostics-20220819-1943.zip

Edited August 20, 2022 by berta123

[JorgeB]

To see if it's FUSE related enable disk shares and transfer directly to the cache share, see if there's any difference.

[berta123]

On 8/20/2022 at 3:13 AM, JorgeB said:

To see if it's FUSE related enable disk shares and transfer directly to the cache share, see if there's any difference.

Same results with SMB cache share. 

[JorgeB]

11 hours ago, berta123 said:

Same results with SMB cache share. 

That suggests a device issue, if you're willing balance the pool to raid0 and test again.

[berta123]

6 hours ago, JorgeB said:

That suggests a device issue, if you're willing balance the pool to raid0 and test again.

Converted to RAID0 and I have the same results. 

 

I also ran this test to verify pool was performing correctly locally:

root@beta:/mnt/cache# dd if=/dev/zero of=test.img bs=1M count=51200
51200+0 records in
51200+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 28.0109 s, 1.9 GB/s

 

[JorgeB]

On 8/13/2022 at 8:30 PM, berta123 said:

From a Windows 11 VM running on my unraid server

I missed this part, did you test from a client PC? Assuming you have one with 10GbE and a fast device to copy from.

[berta123]

Just now, JorgeB said:

I missed this part, did you test from a client PC? Assuming you have one with 10GbE and a fast device to copy from.

I actually had the same idea. dd gives 500+MB/s from Proxmox host machine with 10GbE nic. So it seems to be limited to the VM. I am trying to mount to another VM to see if it is just 1 VM in particular or ALL vms. 

[berta123]

1 minute ago, berta123 said:

I actually had the same idea. dd gives 500+MB/s from Proxmox host machine with 10GbE nic. So it seems to be limited to the VM. I am trying to mount to another VM to see if it is just 1 VM in particular or ALL vms. 

UPDATE:
Tested from Ubuntu VM mounted SMB and I see ~304 MB/s with dd. So less than another machine with a 10 GbE nic but more than Windows 11 VM. 

[berta123]

Bumping this back up

[JorgeB]

Did you test with a non VM client like mentioned?

[berta123]

On 8/22/2022 at 10:47 AM, berta123 said:

I actually had the same idea. dd gives 500+MB/s from Proxmox host machine with 10GbE nic. So it seems to be limited to the VM. I am trying to mount to another VM to see if it is just 1 VM in particular or ALL vms. 

Yep! That’s what I meant by proxmox host. The proxmox host is a separate machine on the network and sees the expected speeds. 

[JorgeB]

It's normal for VM virtual network performance to be slower than using a dedicated NIC, also make sure you are using virtio for the network model, it usually performs better than virtio-net.

[berta123]

On 8/25/2022 at 8:39 AM, JorgeB said:

It's normal for VM virtual network performance to be slower than using a dedicated NIC, also make sure you are using virtio for the network model, it usually performs better than virtio-net.

I tried virtio and got similar results. I’m surprised it’s so slow. I saw this post from a long time ago that seems to suggest it is possible to get 15gbps VM to host but only on 6.8.3? Any ideas what could have changed in newer versions? I’m running a threadripper cpu so I doubt cpu performance is the bottleneck. 

 

 

[berta123]

For others who find this, switching to virtio from virtio-net AND uninstalling the driver from windows and re-installing it seemed to get the speeds up. I now get 4-6 Gb/s which is still much less than the 32 Gb/s I get from a Linux VM, but I suppose that is just the Windows tax?