berta123 Posted August 13, 2022 Share Posted August 13, 2022 (edited) I have 2x MX500 SATA SSDs in a mirror for my cache drive. From a Windows 11 VM running on my unraid server I am only able to get ~160MB/s write speeds and 260 MB/s read. The DiskSpeed docker reports each drive is capable of 500MB/s+. I expect some loss to SMB, but not this severe. While testing I did notice 2 odd things: 1. A share set to cache only is writing directly to the array even though there is >50% space left on the cache drive 2. When using the cache disk share directly it does seem to write to the cache BUT it does so sporadically when watching the disk speeds on the "Main" page. So what I'll observe is writing begins in my test (using the AJA system test) and I see no activity change for several seconds on the drives. Then maybe 10 seconds in I'll start seeing the drives show 300-500MB/s writes. Then it will die down, and then pop back up. It's almost like there is another cache before the cache array that is very slow and so the drives just read out of it when there is enough data. But the transfer speed remains constant despite the fluctuations in actual disk write. Adding to the bizarre readings when doing the read test in AJA the disks show no read occurring with just very minimal background noise (a few KB/s). Anyone have any ideas? @Support? System Specs: AMD Ryzen Threadripper 3960X 24-Core 192GB Ram 10GBe System is basically idle during these tests (~10% load) Edited August 13, 2022 by berta123 Quote Link to comment
itimpi Posted August 13, 2022 Share Posted August 13, 2022 3 minutes ago, berta123 said: A share set to cache only is writing directly to the array even though there is >50% space left on the cache drive The only way I can think of this happening is if you are overwriting a file that already exists there as in this case it is done in situ. It should NEVER happen for new file if the Use Cache=Only setting is set. Quote Link to comment
berta123 Posted August 13, 2022 Author Share Posted August 13, 2022 1 minute ago, itimpi said: The only way I can think of this happening is if you are overwriting a file that already exists there as in this case it is done in situ. It should NEVER happen for new file if the Use Cache=Only setting is set. I can create a test word document in the share and it shows only on the array device. The cache only share doesn't even have a folder on the cache drive! Quote Link to comment
itimpi Posted August 13, 2022 Share Posted August 13, 2022 Just now, berta123 said: I can create a test word document in the share and it shows only on the array device. The cache only share doesn't even have a folder on the cache drive! That does not make sense - something has to be set wrong for that to happen. I would suggest you post your system’s diagnostics zip file and mention which share is showing this behaviour. Quote Link to comment
berta123 Posted August 13, 2022 Author Share Posted August 13, 2022 28 minutes ago, itimpi said: That does not make sense - something has to be set wrong for that to happen. I would suggest you post your system’s diagnostics zip file and mention which share is showing this behaviour. Generating this now... How long should it take? It's been going for almost 30 mins now running commands like: sed -i 's/\/mnt\/disk9\/os\/docker\/gitlab\/data\/postgresql\/data\/base\/16386\/29604/\/\/..4\/.../g' '/beta-diagnostics-20220813-1504/logs/syslog.1.txt' 2>/dev/null What's weird is that file doesn't even exist on disk 9. In fact that docker isn't even in use anymore. Quote Link to comment
itimpi Posted August 13, 2022 Share Posted August 13, 2022 It should only take a minute or two - definitely something going wrong. Quote Link to comment
berta123 Posted August 13, 2022 Author Share Posted August 13, 2022 Finally finished. Share I was using as cache only was called "fast" beta-diagnostics-20220813-1520.zip Quote Link to comment
JorgeB Posted August 14, 2022 Share Posted August 14, 2022 Are you using a rootshare? IIRC someone else had shares issues recently because of that. Quote Link to comment
berta123 Posted August 15, 2022 Author Share Posted August 15, 2022 No rootshare but I did somewhat recently replace my cache (set all shares to cache yes, run mover, disable VMs docker etc, deleted cache, swapped a drive, then re-setup cache, reset shares to proper settings and ran mover again) Quote Link to comment
berta123 Posted August 16, 2022 Author Share Posted August 16, 2022 Bumping this up. Anyone have ideas? Quote Link to comment
JorgeB Posted August 16, 2022 Share Posted August 16, 2022 Create a new test share, set it to cache yes, leave all remaining settings as default, then create a file: touch /mnt/user/test_share/file Then post new diags. Quote Link to comment
berta123 Posted August 20, 2022 Author Share Posted August 20, 2022 (edited) On 8/16/2022 at 1:12 AM, JorgeB said: Create a new test share, set it to cache yes, leave all remaining settings as default, then create a file: touch /mnt/user/test_share/file Then post new diags. Ok test file was created on cache drive. But I am still seeing only HDD speeds on the share. I see identical speeds writing/reading to the share set to cache yes and cache no from a VM. Test share was called fast2 beta-diagnostics-20220819-1943.zip Edited August 20, 2022 by berta123 Quote Link to comment
JorgeB Posted August 20, 2022 Share Posted August 20, 2022 To see if it's FUSE related enable disk shares and transfer directly to the cache share, see if there's any difference. Quote Link to comment
berta123 Posted August 21, 2022 Author Share Posted August 21, 2022 On 8/20/2022 at 3:13 AM, JorgeB said: To see if it's FUSE related enable disk shares and transfer directly to the cache share, see if there's any difference. Same results with SMB cache share. Quote Link to comment
JorgeB Posted August 22, 2022 Share Posted August 22, 2022 11 hours ago, berta123 said: Same results with SMB cache share. That suggests a device issue, if you're willing balance the pool to raid0 and test again. Quote Link to comment
berta123 Posted August 22, 2022 Author Share Posted August 22, 2022 6 hours ago, JorgeB said: That suggests a device issue, if you're willing balance the pool to raid0 and test again. Converted to RAID0 and I have the same results. I also ran this test to verify pool was performing correctly locally: root@beta:/mnt/cache# dd if=/dev/zero of=test.img bs=1M count=51200 51200+0 records in 51200+0 records out 53687091200 bytes (54 GB, 50 GiB) copied, 28.0109 s, 1.9 GB/s Quote Link to comment
JorgeB Posted August 22, 2022 Share Posted August 22, 2022 On 8/13/2022 at 8:30 PM, berta123 said: From a Windows 11 VM running on my unraid server I missed this part, did you test from a client PC? Assuming you have one with 10GbE and a fast device to copy from. Quote Link to comment
berta123 Posted August 22, 2022 Author Share Posted August 22, 2022 Just now, JorgeB said: I missed this part, did you test from a client PC? Assuming you have one with 10GbE and a fast device to copy from. I actually had the same idea. dd gives 500+MB/s from Proxmox host machine with 10GbE nic. So it seems to be limited to the VM. I am trying to mount to another VM to see if it is just 1 VM in particular or ALL vms. Quote Link to comment
berta123 Posted August 22, 2022 Author Share Posted August 22, 2022 1 minute ago, berta123 said: I actually had the same idea. dd gives 500+MB/s from Proxmox host machine with 10GbE nic. So it seems to be limited to the VM. I am trying to mount to another VM to see if it is just 1 VM in particular or ALL vms. UPDATE: Tested from Ubuntu VM mounted SMB and I see ~304 MB/s with dd. So less than another machine with a 10 GbE nic but more than Windows 11 VM. Quote Link to comment
berta123 Posted August 24, 2022 Author Share Posted August 24, 2022 Bumping this back up Quote Link to comment
JorgeB Posted August 25, 2022 Share Posted August 25, 2022 Did you test with a non VM client like mentioned? Quote Link to comment
berta123 Posted August 25, 2022 Author Share Posted August 25, 2022 On 8/22/2022 at 10:47 AM, berta123 said: I actually had the same idea. dd gives 500+MB/s from Proxmox host machine with 10GbE nic. So it seems to be limited to the VM. I am trying to mount to another VM to see if it is just 1 VM in particular or ALL vms. Yep! That’s what I meant by proxmox host. The proxmox host is a separate machine on the network and sees the expected speeds. Quote Link to comment
JorgeB Posted August 25, 2022 Share Posted August 25, 2022 It's normal for VM virtual network performance to be slower than using a dedicated NIC, also make sure you are using virtio for the network model, it usually performs better than virtio-net. Quote Link to comment
berta123 Posted August 29, 2022 Author Share Posted August 29, 2022 On 8/25/2022 at 8:39 AM, JorgeB said: It's normal for VM virtual network performance to be slower than using a dedicated NIC, also make sure you are using virtio for the network model, it usually performs better than virtio-net. I tried virtio and got similar results. I’m surprised it’s so slow. I saw this post from a long time ago that seems to suggest it is possible to get 15gbps VM to host but only on 6.8.3? Any ideas what could have changed in newer versions? I’m running a threadripper cpu so I doubt cpu performance is the bottleneck. Quote Link to comment
berta123 Posted September 2, 2022 Author Share Posted September 2, 2022 For others who find this, switching to virtio from virtio-net AND uninstalling the driver from windows and re-installing it seemed to get the speeds up. I now get 4-6 Gb/s which is still much less than the 32 Gb/s I get from a Linux VM, but I suppose that is just the Windows tax? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.