6.12.13 Extremely slow smb writes

Quote

Hopefully other Windows users can replicate the issue.

 

I cannot, and haven't seen any other reports, so likely something specific to your config/hardware.

 

Start by please posting the diagnostics during a large file transfer, also post the Windows explorer graph.

Just tested my own system, from Win10 to Unraid 6.12.13 and was able to hit multi Gbps speeds to disk and user shares.

I ran into something similar. I have two Windows Server 2022 VMs running on my Unraid box - one for Veeam and the other for BlueIris.  Both of them reach back out via SMB to the array for storage. 

 

Immediately after upgrading to 6.12.13 today all my Veeam backups failed due to timing out writing to the SMB share.  BlueIris also was having a fit and unable to write camera clips to the array complaining that the disk writes could not keep up.  These are both completely separate shares: Veeam writes to the array, BI writes to the SSD cache pool so it was not resource bound.  After rolling back to 6.12.11 everything is back to OK.

 

Unfortunately I wasn't able to troubleshoot before rolling back due to needing BlueIris recording.  If I get more time I upgrade again and try to get diagnostics.

15 hours ago, JorgeB said:

 

I cannot, and haven't seen any other reports, so likely something specific to your config/hardware.

 

Start by please posting the diagnostics during a large file transfer, also post the Windows explorer graph.

When i tested even a 1mb .docx file would stall explorer and would prob take hours to complete. It took about 5 minutes just to get the transfer to cancel. I'll try find some time to upgrade again, what diagnostics file do you need?

5 hours ago, Spokz said:

what diagnostics file do you need?

The complete zip please.

Got exactly the same issue here

 

Good performance before upgrade to 6.12.13 100mb+

 

Now in the low kilobytes range, so have reverted

 

Took the below diag before I rolled back

mordor-diagnostics-20240824_1106.zip

Edited August 24 by mraerosmith
add file

@Spokz please post your diags also, can be from 6.12.11, just to see if you have some hardware in common with the user above.

Upgraded to 6.12.13 again this morning and a few minutes after starting the array, my BlueIris VM (SVR2022) started having problems writing. I RDP into the VM and tried copying a file to the share directly - which also froze up the file explorer copy and eventually the VM. I tried copying externally from my MacBook to/from the same share and had no issues even while the Windows VM was exhibiting the issue.  I've grabbed and attached diagnostics here before rolling back to 6.12.11 (which again resolves the issue). 

 

I did enable SMB logging to syslog and tried to copy a file with the VM and re-created the issue with the logging enabled.  I'd rather not post that diag file here publicly, so if that file would be useful I can PM that to the devs. Or if any other testing I can do would be helpful please let me know.

unraidnas-diagnostics-20240824-0815.zip

@mraerosmith, @WackyWRZ I see that both are using a bond with a 10GbE and a gigabit NIC, please retest without the bond.

I also have the exact same issue with a bond. I will retest this afternoon.... but this sounds like a regression.

Seeing similar.  I have a Windows 10 VM hosted by Unraid, that's typically where I run things like MakeMKV and it is terribly slow as described above, would take hours if not days for a single movie.  If I access the same share from my completely separate physical host, it seems to be running normal or at least acceptable.  Something changed with the latest Unraid version when accessing the user shares from a VM hosted on Unraid.

I'm having the exact same problem. From my Windows 10 VM hosted on the Unraid server I have beyond slow write speed. Every other PC has normal write speed to the Unraid server. Read speed is normal. I was previously on 6.12.10.

kurnous-diagnostics-20240825-1949.zip

Please try creating a new VM just for testing, using all default settings, and see if you have the same issue.

So far doesn't appear to be related to the network interface I select. br0 is my 10gb nic that was bonded. I disabled bonding and still had slow speeds using br0, also tried br1 which is 1gb nic with same results.

I'll try to spin up a new VM as well with all the defaults.

Virtio network model is totally broken. A new (and existing) VM only works properly with virtio-net now. The performance with virtio-net isnt as good as my old setup. Looks like ill be rolling back to 6.12.11.

that would follow as im using virtio on my vm as well

Another finding: Upgrading to the latest virtio-win drivers does not fix the issue. With virtio-net I can can only get like 60 MiB/s transfer rate whereas previously I could get over 200 MiB/s with virtio.

EDIT: Reverted to 6.12.11 and virtio network model works again with no other changes. 450 MiB/s transfers with latest drivers.

Edited August 25 by johnsanc

53 minutes ago, johnsanc said:

EDIT: Reverted to 6.12.11 and virtio network model works again with no other changes. 450 MiB/s transfers with latest drivers.

 

Thanks for testing, I should now be able to reproduce, the initial test I did was with the default settings

Likely this is due to recent bug in Linux kernel. Check if you see "Bad GSO" errors in the Unraid log.
I didn't want to revert Unraid version, so switching to e1000e adapter for Unraid VM helped tremendously, I am getting up to 900 Mbps (110 MBps) SMB write speeds on my 1 Gig network connection.
https://bugzilla.kernel.org/show_bug.cgi?id=219129

Edited August 25 by mtikhov

11 hours ago, mtikhov said:

Check if you see "Bad GSO" errors in the Unraid log.

Thanks, I don't see that in the log, but it does look like the same problem.

Here the same with bond. Slow as hell, went back to 6.12.11 and it works

Confirmed I am also using "virtio" instead of "virtio-net" on all my VMs as well. 

 

Although this looks like a Kernel issue, I did testing w/o the bond and completely removing the 10G NIC - doesn't solve the issue.

I wanted to post that I ran into this (or similar) issue as well when upgrading from 6.12.11 - > 6.12.13. Win10 BlueIris vm, virtio, two nics (one 10g one gigabit, not bonded). The VM became extremely slow/unstable. Like others above, I can't spend much time testing as I need the server to function so I rolled back to 6.12.11 via the web-ui and things immediately went back to normal. I know this is not adding much to the discussion, but wanted to +1 the issue for awareness. Cheers!

Edited August 28 by adamrgolf

This is a kernel issue, but there's no fix yet for kernel 6.1, only for newer LTS kernels.

It would be worth adding this to the release notes , so folks are aware of this before they upgrade