Slow SMB speeds

[David Dernovoj]

I have a RAID controller hooked into my array as storage, and have 4x 4 TB NVMe drives set up as cache.

 

I have a 10 GBe network card (SFP+) in both my workstation and my Unraid server. iperf showed about 6.5 Gbps, which is a bit lackluster but shouldn't affect me that much. The share I'm copying to is set to cache:yes (I have also tried a different share with cache:prefer) and I am getting sub par speeds of 100-150 MB/s. To my understanding, everything I'm copying should go to the cache drives, and then later be moved onto the storage array. So what exactly am I doing wrong? Even though the network throughput is subpar (6.5 of 10 Gbps), I should be getting somewhere around that in copying speeds, not the equivalent of a GBit connection.

 

If I can provide any further information to help resolve this, please let me know what exactly that would entail.

 

Edit#1: I just copied a file from the server and got 700-950 MB/s, so about what I'd expect / hope to obtain. That means something is wrong when I'm trying to write to the server.

Edit#2: I am copying from an NVMe disk locally to the server (which, as mentioned, uses NVMe drives for cache), so my local read speeds aren't the bottleneck.

Edited September 25, 2020 by David Dernovoj

[JorgeB]

See if copying to the disk share is faster, like this one:

 

[David Dernovoj]

I have limited ability to just randomly take my array down every time to change settings, but I'll give this a try tonight and report back, thanks!

[j44rs4]

You are describing exactly the same problem that I'm facing also.

 

Please tell me more about your server/system.

Like what kind of Raid controller are you using, RAM, CPU, Motherboard etc.

[David Dernovoj]

Sorry, I had to put this down for a while and want to revive this. I was able to actually do some maintenance now and try disk shares.

 

For a few specs:

 

On the server (which runs unraid) I am using:

8x 14 TB HDD, in a hardware raid with a  MegaRAID SAS 9361-8i controller (array)

4x 4 TB Western Digital Ultrastar DC SN630 (cache)

AMD Epyc 7702p

512 GB DDR4-2933 ECC

Supermicro H11SSL-i mobo (I believe)

Intel 82599ES dual 10 gbit SFP+ NIC

 

on the client, I am using:

i9 [email protected] GHz

128 GB DDR4-3200 (running at 2800)

Samsung 960 PRO 2 TB

Samsung 970 EVO 1 TB x2

Samsung 860 EVO 2 TB x2

RTX2080Ti

Intel X520 dual 10 gbit SFP+ NIC

 

I have both machines connected via an Ubiquiti UniFi 48-port switch, it has 2 SFP+ ports on it which I am using on each machine. On unraid, I have made the 10 GBit NIC my primary network device, and on my desktop it is, too.

 

I have tried a disk share, and my writes to the cache were very inconsistent (100-350 MB/s), when writing to the storage array, I got about 200-300 MB/s. Not a lot of difference between that and writing to a user share. I noticed that every time I do copy, my CPU usage is quite high, on all 18 cores / 36 threads. Up to 70% at times. Is SMB that taxing? I didn't even copy individual small files, I am trying to do big files (>4 GB individually) to try and get the best sequential speed out of it.

 

If I can provide any more information, should dump any information I can from unraid itself, please let me know.

[JorgeB]

6 hours ago, David Dernovoj said:

I noticed that every time I do copy, my CPU usage is quite high, on all 18 cores / 36 threads. Up to 70% at times. Is SMB that taxing?

CPU usage in the dashboard includes I/O wait, check with top, it will show actual CPU utilization.

[David Dernovoj]

4 hours ago, JorgeB said:

CPU usage in the dashboard includes I/O wait, check with top, it will show actual CPU utilization.

no top in Windows I'm afraid, unless I missed something ^^

[trurl]

4 minutes ago, David Dernovoj said:

I missed something

You can get to the command line in Unraid by clicking the >_ (Terminal) button at upper right

[David Dernovoj]

Just now, trurl said:

You can get to the command line in Unraid by clicking the >_ (Terminal) button at upper right

I might have been a bit unclear in general. I am having 70% cpu load on the client side. On a Windows machine.

[JorgeB]

That's not normal, but also not an Unraid problem.

[David Dernovoj]

3 hours ago, JorgeB said:

That's not normal, but also not an Unraid problem.

That's true. However, I am not sure how to proceed on this SMB problem I have. I can copy from the server relatively fast (I'd be cool if I even got up to 500-700 MB/s), but I am barely scratching 300. And only sometimes. It starts at like 90, and climbs up to the mid 200s and then keeps jumping between 100 and 300. Always in the optimal scenario with large files, not with smaller / many files.

[JorgeB]

If you're using any Antivirus/firewall make to try with those disable and it's been known to cause issues before.

[David Dernovoj]

7 hours ago, JorgeB said:

If you're using any Antivirus/firewall make to try with those disable and it's been known to cause issues before.

Sadly not the solution, already had it disabled. I even exited the glasswire process and it didn't change transfer speeds. I strongly believe a firewall would block the entire traffic, not just slow it down.

[JorgeB]

10 hours ago, David Dernovoj said:

not just slow it down.

This was more about AV software, then best bet is to try a different PC or reformatting/new OS on that one.

[David Dernovoj]

14 hours ago, JorgeB said:

This was more about AV software, then best bet is to try a different PC or reformatting/new OS on that one.

I guess I'm not going to be able to test this for a while then, because I'd need to move the 10G NIC accordingly then.

However, I am getting the appropriate download speeds (copying from the server), but the upload sucks. I also have a proven 4-6 GBit via iperf, so I feel like something's wrong with SMB. But with my inability to test this right now I guess we're just spinning in circles. 

[j44rs4]

Have you tried to connect directly to your UR, without go through the switch?

 

Normally you will have to configure your switch to handle jumboframe, or else the transfer will not work optimal.

 

If your switch is layer 3, I'd add another VLAN than the default VLAN 1 to allow higher jumboframes.

Becaus if you have at least one device on your network that does not support jumbo frames (network printer, SIP phone, etc.) then you cannot use jumbo frames at all. Otherwise you won't be able to talk to that device.

Edited October 28, 2020 by j44rs4

[David Dernovoj]

On 10/28/2020 at 3:20 PM, j44rs4 said:

Have you tried to connect directly to your UR, without go through the switch?

 

Normally you will have to configure your switch to handle jumboframe, or else the transfer will not work optimal.

 

If your switch is layer 3, I'd add another VLAN than the default VLAN 1 to allow higher jumboframes.

Becaus if you have at least one device on your network that does not support jumbo frames (network printer, SIP phone, etc.) then you cannot use jumbo frames at all. Otherwise you won't be able to talk to that device.

I finally managed to take down my array for maintenance. A direct connection doesn't help at all, same results.

I have also tried various attempts at fixing from the famously referenced 10G SMB guide, but nothing helps me.

 

I have noticed that I can get a good 6-8 Gbps of throughput, using parallel copies via the explorer, rather than individual files. I'd rather not start 10 processes though if I can just do one batch. Doesn't help if I'm moving one large file either.

 

Do diagnostics help at this point?