Hello.

 

After an update to the 6.12.11 I got an issue with slow SMB share speeds. For example, I have a nightly backup of my Windows VM, it's about 100 gigabytes. On the 6.12.10 it took about 4 mins every night but after an update it always took 7-8 mins. Which a lot slower than before (NVMe to the NVMe transfer, 10G network). Average speed is about 300 mb/sec, before it was 600+ mb/sec.

 

I had this issue on the 6.12.9 as far as I remember, but after an update to the 6.12.10 it got resolved somehow.

 

Any ideas why this is happening? No hardware or software, network, etc configuration update was made.

 

Iperf3 result is 9.4G in both directions.

 

Here is the network graph on the 6.12.11:

Graph on the 6.12.10:

Edited July 29, 20241 yr by SuberSeb

3 hours ago, SuberSeb said:

I had this issue on the 6.12.9 as far as I remember, but after an update to the 6.12.10 it got resolved somehow.

This is suspicious, if you go back to 6.12.10 does the performance return?

2 hours ago, JorgeB said:

This is suspicious, if you go back to 6.12.10 does the performance return?

Yes, on the 6.12.10 performance is great, I hit 700-800 megabytes per second.

The strange thing is that on the 6.12.9 I had the same problem but it was resolved on the 6.12.10 (there was a some rollback in the change list of the 6.12.10). Can it be a reason?

Edited July 29, 20241 yr by SuberSeb

Please go back to 6.12.10 and confirm the speed improves.

Just now, JorgeB said:

Please go back to 6.12.10 and confirm the speed improves.

Already done this before opening this thread, speed improves instantly.

I forgot to mention that I got 3 servers on Unraid and problem exists on the all of them.

I find this very strange, .9 slow, .10 fast and .11 slow again, but I will make some tests to see if I can duplicate, but will only be able to do this next week since I'm not home this week.

I've checked multiple things. Network seems totally ok (almost 10G between servers in both directions), SMB/Network settings was not changed for years and seems ok too. Router, switches are totally ok. So this thread is my last chance to resolve this issue.

 

Thank you anyway.

8 hours ago, JorgeB said:

I find this very strange, .9 slow, .10 fast and .11 slow again, but I will make some tests to see if I can duplicate, but will only be able to do this next week since I'm not home this week.

While not as extreme as SuberSeb's example i experienced a good drop in speed after having upgraded to the beta.

Specially transfers over the fuse layer are effected.

 

Used to be able to hit 700-800MB/s as well, now it barely goes past 600.

With the beta I did test, vs 6.12.10 which was the most recent v.612 release at the time, and didn't find any significant difference:

 

17 minutes ago, JorgeB said:

With the beta I did test, vs 6.12.10 which was the most recent v.612 release at the time, and didn't find any significant difference:

 

 

That's promising, shows it just my config that might have a small hiccup somewhere.

 

As promised, I'm now back home, ran some tests and I don't find any significant different in SMB speeds for 6.12.11:

 

 

 

Which NICs are you using? I have Mellanox COnnectX-2, 3 and 4

 

P.S. I redid all the tests, to also include the new beta and to make sure they were all done using the same devices/settings.

I'm using Mellanox ConnectX-3 MCX311A.

Then I don't see a reason for the slowdown, and possibly it's not even related to the upgrade, but I thought the same when you mentioned it was also slow with 6.12.9, since the changes from 6.12.9 to 6.12.10 are minimal, mostly bug fixes, nothing that should affect 10GbE performance.

 

 

Well, I don't see what can affect it either. There were no any configuration or hardware updates for months (or even years), servers just working and I do not touch anything. But the strange thing that I had this issue twice and both times right after OS updates...

 

Any ideas what else I should to check?

Can it be related to SMB and/or UD plugin?

Seems unlikely to me, can you test with a different copy operation? Or a different PC just doing a file transfer, with Windows for example.

I will try to test with a NFS mounted share, also I will try to test with a Windows PC but it's unlikely that it will hit 10G due to PC hardware limitations.

I made some tests. First one is SMB, second one is NFS. Very poor speeds for NVMe to NVMe transfer. NFS is slow even for HDD.

pv /mnt/user/Test/image.img > /mnt/remotes/192.168.1.1_Test/image.img
51.9GiB 0:04:30 [ 195MiB/s] [======================================================================================================>] 100%

pv /mnt/user/Test/image.img > /mnt/remotes/192.168.1.1_Test/image.img
51.9GiB 0:05:22 [ 164MiB/s] [======================================================================================================>] 100%

So seems not to be related to SMB.

 

Here is iperf3 result:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.05 GBytes  8.99 Gbits/sec    0    495 KBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec    0    639 KBytes
[  5]   2.00-3.00   sec  1.09 GBytes  9.37 Gbits/sec    0    515 KBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec    0    481 KBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0    427 KBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec    0    475 KBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec    0    433 KBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec    0    438 KBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec    0    481 KBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec    0    424 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.34 Gbits/sec                  receiver

 

Edited August 5, 20241 yr by SuberSeb

After downgrade to 6.12.10 speed improves instantly:

pv /mnt/user/Test/image.img > /mnt/remotes/192.168.1.1_Test/image.img
51.9GiB 0:02:17 [ 385MiB/s] 

 

I'm sorry, no idea what it could be since I cannot reproduce.

Could this be a ZFS thing, as I did see in the change log that ZFS was updated? I am actually seeing a perf degrade when copying large files to my cache pool, which is ZFS. Standard SATA SSD cache pool with 2 Samsung Evo 850s(1TB) and 2 Samsung EVO 870s(1TB) in a RAIDZ1. Whole network is 10G and it struggles to get to 200MB where it used to get into the 300s with no problem. Not critical as this is just a home use case for me but thought I would lend SuberSeb some support as it seems I am seeing something similar.

Edited August 5, 20241 yr by dwoforever
spelling

7 hours ago, dwoforever said:

Could this be a ZFS thing, as I did see in the change log that ZFS was updated? I am actually seeing a perf degrade when copying large files to my cache pool, which is ZFS. Standard SATA SSD cache pool with 2 Samsung Evo 850s(1TB) and 2 Samsung EVO 870s(1TB) in a RAIDZ1. Whole network is 10G and it struggles to get to 200MB where it used to get into the 300s with no problem. Not critical as this is just a home use case for me but thought I would lend SuberSeb some support as it seems I am seeing something similar.

Thanks for the reply and for the help, but in my case I'm not using ZFS at all... By pools are BTRFS pools.

10 hours ago, JorgeB said:

I'm sorry, no idea what it could be since I cannot reproduce.

Anyway, thanks for checking and for help.

I have the same problem with the mellanox connectx-3 nic, after rollback to.10 speed is normal again.

 

Edited August 22, 20241 yr by Marvin_G

21 hours ago, Marvin_G said:

I have the same problem with the mellanox connectx-3 nic, after rollback to.10 speed is normal again.

 

I have MCX311A. Did you try to resolve it somehow without rollback? I tried .12 with no luck, will try .13 today.