10Gbe network - Transfer PC to Unraid fast | Unraid to PC slow - UNRAID 6.9.2

[bowets]

Hi, this is my first time posting here. I've read many posts and watched lots of great set up videos. So far the information has been great, but I ran into a problem which I don't know how to fix or troubleshoot. 

 

I recently bought two Mellanox 10Gbe cards on Ebay link

2x Mellanox MCX311A-XCAT CX311A ConnectX-3 EN Network Card 10GbE 

1x cable SFP

 

I added one to my PC and the other to Unraid and followed the Spaceinvader One video on how to set up a 10Gbe network. The cards work and I can transfer files between my PC and Unraid. However, when I write to Unraid from my PC I get on average 4.5Gbps speeds. It's not great for a 10Gbe network, but ok. When I transfer files from Unraid to my PC the speeds are on average 630 Mbps. I get around the same speed with my normal gigabit network. 

 

I don't know what the issue could be. I did a network test with iperf and got these results. 

 

 

When I copy from Unraid to PC I'm copying to an Nvme drive. I thought maybe when reading from the array it can't read so fast so that's why the transfer is slow. So I created a share on the cache which is an SSD and tried again, but I get the same slow speed. 

 

What haven't I tried? What could be causing this?

 

My configuration is:

 

PC:

Windows 10

CPU: AMD Ryzen 5 2600

RAM: 16GB Corsair Vengence 3200

SSD: WD Blue 1TB

NVME: WD Blue 1TB

MB: MSI Gaming Pro Max

 

Unraid:

Unraid 6.9.2

CPU: AMD Ryzen 7 3700
RAM: 16GB Corsair Vengence 3200

HDD: 24TB Array of 7 (2 parity 8TB drives)

Cache: Samsung 256GB SSD

Edited May 12, 2021 by bowets

[John_M]

Your iperf tests are consistent with the transfer speeds you're seeing, which indicates that the problem is with your network, not with your storage. Maybe you have a bad cable in one direction - you could try turning it round to see if the speed discrepancy is reversed. Are there any error messages logged by the driver?

 

[cwrivers]

Did you change your MTU size? When i upgraded to 10gbe i had a very similar problem. Changing the MTU to 9000 or whatever is closest on both unraid and W10 fixed it for me.

[bowets]

Hi guys, thanks for the replies. 

I tried switching the cable around to see if that is the problem as suggested by John_M, however that didn't work. 

I also increased the MTU on both my PC and Unraid to 9000 and that increased the speed for both, but the difference in speed still remains. 

 

What I'll try to do after work is switch the network cards around to see if the issue is with the card itself. If that's the problem then I guess it means getting a new card. 

I'll come back to report what happened when I switch the cards. 

 

My new speeds after increasing MTU to 9000

 

PC to Unraid:

 

Unraid to PC:

[John_M]

2 hours ago, bowets said:

I tried switching the cable around to see if that is the problem as suggested by John_M, however that didn't work. 

 

I'm not sure how to interpret "that didn't work". I never expected it to fix the problem, just to help with the troubleshooting. Do you mean that your results were the same as before you turned the cable round? If so, it means that the cable is working the same in both directions (not necessarily properly, but at least it's symmetrical - remember, it isn't just a bundle of wires but has active components at each end), which is useful diagnostic information and points to a possible problem with one or both cards.

 

[bowets]

42 minutes ago, John_M said:

 

I'm not sure how to interpret "that didn't work". I never expected it to fix the problem, just to help with the troubleshooting. Do you mean that your results were the same as before you turned the cable round? If so, it means that the cable is working the same in both directions (not necessarily properly, but at least it's symmetrical - remember, it isn't just a bundle of wires but has active components at each end), which is useful diagnostic information and points to a possible problem with one or both cards.

 

 

Thank you for the explanation. "It didn't work" was a poor phrase to use. 

Like you said, I know now that it's not the cable so it could be the network card. I'll try to swap them and see if that changes the direction (slow/fast)

[bowets]

So, I finally found some time and swapped the cards. The card which was in the PC is now in UNRAID and vice versa. 

The results are unchanged. I am still getting the same fast connection writing to UNRAID from the PC and slow reading from UNRAID to PC. 

This means the issue is not with the cable or the cards. 

 

Is there anything else I can try?

 

I'll copy my configurations below to see if there is anything else I am missing 

 

PC:

Windows 10

CPU: AMD Ryzen 5 2600

RAM: 16GB Corsair Vengence 3200

SSD: WD Blue 1TB

NVME: WD Blue 1TB

MB: MSI Gaming Pro Max

 

Unraid:

Unraid 6.9.2

CPU: AMD Ryzen 7 3700
RAM: 16GB Corsair Vengence 3200

HDD: 24TB Array of 7 (2 parity 8TB drives)

Cache: Samsung 256GB SSD

 

[JorgeB]

Do you have another PC you can try with? If not at least try the same PC with a clean OS.

[bowets]

On 5/15/2021 at 11:24 AM, JorgeB said:

Do you have another PC you can try with? If not at least try the same PC with a clean OS.

 

I don't have another PC, unfortunately, but I did reinstall windows and there was no change. 

 

Could it be that reading from a drive is so much slower than writing to it? 

[itimpi]

35 minutes ago, bowets said:

Could it be that reading from a drive is so much slower than writing to it? 

Normally reading is faster than writing.   However when writing RAM can be used to buffer the write giving a higher perceived speed, while reading can never go faster than the drive can deliver.

[JorgeB]

2 hours ago, bowets said:

Could it be that reading from a drive is so much slower than writing to it? 

Iperf just tests the network bandwidth, no drives are involved, so still likely a network related issue.

[John_M]

Could it be a Windows driver issue? Are you using a driver that Windows found for you or are you using one that Mellanox provided?