bowets Posted May 12, 2021 Share Posted May 12, 2021 (edited) Hi, this is my first time posting here. I've read many posts and watched lots of great set up videos. So far the information has been great, but I ran into a problem which I don't know how to fix or troubleshoot. I recently bought two Mellanox 10Gbe cards on Ebay link 2x Mellanox MCX311A-XCAT CX311A ConnectX-3 EN Network Card 10GbE 1x cable SFP I added one to my PC and the other to Unraid and followed the Spaceinvader One video on how to set up a 10Gbe network. The cards work and I can transfer files between my PC and Unraid. However, when I write to Unraid from my PC I get on average 4.5Gbps speeds. It's not great for a 10Gbe network, but ok. When I transfer files from Unraid to my PC the speeds are on average 630 Mbps. I get around the same speed with my normal gigabit network. I don't know what the issue could be. I did a network test with iperf and got these results. When I copy from Unraid to PC I'm copying to an Nvme drive. I thought maybe when reading from the array it can't read so fast so that's why the transfer is slow. So I created a share on the cache which is an SSD and tried again, but I get the same slow speed. What haven't I tried? What could be causing this? My configuration is: PC: Windows 10 CPU: AMD Ryzen 5 2600 RAM: 16GB Corsair Vengence 3200 SSD: WD Blue 1TB NVME: WD Blue 1TB MB: MSI Gaming Pro Max Unraid: Unraid 6.9.2 CPU: AMD Ryzen 7 3700 RAM: 16GB Corsair Vengence 3200 HDD: 24TB Array of 7 (2 parity 8TB drives) Cache: Samsung 256GB SSD Edited May 12, 2021 by bowets Quote Link to comment
John_M Posted May 12, 2021 Share Posted May 12, 2021 Your iperf tests are consistent with the transfer speeds you're seeing, which indicates that the problem is with your network, not with your storage. Maybe you have a bad cable in one direction - you could try turning it round to see if the speed discrepancy is reversed. Are there any error messages logged by the driver? Quote Link to comment
cwrivers Posted May 12, 2021 Share Posted May 12, 2021 Did you change your MTU size? When i upgraded to 10gbe i had a very similar problem. Changing the MTU to 9000 or whatever is closest on both unraid and W10 fixed it for me. Quote Link to comment
bowets Posted May 12, 2021 Author Share Posted May 12, 2021 Hi guys, thanks for the replies. I tried switching the cable around to see if that is the problem as suggested by John_M, however that didn't work. I also increased the MTU on both my PC and Unraid to 9000 and that increased the speed for both, but the difference in speed still remains. What I'll try to do after work is switch the network cards around to see if the issue is with the card itself. If that's the problem then I guess it means getting a new card. I'll come back to report what happened when I switch the cards. My new speeds after increasing MTU to 9000 PC to Unraid: Unraid to PC: Quote Link to comment
John_M Posted May 12, 2021 Share Posted May 12, 2021 2 hours ago, bowets said: I tried switching the cable around to see if that is the problem as suggested by John_M, however that didn't work. I'm not sure how to interpret "that didn't work". I never expected it to fix the problem, just to help with the troubleshooting. Do you mean that your results were the same as before you turned the cable round? If so, it means that the cable is working the same in both directions (not necessarily properly, but at least it's symmetrical - remember, it isn't just a bundle of wires but has active components at each end), which is useful diagnostic information and points to a possible problem with one or both cards. Quote Link to comment
bowets Posted May 12, 2021 Author Share Posted May 12, 2021 42 minutes ago, John_M said: I'm not sure how to interpret "that didn't work". I never expected it to fix the problem, just to help with the troubleshooting. Do you mean that your results were the same as before you turned the cable round? If so, it means that the cable is working the same in both directions (not necessarily properly, but at least it's symmetrical - remember, it isn't just a bundle of wires but has active components at each end), which is useful diagnostic information and points to a possible problem with one or both cards. Thank you for the explanation. "It didn't work" was a poor phrase to use. Like you said, I know now that it's not the cable so it could be the network card. I'll try to swap them and see if that changes the direction (slow/fast) Quote Link to comment
bowets Posted May 15, 2021 Author Share Posted May 15, 2021 So, I finally found some time and swapped the cards. The card which was in the PC is now in UNRAID and vice versa. The results are unchanged. I am still getting the same fast connection writing to UNRAID from the PC and slow reading from UNRAID to PC. This means the issue is not with the cable or the cards. Is there anything else I can try? I'll copy my configurations below to see if there is anything else I am missing PC: Windows 10 CPU: AMD Ryzen 5 2600 RAM: 16GB Corsair Vengence 3200 SSD: WD Blue 1TB NVME: WD Blue 1TB MB: MSI Gaming Pro Max Unraid: Unraid 6.9.2 CPU: AMD Ryzen 7 3700 RAM: 16GB Corsair Vengence 3200 HDD: 24TB Array of 7 (2 parity 8TB drives) Cache: Samsung 256GB SSD Quote Link to comment
JorgeB Posted May 15, 2021 Share Posted May 15, 2021 Do you have another PC you can try with? If not at least try the same PC with a clean OS. Quote Link to comment
bowets Posted May 23, 2021 Author Share Posted May 23, 2021 On 5/15/2021 at 11:24 AM, JorgeB said: Do you have another PC you can try with? If not at least try the same PC with a clean OS. I don't have another PC, unfortunately, but I did reinstall windows and there was no change. Could it be that reading from a drive is so much slower than writing to it? Quote Link to comment
itimpi Posted May 23, 2021 Share Posted May 23, 2021 35 minutes ago, bowets said: Could it be that reading from a drive is so much slower than writing to it? Normally reading is faster than writing. However when writing RAM can be used to buffer the write giving a higher perceived speed, while reading can never go faster than the drive can deliver. Quote Link to comment
JorgeB Posted May 23, 2021 Share Posted May 23, 2021 2 hours ago, bowets said: Could it be that reading from a drive is so much slower than writing to it? Iperf just tests the network bandwidth, no drives are involved, so still likely a network related issue. Quote Link to comment
John_M Posted May 23, 2021 Share Posted May 23, 2021 Could it be a Windows driver issue? Are you using a driver that Windows found for you or are you using one that Mellanox provided? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.