MikroTiK and Mellanox issues . Need someone who owns this hardware to explain to me why 10G Lan is not working


Recommended Posts

So running unraid /cache NVME raid 0 about 13000MB/s  to a NVME on another client running 3000MB/s.     Mellanox MCX311A-XCAT CX311A ConnectX-3 EN Network Card 10GbE SinglePort SFP+

 

I start out at 1.09GB/s  but dies to about 50-100MB/s after 30 secs.  Not sure if there is a setting in the switch MikroTik CRS305-1G-4S+IN that needs to be turned on to stop this issue?   I am running windows 10 to unraid  with issues.  I heard linux is better i.e. Ubuntu? 

MTU size is not the issue and only screws things up with unraid and network shares over SMB.    Increasing Jumbo frames in windows 10 did not do a thing

 

Should I just return this CRAP or and buy a regular 10G network card and use adapters?  What is the deal ???  Maybe the developer for Unraid can help me out. I am really pissed off 

 

Link to comment

I assume you are doing a file transfer and the speed is dropping off? I do not have Mellanox cards, but I do have the same Mikrotik switch. I have no issues sustained reads/writes, as long as the share is set to use cache, which are both NVME as in your setup. For me to get max speed, I have to use an MTU of 9000 for both machines.

 

The fact that your speed is dropping after a period of time, tells me some sort of buffer is being filled up, would be my guess. Have you done testing with iperf3? You could run a test that is longer than 30s or whatever the time your file transfer drops off. That would prove out that its not the NICs or the switch, since it would run entirely from ram.

 

If you aren't familiar with iperf3, the nerdtools plugin for unraid has iperf3 as one of the included packages. Once you have it installed, you'd run "iperf3 -s" on your unraid box. You also need iperf3 for windows of course, then you'd do something like "iperf3 -c <ip of unraid box> -t 60 -P 10" for 60 seconds and 10 streams. Adjust accordingly. To test the other direction add a "-R" to the end, which will just reverse the flow so that your windows box becomes the server.

 

Other things to be sure of are cabling and infrastructure. Are you using copper, fiber, DAC cables? If copper, what type? CAT6, CAT6A? how long are the runs? Lots of factors could come into play. I always start with iperf though. If it tells me I'm good over a particular link, and then I go to do a test file transfer and it is much slower usually a misconfig (not using a share that is assigned to the NVME) or a disk performance issue (transferring a ton of small files like a Plex database kills performance, even on NVME drives).

 

Hope this helps.

Link to comment
  • 7 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.