80 MB/s on 10GB ethernet

[mgranger]

I just installed two ASUS XG-C100C 10G Network Adapter PCI-E cards (one on my windows PC and one on my Unraid server)  I ran CAT6a cable between the two (100 ft cable)  I both seem to be recognized ok but when I go to transfer the files I am only getting 80 MB/s.  I was hoping to get something more around 300-400 MB/s.  Is there something I should be doing different?

Edited April 1, 2018 by mgranger

[JorgeB]

You can use iperf to test max Ethernet bandwidth  and see if that's the problem or it's not lan related.

[mrbilky]

20 minutes ago, mgranger said:

I just installed two ASUS XG-C100C 10G Network Adapter PCI-E cards (one on my windows PC and one on my Unraid server)  I ran CAT6a cable between the two (100 ft cable)  I both seem to be recognized ok but when I go to transfer the files I am only getting 80 MB/s.  I was hoping to get something more around 300-400 MB/s.  Is there something I should be doing different?

Did you map the share with their ip address (and did you assign ip's for both cards) if not your transfer speeds will reflect what you are getting now been down this road the last few weeks and that was what worked for me. Also enable jumbo packets for both your unRAID server and your windows rig to match each other

[mgranger]

1 hour ago, mrbilky said:

Did you map the share with their ip address (and did you assign ip's for both cards) if not your transfer speeds will reflect what you are getting now been down this road the last few weeks and that was what worked for me. Also enable jumbo packets for both your unRAID server and your windows rig to match each other

 

Yes i mapped the ip address and assigned ip's for both cards on a different network.  I am not sure how to enable jumbo packets

 

1 hour ago, johnnie.black said:

You can use iperf to test max Ethernet bandwidth  and see if that's the problem or it's not lan related.

 

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.10.190, port 54292
[  5] local 192.168.10.192 port 5201 connected to 192.168.10.190 port 54294
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   985 MBytes  8.26 Gbits/sec
[  5]   1.00-2.00   sec  1022 MBytes  8.58 Gbits/sec
[  5]   2.00-3.00   sec  1.00 GBytes  8.61 Gbits/sec
[  5]   3.00-4.00   sec  1023 MBytes  8.58 Gbits/sec
[  5]   4.00-5.00   sec  1.00 GBytes  8.61 Gbits/sec
[  5]   5.00-6.00   sec  1019 MBytes  8.55 Gbits/sec
[  5]   6.00-7.00   sec  1.00 GBytes  8.62 Gbits/sec
[  5]   7.00-8.00   sec  1015 MBytes  8.51 Gbits/sec
[  5]   8.00-9.00   sec  1013 MBytes  8.50 Gbits/sec
[  5]   9.00-10.00  sec  1021 MBytes  8.56 Gbits/sec
[  5]  10.00-10.04  sec  37.3 MBytes  8.41 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  9.98 GBytes  8.54 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

 

Here is what i got for iperf

Edited April 1, 2018 by mgranger

[mgranger]

I am getting the expected speed (about 250-300 MB/s) for about 10 seconds but then it goes down to 80 MB/s.

 

Edit:  So it seems to be working kind of.  If I transfer 1 or 2 files at a time then it hits the 400MB/s no problem.  When I did a folder that was 56GB in size it worked for a couple of seconds at 300 MB/s but then eventually dropped to 80 and stayed at 80.  Is this really what is supposed to happen or is there something I am missing where it is getting throttled somehow.  

Edited April 1, 2018 by mgranger

[1812]

perhaps your'e either filing up your ram cache and/or filling up your ssd cache buffer and it is then writing directly to the array... without knowing your setup it's hard to say.

[mgranger]

15 minutes ago, 1812 said:

perhaps your'e either filing up your ram cache and/or filling up your ssd cache buffer and it is then writing directly to the array... without knowing your setup it's hard to say.

What settings do you need to know.  here is my tool diagnostics log file.  Not sure if that would help or give you what you need to know.  obviously the windows side is not included in this.

 

finalizer-diagnostics-20180401-1649.zip

[1812]

1 hour ago, mgranger said:

What settings do you need to know.  here is my tool diagnostics log file.  Not sure if that would help or give you what you need to know.  obviously the windows side is not included in this.

 

finalizer-diagnostics-20180401-1649.zip

 

a quick look through and nothing jumped out at me. are you transferring small files in these folders? do you have a single larger 20GB+ file you could transfer? if the large single file transfer runs fast, and if the folder has lots of smaller files, then that's your problem.

[mrbilky]

1 hour ago, mgranger said:

I am getting the expected speed (about 250-300 MB/s) for about 10 seconds but then it goes down to 80 MB/s.

 

Edit:  So it seems to be working kind of.  If I transfer 1 or 2 files at a time then it hits the 400MB/s no problem.  When I did a folder that was 56GB in size it worked for a couple of seconds at 300 MB/s but then eventually dropped to 80 and stayed at 80.  Is this really what is supposed to happen or is there something I am missing where it is getting throttled somehow.  

Go to network adapters click on the 10 GBE network hit properties tab hit internet protocol version 4 tab hit properties tab ensure you have set your ip correctly then hit the configure button hit advanced tab scroll down to jumbo packet, I set mine to 9000 now go to your unRAID server and hit the settings tab and go to network settings and check that your 10 GBE network is set to the correct ip and the MTU is set for the same 9000 as your windows box Now I'm no expert on diags for sure but if you are bonding a 1Gb and a 10Gb network I'm told that's no good I also made my ip's for my setup as 10.10.10.10 and 10.10.10.20 respectively as I have read on many forums and setups this is good practice to prevent any bottlenecks

[mgranger]

41 minutes ago, 1812 said:

 

a quick look through and nothing jumped out at me. are you transferring small files in these folders? do you have a single larger 20GB+ file you could transfer? if the large single file transfer runs fast, and if the folder has lots of smaller files, then that's your problem.

I have 8 files in the folder i am copying.  All between 5 and 10 gb.  I tried just doing the 10 gb one and it did the same thing where it copied for a little bit at 300 MB/s then dropped to 80 MB/s

[mgranger]

41 minutes ago, mrbilky said:

Go to network adapters click on the 10 GBE network hit properties tab hit internet protocol version 4 tab hit properties tab ensure you have set your ip correctly then hit the configure button hit advanced tab scroll down to jumbo packet, I set mine to 9000 now go to your unRAID server and hit the settings tab and go to network settings and check that your 10 GBE network is set to the correct ip and the MTU is set for the same 9000 as your windows box Now I'm no expert on diags for sure but if you are bonding a 1Gb and a 10Gb network I'm told that's no good I also made my ip's for my setup as 10.10.10.10 and 10.10.10.20 respectively as I have read on many forums and setups this is good practice to prevent any bottlenecks

 

All of these settings seemed to be set like you said except mine was 9014 rather than 9000 and the Unraid was blank so i made it match.  I used 192.168.10.x rather than the 10.10.10.x but this is different from my 1 Gb network so I think it would be ok.  

[mrbilky]

16 minutes ago, mgranger said:

 

All of these settings seemed to be set like you said except mine was 9014 rather than 9000 and the Unraid was blank so i made it match.  I used 192.168.10.x rather than the 10.10.10.x but this is different from my 1 Gb network so I think it would be ok.  

Well the only other suggestion to possibly eliminate something, it was recommended that I created a ram disk for testing and rule out any bottleneck on that front if correctly configured you should saturate transfers with no problem

[mgranger]

25 minutes ago, mrbilky said:

Well the only other suggestion to possibly eliminate something, it was recommended that I created a ram disk for testing and rule out any bottleneck on that front if correctly configured you should saturate transfers with no problem

so for creating a ram disk what does this entail.  I created on one the Windows pc that is about 2.5 GB.  Should i just transfer files to this from the unraid server.  I did this and I was getting about 140 MB/s

Edited April 1, 2018 by mgranger

[mgranger]

5 hours ago, 1812 said:

perhaps your'e either filing up your ram cache and/or filling up your ssd cache buffer and it is then writing directly to the array... without knowing your setup it's hard to say.

 

Is there a way to tell if it is filling up the ram cache or ssd cache buffer?

[mgranger]

So I created a RAM disk using SoftPerfect.  Here is what happens  It transfers at about 1 GB/s for about half of the file then suddenly falls off to 80 MB/s.  This was about a 5.75 GB file and I used a 6GB ram disk.  So it seems to work half way through but then suddenly it reverts to Gigabit speeds.

 

Edited April 2, 2018 by mgranger

[1812]

28 minutes ago, mgranger said:

 

Is there a way to tell if it is filling up the ram cache or ssd cache buffer?

There may be an easier way than this, but:

 

on a fresh server reboot, open system stays and watch the ram/memory graph. If it maxes with “cached” then ram is full. For the ssd, you have to watch the capacity on the main tab. 

 

Again, probably better ways to find out. But with that said, iirc, you have 32gb? or ram and at least a 256 gb 850 evo (don’t remember how much free space) but that should have high sustained write speeds well over the lower number you are getting. 

 

Also note on in the main tab what disks are getting written to, if it’s in the array or on the cache disk. I skimmed the water parts of this , but you’ve verified the share you are using is set to cache=yes?

[mgranger]

11 minutes ago, 1812 said:

There may be an easier way than this, but:

 

on a fresh server reboot, open system stays and watch the ram/memory graph. If it maxes with “cached” then ram is full. For the ssd, you have to watch the capacity on the main tab. 

 

Again, probably better ways to find out. But with that said, iirc, you have 32gb? or ram and at least a 256 gb 850 evo (don’t remember how much free space) but that should have high sustained write speeds well over the lower number you are getting. 

 

Also note on in the main tab what disks are getting written to, if it’s in the array or on the cache disk. I skimmed the water parts of this , but you’ve verified the share you are using is set to cache=yes?

 

Yes that is right 32 gb or ram and the 850 evo is actually 500 gb.  (About 400 free)  Yes this share is using the cache disk.  I verified that.  When I go from Unraid to Windows this is what it looked like.  The windows was not using the ram disk in this instance but it seems like I get the same thing either way.  My Windows PC only has 16 GB or ram in it if that matters.  

 

 

 

[mgranger]

So i saw somewhere online that to help troubleshoot to run a test using AJA System Test.  Not sure if it will help here or not but I will post my results just in case it does.  It looks like I should be cable of getting better than the 80 MB/s that it keeps dropping to. 

MILLENNIUM-FALC.nfo

MILLENNIUM-FALC.pdf

 

EDIT:  So I have transferred a file back and forth a couple of times today just to see if anything has changed.  It seems to be transferring at around 400 MB/s from Unraid to Windows PC and around 300 MB/s from PC to Unraid.  Not sure why all of a sudden it is working now but it seems to be better for some reason.  I will keep an eye on it but I have a feeling something is still not quite right but until then thanks for all the help.

Edited April 2, 2018 by mgranger