Slow Writes to Array AND Cache

[stetay2233]

Hey everyone, new to unRaid enjoying it so far! I have one issue (maybe?) though and, I've been reading everything I can find and just can't seem to find the answer to what has been going on.

 

Setup:

unRaid version 6.5.3
Dual Xeon 2670
64GB ram
1 cache drive (1tb crucial mx500 SSD)
10 disk array WD red 4tb (don't shoot me I already had them)
1 WD red 8tb parity disk (not even using this yet)
3x lsi  9211-8i HBA (it mode)
Intel x540-t2 NIC (only running at 1G currently)

 

I can't figure out why my network writes either to my array or my cache seem to be capped at 50MB/s. The file copy/moves start and end at that speed for both small and large files. However my reads both from the array and the cache disk move along at 100MB/s as expected.

 

I have tried tips and Tweaks, I have checked network equipment from both directions, tried with the cache disk both on/off my HBA, tried copies from multiple computers and network configs, changed the system tunables as recommended and still no improvement.

 

Any body have any insight or suggestions?

 

TL;DR 
Network writes are capped at 50MB/s on a non parity protected array as well as cache disk, while network reads are fine at 100MB/s. 

 

Logs and SMART show no errors or warnings as far as i can tell.

stetay2233_diagnostics-20180628-1118.zip

[JorgeB]

11 minutes ago, stetay2233 said:

The file copy/moves start and end at that speed for both small and large files.

That would suggest a network problem, start of a copy is cached to RAM and only limited by network speed, after that device speed is the limit, you can use iperf to measure the lan bandwidth, or try a different NIC, cable, etc

[stetay2233]

39 minutes ago, johnnie.black said:

That would suggest a network problem, start of a copy is cached to RAM and only limited by network speed, after that device speed is the limit, you can use iperf to measure the lan bandwidth, or try a different NIC, cable, etc

 

Here are my test results as the system is right now. I have tried multiple cables and everything already, i have not tried different nic yet.

Network5GFileTestResults.zip

[JorgeB]

Test confirm the network problem, single stream is only getting around 30MB/s, and a normal not lan limited copy to the server using gigabit looks like this:

 

 

For the first few seconds you should get around 113MB/s, then and in the case the array or device is slower than that it will slow down to the actual array/device speed.

 

[JorgeB]

Also, this is how a single stream iperf test should look like:

 

 

[stetay2233]

41 minutes ago, johnnie.black said:

Test confirm the network problem, single stream is only getting around 30MB/s, and a normal not lan limited copy to the server using gigabit looks like this:

 

 

For the first few seconds you should get around 113MB/s, then and in the case the array or device is slower than that it will slow down to the actual array/device speed.

 

 

sorry, im trying to wrap my head around this, what else could cause this. the cables are tested, the switch is good, everything is negotiated to gigabit speeds, link indicators on all NIC's show they are connected at 1000Mbits/sec.

 

and why would the read speeds be fine? is there anything in unRaid settings that would cause this? i have even tried another nic, both onboard 1G port,s in this machine and its the same issue.

[JorgeB]

Could a source computer problem, do you have any other PC you can use as source?

[stetay2233]

7 minutes ago, johnnie.black said:

Could a source computer problem, do you have any other PC you can use as source?

 

So i ran more tests. Every computer in the house gets similar results running iperf to unRaid. but also weird results between floors in the house. however the actual network trasnfers run full speed. 

 

Other than unRaid, I don't have any non-windows computers in the house right now.

 

Computer 1 (2nd floor office) -> unRaid (basement utility room) = 30MB/s

Computer 2 (2nd floor office) -> unraid = 30MB/s

Computer 3 (Basement HTPC) -> unRaid = 30MB/s

Computer 1 -> Computer 2 = 100MB/s

Computer 2 -> Computer 1 = 100MB/s

Computer 1 -> Computer 3 = 30MB/s (actual network file transfer runs at 100MB/s)

Computer 3 -> Computer 1 = 30MB/s (actual network file transfer runs at 100MB/s)

Computer 2 -> Computer 3 = 30MB/s (actual network file transfer runs at 100MB/s)

Computer 3 -> Computer 2 = 30MB/s (actual network file transfer runs at 100MB/s)

 

Still reading from unRaid to anywhere in the house over the network is fine, it's just the writes. Reads and writes between the other computers in the house run fine (besides the not good iperf results) 

 

[JorgeB]

You can use the attached script to test the server raw write speed, without involving the network, copy to your flash drive root and use like so:

 

/boot/write_speed_test.sh /mnt/cache/test.dat

 

write_speed_test.zip

[stetay2233]

6 hours ago, johnnie.black said:

You can use the attached script to test the server raw write speed, without involving the network, copy to your flash drive root and use like so:

 

/boot/write_speed_test.sh /mnt/cache/test.dat

 

write_speed_test.zip

 

Here are the results:

 

 

[JorgeB]

This confirms, like I mentioned, that the problem is the network, if it's the server NIC, cable, switch or source PC I can't tell you, you need to test and rule things out.

[stetay2233]

1 minute ago, johnnie.black said:

This confirms, like I mentioned, that the problem is the network, if it's the server NIC, cable, switch or source PC I can't tell you, you need to test and rule things out.

 

Thanks, i will do a better and fuller network investigation and report back in the next day or two.

[stetay2233]

On 6/29/2018 at 8:34 AM, johnnie.black said:

This confirms, like I mentioned, that the problem is the network, if it's the server NIC, cable, switch or source PC I can't tell you, you need to test and rule things out.

SOLVED (kinda): but it turns out that the issue was network related. Doing more digging, running iperf tests throughout the house, directly connecting to my unRaid server i was able to track down the issue.

 

Turns out that what was told to us as wired Ethernet in the condo, they are actually using a bunch moca 2.0 adapters in the attic and in the rafters of the basement on coax to get Ethernet to and from wallplates/the basement. 

 

I can't figure out why windows can traverse these things but when copying over the network to my server has issues. but that is where the problem is as connecting to my server and my rack-mounted switch with a different machine directly brought back the speeds i was expecting.