Slow writes to array regardless of write method or cache

[sailorbob134280]

I've been running into a problem with writes and after hours of troubleshooting, I admit I'm stumped. Background info:

 

I'm running a whitebox unRaid server with an old Intel Core i3-540 and 6GB of DDR3 RAM (it gets the job done, or so I thought). Drives are 5x HGST 7.2k 2TB SATA III drives connected through SATA II (motherboard limitation, shouldn't affect throughput?) running in a single parity config. I also have a 120GB SSD as a cache. See the attached diagnostics for full specs. 

 

The problem is that writes to the server (either from my Windows desktop via SMB and FTP or from another unRaid box via NFS) start out at gigabit speeds and eventually settle out at around 50 MB/s or less. Now I know this is normal for read/modify/write operations (those actually top out around 65 MB/s), but this cap applies to reconstruct writes and writes to the SSD cache. Looking at the drive activity indicators on the server itself, the drives only show activity every second or so during turbo writes. Internal transfers from the SSD to the array with turbo write enabled cap out at around ~140 MB/s, which is expected. The activity LEDs light up the entire transfer. Going from the array to the cache is a similar story. 

 

Things I've tried:

 

• Disabling Docker/VM's and any other service that might be accessing the array. The Open Files plugin shows that only Samba is open, which makes sense because I am testing the transfer from my Windows desktop
• Transferring with FTP from my Windows box and NFS from another unRaid box, same symptoms
• Using Teracopy instead of Windows explorer, same symptoms
• Tinkering with the vm.dirty_ratio settings. I was able to make the turbo writes either fluctuate like crazy or settle out at around 40 MB/s. This leads me to believe it might be a memory issue?
• Running the Disk Speed Test plugin to test drive speed, all are capable of much higher transfer speeds than I'm seeing (should be around 140 MB/s if the LAN bottleneck was removed)
• Enabling jumbo frames on both ends (MTU size of 9000), which my switch supports. This produced no change.
• Defragmenting my data drives. This worked wonders for my read speeds off one of the drives (different issue, now solved) but did not impact write performance
• Reverting to previous versions of Samba, this made no change (unsurprising, as the FTP and NFS transfers were similarly affected in the first place)
• Moving SATA ports/cables around to try to isolate a bad port/cable. No change.
• Transferring the test file from my Windows box to the other unRaid box, substained 112 MB/s. 

 

Any ideas? I'm not unwilling to spend a little money on parts if I know it'll help (more ram, cheap HBA, new NIC, etc) but I'd rather get some advice before I resort to shotgun repair. I had initially thought it was the SATA controller becoming saturated since it's an old board and was never top of the line to begin with, but since the internal transfers worked fine, I'm beginning to think its something to do with the memory. 

 

Thanks for the help!

 

Transfer to the cache drive. It starts at 112 MB/s and drops to about 50 MB/s where it stays for the rest of the transfer.

 

Transfer directly to the array with turbo write enabled. RAM cache settings heavily impact the speed graph. Average is around 50 MB/s.

 

Transfer directly to the array with read/modify/write enabled, which is currently the fastest way to transfer files to the server. RAM cache settings don't really make a difference here. 

aurora-diagnostics-20171108-2342.zip

Edited November 9, 2017 by sailorbob134280

[JorgeB]

Strange problem, normal write mode speed is actually very good for the disks used.

 

No idea about the low turbo write speed, as for cache writes, do you regularly trim your SSD?

 

Also, what's the SSD model and specs, SMART only mentions SPCC, if it's a cheap TLC SSD those can have low sustained write performance.

 

 

[JorgeB]

After thinking a little more about pretty sure I know why the low speed with turbo write, you're using an older board with DMI 1.0, and with turbo write enable doens't have enough bandwidth to handle all disks plus the NIC simultaneously, to improve speed you'd need to upgrade the board, use an HBA in one of the x16 CPU slots, or add an external NIC to one of the same x16 slots, those slots don't share the DMI.

 

For the cache writes, it's either an untrimmed or low performance SSD, if it's being trimmed regularly you'd need to replace it with a better performing SSD, e.g., a Samsung 850 EVO, it should allow sustained 100MB/s+ writes.

[sailorbob134280]

Thanks for the help, I had imagined it was a bandwidth issue somewhere in the chain. The SSD is indeed a cheap TLC one that I bought the other day, looks like I'll be returning it and buying a better one, as well as a NIC.