Slow Write Speeds after x Minutes

[alias8818]

Guys,

 

I am really hoping you can help me out with this issue I having. I have a Dell C2100 with an H200 and 6gbps SAS Backplane. I have 5x 2TB, 1x 5TB, 3x 3TB, 2x 73GB (SAS that are currently removed from the array for troubleshooting). The 2TB are 3gbps, and the rest are 6gbps. 

 

After a transfer is started to the server, the write speeds go WAY down after anywhere from 5-50minutes. The dirty cache (grep - i dirty /proc/meminfo) remains full, so I know that the network transfer speed/SMB are working correctly. It just looks like the dumping from dirty cache (memory) to disk becomes slow. I see no errors in the server's SEL or syslog regarding the issue. 

 

I have tried:

• Flashing the H200 to P20, P19, Dell 6Gbps FW, and stock. - Same issue each time
• Starting a Parity Check immediately after the speed drops - Once dirty cache is empty, all drives go to 100MB/s.
  • This tells me its likely not the controller/overheating. If I can go right from the problem happening, to a read test on all drives for 1GB/s. 
  • It really feels as if there is an issue reading from memory and dumping to disk.
• Tried removing the 15k 73GB SAS drives from the cache pool
• Lowering/raising the dirty cache ratios and numbers.
• Tried playing around with nr_request, and other tunables
• Upgrading to 6.5.0 - which it still occurs after the upgrade

 

Please note: md_write_method is to reconstruct write. In my opinion it should be 100MB/s write and read from each drive, but when it has an issue - its between 2MB/s - 25 MB/s. Attached are a few screenshots of it starting to run fine, and then dropping way low. Also attached is my diagnostics report. 

 

Also to note... I find it odd that when the issue happens, only 2 of the drives are reading to calculate parity during turbo write. Per my understanding, it should be reading all of the drives at once. If it makes a difference, these were added after the initial creation of the entire array. Also to note, in "iotop -oPa" -- unraidd has "IO>" at least 80-90% all the time. This might be an issue with adding the drives in after the initial array creation? 

 

tower-diagnostics-20180314-0212.zip

[trurl]

Syslog in diagnostics only covers a few minutes after boot. Post another diagnostic after the problems exhibits but before rebooting.

[alias8818]

@trurl, I believe I've solved my own issue. My parity drive appears to be dumping can causing the whole issue. When it happens, await times for the drive skyrockets. I will be replacing it for a 3TB drive.

[trurl]

Since you didn't post new diagnostics I don't have any recommendations. What makes you think the disk is bad? Bad connections, cables, problems with ports or controllers are a much more common source of problems than an actual bad disk.

[alias8818]

@trurl

 

The drive is an SMR drive, with FW of CC46, shucked from a USB enclosure.. Others posted the information on how SMR drives work. After the cache (on the drive? is filled up, its a very slow write to dump it to disk. It appears that during this time is when the whole system halts in transfer. I am just getting into Unraid and the whole system/setup - so this was news for me.

 

I am doing my best to do the most reading I can in order to lessen the load of others to solve my problems. At the point I posted this, I felt I had went through everything. At one point, I was sure the controller was overheating thus slowing transfers.

 

This drive would be fine in the long run of hosting this file server, however currently I am migrating 9TB worth of data and its taking days to get through the folders, with speeds dropping to 5MB/s at times. I know it's capable of much more given the "Turbo Mode" like transfer. I'd even be fine with 60-70MB/s, but 5-20MB/s is horrible IMO. My how times have changed. 

 

Thanks for your review, though. I appreciate how much you do for this community. I see your name everywhere (been a long time lurker).