Inconsistent Write Speeds

[scaw]

Hi all, I hate to make a post about write speeds because I know there have been a million of them, but I can't figure out why my performance is so bad/inconsistent. I'm running a Dell R710 with a H200, unRAID version 6.3.5.

Full specs are

 

Quote

CPU: Intel® Xeon® CPU X5670 @ 2.93GHz

Cache: 192 kB

Memory: 48 GB (max. installable capacity 288 GB)

Network: bond0: fault-tolerance (active-backup), mtu 1500
 eth0: 1000 Mb/s, full duplex, mtu 1500

 

My problem is that since I set up the entire array (6x8tb WD Reds, 4 data 2 parity) my write speeds have been bad and inconsistent. I get between 6 and 12 MB/s writes most of the time, sometimes going up to 40 or 50 for a few seconds and then back down. I don't have a cache drive right now and I assume that parity is slowing this down, but from all I've read it shouldn't be this slow! I would be happy with a consistent 25-35 MB/s, which seems like a reasonable expectation based on what I've read. All six drives check out and precleared beautifully with speeds between 150 MB/s and 190 MB/s with no errors, so I don't think it's a bad drive. I'm hard wired via a gigabit ethernet switch. I've done iperf tests to the server, and I get full gigabit speeds, so I don't think it's the network either.

 

Here's a screenshot of a transfer with some dynamix stats up if it helps.

 

 

I'm not running a lot of stuff yet -- binhex's deluge docker and a plex server which was siting idle during the transfers. I've run Fix Common Problems and there's no warnings or errors. I've read the tips and tricks page too, and the page on performance.

 

Please let me know if I'm missing something, or if I should upload logs. The one thing I haven't done yet is enable turbo write, but I would rather not have my drives spinning 24/7 if I can avoid it so I'd like to first see if there's anything I can do.

 

Edit: Well, I tried enabling turbo write and the difference is staggering. For one thing, my RAM seems to flush its cache far more regularly now, which helps with the writes. Additionally, the disk write speed reported on dynamix (the bottom right) is around 300-400 MB/s (I'm assuming that's ~130 each on the data drive and the two parity drives. And the real difference -- I just copied 30 GB in about 5 minutes, with an average transfer speed (as given by Windows) of between 80 and 110 MB/s (the latter of which saturated my gigabit network).

 

So the questions are now:

a) What is it about turbo write that causes the RAM cache to behave so differently, and can I replicate it without turbo write? I assumed that if I had extra RAM floating around, unRAID would use it as a cache which would alleviate the need for a cache drive since I have plenty of extra RAM, but until I enabled turbo write the behavior was instead System boots --> First few transfers are fairly fast --> RAM fills w/ cache --> RAM never empties more than a couple gigabytes of cache over several hours and everything slows to a crawl.

b) Does this prove that my slow transfers before were due to parity issues in some way? I don't 100% understand how turbo write works, but I just saw a 2000% increase in my write speeds and I'm baffled. Again, I'd take half the performance most of the time if I didn't have to leave turbo write always enabled, but if it's 5 MB/s or 100, I'd have to keep the latter, drive wear be damned.

 

Any insight would be appreciated...

[pwm]

Turbo write means the machine is reading a block each from all data disks you aren't writing to. Based on this information and the new information you are about to write to one of the data drives the system computes the parity information - and then performs a write to the data disk and a write to the parity disk. So to write a block to one data disk you have one read from each of the other data drives and one write to each parity drive.

In the normal mode, the system will instead read one block from the data drive and one block from each of the parity drives. Then recompute the parity based on the change that is about to happen to the data drive. Then the system will write to the data drive and the parity drives. This means you get multiple transfers of the same address from the target data disk and from the parity disk(s). This read-modify-write is much slower because the target data disk and the parity disk(s) will become the bottlenecks - and having to wait for the platter to rotate a full turn between the read and the write does mean the slowdown isn't just because you transfer twice as much data from each disk - you are also introducing two search latencies which will end up resulting in the largest slowdown.

 

That's also why unRAID have a number of tunable paramters in Settings/Disk settings.

[Frank1940]

One more factor.  The type of data transfers has a big effect on transfer speed more than the amount of data.  A transfer of a large number of small files is much slower than the transfer of the same amount of data in a few large files.  This is because of the overhead required for the file creation and allocation of disk space.  This requires even more reads and writes on the same sectors plus the track-to-track seek times to get from the directory area to the data area on the data disk and parity disks. 

[pwm]

Creating a new file means

- writing a new inode

- updating the directory (possibly making the directory grow enough that the free-space information needs to be consulted and updated)

- writing the file data

- updating the free-space data structures based on the blocks consumed by the file data

 

And all writes must be updated to read-modify-write to know which bits are changing state so the parity can be updated - also using a read-modify-write.

[scaw]

3 hours ago, Frank1940 said:

One more factor.  The type of data transfers has a big effect on transfer speed more than the amount of data.  A transfer of a large number of small files is much slower than the transfer of the same amount of data in a few large files.  This is because of the overhead required for the file creation and allocation of disk space.  This requires even more reads and writes on the same sectors plus the track-to-track seek times to get from the directory area to the data area on the data disk and parity disks. 

 

Yeah, I'm aware of that which is why I did most of my testing with sequences of 2-4 GB files to minimize that issue.

 

48 minutes ago, pwm said:

Creating a new file means

- writing a new inode

- updating the directory (possibly making the directory grow enough that the free-space information needs to be consulted and updated)

- writing the file data

- updating the free-space data structures based on the blocks consumed by the file data

 

And all writes must be updated to read-modify-write to know which bits are changing state so the parity can be updated - also using a read-modify-write.

 

 

Right, I understand that much, so does that mean that 5 MB/s is just what can be expected from a dual parity setup? That seems too slow based on the other things I've read.

[JorgeB]

5 minutes ago, scaw said:

Right, I understand that much, so does that mean that 5 MB/s is just what can be expected from a dual parity setup?

 

No, with those disks you should get around 40-50MB/s without turbo write.

[pwm]

If the CPU isn't the limiting factor for parity computation and you don't have an ancient disk controller then streamed transfers should manage over 40MB as @johnnie.black mentions. It's only when moving small files that all the extra seeks should drop the transfer speeds even further.

 

And in this case, we know that the disk controller and the CPU isn't limiting the transfer speeds since you managed 100 MB/s in turbo write mode.

 

So something is off if you only get 5 MB/s for read/modify/write. I'm not really sure what may cause it. It might be interesting to look at the seek error rate in the SMART data for the drives - in turbo write mode there is very few seeks since no disk has to perform any read/modify/write. The file system tries to hold together the file data in large, continuous blocks where it's possible to do speculative read-ahead. If one of the drives has seek issues, that will affect the read/modify/write mode much more.

 

An alternative is that the system isn't using reasonable values for the tunable parameters.