Turbo write

[RobJ]

Turbo Write

  technically known as "reconstruct write" - a new method for updating parity

 

JonP gave a short description of what "reconstruct write" is, but I thought I would give a little more detail, what it is, how it compares with the traditional method, and the ramifications of using it.

 

First, where is the setting?  Go to Settings -> Disk Settings, and look for Tunable (md_write_method).  The 3 options are read/modify/write (the way we've always done it), reconstruct write (Turbo write, the new way), and Auto which is something for the future but is currently the same as the old way.  To change it, click on the option you want, then the Apply button.  The effect should be immediate.

 

Traditionally, unRAID has used the "read/modify/write" method to update parity, to keep parity correct for all data drives.  Say you have a block of data to write to a drive in your array, and naturally you want parity to be updated too.  In order to know how to update parity for that block, you have to know what is the difference between this new block of data and the existing block of data currently on the drive.  So you start by reading in the existing block, and comparing it with the new block.  That allows you to figure out what is different, so now you know what changes you need to make to the parity block, but first you need to read in the existing parity block.  So you apply the changes you figured out to the parity block, resulting in a new parity block to be written out.  Now you want to write out the new data block, and the parity block, but the drive head is just past the end of the blocks because you just read them.  So you have to wait a long time (in computer time) for the disk platters to rotate all the way back around, until they are positioned to write to that same block.  That platter rotation time is the part that makes this method take so long.  It's the main reason why parity writes are so much slower than regular writes.

 

To summarize, for the "read/modify/write" method, you need to:

* read in the parity block and read in the existing data block (can be done simultaneously)

* compare the data blocks, then use the difference to change the parity block to produce a new parity block (very short)

* wait for platter rotation (very long!)

* write out the parity block and write out the data block (can be done simultaneously)

 

That's 2 reads, a calc, a long wait, and 2 writes.

 

Turbo write is the new method, often called "reconstruct write".  We start with that same block of new data to be saved, but this time we don't care about the existing data or the existing parity block.  So we can immediately write out the data block, but how do we know what the parity block should be?  We issue a read of the same block on all of the *other* data drives, and once we have them, we combine all of them plus our new data block to give us the new parity block, which we then write out!  Done!

 

To summarize, for the "reconstruct write" method, you need to:

* write out the data block while simultaneously reading in the data blocks of all other data drives

* calculate the new parity block from all of the data blocks, including the new one (very short)

* write out the parity block

 

That's a write and a bunch of simultaneous reads, a calc, and a write, but no platter rotation wait!  Now you can see why it can be so much faster!

 

The upside is it can be much faster.  The downside is that ALL of the array drives must be spinning, because they ALL are involved in EVERY write.

 

So what are the ramifications of this?

* For some operations, like parity checks and parity builds and drive rebuilds, it doesn't matter, because all of the drives are spinning anyway.

* For large write operations, like large transfers to the array, it can make a big difference in speed!

* For a small write, especially at an odd time when the drives are normally sleeping, all of the drives have to be spun up before the small write can proceed.

* And what about those little writes that go on in the background, like file system housekeeping operations?  EVERY write at any time forces EVERY array drive to spin up.  So you are likely to be surprised at odd times when checking on your array, and expecting all of your drives to be spun down, and finding every one of them spun up, for no discernible reason.

* So one of the questions to be faced is, how do you want your various write operations to be handled.  Take a small scheduled backup of your phone at 4 in the morning.  The backup tool determines there's a new picture to back up, so tries to write it to your unRAID server.  If you are using the old method, the data drive and the parity drive have to spin up, then this small amount of data is written, possibly taking a couple more seconds than Turbo write would take.  It's 4am, do you care?  If you were using Turbo write, then all of the drives will spin up, which probably takes somewhat longer spinning them up than any time saved by using Turbo write to save that picture (but a couple of seconds faster in the save).  Plus, all of the drives are now spinning, uselessly.

* Another possible problem if you were in Turbo mode, and you are watching a movie streaming to your player, then a write kicks in to the server and starts spinning up ALL of the drives, causing that well-known pause and stuttering in your movie.  Who wants to deal with the whining that starts then?

 

Currently, you only have the option to use the old method or the new (currently the Auto option means the old method).  But the plan is to add the true Auto option that will use the old method by default, *unless* all of the drives are currently spinning.  If the drives are all spinning, then it slips into Turbo.  This should be enough for many users.  It would normally use the old method, but if you planned a large transfer or a bunch of writes, then you would spin up all of the drives - and enjoy faster writing.

 

Tom talked about that Auto mode quite awhile ago, but I'm rather sure he backed off at that time, once he faced the problems of knowing when a drive is spinning, and being able to detect it without noticeably affecting write performance, ruining the very benefits we were trying to achieve.  If on every write you have to query each drive for its status, then you will noticeably impact I/O performance.  So to maintain good performance, you need another function working in the background keeping near-instantaneous track of spin status, and providing a single flag for the writer to check, whether they are all spun up or not, to know which method to use.

 

So that provides 3 options, but many of us are going to want tighter and smarter control of when it is in either mode.  Quite awhile ago, WeeboTech developed his own scheme of scheduling.  If I remember right (and I could have it backwards), he was going to use cron to toggle it twice a day, so that it used one method during the day, and the other method at night.  I think many users may find that scheduling it may satisfy their needs, Turbo when there's lots of writing, old style over night and when they are streaming movies.

 

For awhile, I did think that other users, including myself, would be happiest with a Turbo button on the Main screen (and Dashboard).  Then I realized that that's exactly what our Spin up button would be, if we used the new Auto mode.  The server would normally be in the old mode (except for times when all drives were spinning).  If we had a big update session, backing up or or downloading lots of stuff, we would click the Turbo / Spin up button and would have Turbo write, which would then automatically timeout when the drives started spinning down, after the backup session or transfers are complete.

 

Edit: added what the setting is and where it's located (completely forgot this!)

[aptalca]

Thanks so much for the detailed explanation. I was always wondering what turbo write meant, now I get it.

[DZMM]

Thanks for a good write up.  I'm going back to the old method as I think it'll make my machine quieter and will rely on my cache drive for when I need 'faster' writes until the true Auto is developed.

[WonderfulSlipperyThing]

Thanks for the writeup. Presumably auto in the future will use turbo write if all the drives happen to be spinning already. All of my drives are currently spinning but auto is still using read/modify/write which is why I came to this post, didn't realise that feature wasn't actually implemented yet.

 

It's a great feature and works very well but I'm really looking forward to being able to get better control over it. If all the drives are spun up already it would be great if it just used reconstruct write. Also I think for many people they don't really care that much about write speeds unless they're actually actively doing something themselves so having it just enabled for SMB shares or something would also be really useful.

 

Either way thanks for the explanation, makes a lot of sense and seems to make a huge speed difference!

[Scottathon]

Very interesting, I wasn't aware of this feature. Is there any reason to not enable this if I always keep my disks spinning?

[itimpi]

Very interesting, I wasn't aware of this feature. Is there any reason to not enable this if I always keep my disks spinning?

If you never spin down your disks, then this is likely to improve performance for writes.

[trurl]

This really should be in a different forum, seems more like v6 support than a topic for Lounge. Maybe you could put it in the V6 FAQ.

[RobJ]

This really should be in a different forum, seems more like v6 support than a topic for Lounge.

Yeah, I couldn't decide where to put it, at all, so finally went for the safe choice.  V6 support was the other best choice.  I was hoping more would have opinions on that, yours is the first.  I'll wait just a bit more, then probably move it there.

 

Maybe you could put it in the V6 FAQ.

I did think about that, haven't felt strongly enough about it I guess.  If it starts to get buried pages deep somewhere, then yes, that should probably be done.

[garycase]

As for where in the forum it's placed, here is probably fine for now -- at least until the "Auto" feature is actually implemented.  No real reason to put more focus on it until then ... although it's much quicker than the standard read/modify/write method, it's still a good bit slower than using a cache drive -- but has the advantage that all writes are immediately fault tolerant (unlike with a cache unless it's a fault tolerant btrfs array).

 

I really like the "Auto" setting, since it would make it trivial to switch to "high speed" writes if you're getting ready to add a lot of data to your array (Just click on "Spin Up" in the GUI before doing the writes); and otherwise only the minimal number of drives would be spun up.

 

[gubbgnutten]

Nice writeup. Next time I need to link it I'm sure I will remember where it is

 

Looking forward to see what "auto" can do if and when it is implemented, maybe I can actually stop toggling write mode manually then.

 

... although it's much quicker than the standard read/modify/write method, it's still a good bit slower than using a cache drive -- but has the advantage that all writes are immediately fault tolerant (unlike with a cache unless it's a fault tolerant btrfs array).

 

...and even if it is a fault tolerant BTRFS array, keep in mind that the mover is a weak link in the chain (perfectly illustrated by the recent ReiserFS snafu).

 

[3psus]

* Another possible problem if you were in Turbo mode, and you are watching a movie streaming to your player, then a write kicks in to the server and starts spinning up ALL of the drives, causing that well-known pause and stuttering in your movie.  Who wants to deal with the whining that starts then?

Very interesting, I wasn't aware of this feature. Is there any reason to not enable this if I always keep my disks spinning?

If you never spin down your disks, then this is likely to improve performance for writes.

 

I feel like Scott's question deserve's a little more attention. From what Rob wrote, it seems like turbo write could worsen read operation currently going on while other disks are spun up, like when watching a movie. In my environment, disks are never spun down anyways so spinning them up for a small write isn't an issue.

 

My main question is: if all disks are always spinning, am I going to be penalized in read operations if I turn turbo write on? Is the wait time mentioned in the movie example only caused by the time it takes for other disks to spin up or is there any other variables in play?

 

Thanks for this great article Rob!

[garycase]

If all disks are already spinning you shouldn't notice any slowdowns in streaming.  It WOULD require a few extra seeks on the disk that was currently streaming, but these are fast enough that there shouldn't be any issues with a movie stream.  It's the spin-up delay that can cause problems ... and since you don't have anything to spin up, you'd be fine.

 

 

[3psus]

If all disks are already spinning you shouldn't notice any slowdowns in streaming.  It WOULD require a few extra seeks on the disk that was currently streaming, but these are fast enough that there shouldn't be any issues with a movie stream.  It's the spin-up delay that can cause problems ... and since you don't have anything to spin up, you'd be fine.

 

I understand, thanks for the clarification. Streaming aside, would there be any noticeable impact on reads going on, wether it be in latency or in read speed? Or is it marginal to the point that it wouldn't be noticeable?

[garycase]

As I noted above, you'd likely not notice any practical impact.

 

However, there IS an impact -- it's just that it's likely masked in terms of the "feel" to the user.

 

Consider a stream from a particular disk (any disk).    In the absence of any other activity, the "next" read is almost always on the same track as the last one, or perhaps the adjacent track.    If there is simultaneous write activity on different disk, using the "normal" write mode; then there's zero impact on those reads.    But if turbo write is on; then for each write the disk you're currently streaming from has to seek to the appropriate cylinder for the write activity (to read the current contents); and then seek back to the area of the streaming activity for the next read of that stream.

 

Total access time for a modern disk is typically in the 12-15ms range (seek plus latency plus actual data transfer).  So with two extra accesses, there's an effective delay of about 30ms between reads for the streaming data.  Assume that instead of just streaming a video, the "streaming" activity was copying a file from the server, which would be FAR more demanding of the system.  If it was on a disk that was fast enough to saturate a Gb network, that would mean you were pulling 120MB/sec of data -- or, about 3.6MB in 30 msec.    Most modern multi-TB drives have at least a 64MB, and often larger, cache -- so they would almost certainly have enough data already in the cache to satisfy the next read request with no delay at all.  If the "next" read required a seek (i.e. was on the next cylinder) that might result in a very minor delay, but almost certainly not enough to cause any notable impact.

 

If you forced a LOT of activity you could perhaps force a situation where you MIGHT notice an occasional minor "glitch" in a stream -- i.e. if you were streaming 4 or 5 different streams, all from the same disk; and were simultaneously doing 3 or 4 writes to the array from different clients using turbo write, you MIGHT notice the difference.  But even then I doubt it.

 

[3psus]

Great answer Gary, huge thanks!

 

Definitely turning turbo write on now.

[BartDG]

I'm very curious to know what kind of speed gain can be achieved by using Turbo mode.  Has anybody already gone through the trouble of copying a set of files first old school style, then again using Turbo write and then compare the speed increase?

 

On the other hand, if I get this correctly: this doesn't make any difference when writing to the array when using a cache disk, right?  It only speeds things up when writing to the disk array directly? (I'm guessing because writing to the cache disk doesn't use parity anyway?)

 

[garycase]

Haven't measured this, but I'd expect Turbo write would nearly double your write speeds.    Note that for a "normal" write there are 4 disk operations involved (or 6 with dual parity) ... the current contents of the disk being written to need to be read, as well as the contents of the parity disk(s);  then the actual data is written and the parity disk(s) is/are updated ... with a full rotation of the disks involved between the read and write operations.

 

With turbo write there's only one operation per disk => every disk except the disk being written to and the parity disk(s) are read in the appropriate sector for the parity calculation;  then the disk being written to and the parity disk(s) are updated.  So there's no waiting for an entire disk rotation.

 

Note that there are actually two reads from the "uninvolved" disks on a dual parity system, since each sector needed for the appropriate parity calculation needs to be read ... but these reads can both be done during the same disk rotation.

 

Since rotational latency is a significant component of the access time of a disk (8 - 11 ms for a full rotation depending on the speed of the disk), eliminating a full rotation is a significant gain in net write speeds.    The initial seek and latency is likely slightly longer than what is being saved, so I'd not expect a doubling of write speeds, but it should be pretty close in most cases.

 

You're correct that if you're using cached writes there's no difference in the initial write speed to the array.  Turbo writes would still speed up the mover process, but that happens "behind the scenes" so wouldn't be apparent to the user.

 

 

[BartDG]

Got it!  Thanks very much for your detailed answer Gary!

[ZataH]

Anyone experience slow read from unraid when using the Turbo Write?

 

I get 100+MB/s when i write to unraid (Without cache), but it swings from 50-100 when reading from unraid.

[JorgeB]

Anyone experience slow read from unraid when using the Turbo Write?

 

I get 100+MB/s when i write to unraid (Without cache), but it swings from 50-100 when reading from unraid.

 

See if this helps:

 

http://lime-technology.com/forum/index.php?topic=53689.msg517002#msg517002

[ZataH]

Anyone experience slow read from unraid when using the Turbo Write?

 

I get 100+MB/s when i write to unraid (Without cache), but it swings from 50-100 when reading from unraid.

 

See if this helps:

 

http://lime-technology.com/forum/index.php?topic=53689.msg517002#msg517002

 

Thanks for that I had it to SMB2, and changing it to SMB2_02 fixed it.

I have been running with _02 before. Just read somewhere that SMB2 should be fine, guess it was not.

[CraziFuzzy]

So it seems that aggressive use of a cache drive should really work well with turbo mode, correct?  Those little nuisance writes could be done to the cache, and the whole array only needs to be spun up for turbo writes when the mover runs, correct?  Is this what people see in practice?  With that in mind, would it make sense to have it turn on turbo writes when mover starts, and turn it back off when mover is complete?

Edited March 6, 2017 by CraziFuzzy

[stacyy11]

Thanks for the explanation. Now I get it what turbo write meant.

[RobJ]

On 3/6/2017 at 4:24 AM, CraziFuzzy said:

So it seems that aggressive use of a cache drive should really work well with turbo mode, correct?  Those little nuisance writes could be done to the cache, and the whole array only needs to be spun up for turbo writes when the mover runs, correct?  Is this what people see in practice?  With that in mind, would it make sense to have it turn on turbo writes when mover starts, and turn it back off when mover is complete?

 

I have no personal experience, don't use a cache drive, but what you suggest sounds correct.  Toggling Turbo Write on should speed up Cache:Yes moves (but not Cache:Prefer moves).

 

However, some perspective would be good.  The Mover is designed to do these moves at a time when the system has the least usage, generally an early morning moment when you know everyone of its users will be sleeping.  So do you really care how fast it is then?  You're sleeping!  What you might care more about is that a number of additional drives had to be spun up too, that normally would have been sleeping too.

[CraziFuzzy]

15 hours ago, RobJ said:

 

I have no personal experience, don't use a cache drive, but what you suggest sounds correct.  Toggling Turbo Write on should speed up Cache:Yes moves (but not Cache:Prefer moves).

 

However, some perspective would be good.  The Mover is designed to do these moves at a time when the system has the least usage, generally an early morning moment when you know everyone of its users will be sleeping.  So do you really care how fast it is then?  You're sleeping!  What you might care more about is that a number of additional drives had to be spun up too, that normally would have been sleeping too.

I suppose it would depend on number of drives involved, and how much faster the mover will run with turbo writes vs. traditional read/write.  I mean, if you've got a minimal array of 2 data drives and 1 parity, and using turbo means spinning up 3 drives instead of 2, but those 3 are spun up for half the time, it will end up with fewer drive-hours up and running each night.