Turbo write


RobJ

Recommended Posts

On 3/8/2017 at 3:57 AM, CraziFuzzy said:

I suppose it would depend on number of drives involved, and how much faster the mover will run with turbo writes vs. traditional read/write.  I mean, if you've got a minimal array of 2 data drives and 1 parity, and using turbo means spinning up 3 drives instead of 2, but those 3 are spun up for half the time, it will end up with fewer drive-hours up and running each night.

 

I can't point to the studies that corroborate it, but it's long been my understanding that hours spinning is not as hard on the drive as number of times it has to spin up and down.  If the drive needs to spin up and down a few times, then it's my understanding that the drive will last longer staying spun up.  Minimizing drive hours is not as important as minimizing spin changes.  A spinning drive does use more power, and generate a little more heat, but will probably last longer with fewer spin ups and spin downs.

  • Upvote 1
Link to comment
12 hours ago, RobJ said:

 

I can't point to the studies that corroborate it, but it's long been my understanding that hours spinning is not as hard on the drive as number of times it has to spin up and down.  If the drive needs to spin up and down a few times, then it's my understanding that the drive will last longer staying spun up.  Minimizing drive hours is not as important as minimizing spin changes.  A spinning drive does use more power, and generate a little more heat, but will probably last longer with fewer spin ups and spin downs.

 

I'm not sure it is so much the number of times spinning up and down, as the temperature variations that impact longevity. (Guess they are somewhat correlated.)

 

For that reason, I keep my spindown time set to a relatively high number - 5 hours. Based on my usage pattern, I have found that is a good number and avoids spinning up a disk more than once a day in most circumstances. I think its better to have it run that long than to have it spinning up and down 3 or 4 times during a day and having the associated temperature fluctuations.I Jeff.I think the default is 1 hour. I very much recommend others to adjust based on their usage pattern, regardless of turbo write setting.

 

My opinion is that turbo write is a very good thing for initial loads, file system conversion (from RFS to XFS), backups, and other lengthy disk write operations. Those uses could save hours or even days in some cases. But for everyday I wouldn't use it, even if all the disks were already spinning. Maybe I'd feel different if I had only a handful of drives - but creating the level of activity on the server to write a file and save a few seconds or even minutes here and there seems overkill. My server is being used more and more for varied tasks, and I really enjoy that my accesses only minimally inconvenience each other. I can't remember last time I was drumming my fingers waiting for a write to finish. I do find my RAID0 parity setup, along with unRAIDs tuning that provides a nice burst of speed when a file starts copying, currently keep everyday performance very reasonable.

Link to comment
9 minutes ago, bjp999 said:

I wouldn't use it, even if all the disks were already spinning

 

I can't think of any reason not to use it in this case -- I think that's what the "auto" setting will (when implemented) do.   I agree I wouldn't spin drives up to use turbo write for just a few writes, but certainly if they were already spinning it'd be nice if the system would use the faster write method if it didn't cause any additional spinups.

 

 

Link to comment
 
I can't think of any reason not to use it in this case -- I think that's what the "auto" setting will (when implemented) do.   I agree I wouldn't spin drives up to use turbo write for just a few writes, but certainly if they were already spinning it'd be nice if the system would use the faster write method if it didn't cause any additional spinups.
 
 

There's a script here in the additional scripts for the user scripts plugin that does exactly that, and was going to make it in plugin form this weekend

Sent from my LG-D852 using Tapatalk

Link to comment

Based on discussions in another thread, I'm fairly sure that's what the "auto" setting for md_write_method => it's actually the default setting; but isn't really implemented -- it simply defaults to the standard write method at the moment.    But a plug-in to do it NOW is indeed nice.

 

Regardless of which way it's implemented (plugin or "auto"), the nice thing about that is you then don't have to change the setting to use turbo-write for large data transfers ... just click on "Spin Up" and the system will automatically use turbo write ... and once the drives are spun down writes will revert to the normal method.

 

Link to comment
5 minutes ago, Squid said:

Historically auto in unraid for any setting means enabled or disabled and not changing dynamiclly. In this case auto means it's automatically set to be disabled.

Sent from my LG-D852 using Tapatalk
 

??   Note that the "auto" setting is for a parameter called "md_write_method"  ... there's no "enabled" or "disabled" values for that => it's either "read/modify/write"  or  "reconstruct write"  (the latter is what's referred to as "turbo write").

 

... but even if the choices were "enabled" or "disabled", it's very counter-intuitive to have an "auto" setting unless it's enabling/disabling based on some state of the system.

 

There's NO reason to have a setting called "auto" unless it's actually making some kind of choice -- and the only choice that makes any sense with regards to this setting would be to use reconstruct write if all the disks are spinning; or the read/modify/write method otherwise.   I don't recall the specific thread, but this has been discussed before, and my understanding was the ultimately that's what it was intended to do.

 

Link to comment
2 minutes ago, johnnie.black said:

IIRC Tom mentioned the auto setting is for future enhancement implementation, it defaults to auto so that when that enhancement is made all users take advantage without needing to change the setting.


Yes, that's what I recall as well.   Just can't seem to find the thread where it was discussed.

Link to comment
IIRC Tom mentioned the auto setting is for future enhancement implementation, it defaults to auto so that when that enhancement is made all users take advantage without needing to change the setting.

Must've missed that one. I'll have to double down and re-read every thread on the forum again. I'm slacking off as of yet

Sent from my LG-D852 using Tapatalk

Link to comment
4 hours ago, garycase said:

 

I can't think of any reason not to use it in this case -- I think that's what the "auto" setting will (when implemented) do.   I agree I wouldn't spin drives up to use turbo write for just a few writes, but certainly if they were already spinning it'd be nice if the system would use the faster write method if it didn't cause any additional spinups.

 

 

 

I can think of a big reason ...

 

If I am writing in turbo mode to disk1, and want to play a movie from disk2, disk2 is going to be very busy and the movie will stutter..  If fact the contention on disk2 might seriously slow down the writes to disk1 in this scenario. Turbo write is bad for arrays that do multiple things at once.

 

If I am writing in regular unRAID write mode, the write to disk1 is only affecting disk1 and parity, and reads from disk2 are going to be unaffected.

Link to comment
2 hours ago, bjp999 said:

 

I can think of a big reason ...

 

If I am writing in turbo mode to disk1, and want to play a movie from disk2, disk2 is going to be very busy and the movie will stutter..  If fact the contention on disk2 might seriously slow down the writes to disk1 in this scenario. Turbo write is bad for arrays that do multiple things at once.

 

If I am writing in regular unRAID write mode, the write to disk1 is only affecting disk1 and parity, and reads from disk2 are going to be unaffected.

Not sure if that's a real issue.  If it was, I wouldn't be able to perform a parity check and still be able to watch a movie at the same time.  Modern hard drives transfer rates far exceed the bit rates on a movie.  Add to that most clients buffer at least a couple of seconds of video.

Link to comment

Agree -- any modern disk is easily fast enough that an active movie stream at the same time as a write is NOT going to cause any issues with the stream.   As Squid noted, you can easily see this by streaming a movie during a parity check.     In fact, just stream a movie from a disk you're writing to in normal read/modify/write mode -- which has twice as many disk I/O's as turbo write, so would be even more likely to cause an issue ... and even that almost certainly won't cause any problems.

 

Link to comment

My movies are pretty high bitrate (bluray images) and typically playing HD audio. Definitely had trouble when running a parity check at the same time. But its true that the bandwidth to play a movie is nothing compared to the bandwidth to do a copy. It might take only a few minutes to copy a movie that takes 2 hours to play.

 

But what if I'm doing something more demanding, like running md5 comparisons. (I sometimes run a bunch of whole disk md5 checks in screen sessions as an alternative to a parity check to confirm I'm not seeing any corruption). Or copying new files to a backup server. Or maybe running a transcode to download movies to a lightweight media. Lots of things can demand enough bit rate to cause the I/O to thrash if running in parallel to a lengthy write that demands all hands on deck.

 

But, as I said, for lengthy operations I would definitely turn on turbo write. I did my RFS to XFS conversion with parity disabled. If I'd had an option to do it with turbo write and get rear full speed writes, I might have done that.

Link to comment
  • 1 month later...
  • 8 months later...
  • 2 weeks later...
  • 2 months later...

I use a cache drive but I don't have mover enabled for anything. I like to move things myself once or twice a week. Is there any reason I couldn't turn on Turbo Write before I manually move from cache to array and then turn it off after I'm done?

 

No need to state the obvious reason that it requires more user effort.

Edited by KR1SeS
Link to comment

There's no reason you couldn't do that; but why?   Turbo-write is generally used to get the quickest possible writes directly to the array, so your data is immediately fault-tolerant and doesn't have to go through the cache.

 

Since you're caching all your writes, then the amount of time it takes to do the actual writes to the array is "hidden" from you anyway, since from the user's perspective that data has already been written to the array.    In most cases, those writes are also automatic ... that's the whole idea of the mover script.    ... nevertheless, you can simply turn on turbo-write; initiate the mover; and then later just turn off turbo-write.

 

 

 

Link to comment
  • trurl pinned this topic
  • 3 months later...
7 hours ago, koyaanisqatsi said:

I turned on turbo/reconstruct-write part way through a 600GB move from one disk to another, and this is what I got in stats. Roughly twice the write throughput

 

Turbo.thumb.png.987c966ee094fead612ffc132dbaca29.png

 

Cacti graphs:

sdc is parity

sdh is the source disk

sdi is the destination disk

 

 

 

1625446856_TurboThroughput.thumb.png.8b37631a4b842c5caf76fcf105d450e4.png

 

 

How do you generate these graphics?

Link to comment
  • 4 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.