Jump to content
jonp

Turbo Write

61 posts in this topic Last Reply

Recommended Posts

Tom M, please add details to this feature before we release 6.0, but essentially, this is the ability for smaller arrays to write data faster without caching.

Share this post


Link to post

I would love to have the ability to turn this on/off on demand and per some schedule.

 

For example during a busy part of the day, say 8am to 6pm, I want turbo write enabled.

after 6pm turn off turbo write so drives can spin down per the spin down timer.

 

 

 

Share this post


Link to post

IIRC one of Tom's considered MO's for turbo write would be to have it active if all drives are already spun up, and off otherwise.  Given that rule-set, it seems like your request could be satified by scheduling drive spin policy.  The same could be said for, "I want to be able to toggle turbo-write manually".  The answer would be, manually spin up all drives, get to writing, and then they'll just spin-down on their own and take turbo-write with it.

 

For the people with large arrays for whom turbo-write might actually be slower, there would of course still need to be a master toggle to disable turbo-write.

 

Just random thoughts.

Share this post


Link to post

That might work for me.

 

However, once turbo writes are enabled, the drives that do not get writes may never spin down if other drives are being written.

 

For example.

 

Torrents are writing to disk 1 24/7.

Documents are written to disk 2 from 8am to 9pm.

Music is writing updates to disk 3 from 6am to 9pm.

 

The documents and music disk will never spin down if torrents are written all day long.

 

If we had an auto spin up / down and turbo write schedule manager like the mover it would help.

I still see the need to manually turn off turbo write on a frequency.

Share this post


Link to post

ohhhh I think I see what you mean.  Otherwise unused drives will be kept spun-up because of turbo-write activity on other drives?  Yeah I think you're right then that an explicit turbo-write schedule would be the best way to deal with that.

Share this post


Link to post

Can that be turned on now in V6?  I thought I had tried on an earlier beta, and it just rejected the command.

Share this post


Link to post

Can that be turned on now in V6?  I thought I had tried on an earlier beta, and it just rejected the command.

 

I would assume so.

I don't have v6 running anywhere.

 

You can check with the following command.

 


root@unRAID:# cat /etc/unraid-version 
version=5.0.5

root@unRAID:# find /usr/src -name md.c -exec grep 'md_write_method' {} \; 
int md_write_method       = READ_MODIFY_WRITE; /* write algorithm (either 0 or 1) */
        printk("md_write_method=%d\n", md_write_method);
                if (!strcmp( "md_write_method", name)) {
                                        md_write_method = value;
                                md_write_method = READ_MODIFY_WRITE;

or


root@unRAID: # find /lib -name 'md-mod.ko' -print | xargs strings | grep 'md_write_method'
md_write_method=%d
md_write_method
md_write_method

Share this post


Link to post

I am guessing no...

 

root@Tower:/usr/src# find /usr/src -name md.c -exec grep 'md_write_method' {} \;

int md_write_method      = READ_MODIFY_WRITE; /* write algorithm [no longer tunable] */

 

root@Tower:/usr/src# find /lib -name 'md-mod.ko' -print | xargs strings | grep 'md_write_method'

md_write_method

Share this post


Link to post

Some idea's regarding scheduling.

 

I know many people use the crontab -l  and crontab command directly in scripts.

Another way of doing this is with a static crontab file specifically for this task.

 

If you drive a crontab formatted file in /etc/cron.d/ this will be slurpped up in cron when it is checked for changes.

It is not as immediate as a direct crontab command, yet it lets you create a set of static cron entries and rsync them from a config directory so they are incorporated automatically into the schedule.

 

Example:

 

root@unRAID:/# ls -l /etc/cron.d/

total 8

-rwxrwxrwx 1 root root 92 2014-08-25 13:00 md_write_method*

-rwxrwxrwx 1 root root 41 2014-02-26 22:31 parity_check*

 

root@unRAID:/# more /etc/cron.d/*

::::::::::::::

/etc/cron.d/md_write_method

::::::::::::::

00 14 * * * /root/mdcmd set md_write_method 1

00 23 * * * /root/mdcmd set md_write_method 0

::::::::::::::

/etc/cron.d/parity_check

::::::::::::::

00 00 27 * * /root/mdcmd check NOCORRECT

 

and the respective log entry for a change I just made.

 

Aug 25 14:00:01 unRAID kernel: mdcmd (972): set md_write_method 1

Aug 25 14:00:01 unRAID kernel:

 

Just some ideas for people to build on.

Share this post


Link to post

I am guessing no...

 

root@Tower:/usr/src# find /usr/src -name md.c -exec grep 'md_write_method' {} \;

int md_write_method      = READ_MODIFY_WRITE; /* write algorithm [no longer tunable] */

 

root@Tower:/usr/src# find /lib -name 'md-mod.ko' -print | xargs strings | grep 'md_write_method'

md_write_method

 

Hmm, well I sure wouldn't mind an autotune feature, but not being able to turn it off is going to cause me allot of grief.

It will force all my disks to stay spinning all the time.

 

It also means that if I'm downloading (writing) or converting a file (writing) on one of my disks, the reads on my other disk could potentially suffer.

 

Case in point when I download a torrent at high speed it causes a huge amount of IO.

I would not want turbo write impeding my ability to read documents, mp3's or videos on the other disk.

Yet when I had to do massive mp3 tagging, I turn on turbo write since it can improve my interactive file server experience.

Share this post


Link to post

One simple way to ensure "turbo-writes" don't interfere with normal drive spin-downs is to have UnRAID simply do this:  Automatically use "turbo-write" IF all drives are currently spinning;  otherwise do not.

 

So ... to turn this feature on, you simply click the Spin Up button.    Then, if you're using the system actively, the writes themselves will keep drives from spinning down, since the spindown timer won't expire.  When the system is idle for long enough to spin down the drives, it simply won't use "turbo-write" anymore.

 

Want to turn the feature off?  Just click the Spin Down button  :)

Share this post


Link to post

One simple way to ensure "turbo-writes" don't interfere with normal drive spin-downs is to have UnRAID simply do this:  Automatically use "turbo-write" IF all drives are currently spinning;  otherwise do not.

 

So ... to turn this feature on, you simply click the Spin Up button.    Then, if you're using the system actively, the writes themselves will keep drives from spinning down, since the spindown timer won't expire.  When the system is idle for long enough to spin down the drives, it simply won't use "turbo-write" anymore.

 

Want to turn the feature off?  Just click the Spin Down button  :)

 

You miss the point where one of my disks is busy 24/7/365. 

Using the aforementioned logic forces all drives in the array to stay up even if the other drives are not being used.

 

One write will cause reads on every drive with a subsequent write to parity.

 

 

I can't press spin down for all of my drives. Otherwise all of the network mounted filesystems fail on other machines.

 

NFS mounts will hang, thus hanging processes on other machines.

Also, the timeout of samba and windows will cause the drive letter to go offline, thus causing all my torrents to fail.

It's not that simple when you use your unRAID server as a full fledged file server instead of just a media warehouse / archive server.

Share this post


Link to post

Didn't realize spinning down the drives would cause all those issues.    If I do that while I'm streaming media, it simply causes pauses on the various systems that are doing the streaming (until the impacted drives are spun back up) ... but the network connections are all still active (all SMB).

 

Sounds like dedicated "Turbo-Write On" and "Turbo-Write Off" buttons are a better idea  :)

 

Is there a good reason to even do this, instead of just using a fault-tolerant BTRFS SSD cache pool to get even faster writes?

 

Share this post


Link to post

Didn't realize spinning down the drives would cause all those issues.    If I do that while I'm streaming media, it simply causes pauses on the various systems that are doing the streaming (until the impacted drives are spun back up) ... but the network connections are all still active (all SMB).

 

Sounds like dedicated "Turbo-Write On" and "Turbo-Write Off" buttons are a better idea  :)

 

Is there a good reason to even do this, instead of just using a fault-tolerant BTRFS SSD cache pool to get even faster writes?

 

I have a few HP Microservers. For torrents, I do not want them to be written to an SSD's cache pool.

Plus I don't have the space or equipment to do this at the current moment.

 

Besides, why buy hardware, when the current tune-able software actually works pretty well for me now.

 

On/Off/Auto is a good idea. auto alone will just give me grief.

 

Besides, I could see there being an issue for very wide arrays.

All drives are spun up and now turbo write auto enables, then other performance issues start cropping up.

Share this post


Link to post

At least for the "massive copying case" I could imagine that unRAID toggles the mode on its own.

Since it has to know where to allocate data on the share (high water, etc.), I suppose it knows how big the incoming amount of data is.

Then why not set a threshold (adjustable) for the incoming data where the turbo-mode is automagically set?

Share this post


Link to post

At least for the "massive copying case" I could imagine that unRAID toggles the mode on its own.

Since it has to know where to allocate data on the share (high water, etc.), I suppose it knows how big the incoming amount of data is.

I do not believe that this is the case - just that a new file (of unknown size) is about to be created.    That is the purpose of the Min Free Space setting so that one can ensure that there is enough space on a drive for the largest file that will be written.

Share this post


Link to post

Only the utility doing the actual copying [Windows Explorer;  TeraCopy; or whatever you happen to use]  would actually "know" the size of the file it's copying.  And even then, it wouldn't know how many additional files you might be planning to copy immediately afterwards.

 

But YOU know that info ... so a simple "Turbo Copy ON" button would let you tell UnRAID to use this mode if you were planning to copy a large amount of data.

 

Share this post


Link to post

or scheduled to coincide with backup operations :)

 

Sure, that would be a good time.  Which brings another nice feature to mind => it would be nice if there was a way in Windows (or from a Mac for those who are Mac-centric)  to send a command to UnRAID to turn turbo-write on or off.    Then an automated backup script could start off by turning on turbo write;  then doing the backups; and then turning off turbo write.    My backup script currently turns on the backup server (via WOL);  waits a couple minutes; then does the backup.  It'd be easy to modify it to turn on turbo write after that initial wait.    If the turbo write "auto" mode worked as I noted earlier (i.e. if all drives are spinning, use turbo write), then this would automatically happen in my case anyway (since the server has just been turned on, all the drives will be spinning).

 

 

 

 

 

Share this post


Link to post

or scheduled to coincide with backup operations :)

 

Sure, that would be a good time.  Which brings another nice feature to mind => it would be nice if there was a way in Windows (or from a Mac for those who are Mac-centric)  to send a command to UnRAID to turn turbo-write on or off.

As long as you can do it in a telnet session, you can do it from a windows batch file.

Share this post


Link to post

or scheduled to coincide with backup operations :)

 

Sure, that would be a good time.  Which brings another nice feature to mind => it would be nice if there was a way in Windows (or from a Mac for those who are Mac-centric)  to send a command to UnRAID to turn turbo-write on or off.    Then an automated backup script could start off by turning on turbo write;  then doing the backups; and then turning off turbo write.    My backup script currently turns on the backup server (via WOL);  waits a couple minutes; then does the backup.  It'd be easy to modify it to turn on turbo write after that initial wait.    If the turbo write "auto" mode worked as I noted earlier (i.e. if all drives are spinning, use turbo write), then this would automatically happen in my case anyway (since the server has just been turned on, all the drives will be spinning).

 

This is easier then you think.  There are a few ways to do this.

 

A command line ssh tool from windows, Allows remote execution, exposes root password unless you set up ssh keys.

A telnet scripting tool. As above allows remote execution, exposes root password

 

Another option is a php form/webgui form

Then a curl command on the windows host. exposes password.

 

A simple shell and a config entry in /etc/inetd.conf on  the unRAID side.

Then a curl command on the windows host to send the appropriate command

 

There is also netcat, however I would suggest acquiring curl for windows, it could open up allot of opportunity for automatic unRAID operations remotely or doing other things remotely.

 

Once you learn how to use curl, you'll have allot of potential at your disposal.

 

If I could get clarification from Tom as to the nature of this tunable in unRAID6 (looks like it was removed) then we can build remote access functions.  For all we know there may be a webGui page in the works.

Share this post


Link to post

This option would perfectly fit in some kind of "unRAID Control Panel" utility for windows.

I'm thinking of a small app/widget running in background having all the unRAID stuff onboard.

I had a Qnap that had this kind of utility...

 

Toggling the "turbo mode" would be best in terms of workflow when integrated in the windows explorer context menu (right click).

Share this post


Link to post

Sooo now that we have a much more configurable GUI, what are the chances of adding this to the gui?  I know we talked a lot about ways to autmoate its usage, but that is a for another time. This would just need to be a selectable option with an indicator.  Waddayasay?

Share this post


Link to post

This is what I have so far.

 

I rsync a file called md_write_method into /etc/cron.d from my /boot/local/etc/cron.d/

 

30 08 * * * /root/mdcmd set md_write_method 1

30 23 * * * /root/mdcmd set md_write_method 0

#

# * * * * * <command to be executed>

# | | | | |

# | | | | |

# | | | | +---- Day of the Week  (range: 1-7, 1 standing for Monday)

# | | | +------ Month of the Year (range: 1-12)

# | | +-------- Day of the Month  (range: 1-31)

# | +---------- Hour              (range: 0-23)

# +------------ Minute            (range: 0-59)

 

go script commands.

 

rsync -rv /boot/local/etc/cron.d/ /etc/cron.d/

find /etc/cron.d -type f -exec chmod a-wx {} \;

 

 

I would like to see.

 

1  Current status      Enabled/Disabled.

2.  Button to Enable  NOW

3.  Button to Disable NOW

 

Some type of method to enable/disable on a schedule.

 

Currently I drop a file into /etc/cron.d called md_write_method.

 

a webGui interface could read this file note enable/disable times and provide a gui to update it.

cache it on the flash

restore on reboot.

save to flash and /etc/cron.d  on change.

 

I suppose the quickest implementation would be

 

provide a skeleton file on flash or via ramdrive defaults folder.

restore from flash on reboot if it exists.

general edit text box for changing /etc/cron.d/md_write_method file

save to /etc/cron.d and flash on change.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.