Why is Turbo write being disabled when copying between disks in array?

[TexasUnraid]

Ok, I just got finished building parity and have been cleaning up some things, still figuring out how I am going to setup the drives with unraid.

 

I do have a question though. I will be needing to move things from hard disk to hard disk during this sorting out phase (many TB's worth).

 

I enabled turbo write and did my first move between drives with parity enabled and am getting strange results.

 

I was under the impression that with turbo write, I would basically get full performance of the drives, they would all just need to be spun up and the parity would mirror any writes.

 

I expected this to happen in the case of copying between disks by all drives except the destination drive, reading the same bits to calculate parity and of course destination drive and parity writing at full speed.

 

I never expected any drives to both read and write at the same time.

 

This is what I am seeing on both the drive I am writing to and the parity, they are exactly the same:

 

 

Notice that they go in spurts exactly as expected, full speed with writes only but then they will have equal parts read and write for long periods between??

 

This is another drive in the array:

 

 

As you can see it matches the parity/destination drive reads with no writes, as expected.

 

Here is another strange thing, the source drive is reading at full speed (~160mb/s) the whole time?

 

 

Anyone know what is going on? Why is anything being read on the parity or destination drive?

 

Any way to speed this process up?

 

Before adding the parity I was able to copy files full speed between drives no problem.

Edited July 26, 2020 by TexasUnraid

[JorgeB]

5 hours ago, TexasUnraid said:

I was under the impression that with turbo write, I would basically get full performance of the drives, they would all just need to be spun up and the parity would mirror any writes.

Currently, and against some users wishes, including my own, Unraid disables turbo write when i/o from multiple array disks is detected.

[TexasUnraid]

6 hours ago, johnnie.black said:

Currently, and against some users wishes, including my own, Unraid disables turbo write when i/o from multiple array disks is detected.

Thats what it seemed like was happening, good to know I was not crazy.

 

Why would that be the only option? Seems like transferring between disks is one of the times that turbo write would have the most benefit? Is there no way to force turbo write on?

 

Why does turbo write seem to be working at points during the copy and disabled at others? Nothing else was happening during the copy.

[TexasUnraid]

I am also curious why the parity drive is being read? Seems like it would only ever be written to unless it was rebuilding something? I can't think of a reason to read the parity besides a parity check or rebuild?

 

Does it constantly check parity when other drives are writing? Seems inefficient?

Edited July 26, 2020 by TexasUnraid

[itimpi]

15 minutes ago, TexasUnraid said:

I am also curious why the parity drive is being read? Seems like it would only ever be written to unless it was rebuilding something? I can't think of a reason to read the parity besides a parity check or rebuild?

Any time you are not running in Turbo mode the parity drive is read as part of a write operation.  This section of the online documentation explains how the Unraid write modes work.

 

[JorgeB]

10 minutes ago, TexasUnraid said:

Why would that be the only option?

Because it's not user set unfortunately, I did ask to use the auto option for that, and that if reconstruct write is enable it should only use reconstruct write, there's also a report about it, you can add your opinion there, the more people ask the more likely Tom will take a look at it.

 

15 minutes ago, TexasUnraid said:

Why does turbo write seem to be working at points during the copy and disabled at others? Nothing else was happening during the copy.

It will alternate as it detects multiple device activity in the array.

 

12 minutes ago, TexasUnraid said:

I am also curious why the parity drive is being read?

Because normal writing mode (read/modify/write) requires that parity be read before any block is written.

[TexasUnraid]

34 minutes ago, itimpi said:

Any time you are not running in Turbo mode the parity drive is read as part of a write operation.  This section of the online documentation explains how the Unraid write modes work.

 

Ok, I see now, I was thinking about everything from a turbo write standpoint where it would not need to know what the old parity was.

[TexasUnraid]

34 minutes ago, johnnie.black said:

Because it's not user set unfortunately, I did ask to use the auto option for that, and that if reconstruct write is enable it should only use reconstruct write, there's also a report about it, you can add your opinion there, the more people ask the more likely Tom will take a look at it.

 

It will alternate as it detects multiple device activity in the array.

 

Because normal writing mode (read/modify/write) requires that parity be read before any block is written.

So a tiny amount of activity to the array will switch it out of turbo write mode? It seemed to go in a pattern, which is really strange since there was no other major activity at that time. Just a few little blips you can see in the graph. I am guessing there is a time delay from when it switches modes? If it was only during the active steam that would not be so bad to drop speed for a second at a time.

 

I see it used to allow turbo write in 6.7 when copying disk to disk.

 

I am happy to add my voice to this, I am really liking how I am setting things up right now but it only works if I am able to easily transfer files between disks manually. Too much of a organization freak to let unraid randomly put data wherever it wants lol.

 

Is that old bug thread the best place to voice it?

[TexasUnraid]

I was just experimenting with using the cache and moving files from there to a particular disk in the array but turbo write also seems to be disabled when doing this? The turbo write plugin specifically has the option to enable turbo write during mover?

 

That seems like it would be the primary use of turbo write? This puts a massive damper on unraid, getting ~1/3 the speed it should for no reason plus thrashing the hard drives for no reason. In fact the thrashing of the drives I would say bothers me as much or more then the speed.

[JorgeB]

12 hours ago, TexasUnraid said:

was just experimenting with using the cache and moving files from there to a particular disk in the array but turbo write also seems to be disabled when doing this?

That shouldn't happen, unless there are some reads/writes going to another array disk at the same time.

[TexasUnraid]

4 hours ago, johnnie.black said:

That shouldn't happen, unless there are some reads/writes going to another array disk at the same time.

There should not be anything else happening although I didn't take steps to ensure there was not a few kb blib during this test. That is bound to happen though, if that prevents a 100GB transfer from going into turbo write mode, whats the point of turbo write?

 

In the test transfer turbo write didn't come on at all that I noticed, the whole thing was done with normal writes. To be honest I don't know that turbo write has worked for more then a few seconds at a time no matter what I have tried.

 

I will try some more isolated tests today to see if I can get it working at all. Maybe disable the plugin and set it manually in the disk settings?

[JorgeB]

I use turbo write for all my serves and I can transfer for hours continuously using it, except for a few seconds when it changes the disk it's writing to, if for you it's changing to rmw mode constantly you have something else accessing the disks, e.g dockers, etc.

[JorgeB]

3 minutes ago, johnnie.black said:

you have something else accessing the disks

or are using the most free allocation setting, which will keep changing disks making writes overlap for a few seconds.

[TexasUnraid]

Nothing accesses these disks unless I tell it to really, outside of someone on the network browsing or accessing something.

 

Testing now, if I use krusader and do not use the /user folder turbo write seems to work as expected from the cache to a particular disk. Once this transfer completes I will try it over SMB again and via the /user folder to see if it replicates the results from yesterday. I did restart this morning, so maybe it cleared something out of it's system.

[TexasUnraid]

Did some more testing and over SMB to the /user file system or direct disk share, turbo write is still being disabled and nothing else is accessing the drives, verified by file activity plugin.

 

EDIT: Hmm, made a change mid-transfer by disabling the turbo write plugin and manually enabling reconstruct write and suddenly turbo write started working over smb.

 

So maybe this is the turbo write plugin that is at fault? Does it still work?

 

The strange part then was I started getting seconds at a time where the transfer would just lock up and nothing would happen. Although I think this falls back to the SMB issues I was investigating before.

Edited July 27, 2020 by TexasUnraid

[JorgeB]

11 hours ago, TexasUnraid said:

So maybe this is the turbo write plugin that is at fault? Does it still work?

Don't know, never used it.

[itimpi]

11 hours ago, TexasUnraid said:

So maybe this is the turbo write plugin that is at fault? Does it still work?

I know it has been deprecated for use with 6.9.0 beta 25.

[TexasUnraid]

5 hours ago, itimpi said:

I know it has been deprecated for use with 6.9.0 beta 25.

Interesting, sad that was a nice feature on the surface since I have my drives split up in such a way that most of them should not even need to be spun up on a normal day (part of why I am not just letting unraid handle the folder splitting).

[Squid]

18 hours ago, itimpi said:

I know it has been deprecated for use with 6.9.0 beta 25.

It has?!?!?!

[itimpi]

10 hours ago, Squid said:

It has?!?!?!

Now that you query it, I think I am getting mixed up with the Mover Tuning plugin?   

[TexasUnraid]

I am on the beta now, might try reinstalling the turbo write plugin and see if it works now.

[jwoolen]

On 7/27/2020 at 7:47 AM, JorgeB said:

or are using the most free allocation setting, which will keep changing disks making writes overlap for a few seconds.

This is what was causing turbo write to be disabled for me. Switching to high water allowed turbo write to work as expected. Thanks JorgeB!

[apraetor]

On 9/22/2020 at 8:45 PM, jwoolen said:

This is what was causing turbo write to be disabled for me. Switching to high water allowed turbo write to work as expected. Thanks JorgeB!

THIS. I've been trying to run down an issue and this is the answer. When moving a single large file from cache to array, turbo write performance is great. If there's more than 1 file, then it often drops to a constant ~30 MB/s read + ~30 MB/s write to all disks by the second file. The thrashing really kills throughput. From the looks of it, Turbo Write isn't smart enough to realize that the current multi-disk IO is actually to consecutive writes briefly overlapping. It's a shame -- my reason for using Most-Free Allocation is to maximize read performance by spreading data across as many disks as possible. It also begs the question, why is the OS reporting the mover's write is complete before it's actually finished writing out that file to the array..?!

[trurl]

31 minutes ago, apraetor said:

From the looks of it, Turbo Write isn't smart enough to realize that the current multi-disk IO is actually to consecutive writes briefly overlapping.

Do you know how Unraid parity works, and how Turbo Write differs from the other way parity is updated realtime?

[trurl]

3 hours ago, apraetor said:

using Most-Free Allocation

Most inefficient for writing and likely to keep more disks spinning whether reading or writing.

 

Highwater is default because it is a good compromise to get files distributed eventually without constantly switching between disks just because they temporarily (and maybe only briefly) have the most free.