Save to multiple disk

[Alphahelix]

First I do not know if this is the right place to post this idea, as it kind of sits between unRAID OS and Turbo write.

 

But here goes:

 

The scenario: Turbo write is active and the share you are transferring files to is covering more than one drive. Then it should be possible to have all the drive the share spans to be able to receive data at once. This would be beneficial when transferring large amount of data, like when mover is initiated. I cant see if this is would be beneficial without Turbo write, but it should be. This would benefit write speed a lot as multiple disk as written too at the same time.

 

Without being an expert, so I may be wrong in my assumption here, the only trade off is that the unRAID OS must calculate what data goes where on what disk. So CPU time should go up. Also there could be a lack before data is written.

 

I do not know if my theory is sound, but it should increase writing data to shares spanning multiple disks.

 

/Alphahelix

[Squid]

Top of my head:

 

- You'd have to have mover sit there and analyse where the file(s) are going to wind up.  If they are on different disks then write them concurrently, and thrash the hell out of the parity disk(s)

- Split levels and/or allocation methods chosen dictate the rules of where the files wind up.  The destination disks may wind up being significantly different depending upon the order that the files are written.

- Assume 2 data drives and 1 parity drive, and in a perfect world everything writes at the maximum rated speed of the drive (and the drives are rated for the same speed).  Since you're doing 2 simultaneous  writes to data drives and 2 simultaneous writes to the parity drive, the net total bandwidth is half (because of the need to write to the parity drive).  Ignoring the thrashing and seek time on the parity drive, the total time taken to write the 2 files simultaneously equals the total time taken to write each file sequentially.

[Alphahelix]

Thank you for the elaboration.

Again I am far from an expert. So excuse my ignorance, but if I via shell or Krusader was to copy from one or more locations to two (to keep it simple) independent disks. Would it then half the speed? I always assumed (maybe incorrectly) that the parity worked by read then write. And in Turbo mode it would read from from all disks to calculate the parity, and it was done in the RAM. I just trying to understand it all.  

 

/Alphahelix

[itimpi]

The big difficulty with what you are trying to suggest it can only work if multiple conditions are ALL true:

• There are multiple read or writes occurring at the same time.   This could be extremely difficult to achieve within a single application.
• The different sectors that need to be read/written are all on the same track of all disks involved in the write.  If not then the reads/writes cannot proceed in parallel as a head movement will be required.

This feels like an idea that is almost certainly impossible to achieve in practise.    The Linux kernel will already be trying to do a level of optimisation on disk I/O where it can and is difficult to see how anything working at the application level can do any better.

[Squid]

Yes.  Because the disks only have so much bandwidth available.  And because parity has to be updated for both the data disks, it winds up being the limiting factor.  So effectively it has to do (ignoring the read existing data, seek times etc) 2 writes for every one to the data disks.  Since it can only write at say 200MB/s max and each data drive can write at 200MB/s, your effective speed is 100MB/s

 

Whether turbo write is enabled or not doesn't change the basic aspects.  And my examples are assuming that what Itimipi is saying is not ture (about the data drives writing to the same sector #s), as the possibility of that happening is about the same as winning the lottery.

Edited January 12, 2019 by Squid

[Alphahelix]

9 hours ago, Squid said:

... as the possibility of that happening is about the same as winning the lottery.

I like your humor...

 

And thank again for the clarification.