Please dont kill me....

 

Is there anything that would allow (in the works, or currently exists) the mover to multithread the movement of documents to the spinning rust?

 

Scenario:

I have a 1TB NVME cache drive and 10ish 16TB drives in an array. The cache drive is 100% full and the mover is active. it seems to grab files one at a time and move them. So my overall speed to move the files are reduced to the write speed of a single drive. since my drives are set as "most free" it will copy the next file to the next drive. if it could do them in parallel, that would be tons faster than it currently is. 

 

is this something that is possible or feasible?

Just now, TheSkaz said:

So my overall speed to move the files are reduced to the write speed of a single drive. since my drives are set as "most free" it will copy the next file to the next drive. if it could do them in parallel, that would be tons faster than it currently is. 

Writes to the parity array will always be limited by the speed of the parity drive(s). Not much point in trying to write to multiple data drives simultaneously when the parity drive is the bottleneck.

did not consider that... ok then.

 

thanks!

On 9/22/2021 at 7:05 AM, JonathanM said:

Writes to the parity array will always be limited by the speed of the parity drive(s). Not much point in trying to write to multiple data drives simultaneously when the parity drive is the bottleneck.

While this may be true for cache > array but what about array > cache would it still be the same limitation if the share is spread over multiple drives?

1 hour ago, IronBeardKnight said:

While this may be true for cache > array but what about array > cache would it still be the same limitation if the share is spread over multiple drives?

Not sure what you are asking. If the write destination is to the parity array drives, it's even worse if the share is spread out. Each write to the individual data drives in the parity array is also calculated and written to the parity drive. So if you write to disk1 and disk2 at the same time, the parity drive is thrashing back and forth to the 2 physical locations that map to the data written to the 2 different disks.

 

Pools have no such limitation, the parity array parity disk is not touched for pool writes.

One scenario (which is more and more likely / feasible / cost-effective these days) where multi-threaded write-operations to the Array would indeed yield a *very large* performance increase, is where you have a ton of slow platter drives (I use a bunch of 2 TB, 5400 RPM laptop drives for very low power use, as streaming a media file is a sequental read and does not need high spindle RPM), and one or two very fast SSD / PCIE drives for parity.

 

I have two 4 TB PCIE drives as my parity drives, so I can upgrade my platters to 4 TB eventually and not have to replace my parity drives.

Those PCI drives can do 5GB / sec each, easily, and millions of IO/s (which means they will never, ever be the bottleneck, even with 22 spindle operations going in parallel).

 

So there is already a use case (mine) and with the falling price/GB of SSD / PCIe drives, it seems like this would be a more and more attractive option for unraid users 

 

If only we had a multithreaded write capable array....

 

Since the answer to this question (from what I've seem) has typically been "it won't gain anything due to the parity drive bottleneck...", and that is demonstrably not true, anymore... Is there any other reason why this feature couldn't be added?

Or has this use-case already come to your attention and it is already being worked on? (this would be amazing news for quite a few of us, I imagine).

 

To put it another way, from the company's perspective...

The old adage "Big / Fast / Cheap... pick two" can effectively be retired with the (fairly cheap) hardware combination I mentioned above, combined with Unraid's efficient parity application.... all that is missing is the multi-thread-write ability.

Since the company would have to invest a fairly minor amount of time/effort to make the Array write operations multi-thread capable, and that would *significantly* raise the performance ceiling that has traditionally been the __single__biggest__drawback__ of Unraid, compared to other NAS implementations.... Why aren't they falling all over themselves to get his feature implemented? 

It seems obvious (to me, anyway) that the almost *certain* increase in adoption and corresponding sales revenue (from removing the one metric that keeps unraid from being a contender against other NAS options, in many cases) makes this the easiest cost v. benefit analysis in the history of ever ever...

 

But maybe I'm missing something obvious... who knows.   

I am curious to find out though.

 

Thanks!

-Sean