Hi all,

There seems to be a problem with parallelism when writing to the array using Reconstruct Write. I have data in my NVMe cache that needs to be transferred to the array. My server has a large amount of RAM which is used as a disk cache automatically. The data in the NVMe cache is split by folders to different drives in the array. If I move one folder using Midnight Commander to the user0 folder I get full speed writes:

However, the MC move finishes quickly as it is copying to the RAM cache. When I start another folder move to the array (which uses a different drive), the write performance tanks and the disks are noticeably louder:

It appears that the RAM cache doesn't operate as a FIFO buffer and that Turbo Writes are switched off when two operations are executed.

Now you may say, "simply wait until disk activity has stopped before starting another move", it's difficult to know when this has happened programmatically, which means that when Mover executes, it defaults to Read/Modify/Write if it's moving data to multiple disks.

Is there a way that Mover can stop additional moves until the single disk activity has finished? Or can Unraid be clever enough to calculate parity for all drives for multiple disk writes without resorting to Read/Modify/Write?

Thanks,

Andy.

Mover is intended for idle time. Default is once per day in the middle of the night.

It is impossible to move from fast cache to slow array as fast as you can write to fast cache, and trying to write and move at the same time just makes it worse.

If you need to write more at one time than cache can hold, don't cache.

I assume by "RAM cache" you mean the standard way that Linux uses extra memory for buffering writes. I'm pretty sure it is FIFO, but when it fills, it has to start flushing the buffer by writing the data to the destination disks. So it has to wait on disk writes before the buiffer can take any more.

And if you are doing turbo to multiple disks, it probably won't be faster since there will be seeks when it switches between disks.

4 hours ago, trurl said:

I assume by "RAM cache" you mean the standard way that Linux uses extra memory for buffering writes. I'm pretty sure it is FIFO, but when it fills, it has to start flushing the buffer by writing the data to the destination disks. So it has to wait on disk writes before the buiffer can take any more.

Yes, the Linux disk RAM cache is what I mean. It's not FIFO, you can test this by copying data to a share that's housed on one array disk, then copy to another share that's housed on a different array disk. I wanted it to wait on disk writes before starting to write to the 2nd disk. That to me is the logical way to handle a cache for an Unraid array (due to performance limitations). For an array that's on an NVMe raid, concurrent writes may be possible, but I would still assume that it's best to get the first job done before starting others.

4 hours ago, trurl said:

Mover is intended for idle time. Default is once per day in the middle of the night.

It is impossible to move from fast cache to slow array as fast as you can write to fast cache, and trying to write and move at the same time just makes it worse.

If you need to write more at one time than cache can hold, don't cache.

Not really a great answer, suppose I have data in the NVMe cache that needs to be placed on 14 different disks due to the share setup for the array. Currently that data will be concurrently written to all the drives at the same time using Read Modify Write. Making the cache FIFO means that each disk is written to one at a time.

4 hours ago, trurl said:

And if you are doing turbo to multiple disks, it probably won't be faster since there will be seeks when it switches between disks.

I'm not sure you understand what I am talking about. I'm not doing Turbo to multiple disks. Unraid is due to the way the RAM disk cache works. If what you say is correct, Mover won't ever work with Turbo Writes - just in case the data is split across disks.

I also like large RAM cache, my main Unraid have 256GB memory. For RAM cache, it never work as FIFO, it more like many-to-many.

7 hours ago, AndyS said:

Now you may say, "simply wait until disk activity has stopped before starting another move", it's difficult to know when this has happened programmatically, which means that when Mover executes, it defaults to Read/Modify/Write if it's moving data to multiple disks.

I almost stop using cache since day one, it is not because automatic switch between RMW or RW. In that time there are no SSD, and due to I have limited disk count, so cache never my option.

As my disk count have expand much, so I could try different storage combination, but I still don't like cache ( main reason also because this build not always on ).

One of advantage of large RAM cache when move many file to array in multiple disk ( no parity ) , I will temporary change array allocation to "most-free", then files will cache in RAM and move to array in multiple disk most time, it will reduce overall completion time.

Anyway, I like strip pool more then mover, my current version are no parity array (3 data disks) and two large strip pool (each have 12 disks)

No parity array are make since last yrs, I re-arrange 16 disks. ( change small size disk in array to large disk size in array, so 2p6d to 0p3d )

Edited February 26Feb 26 by Vr2Io

TurboWrite is automatically disabled if there are writes to more than one device. For best results, avoid that, since writes to parity will also overlap, e.g., don't use the most free allocation method, which will require constant disk swtiching and parity overlap.

14 minutes ago, JorgeB said:

TurboWrite is automatically disabled if there are writes to more than one device. For best results, avoid that, since writes to parity will also overlap, e.g., don't use the most free allocation method, which will require constant disk swtiching and parity overlap.

Is it not possible to make the linux RAM cache operate in FIFO mode? Then we can use Turbo Writes.

26 minutes ago, Vr2Io said:

I also like large RAM cache, my main Unraid have 256GB memory. For RAM cache, it never work as FIFO, it more like many-to-many.

I almost stop using cache since day one, it is not because automatic switch between RMW or RW. In that time there are no SSD, and due to I have limited disk count, so cache never my option.

As my disk count have expand much, so I could try different storage combination. But I still don't like cache ( main reason also because this build not always on ).

Anyway, I like strip pool more then mover, my current version are no parity array (4 data disks) and two large strip pool (each have 12 disks). No parity array are make since last yrs, I re-arrange 16 disks. ( change small size disk in array to large disk size in array, so 2p6d to 0p3d )

Have you disabled the RAM cache or are you not using the NVMe/SSD cache?

Just now, AndyS said:

Have you disabled the RAM cache or are you not using the NVMe/SSD cache?

I never touch the behavior of RAM cache ( for my understand it couldn't stop ), the main build are spinner disk only, I need capacity and try my best to got throughput next.

For doing FIFO, the first method in my mind was using Windows file copy from Unraid cache to Array, you can start in several copy task and Windows will queue the copying. But Windows couldn't handle if there are too much file.

** forget it, it couldn't overcome if file have cache in RAM, that's why you ask how to stopping RAM cache. **

Edited February 26Feb 26 by Vr2Io

If you install the "tips and tweaks" plugin you can adjust some RAM caching settings.

56 minutes ago, Kilrah said:

If you install the "tips and tweaks" plugin you can adjust some RAM caching settings.

Thanks, I don't really want to disable the cache, just stop it from writing to multiple disks at the same time.

Then you're probably gonna have to dig into linux kernel workings to see if there's something that can be changed to help with that. Doesn't seem there is from a cursory look but reducing cache size or shortening the flush intervals could potentially help.

https://github.com/firmianay/Life-long-Learner/blob/master/linux-kernel-development/chapter-16.md

Edited February 26Feb 26 by Kilrah