Hello,

 

I'm running a server with 3 x 16TB 7200rpm drives, with one being parity. I get around 270MB/s R/W when benchmarking these drives. I'm running an i3-10100, so I'm in no way CPU limited. I also have 32GB of RAM.

 

I am trying to move all the contents of an old share to a new one. I have used the built-in GUI "move" button, as Unbalanced does not allow for (as far as I know) moving files from one share to another.

 

Since these files combined weigh around 9TB of large Blu Ray remuxes, I have enabled turbo writes to speed up the process, by setting "md_write_method" to "reconstruct write". Since my setup only has 2 drives (excluding parity), the files are individually extremely large (30-50GB each) and data is already present on both drives (through FUSE), there shouldn't be too much IO overhead, and should be pretty much a best case scenario for sequential R/W.

 

There are essentially 2 problems:

 

Initially the transfer speed started out as around 100MB/s - itself nowhere near the 270MB/s the drives should be able to reach on large, sequential files like these. During parity operations, and writing to the array itself, write speeds top out at 80MB/s with read/modify/write, and around 100MB/s with reconstruct write.

 

Moreover, it estimated around 6 hours to finish the job, yet I wake up this morning with the move only 30% completed, and with transfer speeds crashing down to 20-30MB/s. This would take around 2 to 3 more days according to the GUI estimate.

 

Here are my questions:

1. What gives? Why are my 270MB/s drives limited to 100MB/s, when all drives are identical, turbo writes are enabled, and the files themselves are a best case scenario of single, huge files, as close as it can get to sequential R/W tests?
2. What caused the catastrophic drop in transfer speed from 100MB/s to 20-30MB/s?
3. What is the preferred way to move a large amount of data from one share to another? I assumed it would be the built-in GUI "move" button. Does Unbalanced allow from transfers from one share to another? Should I have preferred copying through Windows from one SMB share to another?
4. Why do transfer speeds in the move GUI not match disk R/W speeds in the "Main" tab?
5. Does RAM usage affect transfer speeds? I have 32GB of RAM, and killing a game hosting process which brought total use from 91% to 15%, has caused transfer speed to be much more erratic: highs now reach 70-80MB/s, lows now go down to 10MB/s. It's much more erratic now.

 

I'm not quite sure what info to include here, but I'll answer as soon as I can with any requested info.

 

Thanks in advance.

 

9 minutes ago, SinoBreizh said:

I have enabled turbo writes

Just in case you haven't read the full explanation of all that happens when the array is written, whichever mode is enabled:

https://docs.unraid.net/unraid-os/manual/storage-management/#array-write-modes

 

Moving is copying (a write operation) the data to the destination, and then deleting (also a write operation) the source.

 

So, 2 write operations being done for each move, and both write operations involve updating parity.

Just now, trurl said:

2 write operations being done for each move, and both write operations involve updating parity.

And seeks, the slowest part of that, between both of those write operations.

2 hours ago, trurl said:

Just in case you haven't read the full explanation of all that happens when the array is written, whichever mode is enabled:

I have read that page, which is where I discovered and then decided to enable turbo writes to begin with. However, it doesn't feature any examples or numbers (which I can understand, for several reasons). So I understand the fundamental process, the pros and cons of each approach, yet I still have exactly zero context to understand whether my numbers are expected, or on the contrary, worrying.

 

Correct me if I'm wrong, but a parity sync also involves a seek and write operation, yet my drives fly through those at around 200MB/s, slowing over time to 170-180MB/s. 20MB/s is an order of magnitude of a difference compared to a theoretical maximum of 270MB/s, and a real-world maximum of 200MB/s during parity checks. That can't be right can it?

5 minutes ago, SinoBreizh said:

Correct me if I'm wrong, but a parity sync also involves a seek and write operation, yet my drives fly through those at around 200MB/s

But dew head movements (the slowest part of the process) as the process is working through the drive sequentially.

3 minutes ago, itimpi said:

But dew head movements (the slowest part of the process) as the process is working through the drive sequentially.

So to hearken back to my initial questions:

1. The numbers I'm seeing are expected behaviour? 4 days to move 9TB from one share to another?
2. Using the "move" button in the Unraid GUI is still the most efficient way to transfer files from one share to another?

27 minutes ago, SinoBreizh said:

decided to enable turbo writes to begin with

Turbo write does not work with array to array transfers, though also keep in mind that DFM has been known to underreport the stats or be slower than other apps, but don't expect more than 40 to 60MB/s for array to array transfers with parity enabled.

10 minutes ago, JorgeB said:

Turbo write does not work with array to array transfers

Good to know, thank you.

11 minutes ago, JorgeB said:

keep in mind that DFM has been known to underreport the stats

Is this something that still affects the new "integrated" DFM in Unraid 7?

12 minutes ago, JorgeB said:

slower than other apps

By other apps, do you mean Krusader?

 

Final question: if I write a lot to my array, and have so few drives that having them spun up all the time isn't a dealbreaker, would this be a use case where I could benefit from the new ZFS filesystem?

15 minutes ago, SinoBreizh said:

Is this something that still affects the new "integrated" DFM in Unraid 7?

Yep, it's the same.

 

16 minutes ago, SinoBreizh said:

By other apps, do you mean Krusader?

Never used that one, but I generally recommend rsync or Midnight Commander (mc on the CLI)

 

16 minutes ago, SinoBreizh said:

would this be a use case where I could benefit from the new ZFS filesystem?

You may benefit from zfs in a pool, in the array it will be the same, or even slower since there's currently a known issue with zfs when used in the array.

 

Zfs in a pool, like raid1 or raidz2 would perform much better, though it's not only advantages, each one has some advantages and disadvantages, you can see below for the main ones:

https://forums.unraid.net/topic/131857-soon™️-612-series/?do=findComment&comment=1243850

 

 

1 hour ago, JorgeB said:

I generally recommend rsync or Midnight Commander (mc on the CLI)

Thank you, I'll look into those for the next batch I need to transfer.

 

1 hour ago, JorgeB said:

You may benefit from zfs in a pool, in the array it will be the same

Guess this might be another reason to upgrade to Unraid 7 then (array-less pools).

 

-----

 

On a side note, I stumbled on another post about painfully slow DFM transfers, which you answered back in 2022. It made me do some quick math regarding my own transfer: if the average parity disk write speed I'm seeing in the "Main" tab on the Unraid GUI was accurate (~150MB/s), this whole thing would have been done in around 19 hours.

 

The move has been running for almost exactly 24 hours at this point, and I'm only barely halfway through the 9TB. So the writes I'm seeing on the parity drive can't possibly be accurate, or they're not all contributing to the move order. (I currently have no other process working on the array). So it seems that the speeds reported by DFM are indeed accurate.

 

One thing I noticed looking more attentively, is the DFM speed sometimes spikes to around 60MB/s, which you mentioned was the realistic maximum for an intra-array move; yet it also comes crashing down as low as 5MB/s regulary. When you mentioned this 40-60MB/s maximum, was it sustained or peak speeds? What could explain such speed volatility? Errors in DFM reporting; inefficiencies inherent to the parity process, fragmentation issues (is defragging even a thing in Unraid)?

 

Either way, at least I'm seeing peaks which match your stated real-world maximum. The initial 100MB/s I was seeing must have been attributed to the large cache on these enterprise drives.

 

----

 

I'll run some tests another time: with the other tools you mentioned, without FUSE overhead, and within a pool of two drives to try and single out variables.

 

I'll mark this post as solved for now, as the 40-60MB/s you mentioned give me the context I need to understand the numbers I'm seeing on my end are not completely out of the ordinary.

 

Thanks for taking the time to answer!

2 minutes ago, SinoBreizh said:

f the average parity disk write speed I'm seeing in the "Main" tab on the Unraid GUI was accurate (~150MB/s

If you are referring to parity checks or rebuilds, which are totally sequential, then this is expected.

 

But data writes in general are not sequential, and seek times are the main reason for slow disk access in any system.

16 minutes ago, trurl said:

If you are referring to parity checks or rebuilds

I was referring to the speeds here:

 

Which consistently, throughout the entire run, do not match the speeds I'm seeing here:

 

How can the drives be writing faster than the speed at which the transfer takes place, throughout the entire run? And the math with the elapsed time seems to show the DFM estimate is accurate.

 

8 minutes ago, trurl said:

But data writes in general are not sequential, and seek times are the main reason for slow disk access in any system

In theory I'd agree, but these writes are about as sequential as it gets in real world usage, with large, 30-50GB files. I was surprised to see these large files write slower than when I back up hundreds of 20-30MB photos directly to the array, where I see around 100MB/s consistently. The only difference is this is a move operation, not a write from outside the array. So as JorgeB mentioned, turbo writes don't work within the array itself, and there's an extra initial read operation I assume.

 

So the slowdown has to be some combination of "read/modify/write's" innate slowness, the overhead/extra reads from a move operation rather than writing from an outside source, FUSE overhead from writing one share to another, and/or some funkiness with how DFM works.

 

Which is why I really need to take the time to test variables to see what, specifically, causes the slowdown.

11 hours ago, SinoBreizh said:

When you mentioned this 40-60MB/s maximum, was it sustained or peak speeds?

Sustained/average speeds.

23 minutes ago, JorgeB said:

Sustained/average speeds.

Something's definitely wrong then. DFM speeds have dropped to 9MB/s max this morning, and the time elapsed vs amount of data transferred math backs that up. It's now saying it needs 4 more days for the remaining 4TB. So almost a week in all to move 9TB from one share to another... And that's assuming it doesn't slow down more over time than it already did.

 

Since you seem more familiar than I am with DFM, the way it moves data is Copy and Delete right? So if I cancel the move, what's already been transferred will be in the new share, while the leftovers remain in the old share. Correct?

 

What happens to the file currently being copied? Since DFM is Copy and Delete, the original version of the current file should be intact and not deleted until it has been completed, correct?

 

I've also looked into rsync and MC, and it looks like the big speed improvement is because they first attempt a File Rename rather than Copy and Delete like DFM. Since all my user shares have no excluded disks and are spread out across my entire array, and I don't use disk shares, I should be fine with these limitations, correct? It doesn't matter that it doesn't actually move the file, just rename the path, since all shares are already spread to all disks.

 

Sorry for coming back with more questions, but I'm kind of starting to pull my hair out with how much it's slowing down, for no obvious reason.

 

Thanks in advance.

1 hour ago, SinoBreizh said:

Since you seem more familiar than I am with DFM

Not really, I've never used it, but I believe it uses rsync to copy and then delete the data.