Mover Array to Cache, Small Files

[Kuusoft]

Hi,

 

I am trying to move close to 70TB of small files (approximately 20KB - 1MB) from Array (20 disks with 2 parity) to a ZFS with RAIDZ1 (3 groups of 6 drives).

 

I turned on the mover log and I can see it is moving the file slowly. I researched on the forum where it make sense to "linear" the process when it is moving from Cache --> Array because it needs to recalculate the parity, but no one really mentioned situations where I want to move from Array --> Cache with lots of small files.

 

Is there a better way to do this other than waiting for the mover to move each file? Perhaps, maybe it can move multiple files (especially I have 88 cores controller with 512GB of DDR4 ECC memory).

 

Any suggestion on what I can do?

 

Thank you

[cYnIx]

I would use the rsync terminal command to move that many files. There are lots of flags for rsync. Your command may look something like: `rsync -av --progress --remove-source-files /mnt/disk#/share/ /mnt/cache/share`

When that command is finished you will likely have many empty folders to remove, You can do so with a command like `find /mnt/disk* -type d -empty -delete`

 

If you don't have the rsync command you can get it with the Nerd Tools Plugin.

 

You can also use the Midnight commander program on terminal to move files, but it would not have the checksum verification of the move like rsync does. 

 

Hope that helps.

[Kuusoft]

Hi @cYnIx,

 

It works really well!

 

Now, I have a inverse question for you.

 

After recreating the array, I want to move some of the files (many files) back from Cache --> Array. Instead of using mover to move it slowly, will it be true to use the same command inversely. For example

 

1) Set the Primary storage of a share from a cache to Array

2) Run `rsync -av --progress --remove-source-files /mnt/cache/share /mnt/user/share`

3) Clean up `find /mnt/cache/share -type d -empty -delete`

 

Will that work?

 

Thank you

[itimpi]

25 minutes ago, Kuusoft said:

Hi @cYnIx,

 

It works really well!

 

Now, I have a inverse question for you.

 

After recreating the array, I want to move some of the files (many files) back from Cache --> Array. Instead of using mover to move it slowly, will it be true to use the same command inversely. For example

 

1) Set the Primary storage of a share from a cache to Array

2) Run `rsync -av --progress --remove-source-files /mnt/cache/share /mnt/user/share`

3) Clean up `find /mnt/cache/share -type d -empty -delete`

 

Will that work?

 

Thank you

that could go wrong and lead to data loss as you are mixing a physical device and a User Share in the same command.   The target needs to be of the form /mnt/disk#/share where # represents the disk you want it to end up on.

[Kuusoft]

2 hours ago, itimpi said:

that could go wrong and lead to data loss as you are mixing a physical device and a User Share in the same command.   The target needs to be of the form /mnt/disk#/share where # represents the disk you want it to end up on.

 

Ok, i see, so I simply specify the disks directly instead of going through /user/ folder. I was thinking that if it go through /user/ folder it will let Unraid do the work of finding the optimal disks. But, after reading your comment, it make sense, because it will end up have multiple copies of the file (the original one on cache and the new one at destination, both will appear under user share).

 

So, in this case, it might be better to just let the mover run if it is going from cache --> array with lots of small files, right?

[itimpi]

2 hours ago, Kuusoft said:

But, after reading your comment, it make sense, because it will end up have multiple copies of the file (the original one on cache and the new one at destination, both will appear under user share).

Even worse as the Linux level is not aware of User Shares it might try and overwrite the copy on the cache with itself which ends up truncating the file to 0 length.