Trying to move about 650GB from cache to array to upgrade cache drive. Running for over 12 hours and not even half done yet...is this normal?

[relink]

As the title says I am trying to upgrade my cache disk so I am moving everything to the array. 
 

I stopped docker and VM services first, and set all shares to use cache yes. The mover is running but it’s been over 12 hours and it’s not even half done. I would have expected a transfer time closer to 6 hours. 
 

so far I have used “renice” and “ionice” to set the priorities of the processes involved in moving higher, but it doesn’t seem to have made much difference. 
 

I was hoping to do that cache upgrade this morning, but at this rate I’ll be lucky if I can do it by tonight. 

[JorgeB]

It's normal if there are a lot of small files, like Plex metadata for example.

[relink]

16 minutes ago, JorgeB said:

It's normal if there are a lot of small files, like Plex metadata for example.

Crap...that’s exactly what it is. Tons of tiny little files. 

[JorgeB]

Mover is very inefficient when dealing with those, next time I suggest using the console or a tool like midnight commander.

[relink]

What command would I need to run to move everything from cache to array?

 

unless there’s some reason I shouldn’t, I will gladly stop the mover and do that instead. 

[relink]

I just realized that midnight commander is built in to unraid. Woul I just stop the mover and then use that to move the folders from “/mnt/cache” to “/mnt/user”?

[Dovy6]

I've done that. User "mover stop" in the terminal. Unfortunately millions of tiny files will take forever to move no matter how you do it.

[StevenD]

Whenever I upgrade my cache, I ALWAYS use the CA Backup/Restore plugin.  Its is way, way faster for backing up and restoring all the appdata.  My appdata folder is about 300GB, most of which is Plex metadata. It does it in about 25 minutes.

[itimpi]

1 hour ago, relink said:

I just realized that midnight commander is built in to unraid. Woul I just stop the mover and then use that to move the folders from “/mnt/cache” to “/mnt/user”?

NO!

 

/mnt/user is for User Shares that include the cache drive and if you try this will end up losing your files as they will be truncated to 0 bites as they try and copy on top of themselves.

 

You need to either use a named drive such as /mnt/diskX for the target or  if you want it to be a User Share use /mnt/user0 as the target as that is the user share files that are only on the main array omitting the cache drive

[relink]

9 hours ago, itimpi said:

NO!

Good thing I ended up not doing that at the last second. 
 

instead I decided to add my new nvme drive as a second cache, and use MC to move everything from the old one to the new one and reassign the cache for each share. 
 

however, this is still ungodly slow. Midnight commander “claims” it’s transferring at 15MB/s ...it’s not, there’s no way. It’s maybe 15Mb/s if that. 
 

I calculated out the transfer time beforehand and this should have been done hours ago, yet it’s still going.

Edited March 22, 2021 by relink

[relink]

Well it was a total success! After quadruple the amount of time I thought it was going to take my new cache drive is up and running. I think next time I’ll take the advice above and just restore from my backup. 

[SimonMisc]

On 3/21/2021 at 10:02 AM, itimpi said:

NO!

 

/mnt/user is for User Shares that include the cache drive and if you try this will end up losing your files as they will be truncated to 0 bites as they try and copy on top of themselves.

 

You need to either use a named drive such as /mnt/diskX for the target or  if you want it to be a User Share use /mnt/user0 as the target as that is the user share files that are only on the main array omitting the cache drive

So I just found this thread while remembering vaguely that there's a no-no somewhere when it comes to transferring files manually, in this case the 0-byte truncation issue you mentioned.

 

What's still not clear to me is which directories are safe to move files between. As an example, I'm hoping to move everything from my Cache pool to a new Temp Pool drive. Can I safely move for example /mnt/cache/appdata to (newly created) /mnt/temp/appdata?

[JorgeB]

10 minutes ago, SimonMisc said:

What's still not clear to me is which directories are safe to move files between.

You can move from a disk share to another disk share and from a user share to another user share, just not from a disk share to a user share or vice versa.

 

/mnt/cache and /mnt/temp are both disk shares so that's fine, anything with /mnt/user/share_name is a user share.