Mover is incredibly slow

[showstopper]

I have seen a few posts about this but no real answer. My cache disk is failing and so i have a replacement. I am running the mover to get everything on the array, however, I started this 6 hours ago and it has done 15.4GB. Is there another way that I can do this, otherwise it could take days!

[JorgeB]

If it's mostly small files the mover won't be very efficient, use midnight commander or the Dynamix file explorer.

[showstopper]

1 minute ago, JorgeB said:

If it's mostly small files the mover won't be very efficient, use midnight commander or the Dynamix file explorer.

 

Can i stop the mover now though, or do i need to wait for it?
If i stop it does it cancel the move?

[JorgeB]

type

mover stop

 

[showstopper]

13 minutes ago, JorgeB said:

type

mover stop

 

 

OK - but can i then copy the files from cache to array? Or do some now exist on the array?

[JorgeB]

Some will already have been moved, move the rest.

[vw-kombi]

I just got bit by this - as planning on replacing the ssd cache with an NVME.

I calculated it would take days to move the 195GB.

I even stopped the mover and deleted a shed load of containers I was keeping around - got the size to 125GB.

After another 3 hours and it not even doing 10GB, I cancelled it.

The dynamix file explorer is now in use doing the same thing.

This is a great 'hack' that I wished I knew about yesterday.

 

Just an update - I was waiting 6 hours for a mover that would have taken days to copy to the array as part of spaceinvadersw zfs cache drive conversion process, and instead with the new 6.12 file explorer, I have done the move to array disk in 10 mintes, then re-created as zfs (1 minute), then the move back to the new cache drive in another 10 minutes.

 

I wasted half a day with the mover pain.  I understand its all the small files but still - what is being done under the convers if the file explorer can do it in a fraction of the time, then something should be done to improve this.

Edited October 24, 2023 by vw-kombi

[JonathanM]

7 hours ago, vw-kombi said:

what is being done under the convers if the file explorer can do it in a fraction of the time, then something should be done to improve this.

It is my understanding that while the file manager is "safe" and slower than mc for some actions like moving files from share to share on the same disk, the mover is "extra, extra, triple safe", so moves can take forever with 1000's of small files.

 

Hopefully while the work is being done to support pool to pool moves (as if this writing the release version of mover only works with array <-> pool) maybe the code will be better optimized.

[Rollingsound514]

On 10/24/2023 at 1:19 AM, vw-kombi said:

I just got bit by this - as planning on replacing the ssd cache with an NVME.

I calculated it would take days to move the 195GB.

I even stopped the mover and deleted a shed load of containers I was keeping around - got the size to 125GB.

After another 3 hours and it not even doing 10GB, I cancelled it.

The dynamix file explorer is now in use doing the same thing.

This is a great 'hack' that I wished I knew about yesterday.

 

Just an update - I was waiting 6 hours for a mover that would have taken days to copy to the array as part of spaceinvadersw zfs cache drive conversion process, and instead with the new 6.12 file explorer, I have done the move to array disk in 10 mintes, then re-created as zfs (1 minute), then the move back to the new cache drive in another 10 minutes.

 

I wasted half a day with the mover pain.  I understand its all the small files but still - what is being done under the convers if the file explorer can do it in a fraction of the time, then something should be done to improve this.

 

 

Thanks for the information, 

So to move you just go into cache drive and select all your folders like appdata and then just choose a disk destination like "Disk 3", and then when you replace the cache drive do the opposite? Will Unraid keeps track of the files in the share being moved without actually using "Mover"? Everyone seems to say just use Mover but it's insanely slow, but this method you used just "feels" kinda of risky.

 

Edit: Also, did the MOVE function delete the files from the destination automatically? Thanks

Edited March 16 by Rollingsound514

[vw-kombi]

4 hours ago, Rollingsound514 said:

 

 

Thanks for the information, 

So to move you just go into cache drive and select all your folders like appdata and then just choose a disk destination like "Disk 3", and then when you replace the cache drive do the opposite? Will Unraid keeps track of the files in the share being moved without actually using "Mover"? Everyone seems to say just use Mover but it's insanely slow, but this method you used just "feels" kinda of risky.

 

Edit: Also, did the MOVE function delete the files from the destination automatically? Thanks

maybe disable mover while doing this - mover does not actually 'keep track' of files - just when it runs is moves stuff configured in shared flagged with cache to array.  I have often moved this myself in between the mover times if re-aligning disks etc.