Will this destroy my appdata etc.?

[NLS]

So, I am doing an upgrade to my server and among other things my SSD cache will be swapped for an M.2.

So I did the "cache YES" thing and started mover.

Mover though is very slow (maybe the safety checks it does), so I decided to "assist" it by using server's own local MC to make some moves myself, trying to not conflict with the current working folder of the mover.

 

This more or less goes smooth, but I noticed a thing, even before I started the moves.

The claimed space used by appdata/system etc. (not I use a docker folder NOT docker file), was much larger (double) the space actually used in my cache. This leads me to believe much of the work is done using links.

 

By doing some moves using MC, do I "destroy" those links?

I haven't finished to actally move things to data disk, let alone move back to cache, but I want to be ready in case something is not as it should.

My VMs should be ok.

My appdata, I don't know, but I suspect there is a process to actually "recreate" the whole docker set I have, so this can be "cleaned" up.
What about "system" and the millions of files and folders there?

 

Note that already MC reported some files as already in the destination (which I am 100% sure is not a matter of the mover as it is doing something else), which seem to be the same files so I overwrite them to be sure. Also a single time it reported me a link will not work over disks, which I skipped, so mover can (possibly) handle it.

 

Note also that I don't have "follow links" ticked in move... and I feel I should?

 

Edited July 2, 2022 by NLS

[Squid]

The way that I would move appdata from one drive to another would be to use appdata backup and restore.  It'll take probably probably 1/20th the time of mover.

 

Stop Docker and VM's in settings  Backup, swap drives, restore

 

Anything is system just do a normal copy.  There's only a couple of files

 

Don't delete anything off of the original cache drive until you have the system operational again.

[trurl]

3 minutes ago, Squid said:

Anything is system just do a normal copy.  There's only a couple of files

 

1 hour ago, NLS said:

I use a docker folder NOT docker file

1 hour ago, NLS said:

What about "system" and the millions of files and folders there?

 

 

[NLS]

16 minutes ago, Squid said:

The way that I would move appdata from one drive to another would be to use appdata backup and restore.  It'll take probably probably 1/20th the time of mover.

 

Stop Docker and VM's in settings  Backup, swap drives, restore

 

Anything is system just do a normal copy.  There's only a couple of files

 

Don't delete anything off of the original cache drive until you have the system operational again.

 

Yeah I thought too late about that. You already had the solution.
 

Mover was running and MC was also moving.

And still is, but is close to the end.
Few things remain, possibly things that somehow conflicted. We'll see.

 

I guess I'll wait and see.

 

I noticed a "supervisor.conf" file is reported many times as existing and is overwritten. I guess this is normally a link, not sure if it remains a link.

Also some moves in system report "cannot overwrite directory /mnt/disk4/syst~g/systemd/user - invalid cross-device link (18)... which I skip... Actually do "skip all" so I don't know if it would say this for other files.

 

Not sure what I will have in the end.

 

[Squid]

Missing the use of a folder for the image.  (thx @trurl)  Regardless of whether you're using a docker image or a folder, you're going to be better off always in simply recreating it via Previous Apps.  Even using an image, it tends to not move very well between drives

[NLS]

New problem is that there are things in /system/docker/docker/btrfs/subvolumes/ that I cannot move and mover also cannot move.
My moves finished (with the skips mentioned above), mover "finished" also, so I started again to see if it will in the end move what MC couldn't.

And although there are just like 2GB in cache left behind, it takes its time, not sure if it actually does anything.

 

I'll try copy instead of move and we'll see. I am not sure how docker can recover from various possible errors.

I'll probably also use backup to just backup those remaining things and see if I need to recover them in the end.

 

---

I saw your new reply while writing this. I thought about using "previous apps" (I first need to keep my appdata or something, right?) but the problem is that system also holds some VM data IIRC...

 

---

EDIT: Actually 4GB, 8000 folders and 149000 files remain in cache folder.  (all inside subvolumes folder)
Don't know what to expect...

 

Edited July 2, 2022 by NLS

[trurl]

Not clear, did you disable Docker and VM Manager in Settings? 

 

Nothing can move open files. 

[Squid]

2 hours ago, NLS said:

/system/docker/docker/btrfs/subvolumes/

As I re-corrected myself above, not worth your time and aggravation.  Mover will take literally hours to move the files, and you have to do it twice.  Simply recreate the docker folder and reinstall via Previous Apps.  5 Minutes and you're back and running vs an all day adventure.  Only if you've created custom networking via the command line will you have to recreate that network, which will add another couple seconds to your time.

[NLS]

3 hours ago, trurl said:

Not clear, did you disable Docker and VM Manager in Settings? 

 

Nothing can move open files. 

 

Of course.

 

2 hours ago, Squid said:

As I re-corrected myself above, not worth your time and aggravation.  Mover will take literally hours to move the files, and you have to do it twice.  Simply recreate the docker folder and reinstall via Previous Apps.  5 Minutes and you're back and running vs an all day adventure.  Only if you've created custom networking via the command line will you have to recreate that network, which will add another couple seconds to your time.

 

Well too late, I am already moving files back. At least it is fast (reading from array and writing to M.2, vs, reading from SSD and writing to array before).

BUT READ ON:

 

Even if moving files back works in the end, I *WILL* need to recreate the thing.
Why? Because indeed links seem to have been broken and replaced by actual files, so when I was using 34% of my 500GB SSD, I am already using 39% of the 500 M.2 WITHOUT having finished the unbelievable amount of "subvolumes" files.

But here is where I need your expertise:

 

Deleting appdata and re-installing containers is one thing. (actually I need help here too, see below)
But, AFAIK system is also used for VM config data.

So the questions are:

1) To rebuild my container set I probably need to delete appdata contents and go back to "install Previous Apps", right?

 

2) BUT the main problem is system folder. This is where the huge folders are. How do I clean this up (both for containers to re-install AND re-use my existing VMs)???

 

Do I delete both folders? (appdata + system) Do I delete specific subfolders?

 

EDIT:

An attempt to help myself:
a) VMs will be fine since I moved back "domains" and libvirt that is in "system". (EDIT: Yes this works ok)
b) Emptying (or moving contents) of appdata and system/docker folders PROBABLY will leave me without containers right? And then I can "cleanly" re-populate this by using "install Previous Apps", right? But where are the custom templates stored? Where is the info of previous apps stored?

(problem is I didn't note down some customizations I did - so I probably need to finish the move anyway and then "destroy" after I note down what I needed)

 

 

The good thing is that the actual hardware upgrade went fine (moved from an very old intel mobo with i7 4771, to an X570 mobo with 5600G CPU). More cores, more RAM, more speed, M.2 instead of SSD.
 

Edited July 2, 2022 by NLS

[trurl]

1 hour ago, NLS said:

delete appdata contents

No, the appdata is what you need for your applications to remember how the application was setup, what they know about, and what they were doing.

 

It is docker.img (or in your case the docker folder) that can be reinstalled from previous apps. That is just the executable code for the containers which are simply downloaded and installed again according to the templates (settings on the Docker page) created on your flash drive when you installed them before.

[trurl]

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Installing_Docker_Applications

[NLS]

I did re-create docker folder after all. Indeed much better.

There is a weird thing though.

Although my new cache now has what it had before, is same size M.2 as the previous SSD and docker folder is supposedly clean as the same containers got re-installed... It shows 41% usage instead of 34-35% (when clean from other cache things) that the old had.
Why I have spent 7% more (35GB)?