I'm in the middle of a big server HW upgrade (don't worry, I have backups of everything.) I'm near the end of the process without any major hardware issues. My last major step was to migrate some old btrfs cache drives to new ZFS NVMe drives. I'm following the new-ish SIO video that demonstrates how to migrate a drive to ZFS by using the mover to move everything to the array first, and then back to the newly formatted (on in my case, just new) ZFS pool.

 

However, after a long time, when I return to the machine, mover is still running. I turned on mover logging before the operation, and I can see that hours before I returned to the server, mover's last log entry was an unknown btrfs file in the cache drive.

 

At this point, I attempted to stop mover manually using the "mover stop" command at the CLI witch seemed to work. However, when I went to stop the array for a reboot (seemed like a good first step) the drives would not unmount. Manually calling "umount -l /mnt/docker_cache/" again, seemed to work (returned no errors) but I was unable to stop the array. Ended up doing a hard shutdown.

 

After a night of parity check (no errors), I did a btrfs scrub on the 2 cache drives I have (no errors.) 

 

I can also confirm that docker and VM services were both stopped before manually starting the mover (per the SIO video,) so I dont think there were in use file issues. Could it be a permissions thing?

 

Attached are my diagnostics after completing parity check and having the mover run again and get stuck again (stopped at what I think is about the same place, though the web browser based log doesn't show the exact place the mover stopped last night, but the diagnostics seem like they're at the same place.) I'll admit, I don't fully understand what these files are (something to do with the btrfs FS?)

 

Barring y'all's advice, my next best guess is to give up, and remove the old cache drives from the server, adjust my shares mappings to use the new ZFS drives (as planned) and restore my shares from backup (I backed up all the shares on the cache drives for just this type of contingency.) But I'd really like to know why this happened so I can prevent it in the future?

 

Maybe a permissions thing?

 

wondermutt-diagnostics-20240107-0936.zip

I should add that my first attempt to manually run mover failed miserably. Pressing "the button" in the GUI did nothing. The log never even showed a mover event. Upon research, I found this post.

 

I have the same boot error, and I've had the mover tuning plugin for years. So, I removed it and re-installed it from the app store. Witch didn't fix the issue (still have the bad softlink.) So, since I don't need the tuning right now, just removed the mover tuning plugin altogether. That got mover working again. That's what I've been using when this hang started. I don't think it's relevant, but thought I would share (and I'd love to find a way to get that link fixed since I DO need mover tuning once the server is back up and running from the upgrade.)

Just to close the loop on this if anyone finds it in the future, I never solved this, but had good backups, so I took a risk and just removed the old cache, set up shares on the new pools, and restored from backup. That actually had similar problems, but they were in non-critical files that I was able to manually remove and delete, and successfully complete the restore.

 

FWIW, I think the mover might have been choking on my Kasm container. Kasm uses docker-in-docker and requires bypassing the FUSE FS. So, I think the files in question are related to the docker in docker containers of some of the workspaces I setup in Kasm. Certainly, when  I had issues restoring from backup, it was these container files that caused the issues. I ultimately deleted my Kasm appdata folder from the backup (it didn't have anything important in it since the config was stored in the template on the flash drive) and re-installed Kasm on the new drive without any loss of data or functionality.

 

So, all in all, it's "fixed" though I don't really understand why it failed in the first place, and the way I got back to normal operation does bother me a bit for the future.

Edited January 12, 20242 yr by FirbyKirby