I've been spending hours getting the system back to normal the past day and making good progress. Here's what happened and looking for guidance to complete the recovery and verify I'm in a better state than before. Running 6.12.10 as of right now.

 

Btrfs corrupted

 

Been having a problems for the last 6 months with my btrfs based cache constantly using 5GB of space daily. The first time back in the fall it reached 90% and was impacting applications that did not have enough free space. I was unable to find what was growing. `df` reports it was full and btrfs indicated the same thing. Hardly any metadata space was in use and all was in data. No subvolumes, no snapshots, tools say everything is healthy. Yet it was impossible to find what was growing. I had about 120GB of real data, but btrfs says I was use 360GB at this point. I ended up copying the entire contents of my cache to another drive, deleted what was on cache and it went back to 0%. I copied the data back and now `df` and btrfs report 120GB in use. WTH? The next day the slow growth was back and this week it got to 90% again. After more hours trying to troubleshoot and gave up and decided I needed to clear the cache, reformat it and rebuild it again, assuming there was some kind of underlying corruption from last year.

 

Errors clearing cache

 

Shutdown Docker and set all shared to use array as primary. My appdata would not move for some reason and that was weird. I ended up moving the data manual (rsync). One of my fatal errors was not checking things worked in the new space before deleting what was still left in cache--that's on me. I did manually move docker.img. I then tried to fire everything up and things got really weird. All of my docker containers were acting as if this was the first time I launched them--all of them. 

 

Recovery

 

Started rebuilding data from my CP backups and have most of it in place. Only data lost so far my Plex metadata (not the media in the array). I'm at a good stopping point with the recovery and want to get the less critical components, but I want to make sure I'm doing this right from the beginning. I need to stop docker, set appdata to use cache as primary and let mover do its thing. I need to make sure I'm not using the explicit path of /mnt/cache/appdata for the docker.img (I recall seeing that and I think that was a bad thing to do as it should be associated with a share, right?). The fact that mover would not move everything off the cache in the first place that should have been a big red flag that something was not right and I should have stopped what I was doing right there to analyze things first to prevent data loss.

1 hour ago, snowboardjoe said:

I need to make sure I'm not using the explicit path of /mnt/cache/appdata for the docker.img (I recall seeing that and I think that was a bad thing to do as it should be associated with a share, right?).

Docker image is typically not inside the appdata share, but that's OK if the image will be on cache, but you can also use /mnt/user.