Dockers Not Reading Appdata after new Cache Pool Created/Mover Not Working

[corradojeff]

Hello,

 

I need some help figuring out an issue I am having. I decided to add a second cache drive and create a cache pool and now my dockers open like they are fresh and need to be set up again.

 

To add the cache drive I:

- Stopped docker and VM service

- Set all shares to Cache:Yes

- Completed mover

- Stopped the array

- Added second ssd and formated both for btrfs

- Setup pool with original and new SSDs

- Started array

- Set my appdata, system, and domain shares to perfer cache

- Invoked mover

- Started docker and VM services.

 

Now when I open up any dockers like nginx proxy manager or nextcloud it's like they are a fresh install.

 

It is possible that I started docker service BEFORE mover completed moving appdata back to the cache as I just left it for what I thought was more than enough time (few hours) and assumed it was completed.

 

Of note is that my cache drive is reading full at 498 GB but the shares set to prefer cache should only be about 300GB. I did have shares set to cache:yes but I had not started docker service yet so not sure why they would have moved that data over. to cache. Also, my cache pool size says its full at 498GB but its displaying that it is 1TB in SIZE.

 

While trying to fix/troubleshoot I am currently attempting to move all cache data back to the array by setting all shares to Cache:Yes but the mover has been running for sometime now and the cache drive READS have not increased.

 

Please let me know if there are any additional screenshots or logs that I can provide to help diagnose.

 

Thank you

utopia-diagnostics-20220103-1231.zip

[corradojeff]

utopia-syslog-20220103-2053.zip

[JorgeB]

Cache filesystem is going read-only:

 

Jan  3 10:15:06 Utopia kernel: BTRFS: error (device nvme1n1p1) in cleanup_transaction:1942: errno=-28 No space left
Jan  3 10:15:06 Utopia kernel: BTRFS info (device nvme1n1p1): forced readonly

 

Likely from getting out of space before, best bet it to backup and re-format the pool, you should never completely fill up a COW filesystem like btrfs or zfs.

[corradojeff]

Figured I would loop back on this for posterity.

 

I am still not sure what happened but my downloads share was on cache and that's what filled it up. It must have been a mis-click on my part. I ended up backing up appdata, domains, and system shares and reformatting.

 

After transferring all data back I still had the issue with dockers. I ended up backing up the .confs (and any other db/config files for each docker) and wiped app data and docker.img and started fresh.

 

I am now in the processes of setting up a second unraid server to send across country for backup.