(SOLVED) Docker apps won't start, can't update - image problem?

[stor44]

Hi there. Woke up to see all my Docker apps stopped and when I try to start them it I see "Execution Error: System error". 4 of the apps show updates available, but when I click update I get this error:

Error: open /var/lib/docker/tmp/GetImageBlob711164869: read-only file system

Something wrong with my Docker img file?

Thanks for any help. I searched but I'm not clear what to try. Diagnostics attached. Specs in my sig.

tower-diagnostics-20210301-0530.zip

Edited March 5, 2021 by stor44
SOLVED: Had to format cache pool, restore appdata and reinstall Dockers and VM.

[JorgeB]

Cache filesystem is corrupt, best is to backup and re-format the pool.

[stor44]

Thanks for the quick reply. I'll dive back into this after work today.

[stor44]

Is this kind of corruption pretty common or did I do something wrong?

 

Anyway I got things back to normal. Documenting it here in case I need it again, and in case it helps anyone else:

- first I tried copying some data off the cache pool, but the corruption was too much. Files and folders were literally disappearing as I refreshed the directories.

- I stopped the array which took way longer than usual, probably 5 minutes instead of 40 seconds.

- I didn't know how to format the cache pool drives (2 x 1TB SSD). But I tried doing this:

Just to be sure, I also formatted them using this command: blkdiscard /dev/sd#

 

 

- at this point I upgraded to 6.9, in case there was any advantage to this new SSD formatting mentioned here:

https://wiki.unraid.net/Unraid_OS_6.9.0#SSD_1_MiB_Partition_Alignment

 

- then re-assigned the SSDs and formatted them to create the new cache pool

- then I needed to restore some of the data I had backed up. Fortunately I have an "appdata-backup" folder. I'm helpless when it comes to untarring the tar file on unRAID, so I copied it to my Windows PC and used WinRAR. I extracted the data back to the cache pool, but WinRAR had a ton of errors about needing to run as Administrator, so I did that and ran it again. It still failed on a bunch of my Channels (DVR) and Plex metadata, but it got all the important files for my other Docker apps.

- I tried to stop the array again, but it just hung, so I had to power off the whole PC.

- This of course forces a parity check. 24 hours later, no errors.

- I also had a backup of my Windows 10 VM thanks to this script:

 

so I restored that. But booting it up, it was super slow, and even unRAID was running sluggish overall. Tracked it down to the "Tips and Tweaks" Plugin, I had "CPU Scaling Governor:" set to "Power Save". I set it to "On Demand" and rebooted and things were fine again. Blue Iris (security camera app I run in Win10) was still slow, but I had to re-enable NVIDIA acceleration in its settings.

So, lots more for me to learn. I've attached new diagnostics on the chance there's anything else helpful in there.

Cheers.

tower-diagnostics-20210305-1548.zip

[JorgeB]

10 hours ago, stor44 said:

Is this kind of corruption pretty common or did I do something wrong?

It's not common with good hardware, still wouldn't worry about one time, but if it keeps happening there's likely some hardware issue. 

[stor44]

Thanks, sounds good. This system is a 2014 gaming rig I converted to unRAID. Longer term I'm hoping to build a new system with unRAID in mind from the start.