This morning, all dockers that are usually stopped as part of the CA backup process have failed to restart. 

 

Pages in the UnRAID gui have php error messages on them.

 

Warning: file_put_contents(/var/lib/docker/unraid/images/home-assistant-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 90

Warning: file_put_contents(/var/lib/docker/unraid/images/mariadb-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 90

Warning: file_put_contents(/var/lib/docker/unraid/images/NginxProxyManager-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 90

tower1-diagnostics-20210912-1235.zip

Edited September 12, 20214 yr by jj_uk

I've just restarted the server and all my dockers are gone !  I have no dockers installed! What's going on!

My appdata share is completely empty. Please help ! I'm starting to panic!

 

Edited September 12, 20214 yr by jj_uk

All shares that were on the cache disk are empty. Seems the caches have unmounted or something?

 

How can I instruct unraid to start using the cache disks again?

Your cache pool is unmountable.  (PITA to see where it started)  Wait for @JorgeB for advice

Logged is spammed with GPU related errors, reboot and post new diags after array start.

Attached, diagnostics taken from a reboot a few mins ago.

 

Thank you so much for your time helping me with this. I'm 'completely' stuck.  

 

tower1-diagnostics-20210913-0931.zip

Is there any prospect of getting the cache drives back online?  There were a few dockers used to run a smart home. We're currently sat in the dark.

 

I have a backup of appdata from the day before, created by CA Backup.  It contains a full /appdata/ backup and 'libvirt.img'. 

 

 

Edited September 13, 20214 yr by jj_uk

Sorry, missed your first reply due to an old forum bug, filesystem is crashing immediately after mounting, there are some recovery options here that you can try.

Thanks!!! I'll give this a try.

 

Before I do, docker has created some files on the system share. Should I delete these before attempting the repair? The files will also be in the cache as system share is set to 'Prefer : Cache'

 

I've copied all the data off of the drives following the guide you posted above.

 

I've stopped the array then run this command to erase the cache drives:

blkdiscard -f /dev/sdb

blkdiscard -f /dev/sdc

 

I've stated the array, and fomatted the "unmountable" drives, which are the 2 cache drives.

 

After the format completes, I get this error:

 

"Unmountable: no pool uuid"

 

 

I've removed both drives from the pool, and set slots to zero to delete the pool, then created a pool of 2 and re-added them, but the error persists.

 

I can't mount the drives.

 

 

EDIT: 

 

Solved the issue. I stopped the array, set all shares to 'Use Cache Pool? No', deleted the docker image in settings-docker, deleted the virtlib file by stopping VM and chosing delete from "Settings - VM Manager".

 

Disabled the syslog server in settings, then deleted the syslog server files from the cache using MC.

 

Then rebooted the server, added a cache pool, added the drives, stated the array, formatted the drives.

 

I'm now copying the data back to the cache.

 

Edited September 13, 20214 yr by jj_uk

Managed to restore all the cache, deleted docker image, re-enabled docker.

 

Installed the 1st container from the APPS -> previous Docker tag. The install failed with an error saying couldn't create /mnt/user/ and then all my shares disappeared!

 

 

Any idea how to fix that? The 'user' share seems to be broken.

 

New diagnostics

Attached. The array is currently stopped as i'm a bit worried about losing everything.

tower1-diagnostics-20210914-0845.zip

Sep 14 01:00:01 tower1 kernel: shfs[25263]: segfault at 0 ip 00000000004043b8 sp 0000146026a3d780 error 4 in shfs[402000+b000]

 

shfs segfaulted, rebooting should bring your shares back, but if it continues to happen it might indicate an underlying issue.

Thanks. That's worked and a docker container started ok.

 

I forgot to select encrypted for the cache, so I'm repeating the process  Quicker to redo the copy process from the backed-up cache than to let mover do it.

 

Fingers crossed!

I'm up and running again, but as soon as I clicked MOVE to start the mover, I got a segment fault again. What causes this?

tower1-diagnostics-20210914-1311.zip

Unclear, try to move those files manually this time.

Mover completed successfully after a reboot.  Next I need to fix the AMD spam in the logs. I'll post a new topic for that.