jj_uk Posted September 12, 2021 Share Posted September 12, 2021 (edited) This morning, all dockers that are usually stopped as part of the CA backup process have failed to restart. Pages in the UnRAID gui have php error messages on them. Warning: file_put_contents(/var/lib/docker/unraid/images/home-assistant-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 90 Warning: file_put_contents(/var/lib/docker/unraid/images/mariadb-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 90 Warning: file_put_contents(/var/lib/docker/unraid/images/NginxProxyManager-icon.png): failed to open stream: Read-only file system in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 90 tower1-diagnostics-20210912-1235.zip Edited September 12, 2021 by jj_uk Quote Link to comment
jj_uk Posted September 12, 2021 Author Share Posted September 12, 2021 I've just restarted the server and all my dockers are gone ! I have no dockers installed! What's going on! Quote Link to comment
jj_uk Posted September 12, 2021 Author Share Posted September 12, 2021 (edited) My appdata share is completely empty. Please help ! I'm starting to panic! Edited September 12, 2021 by jj_uk Quote Link to comment
jj_uk Posted September 12, 2021 Author Share Posted September 12, 2021 All shares that were on the cache disk are empty. Seems the caches have unmounted or something? How can I instruct unraid to start using the cache disks again? Quote Link to comment
Squid Posted September 12, 2021 Share Posted September 12, 2021 Your cache pool is unmountable. (PITA to see where it started) Wait for @JorgeB for advice Quote Link to comment
JorgeB Posted September 13, 2021 Share Posted September 13, 2021 Logged is spammed with GPU related errors, reboot and post new diags after array start. Quote Link to comment
jj_uk Posted September 13, 2021 Author Share Posted September 13, 2021 Attached, diagnostics taken from a reboot a few mins ago. Thank you so much for your time helping me with this. I'm 'completely' stuck. tower1-diagnostics-20210913-0931.zip Quote Link to comment
jj_uk Posted September 13, 2021 Author Share Posted September 13, 2021 (edited) Is there any prospect of getting the cache drives back online? There were a few dockers used to run a smart home. We're currently sat in the dark. I have a backup of appdata from the day before, created by CA Backup. It contains a full /appdata/ backup and 'libvirt.img'. Edited September 13, 2021 by jj_uk Quote Link to comment
Solution JorgeB Posted September 13, 2021 Solution Share Posted September 13, 2021 Sorry, missed your first reply due to an old forum bug, filesystem is crashing immediately after mounting, there are some recovery options here that you can try. Quote Link to comment
jj_uk Posted September 13, 2021 Author Share Posted September 13, 2021 Thanks!!! I'll give this a try. Before I do, docker has created some files on the system share. Should I delete these before attempting the repair? The files will also be in the cache as system share is set to 'Prefer : Cache' Quote Link to comment
jj_uk Posted September 13, 2021 Author Share Posted September 13, 2021 (edited) I've copied all the data off of the drives following the guide you posted above. I've stopped the array then run this command to erase the cache drives: blkdiscard -f /dev/sdb blkdiscard -f /dev/sdc I've stated the array, and fomatted the "unmountable" drives, which are the 2 cache drives. After the format completes, I get this error: "Unmountable: no pool uuid" I've removed both drives from the pool, and set slots to zero to delete the pool, then created a pool of 2 and re-added them, but the error persists. I can't mount the drives. EDIT: Solved the issue. I stopped the array, set all shares to 'Use Cache Pool? No', deleted the docker image in settings-docker, deleted the virtlib file by stopping VM and chosing delete from "Settings - VM Manager". Disabled the syslog server in settings, then deleted the syslog server files from the cache using MC. Then rebooted the server, added a cache pool, added the drives, stated the array, formatted the drives. I'm now copying the data back to the cache. Edited September 13, 2021 by jj_uk Quote Link to comment
jj_uk Posted September 14, 2021 Author Share Posted September 14, 2021 Managed to restore all the cache, deleted docker image, re-enabled docker. Installed the 1st container from the APPS -> previous Docker tag. The install failed with an error saying couldn't create /mnt/user/ and then all my shares disappeared! Any idea how to fix that? The 'user' share seems to be broken. Quote Link to comment
trurl Posted September 14, 2021 Share Posted September 14, 2021 New diagnostics Quote Link to comment
jj_uk Posted September 14, 2021 Author Share Posted September 14, 2021 Attached. The array is currently stopped as i'm a bit worried about losing everything. tower1-diagnostics-20210914-0845.zip Quote Link to comment
JorgeB Posted September 14, 2021 Share Posted September 14, 2021 Sep 14 01:00:01 tower1 kernel: shfs[25263]: segfault at 0 ip 00000000004043b8 sp 0000146026a3d780 error 4 in shfs[402000+b000] shfs segfaulted, rebooting should bring your shares back, but if it continues to happen it might indicate an underlying issue. Quote Link to comment
jj_uk Posted September 14, 2021 Author Share Posted September 14, 2021 Thanks. That's worked and a docker container started ok. I forgot to select encrypted for the cache, so I'm repeating the process Quicker to redo the copy process from the backed-up cache than to let mover do it. Fingers crossed! Quote Link to comment
jj_uk Posted September 14, 2021 Author Share Posted September 14, 2021 I'm up and running again, but as soon as I clicked MOVE to start the mover, I got a segment fault again. What causes this? tower1-diagnostics-20210914-1311.zip Quote Link to comment
JorgeB Posted September 14, 2021 Share Posted September 14, 2021 Unclear, try to move those files manually this time. Quote Link to comment
jj_uk Posted September 14, 2021 Author Share Posted September 14, 2021 Mover completed successfully after a reboot. Next I need to fix the AMD spam in the logs. I'll post a new topic for that. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.