Zee-Unraid Posted 18 hours ago Share Posted 18 hours ago (edited) Hello, thanks for viewing. Hopefully you can impart some wisdom on me and I can learn from my mistakes. My diagnostics export is attached at the bottom of the post. It's fair to point out that, compared to most users on this forum, I'm likely a bit of a novice when it comes to both Unraid, coding/ programming etc., so would welcome clear instructions if you need me to check anything or carry out remedial tasks or require information. I would very much appreciate your patience whilst I am learning and navigating. I have unfortunately rebooted my system since the critical failure occurred, (which I now know not to do before taking a download of the syslog!), however I can tell you what I was doing and why - along with sharing the current status of the system. I also have various code snippets along the way, because I was using ChatGPT to (rightly or wrongly), advise and counsel as I went both before and after the issue occurred, however I believe it was my human mistake that caused the failure/ fallout (steps 8-11). What happened: I was installing a Local LLM model on my machine in a Docker container, the first one worked fine but I was not happy with the LLM outputs. I wanted to try a different LLM and it didn't quite work/ go to plan; I think the container ran out of space during download/ installation and it failed, I'd previously checked the capacity of my system and it seemed fine, but something fell over and the install failed. At this point, my other containers and system was still in tact. No worries. This morning I was attempting to assess the size limits and understand what was going on with the container and how I could fix it. I ran this command to review what is taking up space: root@ZOREO:~# docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 21 7 14.73GB 7.569GB (51%) Containers 7 3 6.491GB 23.82MB (0%) Local Volumes 5 1 0B 0B Build Cache 0 0 0B 0B root@ZOREO:~# docker logs 714aab13613e Error response from daemon: configured logging driver does not support reading root@ZOREO:~# docker volume ls DRIVER VOLUME NAME local 10ce6bb36f5e22542789ef6fb46c6d1a326c8679f9cdf8711eb102b83ea9aec5 local 25a4529cd7273688e4bce4b7d8770bd6b97a69c0f959c7c6e5ae631098c2237d local 78bedba53481c28cfec112f0306eaec7419b47e1cec0c35e0c2ee2aa38513528 local a46d302e5ccd00dc10fee30b986f71851fcd270bb918282abd93c2c0090928f2 local eeedfee9170ad5c678f3148d0d368b89f623afe75ede7269f104032a54381a3b I also then ran the following command line, to assess space utilisation: du -sh /var/lib/docker/* What I didn't realise at this point, was that step 6, above, then started to consume 100% of my CPU usage; and at the same time on my other screen, I'd noticed a number of orphaned images appear in the Docker menu page, so I started clicking them and selecting "remove", because I was tidying up some old of the old images and things I had been playing around with yesterday. I think this is what caused my system to go into meltdown. I hadn't realised the terminal had not finished running the du command and this is possibly what caused the following hanging. Whilst the system was hanging, I could access another terminal and putty to review the system utilisation and I attempted to kill the tasks related to the du command to try and bring the system back responsive. It didn't work after 5min or so. I tried typing reboot as a last resort in the terminal, nothing happened after 20mins. By this point I had 2 zombie processes running. I decided I had no other option that to hard press the power button to reset the system and attempt to bring the system back to life. Which worked. I can login to Unraid fine, however my Docker service seems to be fried and unresponsive. I've tried to reinstall it, but I am concerned the issue could be my cache drive or I might need to either reformat it or reconfigure my cache, which is a little out of my comfort zone and hence why I am here before I potentially make further mistakes. I think I may have lost my previous containers; the only two active at the time were Shinobi and NTP, both related to local home CCTV, (which has been working mostly fine). These are the only ones I really care about recovering. When I visit the Docker page from the Unraid menu, I see the error: "Docker Service failed to start." I think I have installed Docker again properly, but for some reason it's not fully working and I would like help to try and reload my previous containers please, assuming they won't reappear automatically in the Docker menu. I think I also took a backup of my Docker data before I attempted the re-install, see below, but I'm not sure how I can recover/ reintegrate it - or frankly if it's still there at all now! Any help or assistance very much appreciated! Thank you very much! Backup command: root@ZOREO:~# sudo tar -czvf docker_backup.tar.gz /var/lib/docker tar: Removing leading /' from member names /var/lib/docker/ /var/lib/docker/runtimes/ /var/lib/docker/tmp/ /var/lib/docker/buildkit/ /var/lib/docker/buildkit/history.db /var/lib/docker/buildkit/cache.db /var/lib/docker/buildkit/executor/ /var/lib/docker/buildkit/metadata_v2.db /var/lib/docker/buildkit/snapshots.db /var/lib/docker/buildkit/containerdmeta.db /var/lib/docker/buildkit/content/ /var/lib/docker/buildkit/content/ingest/ /var/lib/docker/swarm/ /var/lib/docker/network/ /var/lib/docker/network/files/ /var/lib/docker/network/files/local-kv.db /var/lib/docker/image/ /var/lib/docker/image/overlay2/ /var/lib/docker/image/overlay2/repositories.json /var/lib/docker/image/overlay2/distribution/ /var/lib/docker/image/overlay2/distribution/diffid-by-digest/ /var/lib/docker/image/overlay2/distribution/diffid-by-digest/sha256/ /var/lib/docker/image/overlay2/distribution/diffid-by-digest/sha256/c1ec31eb59444d78df06a974d155e597c894ab4cda84f08294145e845394988e /var/lib/docker/image/overlay2/distribution/v2metadata-by-diffid/ /var/lib/docker/image/overlay2/distribution/v2metadata-by-diffid/sha256/ /var/lib/docker/image/overlay2/distribution/v2metadata-by-diffid/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e /var/lib/docker/image/overlay2/imagedb/ /var/lib/docker/image/overlay2/imagedb/metadata/ /var/lib/docker/image/overlay2/imagedb/metadata/sha256/ /var/lib/docker/image/overlay2/imagedb/content/ /var/lib/docker/image/overlay2/imagedb/content/sha256/ /var/lib/docker/image/overlay2/imagedb/content/sha256/d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a /var/lib/docker/image/overlay2/layerdb/ /var/lib/docker/image/overlay2/layerdb/mounts/ /var/lib/docker/image/overlay2/layerdb/mounts/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/ /var/lib/docker/image/overlay2/layerdb/mounts/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/parent /var/lib/docker/image/overlay2/layerdb/mounts/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/init-id /var/lib/docker/image/overlay2/layerdb/mounts/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/mount-id /var/lib/docker/image/overlay2/layerdb/sha256/ /var/lib/docker/image/overlay2/layerdb/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e/ /var/lib/docker/image/overlay2/layerdb/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e/cache-id /var/lib/docker/image/overlay2/layerdb/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e/size /var/lib/docker/image/overlay2/layerdb/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e/diff /var/lib/docker/image/overlay2/layerdb/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e/tar-split.json.gz /var/lib/docker/image/overlay2/layerdb/tmp/ /var/lib/docker/overlay2/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1/lower /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1/work/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1/work/work/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1/link /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1/diff/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/committed /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/lower /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/work/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/work/work/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/link /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/.dockerenv /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/sys/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/proc/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/etc/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/etc/mtab /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/etc/hosts /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/etc/resolv.conf /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/etc/hostname /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/dev/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/dev/console /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/dev/shm/ /var/lib/docker/overlay2/392c87c035536224bfb044458f7299be153bf7a7af62beea2b14a54890fb57b1-init/diff/dev/pts/ /var/lib/docker/overlay2/be927523c8e3f437c5ff47ed9c8d5bae9a8d1f0f34ff9a02d61184f0326a4e70/ /var/lib/docker/overlay2/be927523c8e3f437c5ff47ed9c8d5bae9a8d1f0f34ff9a02d61184f0326a4e70/committed /var/lib/docker/overlay2/be927523c8e3f437c5ff47ed9c8d5bae9a8d1f0f34ff9a02d61184f0326a4e70/link /var/lib/docker/overlay2/be927523c8e3f437c5ff47ed9c8d5bae9a8d1f0f34ff9a02d61184f0326a4e70/diff/ /var/lib/docker/overlay2/be927523c8e3f437c5ff47ed9c8d5bae9a8d1f0f34ff9a02d61184f0326a4e70/diff/hello /var/lib/docker/overlay2/l/ /var/lib/docker/overlay2/l/QIOQQEL7TZ2DNGOVNYURQYIYMR /var/lib/docker/overlay2/l/SKLEOGYY2B4J6Z7QXWEDHIH7OR /var/lib/docker/overlay2/l/6G43L2J7RRIMNMGB66EZSJOCAC /var/lib/docker/engine-id /var/lib/docker/volumes/ /var/lib/docker/volumes/backingFsBlockDev /var/lib/docker/volumes/metadata.db /var/lib/docker/plugins/ /var/lib/docker/plugins/storage/ /var/lib/docker/plugins/storage/ingest/ /var/lib/docker/plugins/tmp/ /var/lib/docker/containers/ /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/ /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/config.v2.json /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/hostconfig.json /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b-json.log /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/mounts/ /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/hostname /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/resolv.conf.hash /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/resolv.conf /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/hosts /var/lib/docker/containers/d8d13d11f6e09760ce668b362257237a163ee89cee4491a013507153ae23c40b/checkpoints/ /var/lib/docker/containerd/ /var/lib/docker/containerd/daemon/ /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/ /var/lib/docker/containerd/daemon/io.containerd.runtime.v2.task/ /var/lib/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ /var/lib/docker/containerd/daemon/io.containerd.metadata.v1.bolt/ /var/lib/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/ /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/ /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.native/ /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.native/snapshots/ /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.blockfile/ /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/ /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/blobs/ /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/blobs/sha256/ /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/blobs/sha256/e2fc4e5012d16e7fe466f5291c476431beaa1f9b90a5c2125b493ed28e2aba57 /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/blobs/sha256/91fb4b041da273d5a3273b6d587d62d518300a6ad268b28628f74997b93171b2 /var/lib/docker/containerd/daemon/io.containerd.content.v1.content/ingest/ /var/lib/docker/containerd/daemon/tmpmounts/ root@ZOREO:~# Current Docker info: root@ZOREO:~# /etc/rc.d/rc.docker start /usr/bin/dockerd is already running root@ZOREO:~# ls /var/log/ btmp dmesg libvirt/ messages packages@ plugins/ removed_scripts@ scripts@ spooler unbalanced.log wtmp cron faillog maillog nfsd/ phplog pwfail/ removed_uninstall_scripts@ secure swtpm/ vfio-pci debug lastlog mcelog nginx/ pkgtools/ removed_packages@ samba/ setup@ syslog vfio-pci-errors root@ZOREO:~# ls -la /var/lib/docker total 4 drwx--x--- 13 root root 280 Oct 5 14:21 ./ drwxr-xr-x 15 root root 300 Apr 3 2024 ../ drwx--x--x 4 root root 180 Oct 5 14:21 buildkit/ drwx--x--x 3 root root 60 Oct 5 14:21 containerd/ drwx--x--- 2 root root 40 Oct 5 14:21 containers/ -rw------- 1 root root 36 Oct 5 14:21 engine-id drwx------ 3 root root 60 Oct 5 14:21 image/ drwxr-x--- 3 root root 60 Oct 5 14:21 network/ drwx--x--- 3 root root 60 Oct 5 14:21 overlay2/ drwx------ 4 root root 80 Oct 5 14:21 plugins/ drwx------ 2 root root 40 Oct 5 14:21 runtimes/ drwx------ 2 root root 40 Oct 5 14:21 swarm/ drwx------ 2 root root 40 Oct 5 16:01 tmp/ drwx-----x 2 root root 80 Oct 5 14:21 volumes/ root@ZOREO:~# docker --version Docker version 24.0.9, build 2936816 root@ZOREO:~# ls -la /var/lib/docker/containers total 0 drwx--x--- 2 root root 40 Oct 5 14:21 ./ drwx--x--- 13 root root 280 Oct 5 14:21 ../ root@ZOREO:~# zoreo-diagnostics-20241005-1603.zip Edited 18 hours ago by Zee-Unraid Quote Link to comment
JorgeB Posted 11 hours ago Share Posted 11 hours ago There are issues with the cached device, replace both cables and post new diags after array start. Quote Link to comment
Zee-Unraid Posted 10 hours ago Author Share Posted 10 hours ago 22 minutes ago, JorgeB said: There are issues with the cached device, replace both cables and post new diags after array start. Thanks so much for taking the time to review it. May I kindly ask, what do you mean replace both cables, and why? Thanks again, Zee Quote Link to comment
JorgeB Posted 2 hours ago Share Posted 2 hours ago 8 hours ago, Zee-Unraid said: what do you mean replace both cables Power and SATA, it's the only two cables that exist. 8 hours ago, Zee-Unraid said: and why? Because there are constant ATA errors logged, to see if they are cables related. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.