January 17, 20251 yr Hi, I'm experiencing persistent issues with Docker containers. Here's my situation: System Details: - unRAID Version: 7.0 - Docker Storage: Directory-based (previously BTRFS) - Storage Driver: overlay2 - Cache Drive: ZFS Issue Description: 1. After a fresh setup (clearing Docker folder and redownloading all containers), everything works normally for a bit (I'm not sure when it starts to break) 2. After a couple of hours I notice some containers become "not available" (orange highlight in UI), despite definitely being available on dockerhub or github 3. Once this happens, I can't update ANY containers (even available ones) and receive the following error: Pulling image: esphome/esphome:latest [Various layer downloads...] Error: failed to register layer: open /var/lib/docker/overlay2/c9d51d50aaab263982141b1cdadc2f28757be5c6be53001702e0d92f705e4bac/link: no such file or directory What I've Tried: - Cleared Docker folder and redownloaded all images (4 times) - Switched from BTRFS to directory-based storage due to previous corruption issues - System reboots (results in "No Docker containers installed" message despite images being present) - Scrubbing my ZFS cache drive (says NO errors, even after the issue) Additional Observations: - No other issues on the cache drive, only docker. - System reboot without fixing the issue first results in Docker UI showing no containers (despite the images existing in the docker folder) - Previous BTRFS setup had similar corruption issues, which led to switching to directory-based storage. When the BTRFS docker image would be scrubbed, it would have unfixable errors Has anyone experienced similar issues? Any suggestions for troubleshooting or potential fixes would be appreciated. Given that I was having similar issues on 6.12.14 with the BTRFS image, perhaps it's not related to unRAID 7?
January 17, 20251 yr Community Expert Nothing obvious that I can see, I would recommend retesting with a docker image, docker folder can sometimes cause strange issues: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
January 17, 20251 yr Author Sorry, just realized these diagnostics were from after a reboot & repair (following the steps you've just linked). I'll share diags again when it happens again (probably a few hours). After I send those diags, I'll test docker image again and post diags if that image also gets corrupted (which happened 3 times before, which is why I switched to docker folder, hoping it would fix it lol)
January 17, 20251 yr Community Expert 6 minutes ago, pureelectricity said: if that image also gets corrupted (which happened 3 times before, The commonest cause for this seems to be RAM issues. If that is what caused it then it will still occur when using a directory for docker.
January 17, 20251 yr Author 6 minutes ago, itimpi said: The commonest cause for this seems to be RAM issues. If that is what caused it then it will still occur when using a directory for docker. I ran a memtest a few months ago for a different issue and things were fine. I'll add it to the list! Should be faster on 7?
January 17, 20251 yr Community Expert Just now, pureelectricity said: ran a memtest a few months ago for a different issue and things were fine. The problem is that memtest is only definitive if it shows errors.
January 18, 20251 yr Author 21 hours ago, itimpi said: The problem is that memtest is only definitive if it shows errors. True, I've been running memtest using Live Memory Tester plugin and it succeeded with 10 rounds without issue. I'll do a full memtest later on... 22 hours ago, JorgeB said: Nothing obvious that I can see, I would recommend retesting with a docker image, docker folder can sometimes cause strange issues: I woke up this morning and noticed it happened again, it seems not all containers are effected. Some can be force updated just fine and some result in the error. It seems it's not realizing the image is there because it tries pulling it again even if there are no updates (I guess that's why it's also "not available". I've attached the correct diagnostics this time without a reboot (sorry ), including a scrub on the ZFS cache. I will switch back to docker image and see if it helps, and provide those diagnostics as well if it doesn't. gilfoyle-diagnostics-20250118-1141.zip
January 23, 20251 yr Author After switching back to the docker image, everything seems to be stable now for 42 hours. This is odd, because the docker image previously kept getting corrupt as well. I'll miss the overlay2 speeds, but glad that things do appear stable. I'm not sure why it's working now, but the only thing that's changed is after boot issues, I plugged my unRAID USB drive into another USB 2.0 slot and it booted and has been smooth since then. This is a common problem I experience across multiple USB Drive attempts, so I didn't suspect it was related. I'll mark the solution after a full week of testing Edited January 23, 20251 yr by pureelectricity
January 28, 20251 yr Author @JorgeB Unfortunately I noticed some weirdness where my container upon reboot would not properly read any updates I made to its appdata config file. This caused me to reboot and upon reboot, the Docker server failed start and was corrupted again, this time with the docker image. I've attached the diagnostics after a reboot where the image started corrupt... gilfoyle-diagnostics-20250127-2054.zip
January 29, 20251 yr Author Ran for about 12 hours, all 6 passed. Same result when I was using the memtest plugin. Not sure where else to look or what to try.
January 29, 20251 yr Community Expert Since memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.
February 1, 20251 yr I am encountering the same issue, and I recently replaced my RAM with brand new modules. I realize this doesn't rule out the possibility of memory errors, but the fact that multiple users are encountering this with Unraid 7.0.0, and that @pureelectricity is not showing any memory errors, it seems unlikely. My issue is the same as the OP except it only applies to specific containers. Most containers can still update perfectly fine. I also never had this type of issue before Unraid 7.0.0
February 2, 20251 yr Author On 1/31/2025 at 7:12 PM, pumavision said: I am encountering the same issue, and I recently replaced my RAM with brand new modules. I realize this doesn't rule out the possibility of memory errors, but the fact that multiple users are encountering this with Unraid 7.0.0, and that @pureelectricity is not showing any memory errors, it seems unlikely. My issue is the same as the OP except it only applies to specific containers. Most containers can still update perfectly fine. I also never had this type of issue before Unraid 7.0.0 I'm curious what filesystem is your cache drive and what docker implementation are you using? Mine is ZFS and I've tried all the docker implementations: XFS image, BTRFS image, and filesystem with both drivers. I get different errors depending on the implementation but it's always related to docker. I am not sure, but I think I actually started getting this issue since 6.12.14, because I remember hoping the kernel upgrade in 7.0 would fix it. I will be trying with different RAM eventually. It could also be a failing SSD (despite me not getting any SMART issues)? Both my RAM and cache disk are ~5 years old.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.