November 8, 20232 yr This issue randomly occurs. I've verified that all my docker containers are writing outside of the docker.img and I've switched to a directory for docker to see if it was a corruption issue or space issue. The space consumed by the docker directory is only 3.3G right now. What is interesting about this situation is that the docker containers maintain reachable from the outside. They fail to communicate with each other. If a container has a WebUI, I can reach it without an issue. In order to recover from this state I will have to set docker to off in settings, send a SIGTERM to all of the docker container process trees and verify they are all in-fact stopped, delete the docker directory contents (or docker.img), restart the docker daemon, and then re-add all the containers I have from existing templates. I've attached a diag taken during a docker daemon crashed state. The error in docker.log that I believe points to the problem is: time="2023-11-06T11:33:57.999315499-06:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b7e682ad7e15cc134d18b833e6df40c70bb50e97c3d7ff2e09459b799a1a2cee pid=23791 runtime=io.containerd.runc.v2 fatal error: unexpected signal during runtime execution [signal SIGBUS: bus error code=0x2 addr=0x3010708 pc=0x45fad1] runtime stack: runtime.throw({0x1bbc3c1?, 0x4590b0?}) /usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x15426c861e48 sp=0x15426c861e18 pc=0x439bbf runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x15426c861e98 sp=0x15426c861e48 pc=0x4504c9 runtime.gentraceback(0x44beb4?, 0xc0013a3380?, 0xc000d081a0?, 0x15426c862258?, 0x4, 0xc000fbe968, 0x20, 0x0, 0xc000683d40?, 0x0) /usr/local/go/src/runtime/traceback.go:139 +0x411 fp=0x15426c862208 sp=0x15426c861e98 pc=0x45fad1 runtime.callers.func1() /usr/local/go/src/runtime/traceback.go:925 +0x4f fp=0x15426c862270 sp=0x15426c862208 pc=0x462b0f runtime.systemstack() /usr/local/go/src/runtime/asm_amd64.s:492 +0x46 fp=0x15426c862278 sp=0x15426c862270 pc=0x46be66 goroutine 858250 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:459 fp=0xc000fbe8c0 sp=0xc000fbe8b8 pc=0x46be00 runtime.callers(0xc00021f438?, {0xc000fbe968?, 0x1d?, 0x1000000010000?}) /usr/local/go/src/runtime/traceback.go:924 +0x98 fp=0xc000fbe928 sp=0xc000fbe8c0 pc=0x462a58 runtime.mProf_Malloc(0xc000e52000, 0x160) /usr/local/go/src/runtime/mprof.go:420 +0x6a fp=0xc000fbeaa0 sp=0xc000fbe928 pc=0x430b4a runtime.profilealloc(0x1542965eea68?, 0x160?, 0x416105?) /usr/local/go/src/runtime/malloc.go:1238 +0x85 fp=0xc000fbead8 sp=0xc000fbeaa0 pc=0x40fb65 runtime.mallocgc(0x150, 0x228fd40, 0x1) /usr/local/go/src/runtime/malloc.go:1114 +0x71b fp=0xc000fbeb50 sp=0xc000fbead8 pc=0x40f71b runtime.newobject(...) /usr/local/go/src/runtime/malloc.go:1202 runtime.mapassign_faststr(0x22b2000, 0xc001d960c0, {0x1b759a4, 0x14}) /usr/local/go/src/runtime/map_faststr.go:221 +0xa8 fp=0xc000fbebc0 sp=0xc000fbeb50 pc=0x4154e8 fatal error: unexpected signal during runtime execution panic during panic [signal SIGBUS: bus error code=0x2 addr=0x3010708 pc=0x45fad1] runtime stack: runtime.throw({0x1bbc3c1?, 0x1eee0a8?}) /usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x15426c861738 sp=0x15426c861708 pc=0x439bbf runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x15426c861788 sp=0x15426c861738 pc=0x4504c9 runtime.gentraceback(0x0?, 0x8?, 0x0?, 0x15426c861cd8?, 0x0, 0x0, 0x64, 0x0, 0x15426c861b78?, 0x0) /usr/local/go/src/runtime/traceback.go:139 +0x411 fp=0x15426c861af8 sp=0x15426c861788 pc=0x45fad1 runtime.traceback1(0xc0013a3380?, 0x15426c861d10?, 0x43be7a?, 0xc0013a3380, 0x15426c861d10?) /usr/local/go/src/runtime/traceback.go:857 +0x1b1 fp=0x15426c861cb8 sp=0x15426c861af8 pc=0x462391 runtime.traceback(...) /usr/local/go/src/runtime/traceback.go:804 runtime.tracebackothers(0xc000d081a0) /usr/local/go/src/runtime/traceback.go:1049 +0x98 fp=0x15426c861d20 sp=0x15426c861cb8 pc=0x463178 runtime.dopanic_m(0xc000d081a0, 0x1eee0a8?, 0x1?) /usr/local/go/src/runtime/panic.go:1270 +0x285 fp=0x15426c861d98 sp=0x15426c861d20 pc=0x43a5c5 runtime.fatalthrow.func1() /usr/local/go/src/runtime/panic.go:1125 +0x48 fp=0x15426c861dd8 sp=0x15426c861d98 pc=0x439f88 runtime.fatalthrow(0x6c861e20?) /usr/local/go/src/runtime/panic.go:1122 +0x6c fp=0x15426c861e18 sp=0x15426c861dd8 pc=0x439f0c runtime.throw({0x1bbc3c1?, 0x4590b0?}) /usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x15426c861e48 sp=0x15426c861e18 pc=0x439bbf runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x15426c861e98 sp=0x15426c861e48 pc=0x4504c9 runtime.gentraceback(0x44beb4?, 0xc0013a3380?, 0xc000d081a0?, 0x15426c862258?, 0x4, 0xc000fbe968, 0x20, 0x0, 0xc000683d40?, 0x0) /usr/local/go/src/runtime/traceback.go:139 +0x411 fp=0x15426c862208 sp=0x15426c861e98 pc=0x45fad1 runtime.callers.func1() /usr/local/go/src/runtime/traceback.go:925 +0x4f fp=0x15426c862270 sp=0x15426c862208 pc=0x462b0f runtime.systemstack() /usr/local/go/src/runtime/asm_amd64.s:492 +0x46 fp=0x15426c862278 sp=0x15426c862270 pc=0x46be66 Syslog has the following from around that time.. not sure if it is related: Nov 6 11:33:56 unraid kernel: docker0: port 8(veth9521bea) entered blocking state Nov 6 11:33:56 unraid kernel: docker0: port 8(veth9521bea) entered disabled state Nov 6 11:33:56 unraid kernel: device veth9521bea entered promiscuous mode Nov 6 11:33:58 unraid kernel: eth0: renamed from veth36d6453 Nov 6 11:33:58 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9521bea: link becomes ready Nov 6 11:33:58 unraid kernel: docker0: port 8(veth9521bea) entered blocking state Nov 6 11:33:58 unraid kernel: docker0: port 8(veth9521bea) entered forwarding state I am running a 4x1 Gbps bond for my unraid's network interface and the switch is configured for the port-channel and has been working fine for quite a while. My SSD cache is ZFS.. I also have a ZFS pool consisting of 6 disks in a raidz1 that I use for appdata and my plex media server's growing data. The docker directory lives on the cache device. Had the same issue with it living on the ZFS pool. I am open to suggestions on troubleshooting. unraid-diagnostics-20231108-0908.zip
November 11, 20232 yr Author I changed my cache from ZFS to BTRFS two days ago and I've run into this issue again. I will remove the bond configuration and revert the cache back to ZFS later today and going down to a single network interface as my next diagnostic step.
November 11, 20232 yr Author Changed the cache back to ZFS from BTRFS and disabled the bond.. using only eth0. Still the same issue after only a few hours. This time though I didn't delete the docker.img - I just rebooted the machine after making the changes and docker came back up.. That's not a solution I like but it works and is less destructive and less hands on overall. I'll reboot later tonight but I'm really scratching my head on this one. The logs aren't really telling that I can see - hopefully someone else sees value in them. Attached is the diagnostics zip from just now. unraid-diagnostics-20231111-1726.zip Edited November 11, 20232 yr by yuka
November 12, 20232 yr Community Expert Nov 11 16:19:42 unraid kernel: sd 2:0:14:0: [sdp] tag#139 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 11 16:19:42 unraid kernel: sd 2:0:14:0: [sdp] tag#139 Sense Key : 0xb [current] Nov 11 16:19:42 unraid kernel: sd 2:0:14:0: [sdp] tag#139 ASC=0x47 ASCQ=0x3 Nov 11 16:19:42 unraid kernel: sd 2:0:14:0: [sdp] tag#139 CDB: opcode=0x2a 2a 00 32 a9 c2 88 00 01 00 00 Nov 11 16:19:42 unraid kernel: I/O error, dev sdp, sector 849986184 op 0x1:(WRITE) flags 0x700 phys_seg 3 prio class 2 Nov 11 16:19:42 unraid kernel: zio pool=cache vdev=/dev/sdp1 error=5 type=2 offset=435191877632 size=131072 flags=184880 Issues with cache device, check/replace cables, also see here to make sure it's using the latest firmware. Nov 11 17:01:34 unraid kernel: SQUASHFS error: xz decompression failed, data probably corrupt Nov 11 17:01:34 unraid kernel: SQUASHFS error: Failed to read block 0x2e95070: -5 Nov 11 17:01:34 unraid kernel: SQUASHFS error: xz decompression failed, data probably corrupt Nov 11 17:01:34 unraid kernel: SQUASHFS error: Failed to read block 0x2e95070: -5 These errors usually mean a bad flash drive.
November 14, 20232 yr Author On 11/12/2023 at 6:03 AM, JorgeB said: Issues with cache device, check/replace cables, also see here to make sure it's using the latest firmware. I saw that too and during the last docker daemon failure I moved docker.img to my ZFS array a few days ago. This morning the docker daemon has failed again. The only shares that are in use on my cache drive as of this failure are nzb client download files such as intermediate and completed, and my plex transcode files. I also have my tdarr-transcode temporary file directory and an rclone VFS cache directory on the cache drive but neither of those are in use or running in any way right now. I'll move everything off of it and try again. On 11/12/2023 at 6:03 AM, JorgeB said: These errors usually mean a bad flash drive. Can you clarify bad flash - are we talking the SSD as a flash device or the unraid boot flash media? I can swap out the unraid boot flash today if that is the case.
November 14, 20232 yr Author I moved everything off the cache drive onto my ZFS pool for now. Reconfigured all of the shares to primary on the ZFS pool. The firmware flash failed with an Input/output error. Pulled a backup.. wrote support about swapping my registration to the new disk GUID (had to restore from backup in September due to long lost flash drive) and shut down. Waiting on the flash create to write the new flash device and then I'll go through the process of pulling this server out of the rack and bringing it back up with the new flash drive. From there I am just going to attempt to start Docker off of the ZFS pool with all my scratch shares (downloads, plex/tdarr transcode, VFS cache) running off the ZFS pool and see what happens. Fingers crossed. I'll write back in a week or whenever docker daemon crashes again with my results - whichever happens first. UPDATE: I pulled the SSD when I swapped the flash storage. The current firmware is M3CR033 which is the latest version for this drive revision: https://www.crucial.com/support/ssd-support/mx500-support There were later revisions of the drive which shipped with a higher #'d firmware but that update is not applicable to this firmware revision. Their tool on Windows (Crucial Storage Executive) identifies the drive's firmware as up to date and attempting to apply M3CR045 results in an error from the tool indicating the firmware is not applicable to this drive. Edited November 14, 20232 yr by yuka
November 21, 20232 yr Hey yuka, This thread is describing behavior I am also experiencing and am similarly puzzled by. I just switched my docker to Directory from BTRFS because I figured it might be more stable, only for the issue to manifest less than 24h later. At one point I got just short of 26 days uptime prior to it returning. I also happened to replace my boot drive this past summer, so I think it's in ok shape? Looking forward to hearing if you have any progress solving it. Best of luck!
November 27, 20232 yr Author I swapped my boot drive and removed my cache (for now) and I've been stable since November 14th. I need to add another drive to my array once parity check is complete and I was planning on re-adding the cache disk but after reading your reply @localcatperson I decided it might be better to give it more time.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.