yuka

I'm also having this problem. It has been occurring for roughly 1 month - I don't know when it first happened. It only seems to be impacting my Radarr instance - Sonarr and Lidarr are having no issues and live on the same ZFS pool.

I swapped my boot drive and removed my cache (for now) and I've been stable since November 14th. I need to add another drive to my array once parity check is complete and I was planning on re-adding the cache disk but after reading your reply @localcatperson I decided it might be better to give it more time.

I moved everything off the cache drive onto my ZFS pool for now. Reconfigured all of the shares to primary on the ZFS pool. The firmware flash failed with an Input/output error. Pulled a backup.. wrote support about swapping my registration to the new disk GUID (had to restore from backup in September due to long lost flash drive) and shut down. Waiting on the flash create to write the new flash device and then I'll go through the process of pulling this server out of the rack and bringing it back up with the new flash drive. From there I am just going to attempt to start Docker off of the ZFS pool with all my scratch shares (downloads, plex/tdarr transcode, VFS cache) running off the ZFS pool and see what happens. Fingers crossed. I'll write back in a week or whenever docker daemon crashes again with my results - whichever happens first. UPDATE: I pulled the SSD when I swapped the flash storage. The current firmware is M3CR033 which is the latest version for this drive revision: https://www.crucial.com/support/ssd-support/mx500-support There were later revisions of the drive which shipped with a higher #'d firmware but that update is not applicable to this firmware revision. Their tool on Windows (Crucial Storage Executive) identifies the drive's firmware as up to date and attempting to apply M3CR045 results in an error from the tool indicating the firmware is not applicable to this drive.

I saw that too and during the last docker daemon failure I moved docker.img to my ZFS array a few days ago. This morning the docker daemon has failed again. The only shares that are in use on my cache drive as of this failure are nzb client download files such as intermediate and completed, and my plex transcode files. I also have my tdarr-transcode temporary file directory and an rclone VFS cache directory on the cache drive but neither of those are in use or running in any way right now. I'll move everything off of it and try again. Can you clarify bad flash - are we talking the SSD as a flash device or the unraid boot flash media? I can swap out the unraid boot flash today if that is the case.

Changed the cache back to ZFS from BTRFS and disabled the bond.. using only eth0. Still the same issue after only a few hours. This time though I didn't delete the docker.img - I just rebooted the machine after making the changes and docker came back up.. That's not a solution I like but it works and is less destructive and less hands on overall. I'll reboot later tonight but I'm really scratching my head on this one. The logs aren't really telling that I can see - hopefully someone else sees value in them. Attached is the diagnostics zip from just now. unraid-diagnostics-20231111-1726.zip

I changed my cache from ZFS to BTRFS two days ago and I've run into this issue again. I will remove the bond configuration and revert the cache back to ZFS later today and going down to a single network interface as my next diagnostic step.

This issue randomly occurs. I've verified that all my docker containers are writing outside of the docker.img and I've switched to a directory for docker to see if it was a corruption issue or space issue. The space consumed by the docker directory is only 3.3G right now. What is interesting about this situation is that the docker containers maintain reachable from the outside. They fail to communicate with each other. If a container has a WebUI, I can reach it without an issue. In order to recover from this state I will have to set docker to off in settings, send a SIGTERM to all of the docker container process trees and verify they are all in-fact stopped, delete the docker directory contents (or docker.img), restart the docker daemon, and then re-add all the containers I have from existing templates. I've attached a diag taken during a docker daemon crashed state. The error in docker.log that I believe points to the problem is: time="2023-11-06T11:33:57.999315499-06:00" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b7e682ad7e15cc134d18b833e6df40c70bb50e97c3d7ff2e09459b799a1a2cee pid=23791 runtime=io.containerd.runc.v2 fatal error: unexpected signal during runtime execution [signal SIGBUS: bus error code=0x2 addr=0x3010708 pc=0x45fad1] runtime stack: runtime.throw({0x1bbc3c1?, 0x4590b0?}) /usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x15426c861e48 sp=0x15426c861e18 pc=0x439bbf runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x15426c861e98 sp=0x15426c861e48 pc=0x4504c9 runtime.gentraceback(0x44beb4?, 0xc0013a3380?, 0xc000d081a0?, 0x15426c862258?, 0x4, 0xc000fbe968, 0x20, 0x0, 0xc000683d40?, 0x0) /usr/local/go/src/runtime/traceback.go:139 +0x411 fp=0x15426c862208 sp=0x15426c861e98 pc=0x45fad1 runtime.callers.func1() /usr/local/go/src/runtime/traceback.go:925 +0x4f fp=0x15426c862270 sp=0x15426c862208 pc=0x462b0f runtime.systemstack() /usr/local/go/src/runtime/asm_amd64.s:492 +0x46 fp=0x15426c862278 sp=0x15426c862270 pc=0x46be66 goroutine 858250 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_amd64.s:459 fp=0xc000fbe8c0 sp=0xc000fbe8b8 pc=0x46be00 runtime.callers(0xc00021f438?, {0xc000fbe968?, 0x1d?, 0x1000000010000?}) /usr/local/go/src/runtime/traceback.go:924 +0x98 fp=0xc000fbe928 sp=0xc000fbe8c0 pc=0x462a58 runtime.mProf_Malloc(0xc000e52000, 0x160) /usr/local/go/src/runtime/mprof.go:420 +0x6a fp=0xc000fbeaa0 sp=0xc000fbe928 pc=0x430b4a runtime.profilealloc(0x1542965eea68?, 0x160?, 0x416105?) /usr/local/go/src/runtime/malloc.go:1238 +0x85 fp=0xc000fbead8 sp=0xc000fbeaa0 pc=0x40fb65 runtime.mallocgc(0x150, 0x228fd40, 0x1) /usr/local/go/src/runtime/malloc.go:1114 +0x71b fp=0xc000fbeb50 sp=0xc000fbead8 pc=0x40f71b runtime.newobject(...) /usr/local/go/src/runtime/malloc.go:1202 runtime.mapassign_faststr(0x22b2000, 0xc001d960c0, {0x1b759a4, 0x14}) /usr/local/go/src/runtime/map_faststr.go:221 +0xa8 fp=0xc000fbebc0 sp=0xc000fbeb50 pc=0x4154e8 fatal error: unexpected signal during runtime execution panic during panic [signal SIGBUS: bus error code=0x2 addr=0x3010708 pc=0x45fad1] runtime stack: runtime.throw({0x1bbc3c1?, 0x1eee0a8?}) /usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x15426c861738 sp=0x15426c861708 pc=0x439bbf runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x15426c861788 sp=0x15426c861738 pc=0x4504c9 runtime.gentraceback(0x0?, 0x8?, 0x0?, 0x15426c861cd8?, 0x0, 0x0, 0x64, 0x0, 0x15426c861b78?, 0x0) /usr/local/go/src/runtime/traceback.go:139 +0x411 fp=0x15426c861af8 sp=0x15426c861788 pc=0x45fad1 runtime.traceback1(0xc0013a3380?, 0x15426c861d10?, 0x43be7a?, 0xc0013a3380, 0x15426c861d10?) /usr/local/go/src/runtime/traceback.go:857 +0x1b1 fp=0x15426c861cb8 sp=0x15426c861af8 pc=0x462391 runtime.traceback(...) /usr/local/go/src/runtime/traceback.go:804 runtime.tracebackothers(0xc000d081a0) /usr/local/go/src/runtime/traceback.go:1049 +0x98 fp=0x15426c861d20 sp=0x15426c861cb8 pc=0x463178 runtime.dopanic_m(0xc000d081a0, 0x1eee0a8?, 0x1?) /usr/local/go/src/runtime/panic.go:1270 +0x285 fp=0x15426c861d98 sp=0x15426c861d20 pc=0x43a5c5 runtime.fatalthrow.func1() /usr/local/go/src/runtime/panic.go:1125 +0x48 fp=0x15426c861dd8 sp=0x15426c861d98 pc=0x439f88 runtime.fatalthrow(0x6c861e20?) /usr/local/go/src/runtime/panic.go:1122 +0x6c fp=0x15426c861e18 sp=0x15426c861dd8 pc=0x439f0c runtime.throw({0x1bbc3c1?, 0x4590b0?}) /usr/local/go/src/runtime/panic.go:1047 +0x5f fp=0x15426c861e48 sp=0x15426c861e18 pc=0x439bbf runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x15426c861e98 sp=0x15426c861e48 pc=0x4504c9 runtime.gentraceback(0x44beb4?, 0xc0013a3380?, 0xc000d081a0?, 0x15426c862258?, 0x4, 0xc000fbe968, 0x20, 0x0, 0xc000683d40?, 0x0) /usr/local/go/src/runtime/traceback.go:139 +0x411 fp=0x15426c862208 sp=0x15426c861e98 pc=0x45fad1 runtime.callers.func1() /usr/local/go/src/runtime/traceback.go:925 +0x4f fp=0x15426c862270 sp=0x15426c862208 pc=0x462b0f runtime.systemstack() /usr/local/go/src/runtime/asm_amd64.s:492 +0x46 fp=0x15426c862278 sp=0x15426c862270 pc=0x46be66 Syslog has the following from around that time.. not sure if it is related: Nov 6 11:33:56 unraid kernel: docker0: port 8(veth9521bea) entered blocking state Nov 6 11:33:56 unraid kernel: docker0: port 8(veth9521bea) entered disabled state Nov 6 11:33:56 unraid kernel: device veth9521bea entered promiscuous mode Nov 6 11:33:58 unraid kernel: eth0: renamed from veth36d6453 Nov 6 11:33:58 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9521bea: link becomes ready Nov 6 11:33:58 unraid kernel: docker0: port 8(veth9521bea) entered blocking state Nov 6 11:33:58 unraid kernel: docker0: port 8(veth9521bea) entered forwarding state I am running a 4x1 Gbps bond for my unraid's network interface and the switch is configured for the port-channel and has been working fine for quite a while. My SSD cache is ZFS.. I also have a ZFS pool consisting of 6 disks in a raidz1 that I use for appdata and my plex media server's growing data. The docker directory lives on the cache device. Had the same issue with it living on the ZFS pool. I am open to suggestions on troubleshooting. unraid-diagnostics-20231108-0908.zip