Hello guys, I'm back to document the (up to now) solution to my problem. Diagnosis and solution supported by Claude (claude.ai)"

---

UPDATE - Root Cause Found & Resolved

After further investigation, the Docker daemon crashes (goroutine dumps, CopyToPipe FIFO blocks, libnetwork DNS resolver hangs) were not primarily a Docker bug — they were a symptom of a failing USB flash drive.

What we found:

Running dmesg revealed the real culprit:

sd 0:0:0:0: [sda] Sense Key : 0x3  ← MEDIUM ERROR
sd 0:0:0:0: [sda] ASC=0x11 ASCQ=0x0  ← Unrecovered Read Error
critical medium error, dev sda, sector 13500440
SQUASHFS error: Failed to read block 0x243b6a8: -5
SQUASHFS error: xz decompression failed, data probably corrupt

sda was the Unraid USB boot drive (SanDisk Cruzer, the low-end model). The failing sectors were part of the OS squashfs — specifically the region containing the dockerd binary. When Docker tried to restart, it couldn't read its own executable from disk, resulting in a Bus error.

The goroutine dump was a snapshot of the daemon's state at the moment of the crash — the FIFO and DNS goroutines were leaked/blocked, but they were a consequence of the abrupt daemon death, not the cause.

Timeline reconstruction:

• Docker daemon degraded for 1+ hour with leaked goroutines (FIFO blocked 13 min, DNS resolver blocked 82 min)

• USB read errors caused dockerd to crash uncleanly

• docker.img remained mounted as an orphan (confirmed by is in-use, cannot mount in syslog)

• Subsequent restart attempts failed with Bus error due to continued USB read failures

Resolution:

1. Backed up /boot/config (including Plus.key) while the system was still running

2. Prepared a new USB drive (SanDisk Extreme 64GB) on Linux with a proper 2GB FAT32 partition

3. Took the opportunity to upgrade directly from Unraid 6.12.14 → 7.2.4 (Docker 24.0.9 → 27.5.1)

4. Transferred the license to the new USB GUID via Unraid support

5. After 15+ hours uptime: zero goroutine dumps, zero disk read errors, all containers stable

Lessons learned:

• If Docker daemon crashes with goroutine dumps, always check dmesg for hardware errors first before assuming it's a Docker bug

• SanDisk Cruzer/basic USB drives are not suitable for continuous Unraid use — prefer SanDisk Extreme, Samsung FIT Plus or similar drives with proper wear leveling

• The Bus error on dockerd restart is a strong indicator of underlying storage/hardware failure

• Keep /boot/config backed up regularly — in this case it saved the migration

Current status: Running stable on Unraid 7.2.4, Docker 27.5.1, new USB flash drive. Parity check completed with only 1 correction (expected after unclean shutdown).

---

I'll try recreating my boot drive then.

Meanwhile, I think we can rule out my memory as being the cause, right? I mean, 15 passes of memtest86+ isn't exactly unstable 😜

I also rebuilt my boot drive, tested it numerous times, ran every hardware stability test out there, ran my Corsair SF750 through a PSU tester, used a different motherboard, etc. and it still crashes.

Out of desperation, I upgraded to 7.3.0 beta 1, and now it fails with the following message:

msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"

Edited March 22Mar 22 by andrebrait

I have now moved to a mirror ZFS boot partition and the license is now on the TPM. This rules out any issues with the flash drive since it's not even present anymore.

It seems to be somehow connected to Tailscale.

I'm taking the time to debug it, but every time my rsync job over Tailscale failed, Docker also crashed.

I disabled Tailscale on unRAID, relying on my Firewall to do the routing for me, so let's see.

Nope. It crashed again.

I'm lost. Gonna try enabling debug logs for dockerd.

The latest logs I got seem to indicate a nil pointer is being dereferenced when trying to run the health check for my crowdsec container.

I have reviewed all my healthcheck commands to ensure they exit properly and timeout properly.

Well, yes, it was a healthcheck. Somehow Docker is not handling it well. Only after I removed all health checks it stopped breaking.

docker (1).log (2).txt

After multiple days of uptime now, I have confirmed the crash was due to some healthcheck in a container.

I disabled healthchecks for all containers both via --no-healthcheck in the additional arguments, for unRAID templates; as well as healthcheck.disable: true in the compose yaml, and all crashes are gone.

I'll begin enabling healthchecks one by one again, but it's scary this can ever happen. Going to report the issue to Docker (moby) as it shouldn't be possible for a bad healthcheck toncause the daemon to crash, after all.

@JorgeB this is resolved. Upstream bug in Docker and interaction with healthchecks.

@JorgeB it turns out it was a failing power supply.

The Corsair SF750 Platinum I was using was probably having a hard time dealing with the bursty loads caused by Docker's recurring maintenance tasks/checks so it ended up happening disproportionally there, but the machine was also turning itself off.

It's been running like a champ since I disabled all power saving features possible to confirm the PSU had issues with low power modes.