Symptoms:
Docker containers stop working.
Cannot stop docker, the docker.img file in use.
I noticed BTRFS errors in the log.
BTRFS scrubs do not detect errors.

 

Tried restarting docker, but couldn't umount the docker.img file.

Forced the umount:

umount /dev/loop2

 

This is happening weekly.  Sometimes the docker.img file is corrupted according to BTRFS scrub.  Not all configuration settings are restored when recreating the file (custom networks, delay start).

 

I've done 5 memory test passes.
I've done the extended tests on all my hard drives, several times.
No hardware issues detected, except for one pool with a bunch of junk drives that have a SMART error.  I don't care about these drives or the data.

 

I also am doing a lot of I/O on my unassigned drives and the junk pool (old external USB drives), that I use for temporary torrents.

 

Another symptom showed up where Unraid froze, keyboard unresponsive, requiring a forced/unclean reboot.

 

Sep 13 12:35:13 Tower emhttpd: spinning down /dev/sdm
Sep 13 12:41:06 Tower kernel: BTRFS critical (device loop2): corrupt node: root=7 block=13634125824 slot=47, bad key order, current (18446744073709551606 128 7868407808) next (9300072153669959670 136 7885029375)
Sep 13 12:41:06 Tower kernel: BTRFS info (device loop2): node 13634125824 level 1 gen 4979 total ptrs 336 free spc 157 owner 7
Sep 13 12:41:06 Tower kernel: #011key 0 (18446744073709551606 128 7183642624) block 13587808256 gen 4942
Sep 13 12:41:06 Tower kernel: #011key 1 (18446744073709551606 128 7193964544) block 13587857408 gen 4942
...
...
Sep 13 15:00:49 Tower emhttpd: shcmd (535023): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 42
Sep 13 15:00:49 Tower avahi-daemon[20138]: Server startup complete. Host name is Tower.local. Local service cookie is 1906093824.
Sep 13 15:00:49 Tower root: '/mnt/user/system/docker/docker.img' is in-use, cannot mount
Sep 13 15:00:49 Tower emhttpd: shcmd (535023): exit status: 1

 

Attaching my logs.  I'm using the syslog server, but still looks like the log isn't saved.  I've attached a log before the reboot, and the one after.

Logs are not private/anonymous.  The log will capture files from my BTRFS subvolume/snaprshots.  If files/photos have lables, they are identifiable PII information.

 

In my logs I've seen appdata backups, drive spindowns, one of the pools has a SMART error, and BiglyBT (not a docker container) is doing a lot of I/O on the non-array pool over the network (from another machine).  Maybe I'm just stressing my machine out too much, and it can't handle it.  Or my machine is too old.  But it seems to happen when BiglyBT is pushing a lot of I/O.  And none of that should be related to docker containers or non-array pools.

 

 

tower-diagnostics-20230913-1858.zip tower-diagnostics-20230913-1711.zip

Edited September 14, 20232 yr by Jaybau

If the docker image keeps getting corrupt it points to a hardware issue, but you can try using an xfs image to see if it's any different.