BTRFS error with docker.img

sr.gomi · July 27

Unraid version: 6.12.10
- Issue presented after rebooting to upgrade to 6.12.11, so I reverted, which didn't fix the issue.
Plugins
- NerdTools.txt dynamix.schedules.txt gpustat.txt tips.and.tweaks.txt
  appdata.backup.txt dynamix.system.autofan.txt gui-links.txt unassigned.devices-plus.txt
  ca.cleanup.appdata.txt dynamix.system.buttons.txt intel-gpu-top.txt unassigned.devices.plg
  ca.update.applications.txt dynamix.system.info.txt open.files.txt unassigned.devices.preclear.txt
  community.applications.txt dynamix.system.temp.txt pluginPending/ unassigned.devices.txt
  dynamix.active.streams.txt fix.common.problems.plg swapfile.txt unbalance.txt
  dynamix.cache.dirs.txt fix.common.problems.txt theme.engine.txt
Hardware
- Odroid-H2+ x86 sbc (Intel Celeron J4115), 8GB RAM

Here's the problem: after upgrading Unraid to 6.12.10, I restarted, and the Docker service failed to start. In the syslog, I see some BTRFS errors calling out /dev/loop2, which I believe is docker.img; there's also some suspicious looking lines about capacity changes, but the disks haven't changed. Here's just that section of the log (full diagnostics attached):

Jul 26 21:19:05 SMITHERS emhttpd: shcmd (94): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 40
Jul 26 21:19:05 SMITHERS kernel: loop2: detected capacity change from 0 to 83886080
Jul 26 21:19:05 SMITHERS kernel: BTRFS: device fsid 2f9e9233-4a93-4fbf-9f45-f4587b26c59a devid 1 transid 1924980 /dev/loop2 scanned by mount (16185)
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): first mount of filesystem 2f9e9233-4a93-4fbf-9f45-f4587b26c59a
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): using free space tree
Jul 26 21:19:05 SMITHERS kernel: BTRFS error (device loop2): bad tree block start, mirror 1 want 23707648 have 8073320464664375584
Jul 26 21:19:05 SMITHERS kernel: BTRFS error (device loop2): bad tree block start, mirror 2 want 23707648 have 7234314203654612783
Jul 26 21:19:05 SMITHERS kernel: BTRFS error (device loop2): failed to read chunk root
Jul 26 21:19:05 SMITHERS root: mount: /var/lib/docker: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error.
Jul 26 21:19:05 SMITHERS root:        dmesg(1) may have more information after failed mount system call.
Jul 26 21:19:05 SMITHERS kernel: BTRFS error (device loop2): open_ctree failed
Jul 26 21:19:05 SMITHERS root: mount error
Jul 26 21:19:05 SMITHERS emhttpd: shcmd (94): exit status: 1
Jul 26 21:19:05 SMITHERS emhttpd: shcmd (107): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Jul 26 21:19:05 SMITHERS kernel: loop2: detected capacity change from 0 to 2097152
Jul 26 21:19:05 SMITHERS kernel: BTRFS: device fsid a3f83ed4-45b5-414a-9289-4c28fb2d08cd devid 1 transid 211 /dev/loop2 scanned by mount (16261)
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): first mount of filesystem a3f83ed4-45b5-414a-9289-4c28fb2d08cd
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): using crc32c (crc32c-intel) checksum algorithm
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): using free space tree
Jul 26 21:19:05 SMITHERS kernel: BTRFS info (device loop2): enabling ssd optimizations
Jul 26 21:19:05 SMITHERS root: Resize device id 1 (/dev/loop2) from 1.00GiB to max
Jul 26 21:19:05 SMITHERS emhttpd: shcmd (109): /etc/rc.d/rc.libvirt start

What may be relevant is that I recently added two SSDs in USB enclosures to the cache pool, bringing it from 1 drive to 3. Those drives have been periodically overheating when a burst of work comes in. That was a few weeks ago and the server has been restarted since then and worked fine, so I don't think I messed up the cache pool config when adding drives, but something may have gone wrong while overheating.

Following a different support thread where someone else had trouble with docker.img, I stopped the Docker service and deleted the bad file before restarting the service and was able to install the Emby app, so I do have a path to get Docker working again. However, the reinstalled Emby behaved like a fresh install; restoring from a backup didn't restore my users/libraries like I'd hoped it would. I really don't want to set up and reconfigure Nextcloud and Vaultwarden and 20 other things from scratch this weekend, so this is a last resort.

So for now I put my old corrupted docker.img back and now I'm hoping for a solution to restore that file. Hopefully the solution is as simple as removing a cache disk or a btrfs check command or something

smithers-diagnostics-20240726-2121.zip