sol Posted February 10, 2022 Share Posted February 10, 2022 (edited) Every Thursday morning for a few months now I wake up to a banner on my docker page (haven't screenshot it sorry) with no icons or access to any docker containers. A reboot seems to fix it. Unraid typically just "runs" for me and has for years, so I wasn't sweating it. Today I had some time to dig around a bit. I can't find any processes/jobs that are running on a Wednesday night/Thrusday morning (There HAS to be something....) but I did find that there were a bunch of btrfs errors in the log. I only have btrfs on my cache drive which is where the docker folder is stored of course. I restarted the system and ran a memtest with no errors reported. I then started the array in maintenance mode and ran a btrfs check on the cache drive and got this, but I don't have any idea what to do from here; [1/7] checking root items [2/7] checking extents data backref 40881405952 root 5 owner 46973646 offset 0 num_refs 0 not found in extent tree incorrect local backref count on 40881405952 root 5 owner 46973646 offset 0 found 1 wanted 0 back 0x1ae7ab40 incorrect local backref count on 40881405952 root 5 owner 46973646 offset 32768 found 0 wanted 1 back 0x183e3cb0 backref disk bytenr does not match extent record, bytenr=40881405952, ref bytenr=0 backpointer mismatch on [40881405952 45056] ref mismatch on [48950460416 8192] extent item 549755813888, found 0 owner ref check failed [48950460416 8192] ERROR: errors found in extent allocation tree or chunk allocation [3/7] checking free space tree [4/7] checking fs roots root 5 inode 74013114 errors 200, dir isize wrong root 5 inode 77896360 errors 1, no inode item unresolved ref dir 74013114 index 3862251 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896362 errors 1, no inode item unresolved ref dir 74013114 index 3862253 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896370 errors 1, no inode item unresolved ref dir 74013114 index 3862255 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896373 errors 1, no inode item unresolved ref dir 74013114 index 3862257 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896374 errors 1, no inode item unresolved ref dir 74013114 index 3862259 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896379 errors 1, no inode item unresolved ref dir 74013114 index 3862261 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896382 errors 1, no inode item unresolved ref dir 74013114 index 3862263 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896383 errors 1, no inode item unresolved ref dir 74013114 index 3862265 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896384 errors 1, no inode item unresolved ref dir 74013114 index 3862267 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896385 errors 1, no inode item unresolved ref dir 74013114 index 3862269 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896388 errors 1, no inode item unresolved ref dir 74013114 index 3862271 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896391 errors 1, no inode item unresolved ref dir 74013114 index 3862273 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896392 errors 1, no inode item unresolved ref dir 74013114 index 3862275 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896393 errors 1, no inode item unresolved ref dir 74013114 index 3862277 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896394 errors 1, no inode item unresolved ref dir 74013114 index 3862279 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896397 errors 1, no inode item unresolved ref dir 74013114 index 3862281 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896400 errors 1, no inode item unresolved ref dir 74013114 index 3862283 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896401 errors 1, no inode item unresolved ref dir 74013114 index 3862285 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896402 errors 1, no inode item unresolved ref dir 74013114 index 3862287 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref root 5 inode 77896403 errors 1, no inode item unresolved ref dir 74013114 index 3862289 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref ERROR: errors found in fs roots Opening filesystem to check... Checking filesystem on /dev/sdf1 UUID: 482f734c-0b01-4faf-8bc0-86c08db6bd62 cache and super generation don't match, space cache will be invalidated found 137627197440 bytes used, error(s) found total csum bytes: 131599768 total tree bytes: 2780954624 total fs tree bytes: 2420736000 total extent tree bytes: 176078848 btree space waste bytes: 704096014 file data blocks allocated: 190648750080 referenced 137950638080 Edited February 10, 2022 by sol spelling Quote Link to comment
sol Posted February 24, 2022 Author Share Posted February 24, 2022 root@tmedia:~# df -h /var/log Filesystem Size Used Avail Use% Mounted on tmpfs 384M 384M 0 100% /var/log root@tmedia:~# du -sm /var/log/* 1 /var/log/apcupsd.events 1 /var/log/apcupsd.events.1 0 /var/log/btmp 0 /var/log/cron 0 /var/log/debug 1 /var/log/dmesg 0 /var/log/docker.log 383 /var/log/docker.log.1 0 /var/log/faillog 1 /var/log/gitflash 0 /var/log/lastlog 0 /var/log/libvirt 1 /var/log/maillog 0 /var/log/messages 0 /var/log/nfsd 1 /var/log/nginx 0 /var/log/packages 1 /var/log/pkgtools 0 /var/log/plugins 0 /var/log/pwfail 0 /var/log/removed_packages 0 /var/log/removed_scripts 0 /var/log/removed_uninstall_scripts 1 /var/log/samba 0 /var/log/scripts 0 /var/log/secure 0 /var/log/setup 0 /var/log/spooler 0 /var/log/swtpm 1 /var/log/syslog 2 /var/log/syslog.1 0 /var/log/unraid-api 0 /var/log/vfio-pci 1 /var/log/wtmp Quote Link to comment
sol Posted February 24, 2022 Author Share Posted February 24, 2022 ime="2022-02-17T09:39:13.452768189-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/11042cc115649e3ab6 time="2022-02-17T09:39:14.623462556-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b561fa76d6448f3866 time="2022-02-17T09:39:22.156585651-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/2797cb4cbee2465f19 time="2022-02-17T09:39:29.803429402-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/46ed75f24b478dac09 time="2022-02-17T09:39:42.249379513-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5d9bc873b3912de1d8 time="2022-02-17T09:41:56.087910866-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/813c029f1297badc0d time="2022-02-17T13:17:05.810879095-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/11042cc115649e3ab6 time="2022-02-17T16:02:40.611782191-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/1e12f74ffa47552535 time="2022-02-17T16:10:26.454018823-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/2917c4264a8d7fa646 time="2022-02-17T16:12:57.570313462-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/6e5057e30f675ad9e1 time="2022-02-17T19:42:31.947233453-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/029848900a6b20de1f time="2022-02-20T03:02:27.631077081-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/074683c5cf36bad42a time="2022-02-20T03:02:27.869498318-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e7fbdfeaf2938bf844 time="2022-02-20T03:02:28.093142878-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e971c0d73df2f126ce time="2022-02-20T03:02:28.759846471-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5fde6164eee377132e time="2022-02-20T18:15:32.855833192-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e7fbdfeaf2938bf844 time="2022-02-20T18:15:35.908244972-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e971c0d73df2f126ce time="2022-02-20T18:21:25.657697206-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/361f9711b624934400 time="2022-02-20T18:30:55.243448330-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/623879e0f6a156ec9f time="2022-02-20T18:31:30.396987709-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/623879e0f6a156ec9f time="2022-02-20T18:31:33.436374626-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/361f9711b624934400 time="2022-02-21T05:16:53.142672669-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/839d03928f88bace48 time="2022-02-21T05:16:53.338638453-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/46b04ec6c0c08c67c0 time="2022-02-21T05:16:53.483946879-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/c683260bf2fce1850a time="2022-02-21T05:16:53.747645779-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/623879e0f6a156ec9f time="2022-02-21T05:16:53.974481072-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/361f9711b624934400 time="2022-02-21T05:16:54.837505517-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/074683c5cf36bad42a time="2022-02-21T05:16:55.094084245-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b561fa76d6448f3866 time="2022-02-21T05:16:55.635014188-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5fde6164eee377132e time="2022-02-21T05:16:55.880927303-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/6e5057e30f675ad9e1 time="2022-02-21T05:16:56.624917396-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/46ed75f24b478dac09 time="2022-02-21T05:16:57.072193293-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4bf9a4eefa4506e462 time="2022-02-21T05:16:57.458395737-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/813c029f1297badc0d time="2022-02-21T05:16:57.846209906-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5d9bc873b3912de1d8 time="2022-02-23T08:04:16.299433190-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/cfac2a914f66fac185 time="2022-02-23T08:04:32.226963630-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/8e13546ef3b1fc375f time="2022-02-23T08:04:45.446979587-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/66012d868342c76e6f time="2022-02-23T08:05:28.088544708-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ade6eb0470838d3937 time="2022-02-24T02:08:39.381326314-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:39.388577844-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:39.388614598-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:39.388655682-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:40.382377322-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:40.389256094-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:40.389308017-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:40.389369289-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:41.340254415-06:00" level=error msg="Error replicating health state for container 6e5057e30f675ad9e12892bdab82a35ccbf6c79520f015c145eb219607b9392f: open /var/lib/doc time="2022-02-24T02:08:41.383229258-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:41.389602197-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e time="2022-02-24T02:08:41.389639522-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68 The beginning of the errors of docker.log.1 are at the bottom here. Quote Link to comment
sol Posted February 24, 2022 Author Share Posted February 24, 2022 tmedia-diagnostics-20220224-0746.zip Quote Link to comment
sol Posted February 24, 2022 Author Share Posted February 24, 2022 (edited) The container throwing errors at 2:08 in the above docker.log.1 is Plex. Which makes sense as it is one of the few containers that do weekly tasks. Like; Optimize database every week Remove old bundles every week Remove old cache files every week Refresh local metadata every three days Update all libraries during maintenance Upgrade media analysis during maintenance Refresh library metadata periodically Perform extensive media analysis during maintenance Fetch missing location names for items in photo sections Analyze and tag photos Edited February 24, 2022 by sol spelling Quote Link to comment
JorgeB Posted February 24, 2022 Share Posted February 24, 2022 Feb 24 02:08:38 tmedia kernel: BTRFS info (device sdf1): forced readonly Corruption on the cache filesystem, best bet is to backup and re-format. Quote Link to comment
sol Posted February 24, 2022 Author Share Posted February 24, 2022 Yeah, I am going to re-format to XFS and check the drive health. Maybe replace the drive while I'm at it. Quote Link to comment
Solution sol Posted March 11, 2022 Author Solution Share Posted March 11, 2022 Replaced cache drive and formatted xfs. Moved everything back and rebuilt docker containers. Up nine days with no issues. No full log, no read only cache. Obviously some kind of cache drive issue even though it tested good on extended smart. Likely btrfs issue that couldn't be easily solved with a solo drive. btrfs likely should not be the default format for solo cache drives in unraid. Solved Quote Link to comment
ChatNoir Posted March 11, 2022 Share Posted March 11, 2022 16 minutes ago, sol said: btrfs likely should not be the default format for solo cache drives in unraid. I get that, but there is no way for the OS to know if a user plans to add additional drives in the future. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.