Docker failure, losing cache, btrfs failure?


sol
Go to solution Solved by sol,

Recommended Posts

Every Thursday morning for a few months now I wake up to a banner on my docker page (haven't screenshot it sorry) with no icons or access to any docker containers. A reboot seems to fix it. Unraid typically just "runs" for me and has for years, so I wasn't sweating it. Today I had some time to dig around a bit. I can't find any processes/jobs that are running on a Wednesday night/Thrusday morning (There HAS to be something....) but I did find that there were a bunch of btrfs errors in the log. I only have btrfs on my cache drive which is where the docker folder is stored of course. I restarted the system and ran a memtest with no errors reported. I then started the array in maintenance mode and ran a btrfs check on the cache drive and got this, but I don't have any idea what to do from here; 
 

[1/7] checking root items
[2/7] checking extents
data backref 40881405952 root 5 owner 46973646 offset 0 num_refs 0 not found in extent tree
incorrect local backref count on 40881405952 root 5 owner 46973646 offset 0 found 1 wanted 0 back 0x1ae7ab40
incorrect local backref count on 40881405952 root 5 owner 46973646 offset 32768 found 0 wanted 1 back 0x183e3cb0
backref disk bytenr does not match extent record, bytenr=40881405952, ref bytenr=0
backpointer mismatch on [40881405952 45056]
ref mismatch on [48950460416 8192] extent item 549755813888, found 0
owner ref check failed [48950460416 8192]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
root 5 inode 74013114 errors 200, dir isize wrong
root 5 inode 77896360 errors 1, no inode item
	unresolved ref dir 74013114 index 3862251 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896362 errors 1, no inode item
	unresolved ref dir 74013114 index 3862253 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896370 errors 1, no inode item
	unresolved ref dir 74013114 index 3862255 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896373 errors 1, no inode item
	unresolved ref dir 74013114 index 3862257 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896374 errors 1, no inode item
	unresolved ref dir 74013114 index 3862259 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896379 errors 1, no inode item
	unresolved ref dir 74013114 index 3862261 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896382 errors 1, no inode item
	unresolved ref dir 74013114 index 3862263 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896383 errors 1, no inode item
	unresolved ref dir 74013114 index 3862265 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896384 errors 1, no inode item
	unresolved ref dir 74013114 index 3862267 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896385 errors 1, no inode item
	unresolved ref dir 74013114 index 3862269 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896388 errors 1, no inode item
	unresolved ref dir 74013114 index 3862271 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896391 errors 1, no inode item
	unresolved ref dir 74013114 index 3862273 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896392 errors 1, no inode item
	unresolved ref dir 74013114 index 3862275 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896393 errors 1, no inode item
	unresolved ref dir 74013114 index 3862277 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896394 errors 1, no inode item
	unresolved ref dir 74013114 index 3862279 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896397 errors 1, no inode item
	unresolved ref dir 74013114 index 3862281 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896400 errors 1, no inode item
	unresolved ref dir 74013114 index 3862283 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896401 errors 1, no inode item
	unresolved ref dir 74013114 index 3862285 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896402 errors 1, no inode item
	unresolved ref dir 74013114 index 3862287 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
root 5 inode 77896403 errors 1, no inode item
	unresolved ref dir 74013114 index 3862289 namelen 45 name 3d18b2ea89fa5357cf6423b4e6985b46fa10195e.json filetype 1 errors 5, no dir item, no inode ref
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/sdf1
UUID: 482f734c-0b01-4faf-8bc0-86c08db6bd62
cache and super generation don't match, space cache will be invalidated
found 137627197440 bytes used, error(s) found
total csum bytes: 131599768
total tree bytes: 2780954624
total fs tree bytes: 2420736000
total extent tree bytes: 176078848
btree space waste bytes: 704096014
file data blocks allocated: 190648750080
 referenced 137950638080

 

Edited by sol
spelling
Link to comment
  • 2 weeks later...

root@tmedia:~# df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           384M  384M     0 100% /var/log
root@tmedia:~# du -sm /var/log/*
1       /var/log/apcupsd.events
1       /var/log/apcupsd.events.1
0       /var/log/btmp
0       /var/log/cron
0       /var/log/debug
1       /var/log/dmesg
0       /var/log/docker.log
383     /var/log/docker.log.1
0       /var/log/faillog
1       /var/log/gitflash
0       /var/log/lastlog
0       /var/log/libvirt
1       /var/log/maillog
0       /var/log/messages
0       /var/log/nfsd
1       /var/log/nginx
0       /var/log/packages
1       /var/log/pkgtools
0       /var/log/plugins
0       /var/log/pwfail
0       /var/log/removed_packages
0       /var/log/removed_scripts
0       /var/log/removed_uninstall_scripts
1       /var/log/samba
0       /var/log/scripts
0       /var/log/secure
0       /var/log/setup
0       /var/log/spooler
0       /var/log/swtpm
1       /var/log/syslog
2       /var/log/syslog.1
0       /var/log/unraid-api
0       /var/log/vfio-pci
1       /var/log/wtmp

Link to comment
ime="2022-02-17T09:39:13.452768189-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/11042cc115649e3ab6
time="2022-02-17T09:39:14.623462556-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b561fa76d6448f3866
time="2022-02-17T09:39:22.156585651-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/2797cb4cbee2465f19
time="2022-02-17T09:39:29.803429402-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/46ed75f24b478dac09
time="2022-02-17T09:39:42.249379513-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5d9bc873b3912de1d8
time="2022-02-17T09:41:56.087910866-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/813c029f1297badc0d
time="2022-02-17T13:17:05.810879095-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/11042cc115649e3ab6
time="2022-02-17T16:02:40.611782191-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/1e12f74ffa47552535
time="2022-02-17T16:10:26.454018823-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/2917c4264a8d7fa646
time="2022-02-17T16:12:57.570313462-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/6e5057e30f675ad9e1
time="2022-02-17T19:42:31.947233453-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/029848900a6b20de1f
time="2022-02-20T03:02:27.631077081-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/074683c5cf36bad42a
time="2022-02-20T03:02:27.869498318-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e7fbdfeaf2938bf844
time="2022-02-20T03:02:28.093142878-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e971c0d73df2f126ce
time="2022-02-20T03:02:28.759846471-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5fde6164eee377132e
time="2022-02-20T18:15:32.855833192-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e7fbdfeaf2938bf844
time="2022-02-20T18:15:35.908244972-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/e971c0d73df2f126ce
time="2022-02-20T18:21:25.657697206-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/361f9711b624934400
time="2022-02-20T18:30:55.243448330-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/623879e0f6a156ec9f
time="2022-02-20T18:31:30.396987709-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/623879e0f6a156ec9f
time="2022-02-20T18:31:33.436374626-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/361f9711b624934400
time="2022-02-21T05:16:53.142672669-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/839d03928f88bace48
time="2022-02-21T05:16:53.338638453-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/46b04ec6c0c08c67c0
time="2022-02-21T05:16:53.483946879-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/c683260bf2fce1850a
time="2022-02-21T05:16:53.747645779-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/623879e0f6a156ec9f
time="2022-02-21T05:16:53.974481072-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/361f9711b624934400
time="2022-02-21T05:16:54.837505517-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/074683c5cf36bad42a
time="2022-02-21T05:16:55.094084245-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b561fa76d6448f3866
time="2022-02-21T05:16:55.635014188-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5fde6164eee377132e
time="2022-02-21T05:16:55.880927303-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/6e5057e30f675ad9e1
time="2022-02-21T05:16:56.624917396-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/46ed75f24b478dac09
time="2022-02-21T05:16:57.072193293-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4bf9a4eefa4506e462
time="2022-02-21T05:16:57.458395737-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/813c029f1297badc0d
time="2022-02-21T05:16:57.846209906-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5d9bc873b3912de1d8
time="2022-02-23T08:04:16.299433190-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/cfac2a914f66fac185
time="2022-02-23T08:04:32.226963630-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/8e13546ef3b1fc375f
time="2022-02-23T08:04:45.446979587-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/66012d868342c76e6f
time="2022-02-23T08:05:28.088544708-06:00" level=info msg="starting signal loop" namespace=moby path=/var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ade6eb0470838d3937
time="2022-02-24T02:08:39.381326314-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:39.388577844-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:39.388614598-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:39.388655682-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:40.382377322-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:40.389256094-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:40.389308017-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:40.389369289-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:41.340254415-06:00" level=error msg="Error replicating health state for container 6e5057e30f675ad9e12892bdab82a35ccbf6c79520f015c145eb219607b9392f: open /var/lib/doc
time="2022-02-24T02:08:41.383229258-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:41.389602197-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68e
time="2022-02-24T02:08:41.389639522-06:00" level=error msg="Failed to log msg \"\" for logger json-file: error writing log entry: write /var/lib/docker/containers/623879e0f6a156ec9f5f2e68

The beginning of the errors of docker.log.1 are at the bottom here. 

Link to comment

The container throwing errors at 2:08 in the above docker.log.1 is Plex. Which makes sense as it is one of the few containers that do weekly tasks. Like; 

 

 Optimize database every week

 Remove old bundles every week

 Remove old cache files every week

 Refresh local metadata every three days

 Update all libraries during maintenance

 Upgrade media analysis during maintenance

 Refresh library metadata periodically

 Perform extensive media analysis during maintenance

 Fetch missing location names for items in photo sections

 Analyze and tag photos

Edited by sol
spelling
Link to comment
  • 2 weeks later...
  • Solution

Replaced cache drive and formatted xfs. Moved everything back and rebuilt docker containers. 

Up nine days with no issues. No full log, no read only cache. 

Obviously some kind of cache drive issue even though it tested good on extended smart. Likely btrfs issue that couldn't be easily solved with a solo drive. 

btrfs likely should not be the default format for solo cache drives in unraid. 

 

Solved

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.