Patb Posted August 5, 2019 Share Posted August 5, 2019 (edited) My server froze last night and l had to do a reboot. The docker service fails to start after reboot and I have what appear to be BTRFS errors. My cache used to be set to BTRFS but I changes it to XFS not too long ago after I had a crash that I couldn't recover from. I don't understand the BTRFS errors when I don't have a BTRFS formatted drive. I am attaching the log since the previous restart and my diagnostics. Help The first error appears to be: Aug 5 02:41:41 Tower kernel: WARNING: CPU: 12 PID: 8042 at fs/btrfs/ctree.c:3611 leaf_space_used+0xc6/0xef Aug 5 02:41:41 Tower kernel: Modules linked in: iptable_mangle tun macvlan xt_nat veth ipt_MASQUERADE iptable_nat nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod bonding bnx2x mdio igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 ipmi_ssif crypto_simd cryptd glue_helper intel_cstate mpt3sas intel_uncore intel_rapl_perf i2c_i801 i2c_core raid_class wmi ipmi_si scsi_transport_sas button pcc_cpufreq [last unloaded: mdio] Aug 5 02:41:41 Tower kernel: CPU: 12 PID: 8042 Comm: btrfs-transacti Tainted: G W 4.19.56-Unraid #1 Aug 5 02:41:41 Tower kernel: Hardware name: IBM System x3630 M4 -[7158AC1]-/00KF924, BIOS -[BEE166CUS-3.10]- 08/29/2018 Aug 5 02:41:41 Tower kernel: RIP: 0010:leaf_space_used+0xc6/0xef Aug 5 02:41:41 Tower kernel: Code: 41 01 c4 44 39 fb 48 89 e1 0f 4e f3 45 6b ed 19 ff ce 48 63 f6 45 01 e5 48 6b f6 19 48 83 c6 65 e8 6b b8 03 00 41 29 c5 79 02 <0f> 0b 48 8b 4c 24 18 65 48 33 0c 25 28 00 00 00 44 89 e8 74 05 e8 Aug 5 02:41:41 Tower kernel: RSP: 0018:ffffc900074ffc00 EFLAGS: 00010283 Aug 5 02:41:41 Tower kernel: RAX: 0000000000778d42 RBX: 00000000000000c7 RCX: ffff888ae609a000 Aug 5 02:41:41 Tower kernel: RDX: 0000000000001000 RSI: 000000016b204000 RDI: 0000000000000000 Aug 5 02:41:41 Tower kernel: RBP: ffff88814ba33a40 R08: ffffc900074ffbb8 R09: ffffc900074ffbc0 Aug 5 02:41:41 Tower kernel: R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000003f9b Aug 5 02:41:41 Tower kernel: R13: 00000000ff88c5c8 R14: 0000000000000065 R15: 00000000000000c7 Aug 5 02:41:41 Tower kernel: FS: 0000000000000000(0000) GS:ffff888667980000(0000) knlGS:0000000000000000 Aug 5 02:41:41 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 5 02:41:41 Tower kernel: CR2: 00001463c3c808e8 CR3: 0000000001e0a006 CR4: 00000000000606e0 Aug 5 02:41:41 Tower kernel: Call Trace: Aug 5 02:41:41 Tower kernel: btrfs_leaf_free_space+0x48/0x82 Aug 5 02:41:41 Tower kernel: btrfs_search_slot+0x64d/0x832 Aug 5 02:41:41 Tower kernel: btrfs_insert_empty_items+0x4e/0x91 Aug 5 02:41:41 Tower kernel: __btrfs_run_delayed_refs+0x7c3/0xb96 Aug 5 02:41:41 Tower kernel: ? __switch_to_asm+0x35/0x70 Aug 5 02:41:41 Tower kernel: btrfs_run_delayed_refs+0x5a/0x15c Aug 5 02:41:41 Tower kernel: ? try_to_del_timer_sync+0x4f/0x6e Aug 5 02:41:41 Tower kernel: btrfs_commit_transaction+0x50/0x76b Aug 5 02:41:41 Tower kernel: ? start_transaction+0x29e/0x30e Aug 5 02:41:41 Tower kernel: transaction_kthread+0xca/0x136 Aug 5 02:41:41 Tower kernel: ? btrfs_cleanup_transaction+0x4a1/0x4a1 Aug 5 02:41:41 Tower kernel: kthread+0x10b/0x113 Aug 5 02:41:41 Tower kernel: ? kthread_park+0x71/0x71 Aug 5 02:41:41 Tower kernel: ret_from_fork+0x35/0x40 Aug 5 02:41:41 Tower kernel: ---[ end trace 1e3060d86793bb3c ]--- Aug 5 02:41:41 Tower kernel: WARNING: CPU: 12 PID: 8042 at fs/btrfs/ctree.c:3611 leaf_space_used+0xc6/0xef Aug 5 02:41:41 Tower kernel: Modules linked in: iptable_mangle tun macvlan xt_nat veth ipt_MASQUERADE iptable_nat nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod bonding bnx2x mdio igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 ipmi_ssif crypto_simd cryptd glue_helper intel_cstate mpt3sas intel_uncore intel_rapl_perf i2c_i801 i2c_core raid_class wmi ipmi_si scsi_transport_sas button pcc_cpufreq [last unloaded: mdio] Aug 5 02:41:41 Tower kernel: CPU: 12 PID: 8042 Comm: btrfs-transacti Tainted: G W 4.19.56-Unraid #1 Aug 5 02:41:41 Tower kernel: Hardware name: IBM System x3630 M4 -[7158AC1]-/00KF924, BIOS -[BEE166CUS-3.10]- 08/29/2018 Aug 5 02:41:41 Tower kernel: RIP: 0010:leaf_space_used+0xc6/0xef Aug 5 02:41:41 Tower kernel: Code: 41 01 c4 44 39 fb 48 89 e1 0f 4e f3 45 6b ed 19 ff ce 48 63 f6 45 01 e5 48 6b f6 19 48 83 c6 65 e8 6b b8 03 00 41 29 c5 79 02 <0f> 0b 48 8b 4c 24 18 65 48 33 0c 25 28 00 00 00 44 89 e8 74 05 e8 Aug 5 02:41:41 Tower kernel: RSP: 0018:ffffc900074ffbf8 EFLAGS: 00010283 Aug 5 02:41:41 Tower kernel: RAX: 0000000000778d42 RBX: 00000000000000c7 RCX: ffff888ae609a000 Aug 5 02:41:41 Tower kernel: RDX: 0000000000001000 RSI: 000000016b204000 RDI: 0000000000000000 Aug 5 02:41:41 Tower kernel: RBP: ffff88814ba33a40 R08: ffffc900074ffbb0 R09: ffffc900074ffbb8 Aug 5 02:41:41 Tower kernel: R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000003f9b Aug 5 02:41:41 Tower kernel: R13: 00000000ff88c5c8 R14: 0000000000000065 R15: 00000000000000c7 Aug 5 02:41:41 Tower kernel: FS: 0000000000000000(0000) GS:ffff888667980000(0000) knlGS:0000000000000000 Aug 5 02:41:41 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 5 02:41:41 Tower kernel: CR2: 00001463c3c808e8 CR3: 0000000001e0a006 CR4: 00000000000606e0 Aug 5 02:41:41 Tower kernel: Call Trace: Aug 5 02:41:41 Tower kernel: btrfs_leaf_free_space+0x48/0x82 Aug 5 02:41:41 Tower kernel: setup_items_for_insert+0xf4/0x352 Aug 5 02:41:41 Tower kernel: btrfs_insert_empty_items+0x7b/0x91 Aug 5 02:41:41 Tower kernel: __btrfs_run_delayed_refs+0x7c3/0xb96 Aug 5 02:41:41 Tower kernel: ? __switch_to_asm+0x35/0x70 Aug 5 02:41:41 Tower kernel: btrfs_run_delayed_refs+0x5a/0x15c Aug 5 02:41:41 Tower kernel: ? try_to_del_timer_sync+0x4f/0x6e Aug 5 02:41:41 Tower kernel: btrfs_commit_transaction+0x50/0x76b Aug 5 02:41:41 Tower kernel: ? start_transaction+0x29e/0x30e Aug 5 02:41:41 Tower kernel: transaction_kthread+0xca/0x136 Aug 5 02:41:41 Tower kernel: ? btrfs_cleanup_transaction+0x4a1/0x4a1 Aug 5 02:41:41 Tower kernel: kthread+0x10b/0x113 Aug 5 02:41:41 Tower kernel: ? kthread_park+0x71/0x71 Aug 5 02:41:41 Tower kernel: ret_from_fork+0x35/0x40 Aug 5 02:41:41 Tower kernel: ---[ end trace 1e3060d86793bb3d ]--- and then I get a series of these: Aug 5 02:41:41 Tower kernel: BTRFS info (device loop2): leaf 6092242944 gen 60401 total ptrs 199 free space 7829971 owner 2 Aug 5 02:41:41 Tower kernel: item 0 key (6089441280 169 0) itemoff 16223 itemsize 60 Aug 5 02:41:41 Tower kernel: extent refs 4 gen 10446 flags 2 Aug 5 02:41:41 Tower kernel: ref#0: tree block backref root 462 Aug 5 02:41:41 Tower kernel: ref#1: tree block backref root 461 Aug 5 02:41:41 Tower kernel: ref#2: tree block backref root 460 Aug 5 02:41:41 Tower kernel: ref#3: tree block backref root 458 Aug 5 02:41:41 Tower kernel: item 1 key (6089457664 169 0) itemoff 16100 itemsize 123 Aug 5 02:41:41 Tower kernel: extent refs 11 gen 10446 flags 2 Aug 5 02:41:41 Tower kernel: ref#0: tree block backref root 469 Aug 5 02:41:41 Tower kernel: ref#1: tree block backref root 468 Aug 5 02:41:41 Tower kernel: ref#2: tree block backref root 467 Aug 5 02:41:41 Tower kernel: ref#3: tree block backref root 466 Aug 5 02:41:41 Tower kernel: ref#4: tree block backref root 465 Aug 5 02:41:41 Tower kernel: ref#5: tree block backref root 464 Aug 5 02:41:41 Tower kernel: ref#6: tree block backref root 463 Aug 5 02:41:41 Tower kernel: ref#7: tree block backref root 462 Aug 5 02:41:41 Tower kernel: ref#8: tree block backref root 461 Aug 5 02:41:41 Tower kernel: ref#9: tree block backref root 460 Aug 5 02:41:41 Tower kernel: ref#10: tree block backref root 458 And then when I reboot I can't start docker and I get the following in the log: Aug 5 07:40:04 Tower kernel: BTRFS: device fsid ba577f11-99fc-487e-a96e-aed9e181a702 devid 1 transid 60400 /dev/loop2 Aug 5 07:40:04 Tower kernel: BTRFS info (device loop2): disk space caching is enabled Aug 5 07:40:04 Tower kernel: BTRFS info (device loop2): has skinny extents Aug 5 07:40:05 Tower kernel: BTRFS critical (device loop2): corrupt leaf: root=2 block=6073548800 slot=131, unexpected item end, have 1810063561 expect 10088 Aug 5 07:40:05 Tower kernel: BTRFS critical (device loop2): corrupt leaf: root=2 block=6073548800 slot=131, unexpected item end, have 1810063561 expect 10088 Aug 5 07:40:05 Tower kernel: BTRFS critical (device loop2): corrupt leaf: root=2 block=6073548800 slot=131, unexpected item end, have 1810063561 expect 10088 Aug 5 07:40:05 Tower kernel: BTRFS: error (device loop2) in __btrfs_free_extent:6803: errno=-5 IO failure Aug 5 07:40:05 Tower kernel: BTRFS: error (device loop2) in btrfs_run_delayed_refs:2935: errno=-5 IO failure Aug 5 07:40:05 Tower kernel: BTRFS: error (device loop2) in btrfs_replay_log:2282: errno=-5 IO failure (Failed to recover log tree) Aug 5 07:40:05 Tower kernel: BTRFS error (device loop2): pending csums is 69632 Aug 5 07:40:05 Tower root: mount: /var/lib/docker: can't read superblock on /dev/loop2. Aug 5 07:40:05 Tower kernel: BTRFS error (device loop2): open_ctree failed Aug 5 07:40:05 Tower root: mount error Aug 5 07:40:05 Tower emhttpd: shcmd (85): exit status: 1 Aug 5 07:40:05 Tower kernel: mdcmd (41): check nocorrect Aug 5 07:40:05 Tower kernel: md: recovery thread: check Q ... Aug 5 07:40:05 Tower kernel: md: using 8192k window, over a total of 9766436812 blocks. Aug 5 07:40:07 Tower unassigned.devices: Mounting 'Auto Mount' Remote Shares... Aug 5 07:40:11 Tower unassigned.devices: Remote SMB mount '192.168.2.207://volume1/Various' is not set to auto mount and will not be mounted... Aug 5 07:40:11 Tower unassigned.devices: Remote SMB mount '192.168.2.206://volume1/photo' is not set to auto mount and will not be mounted... Aug 5 07:40:11 Tower unassigned.devices: Remote SMB mount '192.168.2.206://volume1/video' is not set to auto mount and will not be mounted... Aug 5 07:40:11 Tower unassigned.devices: Remote SMB mount '192.168.2.206://volume1/music' is not set to auto mount and will not be mounted... Aug 5 07:40:11 Tower unassigned.devices: Remote SMB mount '192.168.2.207://volume1/Backup' is not set to auto mount and will not be mounted... Aug 5 07:40:11 Tower unassigned.devices: Remote SMB mount '192.168.2.206://volume1/homes' is not set to auto mount and will not be mounted... Aug 5 07:40:18 Tower nginx: 2019/08/05 07:40:18 [error] 5948#5948: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Aug 5 07:40:18 Tower nginx: 2019/08/05 07:40:18 [error] 5948#5948: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Aug 5 07:40:18 Tower nginx: 2019/08/05 07:40:18 [error] 5948#5948: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. syslog paired down.txt tower-diagnostics-20190805-1215.zip Edited August 5, 2019 by Patb solved Quote Link to comment
JorgeB Posted August 5, 2019 Share Posted August 5, 2019 loop2 is the docker image, and it's corrupt, delete and recreate. Quote Link to comment
trurl Posted August 5, 2019 Share Posted August 5, 2019 I can't tell much about docker image since it doesn't mount. Have you had problems with filling it? Quote Link to comment
Patb Posted August 5, 2019 Author Share Posted August 5, 2019 (edited) ok, I'm trying that. I've deleted it and recreated it. I am now adding the apps back. My docker is back up. Thanks Edited August 5, 2019 by Patb more information Quote Link to comment
trurl Posted August 5, 2019 Share Posted August 5, 2019 2 hours ago, Patb said: I am now adding the apps back. Did you use the Previous Apps feature on the Apps page? Much easier to get things setup as they were that way. 2 hours ago, trurl said: I can't tell much about docker image since it doesn't mount. Have you had problems with filling it? Quote Link to comment
Patb Posted August 5, 2019 Author Share Posted August 5, 2019 Yes I did. I read about it when I searched how to recreate the docker image. Thanks Quote Link to comment
trurl Posted August 5, 2019 Share Posted August 5, 2019 I'm still wondering about this since I haven't received an answer. Filling docker image seems to be the usual way it gets corrupted. This will happen again if you are filling docker image. You had it set at 20G and that is what I always recommend. You should never come close to filling that unless you have application(s) misconfigured. Did you leave it at 20G when you recreated it? Making it larger isn't the solution to filling it, it will only make it take longer to fill and corrupt again. Quote Link to comment
Patb Posted August 5, 2019 Author Share Posted August 5, 2019 (edited) Sorry, I missed that question. It wasn't full but I nevertheless increased the size to 30G. I will monitor it more closely. I am presently at 30% usage. Edited August 5, 2019 by Patb additional information Quote Link to comment
trurl Posted August 5, 2019 Share Posted August 5, 2019 1 hour ago, Patb said: I nevertheless increased the size to 30G. I recommend not increasing beyond the 20 you had before. Increasing it is not a solution to any problem, but increasing it can make it take longer to discover you have a problem. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.