Thanks guys.

 

Edit: Call trace near the bottom of log.

 

Diag is posted below.

tower-diagnostics-20171019-0804.zip

Edited October 19, 20178 yr by MowMdown

Docker image is corrupt, delete and re-create.

4 minutes ago, johnnie.black said:

Docker image is corrupt, delete and re-create.

 

Where can I buy you a coffee/beer because you're more than helpful around here. Do you have paypal?

Edited October 19, 20178 yr by MowMdown

@johnnie.black

 

I'm just curious for future reference, what exactly gives it away that the image is corrupt?

 

What should I be looking for (what causes) this particular call trace?

The call traces were related to btrfs and you can see in the end there's corruption on the loop0 device, that's the docker image:

 

Oct 18 23:36:41 Tower kernel: ------------[ cut here ]------------
Oct 18 23:36:41 Tower kernel: WARNING: CPU: 3 PID: 30191 at fs/btrfs/extent-tree.c:134 btrfs_put_block_group+0x42/0x59
Oct 18 23:36:41 Tower kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables tun iptable_mangle xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod nct6775 hwmon_vid jc42 e1000e igb ptp pps_core i2c_algo_bit x86_pkg_temp_thermal coretemp i2c_i801 i2c_smbus ahci i2c_core libahci video backlight [last unloaded: kvm]
Oct 18 23:36:41 Tower kernel: CPU: 3 PID: 30191 Comm: umount Not tainted 4.9.30-unRAID #1
Oct 18 23:36:41 Tower kernel: Hardware name: Supermicro X10SAE/X10SAE, BIOS 3.0 05/20/2015
Oct 18 23:36:41 Tower kernel: ffffc9000dd63d20 ffffffff813a4a1b 0000000000000000 ffffffff819692b0
Oct 18 23:36:41 Tower kernel: ffffc9000dd63d60 ffffffff8104d0d9 000000860dd63d00 ffff8803cae3c200
Oct 18 23:36:41 Tower kernel: ffff8803cae3c2e8 ffff880408904090 ffff8803cae3c200 ffff8804089040a0
Oct 18 23:36:41 Tower kernel: Call Trace:
Oct 18 23:36:41 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Oct 18 23:36:41 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Oct 18 23:36:41 Tower kernel: [<ffffffff8104d1a1>] warn_slowpath_null+0x18/0x1a
Oct 18 23:36:41 Tower kernel: [<ffffffff812d6c15>] btrfs_put_block_group+0x42/0x59
Oct 18 23:36:41 Tower kernel: [<ffffffff812dcf26>] btrfs_free_block_groups+0x19d/0x39b
Oct 18 23:36:41 Tower kernel: [<ffffffff812ec0f4>] close_ctree+0x1d4/0x2e8
Oct 18 23:36:41 Tower kernel: [<ffffffff81137ff1>] ? evict_inodes+0x128/0x137
Oct 18 23:36:41 Tower kernel: [<ffffffff812c6f4b>] btrfs_put_super+0x14/0x16
Oct 18 23:36:41 Tower kernel: [<ffffffff811232fb>] generic_shutdown_super+0x6a/0xeb
Oct 18 23:36:41 Tower kernel: [<ffffffff8112355e>] kill_anon_super+0xd/0x17
Oct 18 23:36:41 Tower kernel: [<ffffffff812c6e92>] btrfs_kill_super+0x11/0x94
Oct 18 23:36:41 Tower kernel: [<ffffffff81123750>] deactivate_locked_super+0x3b/0x68
Oct 18 23:36:41 Tower kernel: [<ffffffff811240ec>] deactivate_super+0x39/0x3c
Oct 18 23:36:41 Tower kernel: [<ffffffff8113b02e>] cleanup_mnt+0x53/0x71
Oct 18 23:36:41 Tower kernel: [<ffffffff8113b080>] __cleanup_mnt+0xd/0xf
Oct 18 23:36:41 Tower kernel: [<ffffffff81062609>] task_work_run+0x6a/0x7d
Oct 18 23:36:41 Tower kernel: [<ffffffff81002b24>] exit_to_usermode_loop+0x55/0x81
Oct 18 23:36:41 Tower kernel: [<ffffffff81002c61>] syscall_return_slowpath+0x44/0x47
Oct 18 23:36:41 Tower kernel: [<ffffffff8167f5c4>] entry_SYSCALL_64_fastpath+0xa7/0xa9
Oct 18 23:36:41 Tower kernel: ---[ end trace f3942ab63f71241a ]---
Oct 18 23:36:41 Tower kernel: ------------[ cut here ]------------
Oct 18 23:36:41 Tower kernel: WARNING: CPU: 3 PID: 30191 at fs/btrfs/extent-tree.c:10045 btrfs_free_block_groups+0x37a/0x39b
Oct 18 23:36:41 Tower kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables tun iptable_mangle xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod nct6775 hwmon_vid jc42 e1000e igb ptp pps_core i2c_algo_bit x86_pkg_temp_thermal coretemp i2c_i801 i2c_smbus ahci i2c_core libahci video backlight [last unloaded: kvm]
Oct 18 23:36:41 Tower kernel: CPU: 3 PID: 30191 Comm: umount Tainted: G        W       4.9.30-unRAID #1
Oct 18 23:36:41 Tower kernel: Hardware name: Supermicro X10SAE/X10SAE, BIOS 3.0 05/20/2015
Oct 18 23:36:41 Tower kernel: ffffc9000dd63d38 ffffffff813a4a1b 0000000000000000 ffffffff819692b0
Oct 18 23:36:41 Tower kernel: ffffc9000dd63d78 ffffffff8104d0d9 0000273d0dd63d40 ffff880408904000
Oct 18 23:36:41 Tower kernel: ffff8804094a5688 ffff8804094a5600 ffff880408904b30 0000000000000000
Oct 18 23:36:41 Tower kernel: Call Trace:
Oct 18 23:36:41 Tower kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Oct 18 23:36:41 Tower kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Oct 18 23:36:41 Tower kernel: [<ffffffff8104d1a1>] warn_slowpath_null+0x18/0x1a
Oct 18 23:36:41 Tower kernel: [<ffffffff812dd103>] btrfs_free_block_groups+0x37a/0x39b
Oct 18 23:36:41 Tower kernel: [<ffffffff812ec0f4>] close_ctree+0x1d4/0x2e8
Oct 18 23:36:41 Tower kernel: [<ffffffff81137ff1>] ? evict_inodes+0x128/0x137
Oct 18 23:36:41 Tower kernel: [<ffffffff812c6f4b>] btrfs_put_super+0x14/0x16
Oct 18 23:36:41 Tower kernel: [<ffffffff811232fb>] generic_shutdown_super+0x6a/0xeb
Oct 18 23:36:41 Tower kernel: [<ffffffff8112355e>] kill_anon_super+0xd/0x17
Oct 18 23:36:41 Tower kernel: [<ffffffff812c6e92>] btrfs_kill_super+0x11/0x94
Oct 18 23:36:41 Tower kernel: [<ffffffff81123750>] deactivate_locked_super+0x3b/0x68
Oct 18 23:36:41 Tower kernel: [<ffffffff811240ec>] deactivate_super+0x39/0x3c
Oct 18 23:36:41 Tower kernel: [<ffffffff8113b02e>] cleanup_mnt+0x53/0x71
Oct 18 23:36:41 Tower kernel: [<ffffffff8113b080>] __cleanup_mnt+0xd/0xf
Oct 18 23:36:41 Tower kernel: [<ffffffff81062609>] task_work_run+0x6a/0x7d
Oct 18 23:36:41 Tower kernel: [<ffffffff81002b24>] exit_to_usermode_loop+0x55/0x81
Oct 18 23:36:41 Tower kernel: [<ffffffff81002c61>] syscall_return_slowpath+0x44/0x47
Oct 18 23:36:41 Tower kernel: [<ffffffff8167f5c4>] entry_SYSCALL_64_fastpath+0xa7/0xa9
Oct 18 23:36:41 Tower kernel: ---[ end trace f3942ab63f71241b ]---
Oct 18 23:36:41 Tower kernel: BTRFS info (device loop0): space_info 4 has 760315904 free, is not full
Oct 18 23:36:41 Tower kernel: BTRFS info (device loop0): space_info total=1073741824, used=313344000, pinned=0, reserved=16384, may_use=0, readonly=65536
Oct 18 23:36:41 Tower kernel: BTRFS warning (device loop0): page private not zero on page 915849216
Oct 18 23:36:41 Tower kernel: BTRFS warning (device loop0): page private not zero on page 915853312
Oct 18 23:36:41 Tower kernel: BTRFS warning (device loop0): page private not zero on page 915857408
Oct 18 23:36:41 Tower kernel: BTRFS warning (device loop0): page private not zero on page 915861504

Corruption is mostly cased by the docker image (or the cache device) running out of space or when there are any read/write errors to it.

hrm.. neither were anywhere close to being full (roughly 25%)

 

I followed a guide to isolate a NIC (mobo has 2) for a VM and when I did that, I lost access to the webGUI even when booted into webGUI mode on the machine itself.

 

localhost wouldn't load so I reformatted the USB (moved over the super.dat file and the key) and basically set it up like it was new. 

 

I had some strange docker issue where (since the templates went away) I had to basically re-install them to get them to work properly.

 

thats probably what messed it up. everything seemed ok last night and then this morning thats when I saw a call trace issued in the log.

Edited October 19, 20178 yr by MowMdown