I auto-update my dockers (most of them anyhow) every Sunday night and this morning many of them were not working.  The "Fix Common Problem" plugin detected the following errors:

 

Fix Common Problems: Error: Unable to write to Docker Image
Fix Common Problems: Error: Call Traces found on your server

So I took a look at the syslog and found the following entries (end of post).  I'm not entirely sure where to go from here, so any debug help would be greatly appreciated.  I am not at the machine, so I have yet to simply reboot, but I'd like to figure out if this is a real problem or not.  For the record, my setup has four data drives, one parity, and no cache drive.  I have not come across any SMART errors on my disks and the last parity check was run on January 1st (unsure if this is relevant).

 

I have also attached the full diagnostics file.  Thanks in advance for any help!

 

Jan  8 01:41:37 baratheon kernel: WARNING: CPU: 4 PID: 10763 at fs/btrfs/extent-tree.c:6952 __btrfs_free_extent+0x424/0xa1d
Jan  8 01:41:37 baratheon kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net vhost macvtap macvlan iptable_mangle tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod k10temp bonding e1000e igb ptp pps_core i2c_algo_bit x86_pkg_temp_thermal coretemp kvm_intel i2c_i801 kvm mxm_wmi i2c_smbus ahci i2c_core cdc_acm libahci video wmi backlight [last unloaded: pps_core]
Jan  8 01:41:37 baratheon kernel: CPU: 4 PID: 10763 Comm: btrfs-transacti Not tainted 4.9.30-unRAID #1
Jan  8 01:41:37 baratheon kernel: Hardware name: System manufacturer System Product Name/TUF Z270 MARK 1, BIOS 0505 11/08/2016
Jan  8 01:41:37 baratheon kernel: ffffc90004897bd0 ffffffff813a4a1b 0000000000000000 ffffffff819692b0
Jan  8 01:41:37 baratheon kernel: ffffc90004897c10 ffffffff8104d0d9 00001b2800000000 00000000fffffffe
Jan  8 01:41:37 baratheon kernel: ffff8801b5c1a620 ffff88080c868500 0000000000000000 0000000000000000
Jan  8 01:41:37 baratheon kernel: Call Trace:
Jan  8 01:41:37 baratheon kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Jan  8 01:41:37 baratheon kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Jan  8 01:41:37 baratheon kernel: [<ffffffff8104d1a1>] warn_slowpath_null+0x18/0x1a
Jan  8 01:41:37 baratheon kernel: [<ffffffff812d786d>] __btrfs_free_extent+0x424/0xa1d
Jan  8 01:41:37 baratheon kernel: [<ffffffff812dbd43>] __btrfs_run_delayed_refs+0xaaf/0xd21
Jan  8 01:41:37 baratheon kernel: [<ffffffff812dd9c6>] btrfs_run_delayed_refs+0x8f/0x241
Jan  8 01:41:37 baratheon kernel: [<ffffffff812f0d9a>] btrfs_commit_transaction+0x2c/0x4d
Jan  8 01:41:37 baratheon kernel: [<ffffffff812ebe74>] transaction_kthread+0xf0/0x19c
Jan  8 01:41:37 baratheon kernel: [<ffffffff812ebd84>] ? btrfs_cleanup_transaction+0x479/0x479
Jan  8 01:41:37 baratheon kernel: [<ffffffff81063939>] kthread+0xdb/0xe3
Jan  8 01:41:37 baratheon kernel: [<ffffffff8106385e>] ? kthread_park+0x52/0x52
Jan  8 01:41:37 baratheon kernel: [<ffffffff8167f785>] ret_from_fork+0x25/0x30
  
  <--- snipped --->
    
    Jan  8 01:41:37 baratheon kernel: WARNING: CPU: 4 PID: 10763 at fs/btrfs/extent-tree.c:6958 __btrfs_free_extent+0x486/0xa1d
Jan  8 01:41:37 baratheon kernel: BTRFS: Transaction aborted (error -2)
Jan  8 01:41:37 baratheon kernel: Modules linked in: xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net vhost macvtap macvlan iptable_mangle tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod k10temp bonding e1000e igb ptp pps_core i2c_algo_bit x86_pkg_temp_thermal coretemp kvm_intel i2c_i801 kvm mxm_wmi i2c_smbus ahci i2c_core cdc_acm libahci video wmi backlight [last unloaded: pps_core]
Jan  8 01:41:37 baratheon kernel: CPU: 4 PID: 10763 Comm: btrfs-transacti Tainted: G        W       4.9.30-unRAID #1
Jan  8 01:41:37 baratheon kernel: Hardware name: System manufacturer System Product Name/TUF Z270 MARK 1, BIOS 0505 11/08/2016
Jan  8 01:41:37 baratheon kernel: ffffc90004897b78 ffffffff813a4a1b ffffc90004897bc8 ffffffff819692b0
Jan  8 01:41:37 baratheon kernel: ffffc90004897bb8 ffffffff8104d0d9 00001b2e04897c30 00000000fffffffe
Jan  8 01:41:37 baratheon kernel: ffff8801b5c1a620 ffff88080c868500 0000000000000000 0000000000000000
Jan  8 01:41:37 baratheon kernel: Call Trace:
Jan  8 01:41:37 baratheon kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Jan  8 01:41:37 baratheon kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Jan  8 01:41:37 baratheon kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e
Jan  8 01:41:37 baratheon kernel: [<ffffffff812d78cf>] __btrfs_free_extent+0x486/0xa1d
Jan  8 01:41:37 baratheon kernel: [<ffffffff812dbd43>] __btrfs_run_delayed_refs+0xaaf/0xd21
Jan  8 01:41:37 baratheon kernel: [<ffffffff812dd9c6>] btrfs_run_delayed_refs+0x8f/0x241
Jan  8 01:41:37 baratheon kernel: [<ffffffff812f0d9a>] btrfs_commit_transaction+0x2c/0x4d
Jan  8 01:41:37 baratheon kernel: [<ffffffff812ebe74>] transaction_kthread+0xf0/0x19c
Jan  8 01:41:37 baratheon kernel: [<ffffffff812ebd84>] ? btrfs_cleanup_transaction+0x479/0x479
Jan  8 01:41:37 baratheon kernel: [<ffffffff81063939>] kthread+0xdb/0xe3
Jan  8 01:41:37 baratheon kernel: [<ffffffff8106385e>] ? kthread_park+0x52/0x52
Jan  8 01:41:37 baratheon kernel: [<ffffffff8167f785>] ret_from_fork+0x25/0x30
Jan  8 01:41:37 baratheon kernel: ---[ end trace a171acc71f517fdc ]---
Jan  8 01:41:37 baratheon kernel: BTRFS: error (device loop0) in __btrfs_free_extent:6958: errno=-2 No such entry
Jan  8 01:41:37 baratheon kernel: BTRFS info (device loop0): forced readonly
Jan  8 01:41:37 baratheon kernel: BTRFS: error (device loop0) in btrfs_run_delayed_refs:2967: errno=-2 No such entry
Jan  8 01:41:37 baratheon kernel: BTRFS error (device loop0): pending csums is 86016
Jan  8 09:28:03 baratheon kernel: vethaa7a70c: renamed from eth0
Jan  8 09:28:03 baratheon kernel: docker0: port 9(veth859b120) entered disabled state

 

 

baratheon-diagnostics-20180108-0935.zip

Edited January 8, 20188 yr by fronzbot

You have filesystem corruption disk1 (md1)

 

http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems#Drives_formatted_with_XFS

 

After fixing best to delete and recreate de docker image.

Ah, good catch, thank you!  I'll give that a go when I can get to my machine.

 

EDIT - xfs_repair fixed some stuff and it looks like I'm good to go.  Thanks for the help!

Edited January 8, 20188 yr by fronzbot