Call Traces found on your server

[schford]

HI Guys,

 

I have been through the logs but cant see anything wrong and it all seems ok?  Any one point me in the right direction?

 

Thanks

 

Stuart

mbfs01-diagnostics-20170922-0916.zip

[JorgeB]

Your docker image is corrupt, delete and re-create.

 

https://forums.lime-technology.com/topic/36647-official-guide-restoring-your-docker-applications-in-a-new-image-file/

 

[schford]

Thanks - what should I be looking for in the logs to show that?

[JorgeB]

Sep 17 11:35:58 MBFS01 kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 12, rd 0, flush 0, corrupt 0, gen 0
Sep 17 11:35:58 MBFS01 shfs/user: err: shfs_write: write: (28) No space left on device
Sep 17 11:35:58 MBFS01 kernel: loop: Write error at byte offset 11915517952, length 4096.
Sep 17 11:35:58 MBFS01 kernel: blk_update_request: I/O error, dev loop0, sector 23272448
Sep 17 11:35:58 MBFS01 kernel: BTRFS error (device loop0): bdev /dev/loop0 errs: wr 13, rd 0, flush 0, corrupt 0, gen 0
Sep 17 11:35:59 MBFS01 kernel: BTRFS: error (device loop0) in btrfs_commit_transaction:2227: errno=-5 IO failure (Error while writing out transaction)
Sep 17 11:35:59 MBFS01 kernel: BTRFS info (device loop0): forced readonly
Sep 17 11:35:59 MBFS01 kernel: BTRFS warning (device loop0): Skipping commit of aborted transaction.
Sep 17 11:35:59 MBFS01 kernel: ------------[ cut here ]------------
Sep 17 11:35:59 MBFS01 kernel: WARNING: CPU: 3 PID: 10719 at fs/btrfs/transaction.c:1850 cleanup_transaction+0x8c/0x238
Sep 17 11:35:59 MBFS01 kernel: BTRFS: Transaction aborted (error -5)
Sep 17 11:35:59 MBFS01 kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net vhost macvtap macvlan tun xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod igb ptp pps_core fbcon ast bitblit fbcon_rotate fbcon_ccw fbcon_ud fbcon_cw softcursor ttm font drm_kms_helper cfbfillrect x86_pkg_temp_thermal cfbimgblt coretemp cfbcopyarea kvm_intel drm kvm agpgart mpt3sas syscopyarea sysfillrect sysimgblt i2c_i801 fb_sys_fops ahci i2c_algo_bit i2c_smbus fb raid_class libahci i2c_core fbdev scsi_transport_sas ipmi_si video backlight [last unloaded: pps_core]
Sep 17 11:35:59 MBFS01 kernel: CPU: 3 PID: 10719 Comm: btrfs-transacti Not tainted 4.9.30-unRAID #1
Sep 17 11:35:59 MBFS01 kernel: Hardware name: Supermicro X10SL7-F/X10SL7-F, BIOS 3.0 04/24/2015
Sep 17 11:35:59 MBFS01 kernel: ffffc9001239bcd0 ffffffff813a4a1b ffffc9001239bd20 ffffffff8196b262
Sep 17 11:35:59 MBFS01 kernel: ffffc9001239bd10 ffffffff8104d0d9 0000073a1239bd88 ffff8806a8c8cc08
Sep 17 11:35:59 MBFS01 kernel: ffff8807f8227800 ffff8807cca37f00 00000000fffffffb 0000000000000000
Sep 17 11:35:59 MBFS01 kernel: Call Trace:
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff8104d13a>] warn_slowpath_fmt+0x46/0x4e
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff812eeff1>] cleanup_transaction+0x8c/0x238
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff8107c0fd>] ? wake_up_bit+0x25/0x25
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff812f0d59>] btrfs_commit_transaction.part.11+0x912/0x927
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff812f0db4>] btrfs_commit_transaction+0x46/0x4d
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff812ebe74>] transaction_kthread+0xf0/0x19c
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff812ebd84>] ? btrfs_cleanup_transaction+0x479/0x479
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff81063939>] kthread+0xdb/0xe3
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff8106385e>] ? kthread_park+0x52/0x52
Sep 17 11:35:59 MBFS01 kernel: [<ffffffff8167f785>] ret_from_fork+0x25/0x30
Sep 17 11:35:59 MBFS01 kernel: ---[ end trace 09fe540a639a0b58 ]---

 

[schford]

Thanks! Another note hwo do I add to your rep to say thanks!

[JorgeB]

Click on the little heart on the bottom right of the post.

[JorgeB]

Also note that the image got corrupted because it was out of space, either a misconfigured docker, or a full cache device, your cache filesystem is fully allocated, this is not good for performance since trim won't be working on most of the free space and it can also lead to out of space errors, see here for more info:

 

 

[Zangief]

On 22/09/2017 at 9:23 AM, schford said:

HI Guys,

 

I have been through the logs but cant see anything wrong and it all seems ok?  Any one point me in the right direction?

 

Thanks

 

Stuart

mbfs01-diagnostics-20170922-0916.zip

 

Getting the same issue Stuart.... something to do with our Supermicro boards maybe?

 

Anis