January 7, 20179 yr Hi all, I've been having an issue over the last several days and I'm struggling to find a cause for it. That issue being my server seemingly locking up. I can no longer access it on my network, router loses arp etc.. But the fans are still spinning (hence my belief that it's a lockup). When it started, I was running 6.1.9, Today I figured I'd upgrade to latest (6.2.4) to try and fix the issue, but no such luck. I tried turning redoing my entire docker image, in the hope that this may fix the issue, but it's still happening. Even with the docker image turned off, I was transferring ~11GB of small files, and it froze while doing so. I telnetted in and ran 'tail -f /var/log/syslog' and let it run, fortunately? the next crash happened about 5 minutes later. Reading the log, I have a feeling that it may be saying there's a CPU issue, but I'm not exactly an expert on reading syslogs ... If anyone can assist, that'd be brilliant! Cheers Kharn I'll post the syslog over several posts, as it's hitting the character limit; Jan 7 19:34:52 Kryten kernel: XFS (md7): _xfs_buf_find: Block out of range: block 0x8ffffffe8, EOFS 0x3a3812a48 Jan 7 19:34:52 Kryten kernel: ------------[ cut here ]------------ Jan 7 19:34:52 Kryten kernel: WARNING: CPU: 3 PID: 32678 at fs/xfs/xfs_buf.c:472 _xfs_buf_find+0x7f/0x28c() Jan 7 19:34:52 Kryten kernel: Modules linked in: xt_nat veth xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod coretemp kvm_intel kvm mpt3sas i2c_i801 i2c_core ata_piix ahci raid_class r8169 scsi_transport_sas libahci mii pata_jmicron acpi_cpufreq Jan 7 19:34:52 Kryten kernel: CPU: 3 PID: 32678 Comm: kworker/u16:2 Not tainted 4.4.30-unRAID #2 Jan 7 19:34:52 Kryten kernel: Hardware name: Gigabyte Technology Co., Ltd. P55-UD3/P55-UD3, BIOS F8 05/06/2010 Jan 7 19:34:52 Kryten kernel: Workqueue: writeback wb_workfn (flush-9:7) Jan 7 19:34:52 Kryten kernel: 0000000000000000 ffff8801047233e8 ffffffff8136f79f 0000000000000000 Jan 7 19:34:52 Kryten kernel: 00000000000001d8 ffff880104723420 ffffffff8104a4ab ffffffff81274012 Jan 7 19:34:52 Kryten kernel: ffff88011baa13f0 ffff8800da021840 ffff8800da021840 0000000000000000 Jan 7 19:34:52 Kryten kernel: Call Trace: Jan 7 19:34:52 Kryten kernel: [<ffffffff8136f79f>] dump_stack+0x61/0x7e Jan 7 19:34:52 Kryten kernel: [<ffffffff8104a4ab>] warn_slowpath_common+0x8f/0xa8 Jan 7 19:34:52 Kryten kernel: [<ffffffff81274012>] ? _xfs_buf_find+0x7f/0x28c Jan 7 19:34:52 Kryten kernel: [<ffffffff8104a568>] warn_slowpath_null+0x15/0x17 Jan 7 19:34:52 Kryten kernel: [<ffffffff81274012>] _xfs_buf_find+0x7f/0x28c Jan 7 19:34:52 Kryten kernel: [<ffffffff81274243>] xfs_buf_get_map+0x24/0x12b Jan 7 19:34:52 Kryten kernel: [<ffffffff81295be4>] xfs_trans_get_buf_map+0x80/0xa7 Jan 7 19:34:52 Kryten kernel: [<ffffffff81256229>] xfs_btree_get_bufs+0x4b/0x4d Jan 7 19:34:52 Kryten kernel: [<ffffffff81246c1f>] xfs_alloc_fix_freelist+0x174/0x2de Jan 7 19:34:52 Kryten kernel: [<ffffffff812460f2>] ? xfs_alloc_ag_vextent_near+0x78f/0x7f6 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126c5ed>] ? xfs_perag_get+0x3a/0x44 Jan 7 19:34:52 Kryten kernel: [<ffffffff81246fd2>] xfs_alloc_vextent+0x249/0x3a3 Jan 7 19:34:52 Kryten kernel: [<ffffffff81252671>] xfs_bmap_btalloc+0x3c8/0x598 Jan 7 19:34:52 Kryten kernel: [<ffffffff8125284a>] xfs_bmap_alloc+0x9/0xb Jan 7 19:34:52 Kryten kernel: [<ffffffff812530de>] xfs_bmapi_write+0x40e/0x80f Jan 7 19:34:52 Kryten kernel: [<ffffffff8127e1c3>] xfs_iomap_write_allocate+0x1c1/0x2bd Jan 7 19:34:52 Kryten kernel: [<ffffffff8126f066>] xfs_map_blocks+0x134/0x143 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126fd0d>] xfs_vm_writepage+0x287/0x483 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17b9>] __writepage+0xe/0x26 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1c79>] write_cache_pages+0x249/0x351 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17ab>] ? mapping_tagged+0xf/0xf Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1dc9>] generic_writepages+0x48/0x67 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] ? xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c3738>] do_writepages+0x1b/0x24 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112a5d4>] __writeback_single_inode+0x3d/0x151 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ab89>] writeback_sb_inodes+0x20d/0x3ad Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ad9a>] __writeback_inodes_wb+0x71/0xa9 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112af80>] wb_writeback+0x10b/0x195 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] ? wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8105aede>] process_one_work+0x194/0x2a0 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b894>] worker_thread+0x26b/0x353 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b629>] ? rescuer_thread+0x285/0x285 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fb24>] kthread+0xcd/0xd5 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: [<ffffffff81629f7f>] ret_from_fork+0x3f/0x70 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: ---[ end trace b862f8a408519d7e ]--- Jan 7 19:34:52 Kryten kernel: XFS (md7): _xfs_buf_find: Block out of range: block 0x8ffffffe8, EOFS 0x3a3812a48
January 7, 20179 yr Author part 2 Jan 7 19:34:52 Kryten kernel: ------------[ cut here ]------------ Jan 7 19:34:52 Kryten kernel: WARNING: CPU: 3 PID: 32678 at fs/xfs/xfs_buf.c:472 _xfs_buf_find+0x7f/0x28c() Jan 7 19:34:52 Kryten kernel: Modules linked in: xt_nat veth xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod coretemp kvm_intel kvm mpt3sas i2c_i801 i2c_core ata_piix ahci raid_class r8169 scsi_transport_sas libahci mii pata_jmicron acpi_cpufreq Jan 7 19:34:52 Kryten kernel: CPU: 3 PID: 32678 Comm: kworker/u16:2 Tainted: G W 4.4.30-unRAID #2 Jan 7 19:34:52 Kryten kernel: Hardware name: Gigabyte Technology Co., Ltd. P55-UD3/P55-UD3, BIOS F8 05/06/2010 Jan 7 19:34:52 Kryten kernel: Workqueue: writeback wb_workfn (flush-9:7) Jan 7 19:34:52 Kryten kernel: 0000000000000000 ffff8801047233e8 ffffffff8136f79f 0000000000000000 Jan 7 19:34:52 Kryten kernel: 00000000000001d8 ffff880104723420 ffffffff8104a4ab ffffffff81274012 Jan 7 19:34:52 Kryten kernel: 0000000000000000 ffff8800da021840 ffff8800da021840 ffff8801e8c03200 Jan 7 19:34:52 Kryten kernel: Call Trace: Jan 7 19:34:52 Kryten kernel: [<ffffffff8136f79f>] dump_stack+0x61/0x7e Jan 7 19:34:52 Kryten kernel: [<ffffffff8104a4ab>] warn_slowpath_common+0x8f/0xa8 Jan 7 19:34:52 Kryten kernel: [<ffffffff81274012>] ? _xfs_buf_find+0x7f/0x28c Jan 7 19:34:52 Kryten kernel: [<ffffffff8104a568>] warn_slowpath_null+0x15/0x17 Jan 7 19:34:52 Kryten kernel: [<ffffffff81274012>] _xfs_buf_find+0x7f/0x28c Jan 7 19:34:52 Kryten kernel: [<ffffffff812730e7>] ? xfs_buf_allocate_memory+0x161/0x299 Jan 7 19:34:52 Kryten kernel: [<ffffffff8127428f>] xfs_buf_get_map+0x70/0x12b Jan 7 19:34:52 Kryten kernel: [<ffffffff81295be4>] xfs_trans_get_buf_map+0x80/0xa7 Jan 7 19:34:52 Kryten kernel: [<ffffffff81256229>] xfs_btree_get_bufs+0x4b/0x4d Jan 7 19:34:52 Kryten kernel: [<ffffffff81246c1f>] xfs_alloc_fix_freelist+0x174/0x2de Jan 7 19:34:52 Kryten kernel: [<ffffffff812460f2>] ? xfs_alloc_ag_vextent_near+0x78f/0x7f6 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126c5ed>] ? xfs_perag_get+0x3a/0x44 Jan 7 19:34:52 Kryten kernel: [<ffffffff81246fd2>] xfs_alloc_vextent+0x249/0x3a3 Jan 7 19:34:52 Kryten kernel: [<ffffffff81252671>] xfs_bmap_btalloc+0x3c8/0x598 Jan 7 19:34:52 Kryten kernel: [<ffffffff8125284a>] xfs_bmap_alloc+0x9/0xb Jan 7 19:34:52 Kryten kernel: [<ffffffff812530de>] xfs_bmapi_write+0x40e/0x80f Jan 7 19:34:52 Kryten kernel: [<ffffffff8127e1c3>] xfs_iomap_write_allocate+0x1c1/0x2bd Jan 7 19:34:52 Kryten kernel: [<ffffffff8126f066>] xfs_map_blocks+0x134/0x143 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126fd0d>] xfs_vm_writepage+0x287/0x483 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17b9>] __writepage+0xe/0x26 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1c79>] write_cache_pages+0x249/0x351 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17ab>] ? mapping_tagged+0xf/0xf Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1dc9>] generic_writepages+0x48/0x67 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] ? xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c3738>] do_writepages+0x1b/0x24 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112a5d4>] __writeback_single_inode+0x3d/0x151 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ab89>] writeback_sb_inodes+0x20d/0x3ad Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ad9a>] __writeback_inodes_wb+0x71/0xa9 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112af80>] wb_writeback+0x10b/0x195 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] ? wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8105aede>] process_one_work+0x194/0x2a0 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b894>] worker_thread+0x26b/0x353 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b629>] ? rescuer_thread+0x285/0x285 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fb24>] kthread+0xcd/0xd5 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: [<ffffffff81629f7f>] ret_from_fork+0x3f/0x70 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: ---[ end trace b862f8a408519d7f ]--- Jan 7 19:34:52 Kryten kernel: BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8 Jan 7 19:34:52 Kryten kernel: IP: [<ffffffff81295eee>] xfs_trans_binval+0x7/0x80 Jan 7 19:34:52 Kryten kernel: PGD 1ef4e1067 PUD 1ef4e0067 PMD 0 Jan 7 19:34:52 Kryten kernel: Oops: 0000 [#1] PREEMPT SMP Jan 7 19:34:52 Kryten kernel: Modules linked in: xt_nat veth xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod coretemp kvm_intel kvm mpt3sas i2c_i801 i2c_core ata_piix ahci raid_class r8169 scsi_transport_sas libahci mii pata_jmicron acpi_cpufreq Jan 7 19:34:52 Kryten kernel: CPU: 3 PID: 32678 Comm: kworker/u16:2 Tainted: G W 4.4.30-unRAID #2 Jan 7 19:34:52 Kryten kernel: Hardware name: Gigabyte Technology Co., Ltd. P55-UD3/P55-UD3, BIOS F8 05/06/2010 Jan 7 19:34:52 Kryten kernel: Workqueue: writeback wb_workfn (flush-9:7) Jan 7 19:34:52 Kryten kernel: task: ffff88005fe4d600 ti: ffff880104720000 task.ti: ffff880104720000 Jan 7 19:34:52 Kryten kernel: RIP: 0010:[<ffffffff81295eee>] [<ffffffff81295eee>] xfs_trans_binval+0x7/0x80 Jan 7 19:34:52 Kryten kernel: RSP: 0018:ffff880104723500 EFLAGS: 00010246 Jan 7 19:34:52 Kryten kernel: RAX: 0000000000000000 RBX: ffff880104723650 RCX: 0000000016618103 Jan 7 19:34:52 Kryten kernel: RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffff88011baa13f0 Jan 7 19:34:52 Kryten kernel: RBP: ffff880104723510 R08: 0000000000018ed0 R09: ffffffff81273ac8 Jan 7 19:34:52 Kryten kernel: R10: ffffea0007a30080 R11: 000000000001c2b0 R12: ffff88011baa13f0 Jan 7 19:34:52 Kryten kernel: R13: ffff8800da0219c0 R14: ffff880209a2f800 R15: 0000000000000006 Jan 7 19:34:52 Kryten kernel: FS: 0000000000000000(0000) GS:ffff88021fcc0000(0000) knlGS:0000000000000000 Jan 7 19:34:52 Kryten kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jan 7 19:34:52 Kryten kernel: CR2: 00000000000000f8 CR3: 00000002149f8000 CR4: 00000000000006e0 Jan 7 19:34:52 Kryten kernel: Stack: Jan 7 19:34:52 Kryten kernel: ffff880104723650 ffff88011baa13f0 ffff8801047235e0 ffffffff81246c2a Jan 7 19:34:52 Kryten kernel: 000000018125605e ffffffff142a55f0 ffff8800c14c3380 0000000000000000 Jan 7 19:34:52 Kryten kernel: ffff8801047235c8 ffffffff812460f2 0000000000000200 ffff880098b5b050 Jan 7 19:34:52 Kryten kernel: Call Trace: Jan 7 19:34:52 Kryten kernel: [<ffffffff81246c2a>] xfs_alloc_fix_freelist+0x17f/0x2de Jan 7 19:34:52 Kryten kernel: [<ffffffff812460f2>] ? xfs_alloc_ag_vextent_near+0x78f/0x7f6 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126c5ed>] ? xfs_perag_get+0x3a/0x44 Jan 7 19:34:52 Kryten kernel: [<ffffffff81246fd2>] xfs_alloc_vextent+0x249/0x3a3 Jan 7 19:34:52 Kryten kernel: [<ffffffff81252671>] xfs_bmap_btalloc+0x3c8/0x598 Jan 7 19:34:52 Kryten kernel: [<ffffffff8125284a>] xfs_bmap_alloc+0x9/0xb Jan 7 19:34:52 Kryten kernel: [<ffffffff812530de>] xfs_bmapi_write+0x40e/0x80f Jan 7 19:34:52 Kryten kernel: [<ffffffff8127e1c3>] xfs_iomap_write_allocate+0x1c1/0x2bd Jan 7 19:34:52 Kryten kernel: [<ffffffff8126f066>] xfs_map_blocks+0x134/0x143 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126fd0d>] xfs_vm_writepage+0x287/0x483 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17b9>] __writepage+0xe/0x26 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1c79>] write_cache_pages+0x249/0x351 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17ab>] ? mapping_tagged+0xf/0xf Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1dc9>] generic_writepages+0x48/0x67 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] ? xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c3738>] do_writepages+0x1b/0x24 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112a5d4>] __writeback_single_inode+0x3d/0x151 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ab89>] writeback_sb_inodes+0x20d/0x3ad Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ad9a>] __writeback_inodes_wb+0x71/0xa9 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112af80>] wb_writeback+0x10b/0x195 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] ? wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8105aede>] process_one_work+0x194/0x2a0 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b894>] worker_thread+0x26b/0x353 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b629>] ? rescuer_thread+0x285/0x285 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fb24>] kthread+0xcd/0xd5 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: [<ffffffff81629f7f>] ret_from_fork+0x3f/0x70 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: Code: 01 8b 50 78 89 d6 83 ce 0a 80 e2 80 89 70 78 75 12 55 89 ca 44 89 ce 48 89 c7 48 89 e5 e8 c8 94 ff ff 5d c3 55 48 89 e5 41 54 53 <48> 8b 9e f8 00 00 00 f6 43 78 04 75 67 49 89 fc 48 89 f7 e8 fd Jan 7 19:34:52 Kryten kernel: RIP [<ffffffff81295eee>] xfs_trans_binval+0x7/0x80 Jan 7 19:34:52 Kryten kernel: RSP <ffff880104723500> Jan 7 19:34:52 Kryten kernel: CR2: 00000000000000f8 Jan 7 19:34:52 Kryten kernel: ---[ end trace b862f8a408519d80 ]---
January 7, 20179 yr Author Third and final part. Jan 7 19:34:52 Kryten kernel: ------------[ cut here ]------------ Jan 7 19:34:52 Kryten kernel: WARNING: CPU: 3 PID: 32678 at kernel/softirq.c:150 __local_bh_enable_ip+0x3b/0x85() Jan 7 19:34:52 Kryten kernel: Modules linked in: xt_nat veth xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod coretemp kvm_intel kvm mpt3sas i2c_i801 i2c_core ata_piix ahci raid_class r8169 scsi_transport_sas libahci mii pata_jmicron acpi_cpufreq Jan 7 19:34:52 Kryten kernel: CPU: 3 PID: 32678 Comm: kworker/u16:2 Tainted: G D W 4.4.30-unRAID #2 Jan 7 19:34:52 Kryten kernel: Hardware name: Gigabyte Technology Co., Ltd. P55-UD3/P55-UD3, BIOS F8 05/06/2010 Jan 7 19:34:52 Kryten kernel: fffffffffffffff2 ffff880104723208 ffffffff8136f79f 0000000000000000 Jan 7 19:34:52 Kryten kernel: 0000000000000096 ffff880104723240 ffffffff8104a4ab ffffffff8104d87e Jan 7 19:34:52 Kryten kernel: 0000000000000201 ffff88005fe4d600 ffff8801047232d0 0000000000000000 Jan 7 19:34:52 Kryten kernel: Call Trace: Jan 7 19:34:52 Kryten kernel: [<ffffffff8136f79f>] dump_stack+0x61/0x7e Jan 7 19:34:52 Kryten kernel: [<ffffffff8104a4ab>] warn_slowpath_common+0x8f/0xa8 Jan 7 19:34:52 Kryten kernel: [<ffffffff8104d87e>] ? __local_bh_enable_ip+0x3b/0x85 Jan 7 19:34:52 Kryten kernel: [<ffffffff8104a568>] warn_slowpath_null+0x15/0x17 Jan 7 19:34:52 Kryten kernel: [<ffffffff8104d87e>] __local_bh_enable_ip+0x3b/0x85 Jan 7 19:34:52 Kryten kernel: [<ffffffff81629808>] _raw_spin_unlock_bh+0x19/0x1b Jan 7 19:34:52 Kryten kernel: [<ffffffff810a35c2>] cgroup_exit+0x48/0x94 Jan 7 19:34:52 Kryten kernel: [<ffffffff8104c946>] do_exit+0x3b2/0x87d Jan 7 19:34:52 Kryten kernel: [<ffffffff8100eb90>] oops_end+0x92/0x97 Jan 7 19:34:52 Kryten kernel: [<ffffffff810401d0>] no_context+0x2c6/0x325 Jan 7 19:34:52 Kryten kernel: [<ffffffff810402ac>] __bad_area_nosemaphore+0x7d/0x1cb Jan 7 19:34:52 Kryten kernel: [<ffffffff81040408>] bad_area_nosemaphore+0xe/0x10 Jan 7 19:34:52 Kryten kernel: [<ffffffff81040704>] __do_page_fault+0x170/0x335 Jan 7 19:34:52 Kryten kernel: [<ffffffff8104090c>] do_page_fault+0x22/0x27 Jan 7 19:34:52 Kryten kernel: [<ffffffff8162b718>] page_fault+0x28/0x30 Jan 7 19:34:52 Kryten kernel: [<ffffffff81273ac8>] ? xfs_buf_free+0x9e/0xb6 Jan 7 19:34:52 Kryten kernel: [<ffffffff81295eee>] ? xfs_trans_binval+0x7/0x80 Jan 7 19:34:52 Kryten kernel: [<ffffffff81246c2a>] xfs_alloc_fix_freelist+0x17f/0x2de Jan 7 19:34:52 Kryten kernel: [<ffffffff812460f2>] ? xfs_alloc_ag_vextent_near+0x78f/0x7f6 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126c5ed>] ? xfs_perag_get+0x3a/0x44 Jan 7 19:34:52 Kryten kernel: [<ffffffff81246fd2>] xfs_alloc_vextent+0x249/0x3a3 Jan 7 19:34:52 Kryten kernel: [<ffffffff81252671>] xfs_bmap_btalloc+0x3c8/0x598 Jan 7 19:34:52 Kryten kernel: [<ffffffff8125284a>] xfs_bmap_alloc+0x9/0xb Jan 7 19:34:52 Kryten kernel: [<ffffffff812530de>] xfs_bmapi_write+0x40e/0x80f Jan 7 19:34:52 Kryten kernel: [<ffffffff8127e1c3>] xfs_iomap_write_allocate+0x1c1/0x2bd Jan 7 19:34:52 Kryten kernel: [<ffffffff8126f066>] xfs_map_blocks+0x134/0x143 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126fd0d>] xfs_vm_writepage+0x287/0x483 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17b9>] __writepage+0xe/0x26 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1c79>] write_cache_pages+0x249/0x351 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c17ab>] ? mapping_tagged+0xf/0xf Jan 7 19:34:52 Kryten kernel: [<ffffffff810c1dc9>] generic_writepages+0x48/0x67 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff8126ef15>] ? xfs_vm_writepages+0x3f/0x47 Jan 7 19:34:52 Kryten kernel: [<ffffffff810c3738>] do_writepages+0x1b/0x24 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112a5d4>] __writeback_single_inode+0x3d/0x151 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ab89>] writeback_sb_inodes+0x20d/0x3ad Jan 7 19:34:52 Kryten kernel: [<ffffffff8112ad9a>] __writeback_inodes_wb+0x71/0xa9 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112af80>] wb_writeback+0x10b/0x195 Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8112b492>] ? wb_workfn+0x157/0x22b Jan 7 19:34:52 Kryten kernel: [<ffffffff8105aede>] process_one_work+0x194/0x2a0 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b894>] worker_thread+0x26b/0x353 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105b629>] ? rescuer_thread+0x285/0x285 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fb24>] kthread+0xcd/0xd5 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: [<ffffffff81629f7f>] ret_from_fork+0x3f/0x70 Jan 7 19:34:52 Kryten kernel: [<ffffffff8105fa57>] ? kthread_worker_fn+0x137/0x137 Jan 7 19:34:52 Kryten kernel: ---[ end trace b862f8a408519d81 ]---
January 7, 20179 yr Author It just died again ... just shy of 3mins after boot. Jan 7 19:56:41 Kryten kernel: device veth796b72c entered promiscuous mode Jan 7 19:56:41 Kryten kernel: docker0: port 7(veth796b72c) entered forwarding state Jan 7 19:56:41 Kryten kernel: docker0: port 7(veth796b72c) entered forwarding state Jan 7 19:56:41 Kryten kernel: docker0: port 7(veth796b72c) entered disabled state Jan 7 19:56:42 Kryten kernel: eth0: renamed from vethaf1c14d Jan 7 19:56:42 Kryten kernel: docker0: port 7(veth796b72c) entered forwarding state Jan 7 19:56:42 Kryten kernel: docker0: port 7(veth796b72c) entered forwarding state Jan 7 19:56:50 Kryten sshd[31439]: Accepted password for root from 10.0.0.181 port 54657 ssh2 Jan 7 19:56:57 Kryten kernel: docker0: port 7(veth796b72c) entered forwarding state Jan 7 19:57:01 Kryten crond[1584]: exit status 127 from user root /usr/lib/sa/sa1 1 1 &> /dev/null Jan 7 19:57:12 Kryten sshd[31652]: Accepted password for root from 10.0.0.181 port 54662 ssh2 Jan 7 19:58:01 Kryten crond[1584]: exit status 127 from user root /usr/lib/sa/sa1 1 1 &> /dev/null Jan 7 19:59:01 Kryten crond[1584]: exit status 127 from user root /usr/lib/sa/sa1 1 1 &> /dev/null Jan 7 19:59:26 Kryten kernel: XFS (md7): _xfs_buf_find: Block out of range: block 0x8ffffffe8, EOFS 0x3a3812a48 Jan 7 19:59:26 Kryten kernel: ------------[ cut here ]------------ Jan 7 19:59:26 Kryten kernel: WARNING: CPU: 6 PID: 1244 at fs/xfs/xfs_buf.c:472 _xfs_buf_find+0x7f/0x28c()
January 7, 20179 yr Community Expert Some of that makes me think you need to check filesystem on disk7. But if it won't stay up long enough you can't. Have you done a memtest recently?
January 7, 20179 yr Edit on the flash drive config/disk.cfg and change startArray="yes" to be startArray="no" and reboot Then run filesystem checks on disk7 (all assuming of course that memtest passes fine as per trurl's advice)
January 7, 20179 yr Author Awesome, but just wish to clarify before I do anything; Memtest: Grab memtest ISO, plug monitor & keyboard in, run test for 24 hours? or is there a magical easy unraid way to do it? Filesystem: fsck? I'll wait until I hear back so I don't break anything .. I have it on with the array unmounted, as per your instructions, see if it runs for a while (19ish minutes so far) ..
January 7, 20179 yr Author sweet, I'll rip out the old monitor then thanks! oh, and what of the file check?!
January 8, 20179 yr Author Hey guys, As it was going to take me a while to get my spare monitor (in storage) and my normal monitor is DP only .. I figured I'd run the fs check before I headed out .. and Trurl, I think your gut instinct was correct... it came up with an error almost immediately .. also, the server is still running now, since I turned the array off (well, it's in maintenance mode) some 4 hours straight... Here's the results.. Phase 1 - find and verify superblock... - block cache size set to 639272 entries Phase 2 - using internal log - zero log... zero_log: head block 1724587 tail block 1719557 - scan filesystem freespace and inode maps... Metadata corruption detected at xfs_agf block 0xfffffff1/0x200 flfirst 118 in agf 2 too large (max = 118) agf 118 freelist blocks bad, skipping freelist scan sb_icount 3512768, counted 3528640 sb_ifree 1134, counted 324 sb_fdblocks 597453202, counted 595057080 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 imap claims a free inode 5336355733 is in use, would correct imap and clear inode - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 entry "WiredTiger.turtle.set" at block 0 offset 2192 in directory inode 4385371913 references free inode 5336355733 would clear inode number in entry at offset 2192... No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 entry "WiredTiger.turtle.set" in directory inode 4385371913 points to free inode 5336355733, would junk entry bad hash table for directory inode 4385371913 (no data entry): would rebuild - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Sun Jan 8 07:54:35 2017 Phase Start End Duration Phase 1: 01/08 07:46:43 01/08 07:46:44 1 second Phase 2: 01/08 07:46:44 01/08 07:46:49 5 seconds Phase 3: 01/08 07:46:49 01/08 07:50:47 3 minutes, 58 seconds Phase 4: 01/08 07:50:47 01/08 07:50:48 1 second Phase 5: Skipped Phase 6: 01/08 07:50:48 01/08 07:54:35 3 minutes, 47 seconds Phase 7: 01/08 07:54:35 01/08 07:54:35 Total run time: 7 minutes, 52 seconds
January 8, 20179 yr Author Any XFS gurus here? I tried running the repair tool, and got an error. System is running in maintenance mode. Phase 1 - find and verify superblock... - block cache size set to 639272 entries Phase 2 - using internal log - zero log... zero_log: head block 1724587 tail block 1719557 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. and running "mount" had a whinge; mount: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. and dmesg [16197.347989] UDF-fs: warning (device sdb): udf_fill_super: No partition found (2) I know this is heaps of info, but I don't wanna be that guy who has to be asked 100 questions because I've missed something Cheers
January 8, 20179 yr Author Reading a bunch of other posts .. "-L" is my option ... worst case, I have parity ..
January 8, 20179 yr Reading a bunch of other posts .. "-L" is my option ... worst case, I have parity .. Parity does not keep individual files, it's only able to recreate the whole drive as is, filesystem errors and all. Your random lockups may have caused the filesystem corruption, not the other way around, so after you get the corruption repaired, I'd still run the fix common problems plugin using troubleshooting mode for a couple days to see if the root cause is gone.
January 8, 20179 yr Author Hey Johathanm, do I "run the system as normal" while running that plugin, or should I leave it in maintenance mode? if I'm to run as normal, should I do a parity check?
January 8, 20179 yr Author xfs_repair fixed all issues (using -L) ... when I do a normal check -nv it is clean now .. Good news there ..
January 8, 20179 yr Hey Johathanm, do I "run the system as normal" while running that plugin, or should I leave it in maintenance mode? if I'm to run as normal, should I do a parity check? After the file system checks come back ok, you should be able to run normally. A non-correcting parity check wouldn't be a bad idea, it will stress the system and if it fails, the fcp troubleshooting mode will capture the syslog live right until the system falls over. You shouldn't run troubleshooting mode continuously for more than a day or so at a time.
January 8, 20179 yr Author Brilliant, I've fired it up normal, and I'm running a parity check with no write .. I've put FCP in to troubleshooting mode .. so .. now we wait Hopefully it lasts .. Will update if it borks again, or after a couple of days ..
January 8, 20179 yr Brilliant, I've fired it up normal, and I'm running a parity check with no write .. I've put FCP in to troubleshooting mode .. so .. now we wait Hopefully it lasts .. Will update if it borks again, or after a couple of days .. If the parity check shows any errors, post diagnostics and wait for advice. Theoretically it should come back with exactly zero errors.
January 8, 20179 yr Brilliant, I've fired it up normal, and I'm running a parity check with no write .. I've put FCP in to troubleshooting mode .. so .. now we wait Hopefully it lasts .. Will update if it borks again, or after a couple of days .. If / when you're satisfied the system is running good, you'll have to reboot it to cancel FCP's troubleshooting mode. (Leaving that running for months on end will cause issues due to the excessive logging it performs to try and locate problems / trends)
January 8, 20179 yr Author Parity check is complete, no errors.. Nothing on FCP so far .. (other than some warnings about shared folders split between cache and drives).. Cheers for the heads up with the reboot, squid; I'll do that in a few days (all going well) ..
January 12, 20179 yr Author Just wanted to say a big thank you to everyone who assisted me on this .. No errors from FCS No lockups With no actually metric .. It "feels" like it's running faster.. I didn't actually to the memtest, just the FS check .. But that solved it .. :-*
Archived
This topic is now archived and is closed to further replies.