Kuroyukihimeeee Posted July 17, 2016 Share Posted July 17, 2016 Hey all, recently upgraded to RC2 and RC1, pretty sure this is unrelated but at the same time my array started having issues My log is filled with constant errors shown below. I/O errors on Windows based machines trying to write to the array aswell "Jul 16 12:00:58 AVALON kernel: XFS (md3): xfs_log_force: error -5 returned." All disks were showing as active but an entire disk worth of files was just missing. Rebooted array and it came back, but im going to keep a close eye on it now All SMART tests come back clear, drives are 3-4 months old yet an entire drive dropped without unraid even noticing... I kept logs open and watch Plex as normal, about 10 mins after booting array this kicks in with red. Jul 16 19:46:44 AVALON kernel: XFS (md3): Internal error XFS_WANT_CORRUPTED_GOTO at line 1635 of file fs/xfs/libxfs/xfs_alloc.c. Caller xfs_alloc_fix_freelist+0x155/0x2de Jul 16 19:46:44 AVALON kernel: CPU: 7 PID: 22360 Comm: kworker/u34:7 Tainted: G IO 4.4.15-unRAID #1 Jul 16 19:46:44 AVALON kernel: Hardware name: FUJITSU CELSIUS R570-2 /D2628-C1, BIOS 6.00 R1.22.2628.C1 12/19/2011 Jul 16 19:46:44 AVALON kernel: Workqueue: writeback wb_workfn (flush-9:3) Jul 16 19:46:44 AVALON kernel: 0000000000000000 ffff88050f8cb480 ffffffff81369dfe ffff8804ea7f8000 Jul 16 19:46:44 AVALON kernel: 0000000000000001 ffff88050f8cb498 ffffffff81274baa ffffffff812457e0 Jul 16 19:46:44 AVALON kernel: ffff88050f8cb510 ffffffff81244eb1 0000000300000001 ffff8801ad7ec780 Jul 16 19:46:44 AVALON kernel: Call Trace: Jul 16 19:46:44 AVALON kernel: [<ffffffff81369dfe>] dump_stack+0x61/0x7e Jul 16 19:46:44 AVALON kernel: [<ffffffff81274baa>] xfs_error_report+0x32/0x35 Jul 16 19:46:44 AVALON kernel: [<ffffffff812457e0>] ? xfs_alloc_fix_freelist+0x155/0x2de Jul 16 19:46:44 AVALON kernel: [<ffffffff81244eb1>] xfs_free_ag_extent+0x13d/0x558 Jul 16 19:46:44 AVALON kernel: [<ffffffff81243ee2>] ? xfs_alloc_get_freelist+0x135/0x14d Jul 16 19:46:44 AVALON kernel: [<ffffffff812457e0>] xfs_alloc_fix_freelist+0x155/0x2de Jul 16 19:46:44 AVALON kernel: [<ffffffff8126b1cd>] ? xfs_perag_get+0x3a/0x44 Jul 16 19:46:44 AVALON kernel: [<ffffffff81245bb2>] xfs_alloc_vextent+0x249/0x3a3 Jul 16 19:46:44 AVALON kernel: [<ffffffff81251251>] xfs_bmap_btalloc+0x3c8/0x598 Jul 16 19:46:44 AVALON kernel: [<ffffffff8125142a>] xfs_bmap_alloc+0x9/0xb Jul 16 19:46:44 AVALON kernel: [<ffffffff81251cbe>] xfs_bmapi_write+0x40e/0x80f Jul 16 19:46:44 AVALON kernel: [<ffffffff8127cd9d>] xfs_iomap_write_allocate+0x1c1/0x2bd Jul 16 19:46:44 AVALON kernel: [<ffffffff8126dc40>] xfs_map_blocks+0x134/0x143 Jul 16 19:46:44 AVALON kernel: [<ffffffff8126e8e7>] xfs_vm_writepage+0x287/0x483 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c0eb2>] __writepage+0xe/0x26 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c1372>] write_cache_pages+0x249/0x351 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c0ea4>] ? mapping_tagged+0xf/0xf Jul 16 19:46:44 AVALON kernel: [<ffffffff810c14c2>] generic_writepages+0x48/0x67 Jul 16 19:46:44 AVALON kernel: [<ffffffff8137a240>] ? find_next_bit+0x15/0x1b Jul 16 19:46:44 AVALON kernel: [<ffffffff8136a14d>] ? fprop_fraction_percpu+0x32/0x72 Jul 16 19:46:44 AVALON kernel: [<ffffffff8126daef>] xfs_vm_writepages+0x3f/0x47 Jul 16 19:46:44 AVALON kernel: [<ffffffff8126daef>] ? xfs_vm_writepages+0x3f/0x47 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c2e31>] do_writepages+0x1b/0x24 Jul 16 19:46:44 AVALON kernel: [<ffffffff81129a8f>] __writeback_single_inode+0x3d/0x151 Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a044>] writeback_sb_inodes+0x20d/0x3ad Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a255>] __writeback_inodes_wb+0x71/0xa9 Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a43b>] wb_writeback+0x10b/0x195 Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a984>] wb_workfn+0x18e/0x22b Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a984>] ? wb_workfn+0x18e/0x22b Jul 16 19:46:44 AVALON kernel: [<ffffffff8105acd4>] process_one_work+0x194/0x2a0 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105b68a>] worker_thread+0x26b/0x353 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105b41f>] ? rescuer_thread+0x285/0x285 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105f910>] kthread+0xcd/0xd5 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105f843>] ? kthread_worker_fn+0x137/0x137 Jul 16 19:46:44 AVALON kernel: [<ffffffff816232bf>] ret_from_fork+0x3f/0x70 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105f843>] ? kthread_worker_fn+0x137/0x137 Jul 16 19:46:44 AVALON kernel: XFS (md3): Internal error xfs_trans_cancel at line 990 of file fs/xfs/xfs_trans.c. Caller xfs_iomap_write_allocate+0x298/0x2bd Jul 16 19:46:44 AVALON kernel: CPU: 7 PID: 22360 Comm: kworker/u34:7 Tainted: G IO 4.4.15-unRAID #1 Jul 16 19:46:44 AVALON kernel: Hardware name: FUJITSU CELSIUS R570-2 /D2628-C1, BIOS 6.00 R1.22.2628.C1 12/19/2011 Jul 16 19:46:44 AVALON kernel: Workqueue: writeback wb_workfn (flush-9:3) Jul 16 19:46:44 AVALON kernel: 0000000000000000 ffff88050f8cb820 ffffffff81369dfe ffff8804ba061790 Jul 16 19:46:44 AVALON kernel: 0000000000000101 ffff88050f8cb838 ffffffff81274baa ffffffff8127ce74 Jul 16 19:46:44 AVALON kernel: ffff88050f8cb860 ffffffff81288dc0 ffff8800cc6ef800 0000000000053e82 Jul 16 19:46:44 AVALON kernel: Call Trace: Jul 16 19:46:44 AVALON kernel: [<ffffffff81369dfe>] dump_stack+0x61/0x7e Jul 16 19:46:44 AVALON kernel: [<ffffffff81274baa>] xfs_error_report+0x32/0x35 Jul 16 19:46:44 AVALON kernel: [<ffffffff8127ce74>] ? xfs_iomap_write_allocate+0x298/0x2bd Jul 16 19:46:44 AVALON kernel: [<ffffffff81288dc0>] xfs_trans_cancel+0x49/0xbf Jul 16 19:46:44 AVALON kernel: [<ffffffff8127ce74>] xfs_iomap_write_allocate+0x298/0x2bd Jul 16 19:46:44 AVALON kernel: [<ffffffff8126dc40>] xfs_map_blocks+0x134/0x143 Jul 16 19:46:44 AVALON kernel: [<ffffffff8126e8e7>] xfs_vm_writepage+0x287/0x483 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c0eb2>] __writepage+0xe/0x26 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c1372>] write_cache_pages+0x249/0x351 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c0ea4>] ? mapping_tagged+0xf/0xf Jul 16 19:46:44 AVALON kernel: [<ffffffff810c14c2>] generic_writepages+0x48/0x67 Jul 16 19:46:44 AVALON kernel: [<ffffffff8137a240>] ? find_next_bit+0x15/0x1b Jul 16 19:46:44 AVALON kernel: [<ffffffff8136a14d>] ? fprop_fraction_percpu+0x32/0x72 Jul 16 19:46:44 AVALON kernel: [<ffffffff8126daef>] xfs_vm_writepages+0x3f/0x47 Jul 16 19:46:44 AVALON kernel: [<ffffffff8126daef>] ? xfs_vm_writepages+0x3f/0x47 Jul 16 19:46:44 AVALON kernel: [<ffffffff810c2e31>] do_writepages+0x1b/0x24 Jul 16 19:46:44 AVALON kernel: [<ffffffff81129a8f>] __writeback_single_inode+0x3d/0x151 Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a044>] writeback_sb_inodes+0x20d/0x3ad Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a255>] __writeback_inodes_wb+0x71/0xa9 Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a43b>] wb_writeback+0x10b/0x195 Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a984>] wb_workfn+0x18e/0x22b Jul 16 19:46:44 AVALON kernel: [<ffffffff8112a984>] ? wb_workfn+0x18e/0x22b Jul 16 19:46:44 AVALON kernel: [<ffffffff8105acd4>] process_one_work+0x194/0x2a0 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105b68a>] worker_thread+0x26b/0x353 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105b41f>] ? rescuer_thread+0x285/0x285 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105f910>] kthread+0xcd/0xd5 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105f843>] ? kthread_worker_fn+0x137/0x137 Jul 16 19:46:44 AVALON kernel: [<ffffffff816232bf>] ret_from_fork+0x3f/0x70 Jul 16 19:46:44 AVALON kernel: [<ffffffff8105f843>] ? kthread_worker_fn+0x137/0x137 Jul 16 19:46:44 AVALON kernel: XFS (md3): xfs_do_force_shutdown(0x8) called from line 991 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff81288dd9 Jul 16 19:46:44 AVALON kernel: XFS (md3): Corruption of in-memory data detected. Shutting down filesystem Jul 16 19:46:44 AVALON kernel: XFS (md3): Please umount the filesystem and rectify the problem(s) Jul 16 19:46:44 AVALON shfs/user0: shfs_read: read: (5) Input/output error Jul 16 19:46:44 AVALON shfs/user0: shfs_read: read: (5) Input/output error Jul 16 19:46:44 AVALON shfs/user0: shfs_read: read: (5) Input/output error Jul 16 19:46:44 AVALON shfs/user0: shfs_read: read: (5) Input/output error Jul 16 19:46:44 AVALON shfs/user0: shfs_read: read: (5) Input/output error Jul 16 19:46:44 AVALON shfs/user0: shfs_read: read: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Jul 16 19:46:45 AVALON shfs/user0: shfs_write: write: (5) Input/output error Ran memtest all night, came back with 3 passes and 0 errors so still running now but does anybody have any ideas as to what the problem could be? Never had any issues in the past, array keeps running like normal but the drive that is dropping out obviously means the files on that cant be accessed. Can still open the folders but the files inside them are missing. If i take down the array and then reboot it it works perfectly though for another 10 mins or so until the error kicks in again avalon-diagnostics-20160716-1447.zip Quote Link to comment
JorgeB Posted July 17, 2016 Share Posted July 17, 2016 You have file system corruption on disk3, check filesystem: https://lime-technology.com/wiki/index.php/Check_Disk_Filesystems#Drives_formatted_with_XFS Quote Link to comment
Kuroyukihimeeee Posted July 17, 2016 Author Share Posted July 17, 2016 You have file system corruption on disk3, check filesystem: https://lime-technology.com/wiki/index.php/Check_Disk_Filesystems#Drives_formatted_with_XFS Perfect thanks, ran the repair and it found some corrupted metadata. Will keep an eye on it over the next few hours to see if it drops again. Any ideas what usually causes this? Been rock solid over the past few weeks Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.