December 14, 20178 yr Hi, I'm having issues with unraid. My second disk appears to be okay from a SMART perspective but it is showing an input/ouput error and a high number of reads. I ran DMESG and got the following, which looks like XFS has crashed: [58253.774556] XFS (md2): Internal error xfs_trans_cancel at line 983 of file fs/xfs/xfs_trans.c. Caller xfs_create+0x3a7/0x467 [58253.778728] CPU: 1 PID: 3942 Comm: shfs Not tainted 4.9.30-unRAID #1 [58253.778730] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To Be Filled By O.E.M., BIOS P2.80 12/04/2014 [58253.778732] ffffc900112fbb68 ffffffff813a4a1b ffff880185518828 0000000000000000 [58253.778736] ffffc900112fbb80 ffffffff8129c98d ffffffff812a8f3b ffffc900112fbba8 [58253.778739] ffffffff812b2f96 ffff88085b9cd000 0000000000000000 ffff88059460d640 [58253.778743] Call Trace: [58253.778751] [<ffffffff813a4a1b>] dump_stack+0x61/0x7e [58253.778755] [<ffffffff8129c98d>] xfs_error_report+0x32/0x35 [58253.778758] [<ffffffff812a8f3b>] ? xfs_create+0x3a7/0x467 [58253.778760] [<ffffffff812b2f96>] xfs_trans_cancel+0x49/0xbf [58253.778762] [<ffffffff812a8f3b>] xfs_create+0x3a7/0x467 [58253.778765] [<ffffffff812a6a46>] xfs_generic_create+0xae/0x24b [58253.778768] [<ffffffff81054110>] ? capable_wrt_inode_uidgid+0x3a/0x47 [58253.778771] [<ffffffff812a6c08>] xfs_vn_mknod+0xf/0x11 [58253.778773] [<ffffffff812a6c2b>] xfs_vn_create+0xe/0x10 [58253.778776] [<ffffffff8112df93>] path_openat+0x7c5/0xca8 [58253.778779] [<ffffffff8112ce40>] ? filename_parentat+0xd4/0xef [58253.778782] [<ffffffff8112e4be>] do_filp_open+0x48/0x9e [58253.778785] [<ffffffff811352ca>] ? d_lookup+0x29/0x3d [58253.778787] [<ffffffff8113b6c4>] ? mntput_no_expire+0x27/0x17a [58253.778790] [<ffffffff81120a71>] do_sys_open+0x137/0x1c6 [58253.778793] [<ffffffff81120a71>] ? do_sys_open+0x137/0x1c6 [58253.778795] [<ffffffff8112e7d9>] ? SyS_mkdirat+0x3a/0xac [58253.778798] [<ffffffff81120b19>] SyS_open+0x19/0x1b [58253.778801] [<ffffffff8167f537>] entry_SYSCALL_64_fastpath+0x1a/0xa9 [58253.778805] XFS (md2): xfs_do_force_shutdown(0x8) called from line 984 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff812b2faf [58253.889353] XFS (md2): Corruption of in-memory data detected. Shutting down filesystem [58253.891506] XFS (md2): Please umount the filesystem and rectify the problem(s) [58253.933206] XFS (md2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5. The filesystem is looking to be full too  Disk 2 WDC_WD40EFRX-68WT0N0_WD-WCC4E1552391 - 4 TB (sdt) 29 C 743,666 72,958 0 xfs 4 TB 4 TB 36.9 KB but I configured it to leave space so I'm not sure how its managed to fill up to 36.9KB. I've attached the syslog and dmesg. This starter yesterday when i left my house and travelled to the uk, i forced a reboot, and XFS repaired and now this has reoccured. From an array perspective I am concerned because instead of showing a red ball that the drive is broken, files have dissappeared from my array! logs.tar.gz Edited December 14, 20178 yr by spamalam
December 14, 20178 yr Author root@beyonder-nas:/tmp# cd /mnt/disk2 root@beyonder-nas:/mnt/disk2# ls -l /bin/ls: cannot open directory '.': Input/output error And from Smart: # Attribute Name Flag Value Worst Threshold Type Updated Failed Raw Value 1 Raw read error rate 0x002f 200 200 051 Pre-fail Always Never 0 3 Spin up time 0x0027 184 179 021 Pre-fail Always Never 7775 4 Start stop count 0x0032 096 096 000 Old age Always Never 4944 5 Reallocated sector count 0x0033 200 200 140 Pre-fail Always Never 0 7 Seek error rate 0x002e 200 200 000 Old age Always Never 0 9 Power on hours 0x0032 059 059 000 Old age Always Never 30351 (3y, 5m, 15d, 15h) 10 Spin retry count 0x0032 100 100 000 Old age Always Never 0 11 Calibration retry count 0x0032 100 253 000 Old age Always Never 0 12 Power cycle count 0x0032 100 100 000 Old age Always Never 93 192 Power-off retract count 0x0032 200 200 000 Old age Always Never 23 193 Load cycle count 0x0032 196 196 000 Old age Always Never 12782 194 Temperature celsius 0x0022 123 079 000 Old age Always Never 29 196 Reallocated event count 0x0032 200 200 000 Old age Always Never 0 197 Current pending sector 0x0032 200 200 000 Old age Always Never 0 198 Offline uncorrectable 0x0030 100 253 000 Old age Offline Never 0 199 UDMA CRC error count 0x0032 200 200 000 Old age Always Never 0 200 Multi zone error rate 0x0008 100 253 000 Old age Offline Never 0
December 14, 20178 yr 1 minute ago, spamalam said: a red ball that the drive is broken, files have dissappeared from my array! Very big difference between a broken drive ("red ball") and file system corruption caused by what appears to be a forced reboot resulting in an unclean shutdown You should upload the diagnostics.
December 14, 20178 yr Author 5 minutes ago, Squid said: Very big difference between a broken drive ("red ball") and file system corruption caused by what appears to be a forced reboot resulting in an unclean shutdown You should upload the diagnostics. XFS crashed first, the reboot i forced yesterday only after files dissappeared. unraid had been running for three weeks previously when it was a clean shutdown on that run. Can system corruption hide like that for months? I think the last unclear shutdown was back in September. XFS has crashed again today after recovery from the last boot. I've just forced another reboot, I will attach the diagnostics when it returns. Will post asap. Thanks Edited December 14, 20178 yr by spamalam clearer language
December 14, 20178 yr Author Attaching diagnostics. Stopping the array and running a filesystem check. I lost data in this, because the disk went input/output read error and not red ball the /mnt/user shares simply saw files dissappear. This is really dangerous no? Surely if a disk is input/output error it should stop the array? The application syncthing saw this as deleted files, and then wiped all the clients. I lost files, encryption keys, etc. Luckily I had a client offline and was able to restore, but I think this is rather concerning and might be worthwhile considering a bug? beyonder-nas-diagnostics-20171214-0851.zip
December 14, 20178 yr Author From the filesystem check it does look like there's something up with xfs: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... Metadata corruption detected at xfs_agf block 0xe8e05f31/0x200 fllast 118 in agf 2 too large (max = 118) agf 113 freelist blocks bad, skipping freelist scan sb_fdblocks 739441, counted 739435 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. Edited December 14, 20178 yr by spamalam
December 14, 20178 yr Community Expert Disk2 is completely filled up, only 6MB free, you might not have enough free space to run xfs_repair.
December 14, 20178 yr Author 5 minutes ago, johnnie.black said: Disk2 is completely filled up, only 6MB free, you might not have enough free space to run xfs_repair. Seems to have gone through: xfs_repair status: Phase 1 - find and verify superblock... - block cache size set to 3020352 entries Phase 2 - using internal log - zero log... zero_log: head block 2870652 tail block 2870652 - scan filesystem freespace and inode maps... Metadata corruption detected at xfs_agf block 0xe8e05f31/0x200 fllast 118 in agf 2 too large (max = 118) - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... XFS_REPAIR Summary Thu Dec 14 09:13:31 2017 Phase Start End Duration Phase 1: 12/14 09:07:57 12/14 09:07:57 Phase 2: 12/14 09:07:57 12/14 09:08:00 3 seconds Phase 3: 12/14 09:08:00 12/14 09:10:30 2 minutes, 30 seconds Phase 4: 12/14 09:10:30 12/14 09:10:31 1 second Phase 5: 12/14 09:10:31 12/14 09:10:31 Phase 6: 12/14 09:10:31 12/14 09:12:54 2 minutes, 23 seconds Phase 7: 12/14 09:12:54 12/14 09:12:54 Total run time: 4 minutes, 57 seconds done
December 14, 20178 yr Author and it passed a second check! Thank you very much! I will also use unbalance to re-balance the files around a bit, and start a parity check/rebuild once I know its all okay. Okay, so XFS corrupted. Let's assume this was a historical reboot and stayed hidden until the hard drive got close to full. I'll be adding filesystem checks into the reboot procedure if I lose power again. I have a really big concern with UNRAID about this whole thing though. Unraid didn't care that a filesystem had corrupted, and presented all the shares, minus the files. My expectation is that the array should have been put offline or paused. This actually caused data loss to an application and I personally think this is pretty catastrophic in potential. Is this something that the unraid team is aware of? Is there a fix? I don't want this to reoccur as I was lucky a client was offline at the time. Edited December 14, 20178 yr by spamalam
December 14, 20178 yr Community Expert unRAID or any RAID protects against one or more device failures, it can't protect you against file system corruption.
December 14, 20178 yr Author My hardware NAS has filesystem integrity checks, but surely a simple read/write test could solve that problem ? Wouldn't it be wise for unraid to be doing something similar? The errors behind the curtain trying to read by shfs must have been there, right? Edited December 14, 20178 yr by spamalam
December 14, 20178 yr Community Expert 17 minutes ago, spamalam said: My hardware NAS has filesystem integrity checks, but surely a simple read/write test could solve that problem ? You can install the fix common problems plugin, it will warn you a disk goes read only, still can't protect you from serious filesystem corruption, but at least with unRAID and unless something catastrophic happens, filesystem corruptiont will be limited to a single disks instead of the whole pool.
December 15, 20178 yr Author No dice, the disk went again... unraid doesn't care and pretends everything is fine. root@beyonder-nas:~# ls -l /mnt/disk{1..15} /bin/ls: cannot access '/mnt/disk2': Input/output error Is this really the correct behaviour for a NAS product? [64557.473458] XFS (md2): Internal error xfs_trans_cancel at line 983 of file fs/xfs/xfs_trans.c. Caller xfs_create+0x3a7/0x467 [64557.478616] CPU: 1 PID: 20269 Comm: shfs Not tainted 4.9.30-unRAID #1 [64557.478618] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./To Be Filled By O.E.M., BIOS P2.80 12/04/2014 [64557.478620] ffffc90003e27b68 ffffffff813a4a1b ffff88080ce5ed98 0000000000000000 [64557.478624] ffffc90003e27b80 ffffffff8129c98d ffffffff812a8f3b ffffc90003e27ba8 [64557.478628] ffffffff812b2f96 ffff88081c6d5000 0000000000000000 ffff8805913a2d00 [64557.478631] Call Trace: [64557.478639] [<ffffffff813a4a1b>] dump_stack+0x61/0x7e [64557.478643] [<ffffffff8129c98d>] xfs_error_report+0x32/0x35 [64557.478646] [<ffffffff812a8f3b>] ? xfs_create+0x3a7/0x467 [64557.478649] [<ffffffff812b2f96>] xfs_trans_cancel+0x49/0xbf [64557.478651] [<ffffffff812a8f3b>] xfs_create+0x3a7/0x467 [64557.478654] [<ffffffff812a6a46>] xfs_generic_create+0xae/0x24b [64557.478658] [<ffffffff81054110>] ? capable_wrt_inode_uidgid+0x3a/0x47 [64557.478660] [<ffffffff812a6c08>] xfs_vn_mknod+0xf/0x11 [64557.478662] [<ffffffff812a6c2b>] xfs_vn_create+0xe/0x10 [64557.478666] [<ffffffff8112df93>] path_openat+0x7c5/0xca8 [64557.478669] [<ffffffff8112ce40>] ? filename_parentat+0xd4/0xef [64557.478671] [<ffffffff8112e4be>] do_filp_open+0x48/0x9e [64557.478675] [<ffffffff811352ca>] ? d_lookup+0x29/0x3d [64557.478678] [<ffffffff8113b6c4>] ? mntput_no_expire+0x27/0x17a [64557.478681] [<ffffffff81120a71>] do_sys_open+0x137/0x1c6 [64557.478683] [<ffffffff81120a71>] ? do_sys_open+0x137/0x1c6 [64557.478686] [<ffffffff8112e7d9>] ? SyS_mkdirat+0x3a/0xac [64557.478688] [<ffffffff81120b19>] SyS_open+0x19/0x1b [64557.478692] [<ffffffff8167f537>] entry_SYSCALL_64_fastpath+0x1a/0xa9 [64557.478697] XFS (md2): xfs_do_force_shutdown(0x8) called from line 984 of file fs/xfs/xfs_trans.c. Return address = 0xffffffff812b2faf [64557.757668] XFS (md2): Corruption of in-memory data detected. Shutting down filesystem [64557.759839] XFS (md2): Please umount the filesystem and rectify the problem(s) [64557.762566] XFS (md2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
December 15, 20178 yr Author Here's the latest diagnostics, can anyone help? the curious line is the following, where shfs says not tainted but many files have dissappeared? [64557.473458] XFS (md2): Internal error xfs_trans_cancel at line 983 of file fs/xfs/xfs_trans.c. Caller xfs_create+0x3a7/0x467 [64557.478616] CPU: 1 PID: 20269 Comm: shfs Not tainted 4.9.30-unRAID #1 beyonder-nas-diagnostics-20171215-2202.zip
December 15, 20178 yr Community Expert Filesystem getting corrupted again after so little time is not good, either xfs_repair coudn't completely fix the original issue or you have some hardware issue like bad RAM. I would recommend running memtest for a few hours and if no errors are found fix that disk again, move data to other disk(s) and re-format it.
December 15, 20178 yr Author Re-ran xfs, and have now copied 30 GB from this full hard drive to another in case this is an XFS bug related to a full disk. Running a memory scan will be hard as I'm currently offsite for a few weeks, and ilo management is a pain with this internet connection. Should this second xfs fix and reduction of files not work, I will look at hotswapping a spare disk thats outside the array or running a ram test remotely via iLO. How can i officially get attention of the unraid development team and raise this issue for analysis/fix? I really do think that files disappearing silently from an array when a disk filesystem is gone should be considered a critical defect for a NAS platform and be handled in a similar way to a SMART failure. I had a look at the FAQ and it just says raise a thread, but i'm used to issue trackers at github/jira/etc. and want to make sure i've provided all the data to get it fixed. I've got all the diagnostic logs attached, and am happy to provide anything else.
December 15, 20178 yr Community Expert 26 minutes ago, spamalam said: How can i officially get attention of the unraid development team and raise this issue for analysis/fix? You can ask for it in the feature request forum.
December 15, 20178 yr 1 hour ago, spamalam said: and ilo management is a pain with this internet connection. what are you running unraid on? HP server?
December 15, 20178 yr Author 12 minutes ago, 1812 said: what are you running unraid on? HP server? Sorry that's a habit from work, its ASROCK java web console. The whole webconsole is very, very slow remotely. Not sure why. I'm running on this: http://www.asrockrack.com/general/productdetail.asp?Model=C2550D4I Quote 00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02) 00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02) 00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02) 00:04.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 4 (rev 02) 00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02) 00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02) 00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02) 00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02) 00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 02) 00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02) 00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02) 00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02) 01:00.0 RAID bus controller: Areca Technology Corp. ARC-1280/1280ML 24-Port PCI-Express to SATA II RAID Controller 02:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 03:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 03:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 03:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 03:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) 04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 02) 06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 21) 07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11) The RAM is ECC, which is why I'd be surprised if that was causing me issues. I would expect it to be telling me 'no' very loudly Edited December 15, 20178 yr by spamalam
December 16, 20178 yr Community Expert The RAM is ECC, which is why I'd be surprised if that was causing me issues. I would expect it to be telling me 'no' very loudly [emoji4] If you're using ECC you can skip the memtest, I did try to check your board model before to see if it was a server model but unfortunately Asrock forgot to add the model where it's supposed to be, all I could see is that it was an Asrock: Dec 14 11:08:36 beyonder-nas kernel: DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./To Be Filled By O.E.M., BIOS P2.80 12/04/2014 See if freeing some space helps, I assumed you had done that already when I mentioned the only 6MB free, that's too little free space and can cause issues, if not your best bet it to re-format that disk after moving all data.
December 17, 20178 yr Author I re-balanced the array, all drives now have several GB. No occurrence for a day.
Archived
This topic is now archived and is closed to further replies.