November 5, 20169 yr I have a single XFS drive in the array and it appears to have failed silently. No warning or parity issues. I ran a parity check and it only found 22 errors, but I suspect that was because of the disk being unavailable. I upgraded to 6.2.3, rebooted, and now the disk is reporting as unmountable. So I tried the following: root@unraid:~# mkdir /mnt/tmp root@unraid:~# mount /dev/md13 /mnt/tmp mount: mount /dev/md13 on /mnt/tmp failed: Structure needs cleaning root@unraid:~# xfs_repair -v /dev/md13 Phase 1 - find and verify superblock... - block cache size set to 343656 entries Phase 2 - using internal log - zero log... zero_log: head block 1050306 tail block 1050190 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. root@unraid:~# root@unraid:~# root@unraid:~# root@unraid:~# root@unraid:~# root@unraid:~# xfs_repair -nv /dev/md13 Phase 1 - find and verify superblock... - block cache size set to 343656 entries Phase 2 - using internal log - zero log... zero_log: head block 1050306 tail block 1050190 - scan filesystem freespace and inode maps... Metadata corruption detected at xfs_agf block 0xaea86661/0x200 flfirst 118 in agf 3 too large (max = 118) agf 118 freelist blocks bad, skipping freelist scan sb_icount 42816, counted 42880 sb_ifree 75, counted 57 sb_fdblocks 239938275, counted 239780591 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Sat Nov 5 08:42:58 2016 Phase Start End Duration Phase 1: 11/05 08:42:45 11/05 08:42:45 Phase 2: 11/05 08:42:45 11/05 08:42:46 1 second Phase 3: 11/05 08:42:46 11/05 08:42:56 10 seconds Phase 4: 11/05 08:42:56 11/05 08:42:57 1 second Phase 5: Skipped Phase 6: 11/05 08:42:57 11/05 08:42:58 1 second Phase 7: 11/05 08:42:58 11/05 08:42:58 Total run time: 13 seconds Where do I go from here?
November 5, 20169 yr Author If I need to, I can go and buy another 2TB drive. My concern is that the array is not indicating a problem, so would a rebuild from parity still work?
November 5, 20169 yr ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. You need to follow the instructions. You tried to mount the file system and it failed so it can't replay the log, so you have to use the -L option. You have file system corruption. That is not the same as a failed disk. Parity protection won't help you here. If you rebuild onto a new disk you'll simply rebuild the same corrupted file system.
November 5, 20169 yr Author That brought the disk back online once I started the array. Pretty sure I'm missing things, and nothing in a lost+found. Should I be suspicious of this drive now?
November 5, 20169 yr That brought the disk back online once I started the array. Pretty sure I'm missing things, and nothing in a lost+found. Should I be suspicious of this drive now? Yes, you've lost whatever transactions were not replayed from the log. You'll need to check the contents against your backups. Was there a power outage? It's worth checking the power cable to the disk. What does its SMART report say? Run a long self test on it. I'd run a parity check too, in case there was a power glitch.
November 6, 20169 yr Author Extended SMART test was clean. Since this is the only XFS formatted drive I have, should I be concerned? I definitely lost files, with no indication of which ones. The strange thing is, the lost files seem to have been from a time period of a year ago. Newer files are still present. I'm referring only to the files on this disk. I'm pretty sure this was a replacement disk for a smaller disk that was formerly RFS and failed catastrophically. Formatted as XFS and rebuilt from parity within the last year. Is it possible the corruption was in place during the initial rebuild? My primary concern at this point is trusting my system. It's worked great for something like a decade or more. I've had failures and had to go into lost+found before, but it's never been a silent failure like this. Is there something I should be doing differently with the XFS drive to be alerted to an issue, especially if Unraid itself doesn't see it the problem? Thanks for your help!
November 6, 20169 yr If you're happy with the SMART values and the test then the disk itself is very likely to be good. The problem was file system corruption and a likely cause is loss of power (possibly momentary) during a write operation. So you'll want to check your power cabling. If you're unhappy about the disk then replace it, for your own peace of mind It's possible that there could have been some file system corruption there for some time that wasn't bad enough to prevent it mounting. Unless you do regular file system checks this would go unnoticed. It has nothing to do with parity, so regular parity checks would not reveal anything. I don't understand your description of how you converted from ReiserFS to XFS though. If you removed the failed ReiserFS disk and replaced it with a new one and rebuilt from parity you would end up with a ReiserFS formatted disk, not an XFS one, even if you formatted it XFS before rebuilding. That's because a rebuild completely replaces the contents of the disk. People tend not to do regular file system checks because with XFS and ReiserFS the file system has to be unmounted, which takes the server off-line. More sophisticated file systems, such as BTRFS and ZFS, do their housekeeping and maintenance on-line. BTRFS is, in fact, an option for unRAID data disks but not many people use it, considering it unreliable and not yet mature. I ran one BTRFS formatted data disk in one server, alongside XFS disks for several months as a test and had no problems with it.
November 6, 20169 yr Author John, thanks for all of your quick responses! I'll probably sit on this for a while, and consider using BTRFS for any new drives I install. As for the XFS drive, I haven't had to add any drives for space in a while. I'm pretty sure this particular drive was a 2TB replacing a failed 750GB. Whatever it came from, it was definitely a replacement for a failed drive.
Archived
This topic is now archived and is closed to further replies.