March 3, 20197 yr Hey guys! On of my server's external HDD's has had a loss of power last night. This morning, all HDDs are seen by Unraid, but my 8TB eHDD is showing as Unmountable: No File System. I've already searched the forum and tried the following command in maintenance mode: "xfs_repair -v /dev/md1" However, I get a "Cannot open /dev/md1: Device or ressource busy" error. Oh, also, I have a Single Parity setup. Fairly noobish here so please go easy on me! Any ideas? Thanks for any assistance guys. (Before saying it, I know I should have other backups. But Having a spare 19TB of storage is costly :( ) Edited March 3, 20197 yr by plm42
March 3, 20197 yr Community Expert Don't copy past from the forum, sometimes extra characters are inserted, either that or you're using encryption and if so that's not the correct path. Alternatively use the GUI.
March 3, 20197 yr Author I am indeed using encryption! And I haven't copied-paste from the forum. I typed it in manually. Picture attached FYI
March 3, 20197 yr Community Expert Just now, plm42 said: I am indeed using encryption! Then the correct command is: xfs_repair -v /dev/mapper/md1
March 3, 20197 yr Community Expert BTW, the disk is disabled, so you'll be running xfs_repair on the emulated disk, disk will need to be rebuilt.
March 3, 20197 yr Community Expert Just now, johnnie.black said: Then the correct command is: xfs_repair -v /dev/mapper/md1 And if you do it from the webUI then it would do it correctly for you and you wouldn't need to know the path. Just click on the disk and go to Check Filesystem Status
March 3, 20197 yr Community Expert 26 minutes ago, plm42 said: (Before saying it, I know I should have other backups. But Having a spare 19TB of storage is costly :( ) You don't have to backup everything, but you absolutely must have another copy of anything important and irreplaceable. You must do this ASAP! Copy whatever you can that you think qualifies as important and irreplaceable before attempting to do anything else. If some of that is on the unmountable disk then you won't be able to get that part until you repair its filesystem.
March 3, 20197 yr Author trurl - Currently at: "Phase 1 - find and verify superblock..." on GUI johnnie_black - See attached screenshot
March 3, 20197 yr Community Expert @plm42 Not quite sure from the screenshots what you have done so far? You need to rerun the check using the -L option because of the metadata warning (it is virtually always safe to ignore that warning and without the -n option (or nothing gets repaired). as was mentioned the disk is marked as disabled which means Unraid is only checking the emulated disk and is not writing to the physical disk. This is useful as if anything goes wrong repairing the emulated disk you still have the physical disk as it was at the point it got disabled to use as an emergency fall-back. Assuming the repair of the emulated disk goes well then you will have to do a’rebuild’ to get the physical disk back inline with the emulated one.
March 3, 20197 yr Author Just now, itimpi said: @plm42 Not quite sure from the screenshots what you have done so far? You need to rerun the check using the -L option because of the metadata warning (it is virtually always safe to ignore that warning and without the -n option (or nothing gets repaired). as was mentioned the disk is marked as disabled which means Unraid is only checking the emulated disk and is not writing to the physical disk. This is useful as if anything goes wrong repairing the emulated disk you still have the physical disk as it was at the point it got disabled to use as an emergency fall-back. Assuming the repair of the emulated disk goes well then you will have to do a’rebuild’ to get the physical disk back inline with the emulated one. Only ran check with -n. Will rerun with -L right away and post back.
March 3, 20197 yr Author Ran with -L : Quote Phase 1 - find and verify superblock... sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with calculated value 96 resetting superblock root inode pointer to 96 sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent with calculated value 97 resetting superblock realtime bitmap ino pointer to 97 sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent with calculated value 98 resetting superblock realtime summary ino pointer to 98 Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... sb_icount 0, counted 34752 sb_ifree 0, counted 448 sb_fdblocks 1952984353, counted 118851139 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 99 Bad flags2 set in inode 99 inode 99 is marked reflinked but file system does not support reflink bad CRC for inode 146 bad (negative) size -7726908823766136962 on inode 146 bad CRC for inode 99, will rewrite imap claims a free inode 99 is in use, correcting imap and clearing inode cleared inode 99 bad CRC for inode 146, will rewrite bad (negative) size -7726908823766136962 on inode 146 cleared inode 146 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 2 - agno = 0 - agno = 3 - agno = 4 entry "UNRAID_NAS" in shortform directory 96 references free inode 99 junking entry "UNRAID_NAS" in directory inode 96 - agno = 1 entry "The Blacklist S03E01.mkv" at block 0 offset 136 in directory inode 144 references free inode 146 clearing inode number in entry at offset 136... - agno = 5 - agno = 6 - agno = 7 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 144 (no data entry): rebuilding rebuilding directory inode 144 - traversal finished ... - moving disconnected inodes to lost+found ... disconnected dir inode 2147483744, moving to lost+found disconnected dir inode 10375843511, moving to lost+found disconnected dir inode 14501479126, moving to lost+found Phase 7 - verify and correct link counts... resetting inode 99 nlinks from 2 to 5 Metadata corruption detected at 0x460c82, xfs_dir3_block block 0x2380080/0x1000 libxfs_writebufr: write verifer failed on xfs_dir3_block bno 0x2380080/0x1000 Maximum metadata LSN (1:249216) is ahead of log (1:2). Format log to cycle 4. releasing dirty buffer (bulk) to free list!done
March 3, 20197 yr Author 22 minutes ago, itimpi said: @plm42 Not quite sure from the screenshots what you have done so far? You need to rerun the check using the -L option because of the metadata warning (it is virtually always safe to ignore that warning and without the -n option (or nothing gets repaired). as was mentioned the disk is marked as disabled which means Unraid is only checking the emulated disk and is not writing to the physical disk. This is useful as if anything goes wrong repairing the emulated disk you still have the physical disk as it was at the point it got disabled to use as an emergency fall-back. Assuming the repair of the emulated disk goes well then you will have to do a’rebuild’ to get the physical disk back inline with the emulated one. So I guess next step is to run the check with no modifier options (Leaving blank) and then rebuilding the drive? Thanks again!
March 3, 20197 yr Author I've run the cmd with the -L modifier. What next? Still shows HDD as an umountable xfs filesystem. Is this when I rebuild? If so, how do I do so? Thank you guys again immensely. root@Tower:~# xfs_repair -v /dev/mapper/md1 -L Phase 1 - find and verify superblock... - block cache size set to 899432 entries Phase 2 - using internal log - zero log... zero_log: head block 0 tail block 0 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 1 - agno = 6 - agno = 7 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:248420) is ahead of log (1:2). Format log to cycle 4. XFS_REPAIR Summary Sun Mar 3 12:53:38 2019 Phase Start End Duration Phase 1: 03/03 12:51:44 03/03 12:51:45 1 second Phase 2: 03/03 12:51:45 03/03 12:52:16 31 seconds Phase 3: 03/03 12:52:16 03/03 12:52:30 14 seconds Phase 4: 03/03 12:52:30 03/03 12:52:30 Phase 5: 03/03 12:52:30 03/03 12:52:30 Phase 6: 03/03 12:52:30 03/03 12:52:38 8 seconds Phase 7: 03/03 12:52:38 03/03 12:52:38 Total run time: 54 seconds done
March 3, 20197 yr Community Expert Should mount now, if it doesn't post the diagnostics after normal array start.
March 7, 20197 yr Author On 3/3/2019 at 6:57 PM, johnnie.black said: Should mount now, if it doesn't post the diagnostics after normal array start. Sorry for the late response. I ended up rebuilding the drive, due to the drive still not mounting. Took over 3 days (Some HDD'S running over USB 2.0) Fortunately, no data seems to have been lost. Thanks for the assistance anyhow! Now, I'm off to shuck my eHDD's to install them internal! Loads of fun to come haha.
March 8, 20197 yr 1 hour ago, plm42 said: I ended up rebuilding the drive, due to the drive still not mounting. You had two problems: a disabled disk and a damaged file system. You were always going to have to rebuild the drive, once you'd fixed the file system: On 3/3/2019 at 4:40 PM, johnnie.black said: BTW, the disk is disabled, so you'll be running xfs_repair on the emulated disk, disk will need to be rebuilt.
March 8, 20197 yr Author 15 hours ago, John_M said: You had two problems: a disabled disk and a damaged file system. You were always going to have to rebuild the drive, once you'd fixed the file system: My misunderstanding then. That being said, noob here: Wouldn't have been just as efficient if I had formatted the drive entirely, then rebuilt it? I don't quite see how it would have changed the end result... Thanks again for not being too harsh regarding my lack of knowledge. Love to learn, but there's a lot to learn!
March 8, 20197 yr Community Expert 35 minutes ago, plm42 said: Wouldn't have been just as efficient if I had formatted the drive entirely, then rebuilt it? No, formatting deletes everything on that disk and updates parity, a rebuild then would result in the same empty disk.
March 8, 20197 yr The only thing you could have done differently would be to rebuild the disk first. But then you would have rebuilt a corrupt file system and would still need to repair it. Two different problems, two different fixes but it doesn't really matter which order you do them in. Personally, I'd do the file system repair first, as you did, because it's often quick and it's reassuring to be able to see your files on the emulated disk.
March 8, 20197 yr Community Expert 3 hours ago, plm42 said: formatted the drive entirely, then rebuilt it? This seems to be a very common idea, and always a bad one. And the webUI will warn you not to do this, but people do it anyway. Format is a write operation. It writes an empty filesystem to the disk. When you format a disk in the parity array, Unraid treats that write operation just the same as it does any other. It updates parity. So, after formatting a disk in the parity array, parity agrees that the disk has an empty filesystem. Then rebuilding the disk from parity results in an empty filesystem.
March 8, 20197 yr Author 1 minute ago, trurl said: This seems to be a very common idea, and always a bad one. And the webUI will warn you not to do this, but people do it anyway. Format is a write operation. It writes an empty filesystem to the disk. When you format a disk in the parity array, Unraid treats that write operation just the same as it does any other. It updates parity. So, after formatting a disk in the parity array, parity agrees that the disk has an empty filesystem. Then rebuilding the disk from parity results in an empty filesystem. Huh. I understand! Just to further understand, that means that if I was to format the drive on another computer, and put it back in my Unraid server, it would rebuild it, considering that's it's akin to a new drive? i.e. it won't update the parity drive and consider that the (formatted) HDD should be blank, but rather rebuild it?
March 8, 20197 yr Community Expert 6 minutes ago, plm42 said: Huh. I understand! Just to further understand, that means that if I was to format the drive on another computer, and put it back in my Unraid server, it would rebuild it, considering that's it's akin to a new drive? i.e. it won't update the parity drive and consider that the (formatted) HDD should be blank, but rather rebuild it? But formatting the disk in another system would not actually accomplish anything at all since the complete disk would be overwritten by the rebuild. Doesn't matter at all what was on the disk. You could clear it, format it, even fill it up with baby pictures. It wouldn't matter to the rebuild. And if you hadn't already repaired the emulated filesystem, then the rebuild would still be unmountable.
March 8, 20197 yr Author 9 minutes ago, trurl said: But formatting the disk in another system would not actually accomplish anything at all since the complete disk would be overwritten by the rebuild. Doesn't matter at all what was on the disk. You could clear it, format it, even fill it up with baby pictures. It wouldn't matter to the rebuild. And if you hadn't already repaired the emulated filesystem, then the rebuild would still be unmountable. Ohhh. So in my specific scenario, if the HDD was rebuilt from the parity drive after a format on a separate PC, it would rebuild the disk using the broken FS, thus still giving me a drive with said broken FS? PS: Not looking for repair advice here, just trying to learn Edited March 8, 20197 yr by plm42
March 8, 20197 yr Community Expert The emulated filesystem we are referring to is the contents of the disk as far as the array is concerned, but without the actual disk. When the array is started, and a disk is missing or disabled, its contents are emulated by using parity PLUS ALL remaining disks to calculate the contents of the missing/disabled disk. You can't have more missing/disabled disks than you have parity disks for the disk to be emulated. Note that this means that all the other disks must be read to emulate the disk. And it means all those disks must give the correct results for the emulation to be correct. So as you can see, the health of all disks is important, even disks that you think don't have important data on them. Unraid disables a disk when a write to it fails. But the failed write still updates parity, so that write is still in the array, with the emulated disk. Unraid will not use a disabled disk again until it is rebuilt. It uses the emulated disk instead, whether for reading or for writing. Unraid had disabled the disk, and the emulated disk had filesystem corruption and so was unmountable. The rebuild is simply going to write the contents of the emulated disk to the actual disk. So the filesystem had to be repaired, either before rebuild or after.
Archived
This topic is now archived and is closed to further replies.