Disk Unmountable: No File System

[markswam]

Hello guys,

I'm sorry if I don't articulate things properly here. I've been under a lot of stress for the last few weeks with moving, having to work from home due to the whole COVID thing, and some transmission problems with my truck. So I'm already pretty frazzled, and this latest issue is just another stressor to be added to the pile. I appreciate your patience, and I'll try to be helpful.

Today I booted up my server, and was greeted by Disk 5 in my array listed as "Device Disabled: Contents Emulated," with the error "Unmountable: No File System" shown. I restarted the array in Maintenance mode and ran a SMART check on the drive, which reported no errors.

Can someone please give me guidance on what my next steps should be so I don't lose the data on this drive? Nothing on it is particularly "critical," (it's mostly just movies and TV shows), but I'd prefer to lose as little data as possible.

I assume I'm going to have to rebuild from parity, but I don't know what I should try first.

Again, thank you for your patience.

 

Edit: attached are my diagnostics. I know I'm running an old version of unRAID, but as can be seen from prior posts, attempting to update to a newer version broke my shares; so I reverted to an old version and just haven't tried updating again.

tower-diagnostics-20200403-2302.zip

Edited April 4, 2020 by markswam

[JorgeB]

Check filesystem on disk5:

https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui

 

If it's fixed and the emulated disk mounts and data looks correct you can rebuild on top, recommend replacing/swapping cables first just to rule them out if it happens again to the same disk.

[markswam]

Ran xfs_repair with the -n flag, and got the following output: 

 

So I ran xfs_repair -n, and got...a lot of concerning output. I don't want to post all of it because the output includes a lot of file and folder names, but I'm getting a lot of messages like:

invalid length 0 in record 269 of bno btree block 0/1 out-of-order bno btree record 274 (5045679 10542) block 0/1

out-of-order bno btree record 274 (5045679 10542) block 0/1

block (1,64198264-64198264) multiply claimed by cnt space tree, state - 2

invalid start block 222524845 in record 260 of cnt btree block 1/2

out-of-order cnt btree record 397 (40013591 1) block 1/2

undiscovered finobt record, ino 2430466816 (1/282983168)

inode chunk claims untracked block, finobt block - agno 1, bno 48417490, inopb 8

Metadata corruption detected at 0x450110, xfs_inode block 0xd6ab6a0/0x4000

data fork in ino 1344719 claims free block 371073847

bad CRC for inode 225098164

bad magic number 0x2d0a on inode 225098164

bad version number 0xffffff84 on inode 225098164

entry "South Park" at block 0 offset 1664 in directory inode 100 references non-existent inode 2549256007 would clear inode number in entry at offset 1664...

 

Those all appear thousands of times, and there are more besides.

 

Did I just lose all of the data on this drive?

Edited April 4, 2020 by markswam

[markswam]

The very last lines of the output are:

 

Maximum metadata LSN (120:2485520) is ahead of log (1:2).

Would format log to cycle 123.

No modify flag set, skipping filesystem flush and exiting.

[markswam]

...well, I just learned something troubling.

Apparently one of my roommates started up the server, saw the unmounted drive, did "xfs_repair -L" to try and fix it, and claims it wound up aborting with the message "xfs_repair: phase6.c:1410: longform_dir2_rebuild: Assertion `done' failed."

No data has been written to the array since that happened. Is there any chance that this can be fixed?

Edited April 4, 2020 by markswam

[JorgeB]

You need to update Unraid, some older releases have a bug on xfs-progs.

[markswam]

So after I update, I'm assuming I should run xfs_repair again? Should I use the -L flag, or something else?

 

I'm so sorry for asking such basic questions, but it's 2:19am here and I'm more caffeine than man right now, so I don't want to do something stupid.

[JorgeB]

First run with just -v, if it asks for -L, and it likely will, use it.

[markswam]

Will do. I'm guessing since my roommate already ran it with -L earlier as a first resort, it's probably a moot point.

[markswam]

Ran xfs_repair with just -v and it appears to have worked. The disk, however, is showing approximately 175gb of missing files (went from 749GB available to 922GB available). I'm not sure if those files were deleted by one of my users before this happened, or if the data has been lost. I guess I'll figure that out when the parity rebuild completes. Even if that 175GB out of the 2.25TB of data on the drive is completely lost, I'm counting this as a win. There's nothing stored on this array that can't be redownloaded; or even just re-synced from my desktop.

 

Thank you so much for your help, my friend, even this late at night.

[JorgeB]

If the emulated disk is not showing all the data you can mount the actual disk with UD and see if mounts and looks OK (note that array must be stopped and the disk unassigned), and if it does you could do a new config instead of rebuilding.

[markswam]

Unfortunately I’ve already kicked off a rebuild, so that ship has sailed, I’m guessing. 
 

Oh well. Better a handful of movies than the all the data on the disk. 

[JorgeB]

OK, but that's why I mentioned to only rebuild on top if the data on the emulated disk looked correct.

[markswam]

True. It was entirely my mistake. Like I said, I’ve been under a lot of stress lately, and it being so late certainly didn’t help. But what’s done is done, I guess. I’ll live with whatever consequences come from this. 
 

I appreciate the help you’ve so freely given, even if I wound up getting ahead of myself and messing up. 
 

This drive had been largely static (it was at a high water mark, so all write ops that i know of had been on a different drive) for the last couple months, and my last parity check was from just a couple days ago, so I’m hopeful the missing data is just the result of someone deleting stuff before this whole debacle. I guess I’ll see in the morning. 

Edited April 4, 2020 by markswam

[JorgeB]

 

10 minutes ago, markswam said:

I’ve been under a lot of stress lately

Tell me about it, very strange times, just what the world needed, an f*ing pandemic.

 

Hopefully the data is mostly all there.

[markswam]

I should probably mention that before I started the rebuild, I pulled all of the data from the lost+found directory on the disk to my local machine. So everything that was moved there should still exist. It'll be tedious to go through and try to figure out where the hell it all goes, but it's better than nothing.

[markswam]

Well, update. Upon browsing my shares, it looks like a lot of data is actually missing from Disk5. Hopefully I'll be able to recover it from the Lost+Found backup I made. If not...well, time to go redownload a bunch of files.

 

Again, I really, really appreciate your help with this. I don't know why an emulated version of the disk would be missing all this data if parity was up to date, but who knows. 

 

Edit: Looks like I backed most of these files up either locally, or on Google Drive. So that's nice, at least.

Edited April 4, 2020 by markswam

[markswam]

Well, a little over a month later, and it's happened again, to the same disk. I'm totally at a loss. SMART report for the drive still indicates that it's healthy, so I really don't know what's going on.

 

Could this be the fault of a bad SATA controller? Disk 5 is on a PCI-E SATA card.

Edited May 6, 2020 by markswam

[markswam]

And just my luck, it's showing lost data again. I'm not going to kick off a rebuild this time, since last time that was premature of me.

I'm still waiting on xfs_repair -v to finish up and see what it shows, but this is getting frustrating. I have this server with absolutely no problems, and now a drive loses its mind on me twice in a month.

Edited May 6, 2020 by markswam

[markswam]

So xfs_repair has finished, and moved a bunch of stuff into lost+found. What should my next steps be? Right now I'm in the process of pulling all of the L+F data from the server to my local machine to try and manually go through...again...

[JorgeB]

That's not normal, either there's a hardware problem somewhere or the filesystem still had issues, you can re-format the disk (after backup) to make sure filesystem is recreated from scratch, and if it happens again to the same disk the it's likely hardware related.

[markswam]

Would it be best just to physically remove Disk 5, swap in an empty spare and let it rebuild from that? Or should I try again with the same physical disk?

 

Also, would it be a good idea to try swapping what PCI slot I have my SATA controller plugged into, just in case that slot has suddenly gone wonky?

[JorgeB]

23 minutes ago, markswam said:

Would it be best just to physically remove Disk 5, swap in an empty spare and let it rebuild from that? Or should I try again with the same physical disk?

IMHO there's no much point in rebuilding, just formatting to create a new filesystem.

 

24 minutes ago, markswam said:

Also, would it be a good idea to try swapping what PCI slot I have my SATA controller plugged into, just in case that slot has suddenly gone wonky?

Yes, but I would recommend trying one thing at a time or you'll don't know what fixed it, if it does indeed fix it.

[markswam]

5 minutes ago, johnnie.black said:

IMHO there's no much point in rebuilding, just formatting to create a new filesystem.

Oh, so just not relying on parity at all? Just starting from scratch (whether that be reformatting the current drive or just installing a fresh one) and uploading the data to the drive? 

Edited May 6, 2020 by markswam

[JorgeB]

Parity doesn't help with filesystem corruption, it helps when a disk fails.