Drive Failure - now 2nd 'Unmountable: not mounted' after reboot.

[MooTheKow]

And immediately after posting that/running diagnostics -- it did this.  Would running the diagnostics utility have actually triggered it being disabled?   Just sheer curiosity at this point.   Glad it waited until after the 19 hour rebuild _just_ finished

 

[trurl]

It was struggling so much it filled syslog, everything after that isn't in diagnostics. Simplest way to clear syslog is rebooting.

 

You can see the I/O errors for disk1 in the screenshot. When the diagnostics were taken the disk wasn't yet disabled, but screenshot indicates emulated disk1 has contents.

 

Since it is disabled now, all access to the disk for reading and writing will be emulated and the disk itself won't be accessed.

 

Of course, with a disabled disk and only single parity, you have no redundancy.

[trurl]

Diagnostics doesn't do anything unusual that would cause the disk to be disabled. Diagnostics might access the disks though.

 

A disk is only disabled when a write to it fails, but a failed read can cause Unraid to try to get the data from the parity calculation and write it back to the disk, which might fail and disable the disk.

 

And anything else reading or writing disk1 could have the same effect.

[MooTheKow]

Thanks for the info.  So - i think this is a successful result? :-).  At this point all that is left/can be done is to wait for my replacement drive to arrive, remove disk 1 and then add the new drive to the array in the disk 1 position, correct?  At this point it will again do a rebuild.

 

 

[trurl]

And for the exact timing of disk1 being disabled. Since you only have single parity, Unraid could not disable disk1 no matter what since you already had invalid disk3 rebuilding. And it couldn't try to get any disk1 data from the parity calculation since disk3 would be needed for that and it was still invalid. It couldn't disable disk1 until disk3 rebuild was complete.

 

Even though rebuilt disk3 is mountable and has contents, I don't think you can assume it is all good. Probably it had to assume any unreadable data from disk1 was zeros, just so it could continue with rebuild.

 

You should check filesystem on disk3.

[MooTheKow]

Ah - that makes a lot more sense (re: timing of being disabled).

 

So far as the check filesystem on disk 3 -- here are the results.   Anything look note-worthy? 

 

Disk3 Check.txt

 

Also - even assuming there is some bad data on the rebuilt disk 3 --- I'm still calling it a win that it wasn't a complete loss.  

Edited December 26, 2022 by MooTheKow

[trurl]

Doesn't look bad. If you run it again without -n it is going to put any lost+found in the same folder as the earlier ones, so unless you move those somewhere it might be difficult to see if there are any new ones.

[trurl]

You could just rename lost+found share in the Unraid webUI before doing it again without -n. Then it will create a new lost+found for the new ones.

[MooTheKow]

Will do.  Just realized after doing that 'new config' (i think?) I lost my share.  

 

How can I re-add that back?   I tried going to the 'Shares' and adding a new share with the same name I used to have - but it doesn't seem to be sticking.  AFter clicking 'ADd Share' it just goes back to a blank list of shares.  Been a long time since I set up the original -- so what else do I need to be doing?

[trurl]

16 minutes ago, MooTheKow said:

Just realized after doing that 'new config' (i think?) I lost my share. 

New Config can't do anything to shares. It doesn't change disks in any way, except (by default) it rebuilds parity, so it can't affect any data at all.

 

User Shares are simply the combined top level folders on pools and array. If a top level folder exists, however it was created, a share with that name exists.

 

If you create a top level folder on pool or array, it is automatically a user share with default settings, named for the folder.

 

If you create a user share in the webUI, Unraid creates a top level folder named for the share on pool or array in accordance with settings for the share.

 

Your diagnostics had configuration files for these user shares:

 

The one starting K ending V had no contents, which would happen if you created a share and it no longer exists. Maybe it is all in lost+found now.

 

Or do you mean there are no User Shares at all? That would suggest something broken somewhere, possibly filesystem corruption.

 

Have you rebooted yet to clear syslog? New Diagnostics might not tell much unless syslog is working again.

[MooTheKow]

Correct, it listed nothing in user shares at all.  I did reboot, trying again.

 

in windows I used to be able to access \\kowunraid\Kowdata, but now the only thing I’m seeing is \\kowunraid\flash.

 

I can use the browse button next to individual drives and see a “Kowdata” I’m them and list (and download) files.

 

have some family obligations to attend to now, but will be back later this evening to investigate further.

Edited December 26, 2022 by MooTheKow

[trurl]

5 minutes ago, MooTheKow said:

nothing in user shares at all.  I did reboot

attach diagnostics to your NEXT post in this thread

[MooTheKow]

Here are the diagnostics:

 

kowunraid-diagnostics-20221226-1311.zip

 

 

And a screenshot of the shares:

 

 

Examples of browsing individual disks.

 

Edited December 26, 2022 by MooTheKow

[trurl]

Corruption on (emulated) disk1.

 

Did you check filesystem on rebuilt disk3 without -n yet?

[MooTheKow]

Haven’t had a chance to yet, will do that as soon as I get home, thanks

[MooTheKow]

Ok - ran it:

results.txt

 

Still same behavior though. here are diagnostics i just ran after that check file system without -n

 

kowunraid-diagnostics-20221226-1416.zip

[trurl]

1 hour ago, MooTheKow said:

Ok - ran it

Looks like it may have had some lost+found.

 

Check filesystem on emulated disk1

 

 

[MooTheKow]

Welp, lot more lost and found now 🙂, but share is available again now.    (595 objects: 317 directories, 278 files (102 GB total) in the lost+found folder on emulated disk 1).  So far looks like it's not too bad.. browsing into folders and will find all the contents were a single folder like 'Season 02' of House with all the filenames intact.. so for now just creating a 'House' folder, moving the folder into it, then renaming it to Season 02.

 

Thank you so much for the help.  IS there a best-practice way for me to move files from the 'lost+found' to my sorted folders on in 'KowData'?  Looks like it wants to actually copy the files if I try to move them between folders in windows.

 

Disk 1 NonTrial - Attempt 2.txt

Disk 1 NonTrial - Attempt 1.txt

 

kowunraid-diagnostics-20221226-1937.zip

Edited December 27, 2022 by MooTheKow

[trurl]

You could try to mount physical disk1 as an Unassigned Device, but might as well just wait on the replacement and not mess with that bad disk anymore. After you get disk1 rebuilt maybe you can see if there is still anything to be gained by trying to work with that disk again.

 

[MooTheKow]

Sorry - for clarity:

 

I had already started to go through the lost+found on the emulated disk 1's lost+found.   Is this a bad idea/waste of time at this point or may it potentially hose things up further?   Are you saying I shouldn't do that until I actually have the replacement disk installed?      When I install the replacement disk -- it should build everything on the emulated disk 1 (including the lost+found files) onto the replacement drive, correct?

[trurl]

We usually try to get emulated disk mountable and with no corruption before rebuilding. Especially when rebuilding to the same disk, but you aren't. In your case, you can repair before or after rebuild since the original disk (such as it is) will still have its contents (if they can be read).

 

9 minutes ago, MooTheKow said:

When I install the replacement disk -- it should build everything on the emulated disk 1 (including the lost+found files) onto the replacement drive, correct?

Correct.

[MooTheKow]

Sorry - "We usually try to get emulated disk mountable and with no corruption before rebuilding. Especially when rebuilding to the same disk, but you aren't. In your case, you can repair before or after rebuild since the original disk (such as it is) will still have its contents (if they can be read)."

 

Right now the emulated disk is mounted, isn't it?  and is there still corruption?  I thought the check filesystem utility took care of that and that's why I have a bunch of stuff in the lost+Found now.    Sorry for all the questions - but for as much work as you've helped me with on this I want to make sure I get this right down the stretch :-).

 

[trurl]

Should be OK since you repaired it, but not entirely clear why it needed repair again. Things will be clearer when there are no bad disks involved.

[MooTheKow]

Unclear - I just ran it a 2nd time because the first time ended with a log of:

 

fatal error -- File system metadata writeout failed, err=117.  Re-run xfs_repair

[trurl]

28 minutes ago, MooTheKow said:

ran it a 2nd time because the first time

I thought you had repaired disk1 earlier in the thread but after review I see repair wasn't needed.

 

Some other things I noticed on review

On 12/23/2022 at 12:52 PM, JorgeB said:

Current release only lets you do the test if the disk is set to never spin down.

You aren't on the current release.

28 minutes ago, MooTheKow said:

fatal error -- File system metadata writeout failed, err=117

xfsprogs has been updated so newer release might not have done that but it seems to have worked out.