So, I think I screwed myself tonight, looking for help. 6.11.5+

[trurl]

7 minutes ago, mattw said:

Should I just add it as a blank drive at this point?

NO

 

The filesystem on the emulated disk is currently set to auto since it doesn't know. Change it to XFS then see if it will give you the option to check filesystem on the emulated disk. If not we can try it from the command line.

 

Be sure to capture the output of check filesystem so you can post it.

 

[mattw]

It was pretty long, so I zipped it up.

CheckFilesystem.zip

[trurl]

Just to give you an idea where we are going with this. We will repair filesystem on the emulated disk. And we will also repair filesystem on the physical disk as an Unassigned Device. That should give us 2 versions of the disk contents, and we can decide where to go from there.

 

For now, check filesystem on the emulated disk, without the -n. If it asks for it use -L. Capture output and post it.

 

Then, start the array in normal mode, still with nothing assigned to that disk slot, and post new diagnostics so we can see if the repaired emulated disk mounts, and you can look at its contents.

 

I'll check back in the morning.

 

[itimpi]

10 hours ago, mattw said:

I should add that files that were on it are not being emulated... they are just not there.  Should I just add it as a blank drive at this point?

You should stop the array; explicitly set the file system to be xfs; and then restart the array in maintenance mode.   This should now give you the xfs repair option so you can see if that works.

[mattw]

So, yes I did that above and posted the results.  I blows my mind that you can test an emulated drive!

[JorgeB]

Run it again without -n or nothing will be done.

[trurl]

10 hours ago, trurl said:

check filesystem on the emulated disk, without the -n. If it asks for it use -L. Capture output and post it.

 

Then, start the array in normal mode, still with nothing assigned to that disk slot, and post new diagnostics so we can see if the repaired emulated disk mounts, and you can look at its contents.

 

[mattw]

Ok, has been run without -n.

 

TowerReport.txt

[JorgeB]

Start the array in normal mode and check emulated disk contents.

[trurl]

14 minutes ago, JorgeB said:

Start the array in normal mode and check emulated disk contents.

And post new diagnostics

[mattw]

So upon stopping the array in maintenance mode so that I could start it in normal mode, it seems to have gotten stuck.  Dashboard indicates stopped, line at the bottom of the screen indicates " Array Stopping•Retry unmounting user share(s)..." and array operations screen has the stop button greyed out and indicates "stopping".

 

I have about a 30 minute drive home from work and I will check on it when I get home.

 

Update, just got home and it is still stuck.  I think it really is stopped.  Guess I could try a reboot.

tower-diagnostics-20221221-1622.zip

Edited December 21, 2022 by mattw

[mattw]

So, I waited over an hour for the array to be stopped and finally did a server reboot from the GUI.  Upon boot up it reported unclean shutdown, checked for what was stored on emulated disk 3 and did not find any gains.  This diag file is post reboot, the one in the last message is pre-reboot.

tower-diagnostics-20221221-1851.zip

[trurl]

3 hours ago, mattw said:

checked for what was stored on emulated disk 3 and did not find any gains

Don't know what you mean by "gains". Emulated disk3 mounts now and is half full, with files on 4 user shares. You had lost all of disk3 before repair.

 

One of those user shares is lost+found, might be a problem trying to sort that out.

 

These are the emulated contents, which is the results you would get from rebuild. Do you have a spare disk you can use to rebuild disk3?

 

The next step is to attempt to mount the disk that was assigned as disk3 as an Unassigned Device. Then we can try repair on it if necessary and compare its contents with the contents of emulated disk3.

 

Post a screenshot of Main - Unassigned Devices.

[mattw]

So, I do not see the note that I have an emulated disk after the reboot...  That is why I did not see gains in the Media folder?  I never physically removed disk 3 from the system, so yes it is sitting there ready to be mounted.  I do not have another disk to install, well I do but it was my previous parity drive and it had begun to pop smart errors, was pretty old.

 

 

The message for each share is "Some or all files unprotected"  I also have no disk shares?

 

Edited December 22, 2022 by mattw

[JorgeB]

Mount old disk with UD to compare contents, need to change the XFS UUID first (UD settings)

[trurl]

8 hours ago, mattw said:

I do not see the note that I have an emulated disk after the reboot...

Post a screenshot of Main - Array Devices.

 

8 hours ago, mattw said:

The message for each share is "Some or all files unprotected" 

Since you have a disabled disk and single parity, that is true. Until you have fewer disabled disks than you have parity disks, any share with files on the array are indeed unprotected.

 

8 hours ago, mattw said:

I also have no disk shares?

Neither do I since I never share disks.

[trurl]

8 hours ago, mattw said:

That is why I did not see gains in the Media folder?

Emulated disk3 doesn't have that share currently. Probably any files for that share that were on disk3 are now in lost+found after repair.

 

Next step

4 hours ago, JorgeB said:

Mount old disk with UD to compare contents, need to change the XFS UUID first (UD settings)

 

[mattw]

@trurl I did find the emulated disk 3 and the contents... and yes lost and found is loaded and looks pretty good, almost all files have names and appear to be the right size'ish.  I have physical disk 3 UUID changed, or at least I selected it in UD settings and clicked "Change UUID". 

 

The drive is visible, but when clicking moung the system thinks a few seconds and it does not come back mounted.

 

 

tower-diagnostics-20221222-0822.zip

[JorgeB]

48 minutes ago, mattw said:

but when clicking moung the system thinks a few seconds and it does not come back mounted.

It needs a filesystem check, click this and follow the instructions:

 

 

 

[mattw]

Running it now... but how does one just know that from that screen? 

 

So, once I have both sources of data online, can I just copy the stuff I want to keep to one of my other shares or make a new share and recover?  None of my drives have been showing errors and I am quickly filling up random storage spaces.  I have been thinking about modernizing this server with a new mobo and cpu, but I may end up keeping it as is and building a second server with bigger and lower numbers of drives as an in house replication location.

 

We have this little winter storm headed our way, I may have to give up for a day or 2 because we will have power issues in the flatlands of IL with this wind and my UPS is not big.  I dragged the server down to where it was more convenient to work on.

Edited December 22, 2022 by mattw

[trurl]

4 minutes ago, mattw said:

how does one just know that

last lines of syslog in your diagnostics

Dec 22 08:20:21 Tower unassigned.devices: Mount of 'sdf1' failed: 'mount: /mnt/disks/WD-WCC1P3KLCPP5: mount(2) system call failed: Structure needs cleaning.        dmesg(1) may have more information after failed mount system call. '
Dec 22 08:20:21 Tower unassigned.devices: Partition 'WD-WCC1P3KLCPP5' cannot be mounted.

[trurl]

6 minutes ago, mattw said:

So, once I have both sources of data online

Lets wait and see the results of repairing the filesystem on the physical disk before deciding how to proceed.

[mattw]

The check completed with the following at the end of the run.

No modify flag set, skipping phase 5
Inode allocation btrees are too corrupted, skipping phases 6 and 7
No modify flag set, skipping filesystem flush and exiting.

File system corruption detected!

 

I now have the option to run with correction.  I assume that is the next step?

[trurl]

44 minutes ago, JorgeB said:

filesystem check, click this and follow the instructions

Capture and post output

[mattw]

Here it is.

disk check.txt