[SOLVED ] - Help a newbie - 3 drives redballed

[trurl]

The lost+found user share has files on both disks now. Don't know if which disk something is on would be a clue for you or not. You can work with the disks separately with Dynamix File Manager.

 

Often things work out better than this.

 

Reviewing the thread, you had already rebuilt other disks. Since you have been running on bad disks, I wonder how good those rebuilds actually were. Might be useful to check filesystem on other disks in the array just to see if they are all OK.

 

On 2/19/2023 at 12:51 PM, Deryg said:

this started as 2 redballs so I started rebuilding a drive

Was this incomplete rebuild onto the original disk or onto a new disk? If a new disk was used and rebuild never completed, then that might explain why everything else didn't work well since that incomplete rebuild would be very out-of-sync.

 

Which disk exactly were you rebuilding in that instance? If you were rebuilding onto a new disk, do you still have that original disk? Maybe we could start all this over with that original disk. 

 

 

[Deryg]

7 minutes ago, trurl said:

both disks now. Don't know if which disk something is on would be a clue for you or not. You can work with the disks separately with Dynamix File Manager.

 

Often things work out better than this.

Seems to be minor issues with the other drives but nothing to the same extent. See attached file checks for all drives except parity. 

 

Incomplete rebuild was for drive 6.  I was sure that I was rebuilding onto itself but that may be where the confusion first started. I still have the drives though and nothing got written to them since the aborted . WE could stick them in to have a look at them.

 

Drive 5 never got touched at any rate

Drive 1 -n filecheck.txt Drive 2 -n filecheck.txt Drive 3 -n filecheck.txt Drive 4 -n filecheck.txt Drive 7 -n filecheck.txt Drive 8 -n filecheck.txt

[trurl]

11 minutes ago, Deryg said:

See attached file checks for all drives except parity. 

That doesn't look bad at all. Maybe we will come back to those later. As you probably know, parity doesn't have a filesystem to check.

 

3 minutes ago, Deryg said:

Incomplete rebuild was for drive 6.  I was sure that I was rebuilding onto itself

Probably was rebuilding onto itself since when we first got involved disk6 was not a good disk. And we haven't used physical disk6 during any of this anyway.

 

If you had an incomplete rebuild onto a disk that already had the data you were rebuilding, then the part that hadn't been rebuilt yet would still have whatever data was there.

 

Here I'm speaking of the data "bits", which is how parity sees things. How the bits on a disk correspond to files is what the filesystem is all about.

 

We could try original disks 5, 6 as Unassigned Devices and see if we can work with them that way. Since they aren't in good condition, we could try to clone them to good disks with ddrescue, but of course, that would require more disks.

[Deryg]

So I have 3 spare new disks (the ones I precleared). possible to do these one at a time?

 

I have the room in the system to keep 2 of the new drives plus the 2 bad drives in unassigned.

Should I be rebuilding drives 5 and 6 onto new drives at this point?

 

Also, should I be correcting the issues on the other drives or leaving them the hell alone for now?

 

[trurl]

4 minutes ago, Deryg said:

Should I be rebuilding drives 5 and 6 onto new drives at this point?

This whole time we have been emulating disks 5, 6, you have had no redundancy.

 

People around here tend to get a little paranoid about running with a degraded array, but everyone runs computers all the time with no redundancy and never give it a second thought. Backups are more important than redundancy anyway.

 

Do you have another copy of anything important and irreplaceable?

 

6 minutes ago, Deryg said:

I have the room in the system to keep 2 of the new drives plus the 2 bad drives in unassigned.

If we rebuild disks 5, 6 to new drives, does that mean you only have room for 2 other disks?

 

8 minutes ago, Deryg said:

possible to do these one at a time?

yes

 

8 minutes ago, Deryg said:

correcting the issues on the other drives or leaving them the hell alone for now?

Let's just leave them alone for now. Nothing should change on those while we work on these others as long as you aren't writing to your server.

[Deryg]

actually system has the 3 new drives plus old drives 5 and 6 from when it went bad and an ancient drive that had a drive 6 label on it. Gotta love having 15 hotswap bays.

 

All above are sitting in unassigned devices right now and array is stopped

 

[trurl]

See if original disks 5, 6 can be mounted as Unassigned Devices.

[Deryg]

AS far as backup of irrepleacable files, I believe I have most of them, will find out what's missing when my wife returns home from her trip and starts screaming at me

[trurl]

5 minutes ago, trurl said:

See if original disks 5, 6 can be mounted as Unassigned Devices.

Should be a Mount button for any disks it recognizes a filesystem on.

[Deryg]

The drives mount. One of them shows as completely empty.

Old drive 5 looks great, I can browse directory and see all the files and file structure

 

Drive 6 which was rebuilding is completely trashed. It mounts but shows as empty

 

Ancient drive 6 is fully browsable but is probably out of date

 

 

 

 

[trurl]

Post new diagnostics and tell me the sdX letter designations of each of those disks.

[Deryg]

sdk = drive 5

sdh = ancient drive 6

sdp = drive 6 that trashed during rebuild

mediawhore-diagnostics-20230223-1505.zip

[trurl]

2 minutes ago, Deryg said:

sdk = drive 5

That drive looks OK, but it is not disk5 from the first diagnostics on this thread, which was a different brand and in terrible condition. It appears to be sdp that was unassigned in those diagnostics.

 

I don't know how we got to that point where the thread started. Maybe it was originally disk5 but had a bad connection or something so you rebuilt it onto that bad disk5 we saw there. Can you tell if those contents are the disk5 contents?

 

[trurl]

15 minutes ago, Deryg said:

sdh = ancient drive 6

That doesn't look bad either

22 minutes ago, Deryg said:

Ancient drive 6 is fully browsable but is probably out of date

out-of-date might be better than lost+found

[Deryg]

so here comes another newbie question.

Can we:

- move the lost and found data from the emulated drive 5 and 6 somewhere to sort later and hunt for missed data and screw-ups on my part (like the third spare disk)

- copy the old drive 5 and ancient 6 (since the aborted rebuild is trashed) to 2 of the new drives.

- assign those drives as 5 and 6 

 

- do a new config, rebuild parity

- fix the odds and sods issues on the other drive filesystems

(not sure of appropriate order for last 2 steps)

[trurl]

18 minutes ago, Deryg said:

sdp = drive 6 that trashed during rebuild

That one failed extended self-test.

 

Maybe I was being a little too clever trying to emulate those disks. Looks like things were too out-of-sync to let that work well.

 

Perhaps we should have started with this:

On 2/19/2023 at 8:50 PM, mathomas3 said:

if you willing to rely on some of these 'failed' disks... I would expect that you could build a new array and copy the data from these disks that failed... 

And, that may be where we have ended up anyway.

 

If you want to just forget about those 2 emulated disks which have nothing on them but lost+found, we could New Config with the new disks in place of those, rebuild parity, format those new disks, and copy what we can from the unassigned original disks.

 

Or, if you don't want to give up on the lost+found, we could rebuild those disks with their lost+found, then copy what we can from the unassigned original disks and try to sort out lost+found later.

 

 

[trurl]

2 minutes ago, Deryg said:

Can we

You posted while I was posting. Let me consider what you said and you can consider what I said.

[trurl]

OK, I have read your post. I think what I suggested as my second alternative:

2 minutes ago, trurl said:

we could rebuild those disks with their lost+found, then copy what we can from the unassigned original disks and try to sort out lost+found later.

Gets us where you are wanting to go in a simpler way.

[Deryg]

sounds like we are kinda looking at the same thing from different perspectives. You are the guru so I bow to your superior knowledge and experience. Let's do it your way.

 

Where do we start?

 

[Deryg]

stop array, assign drives, start array, rebuild (do both at same time) and we talk tomorrow cause that rebuild will take a good while?

 

[trurl]

Correct.

 

You should see lots of writes to the rebuilding disks, lots of reads from all other array disks, zeros in the Errors column for all disks.

 

If it looks like it is having problems, post new diagnostics.

 

If rebuild completes, post new diagnostics.

 

 

[Deryg]

Started. Will talk in 16 hours or more

[Deryg]

well, that was way quicker than previous rebuilds. all done, no errors

mediawhore-diagnostics-20230224-0703.zip

[Deryg]

Now at what point do I try to move files over from the old drives or fix filesystem on existing drives and redo parity?

 

More importantly, in what order?

 

[trurl]

No need to redo parity, it is already in sync since we rebuilt those disks with their lost+found.

 

Go ahead and check filesystem on the remaining disks. I expect that will be OK and shouldn't take long.

 

After that you can consider the array stable and it should be OK to enable Docker and VM Manager again.

 

Then you can work on copying data from the unassigned disks back to the array, and sorting out the lost+found.