[SOLVED ] - Help a newbie - 3 drives redballed

[Deryg]

1 hour ago, trurl said:

Federal holiday, but many stores have President's day sales.

Not here in Ontario Canada. Everything is closed. Amazon.ca next day delivery to the rescue

[trurl]

What holiday in Canada? Google doesn't mention any Canadian holiday today.

 

2 hours ago, trurl said:

rebuilding onto the same disk?

47 minutes ago, Deryg said:

I unfortunately was... i'll be chalking that one up to lessons learned

It is always safer to rebuild to another disk, but disks aren't free. So sometimes you consider the pros and cons. If everything is working well, maybe OK to rebuild on top.

[Deryg]

1 hour ago, trurl said:

What holiday in Canada? Google doesn't mention any Canadian holiday today.

 

3 hours ago, trurl said:

rebuilding onto the same disk?

2 hours ago, Deryg said:

I unfortunately was... i'll be chalking that one up to lessons learned

It is always safer to rebuild to another disk, but disks aren't free. So sometimes you consider the pros and cons. If everything is working well, maybe OK to rebuild on top.

It's provincial. Family day. All the stores are closed and I have kids at home underfoot while I try to get some work done .

As far as rebuilding on the same disk, that is a definite lesson learned. I was in a rush and figured if it went to crap, i'd just rebuild again with a different drive. Was not expecting to drop more drives during the rebuild. Never again. I have 3 drives on the way so I'll have a spare 4TB. Gonna need to go shop for a spare 8TB as well. I will NEVER rebuild on same drive again.

 

Drives will arrive tomorrow. I assume I preclear them and then advise and wait for further instructions?

 

[trurl]

3 minutes ago, Deryg said:

preclear them and then advise and wait for further instructions?

That will work. Preclear is just for testing, Unraid doesn't need a clear disk for rebuilds.

 

I never preclear, but I also never introduce more than one new disk into my small array at a time, and I have good (enough) backups.

[Deryg]

52 minutes ago, trurl said:

That will work. Preclear is just for testing, Unraid doesn't need a clear disk for rebuilds.

 

This I knew but im still unsure how I can rebuild with 3 drives dropped from array

[trurl]

22 minutes ago, Deryg said:

unsure how I can rebuild with 3 drives dropped from array

  

17 hours ago, trurl said:

Just need to get all the disks back into the array, disable the bad disks, then work with the emulated disks to see where to go from there.

This starts with New Config. We will get into the details when you are ready, and how we proceed will depend on how things work out.

  

1 hour ago, Deryg said:

preclear them and then advise and wait for further instructions

 

Don't do anything else without further advice

[Frank1940]

3 hours ago, trurl said:

I never preclear, but I also never introduce more than one new disk into my small array at a time, and I have good (enough) backups.

 

While this advice might OK when you are dealing with new disks, I think it should be modified when recycling used disks.  Disks with defective areas that work perfectly (and will may well continue to function for years) in normal PC's can cause massive headaches when used in a server.  A single preclear cycle will usually indicate any potential issues with these disks.  Remember that every sector on an Unraid data disk has to be read-and-writeable. 

[Deryg]

new drives are in. all sata cables replaced with decent latching cables. faulty molex connector replaced (the things you find when you open the case). I pulled the unassigned devices I had sitting there and used the cages to put in the new drives. They are now preclearing (51 percent pre-read). Will advise when the preclear is done.

 

Side note, I did not procure another 8 TB drive as drive 7 is brand new and I have no idea why it dropped but am fairly certain it is fine (also annoyingly enough, it was put in last week to replace a 3TB drive which had gone sideways so data content on it is less than 3 TB) 

[trurl]

9 minutes ago, Deryg said:

latching cables

https://support-in.wd.com/app/answers/detailweb/a_id/15954/~/internal-drive%3A-sata-cable-latch-not-fitting-wd-drive

[Deryg]

Other side note, now that it's rebooted (I just powered down to put in the new drives and replace the cables) It no longer says too many missing disk. Array shows  stopped configuration valid rather than the invalid configuration it used to show

[Deryg]

1 minute ago, trurl said:

https://support-in.wd.com/app/answers/detailweb/a_id/15954/~/internal-drive%3A-sata-cable-latch-not-fitting-wd-drive

All the drives are in icy dock trays and cable goes from mobo to 5-in-3 cage. it's a much more secure connection now with the new cables too. some of the old cables were literally wobbling in their sockets. yikes

[trurl]

24 minutes ago, Deryg said:

now preclearing

Since there were some hardware changes, might as well get some new diagnostics while we wait so we know where we are starting from.

[Deryg]

7 minutes ago, trurl said:

Since there were some hardware changes, might as well get some new diagnostics while we wait so we know where we are starting from.

Et voila

mediawhore-diagnostics-20230221-1928.zip

[Deryg]

OK, all 3 drives have now finished preclear. 0 issues on the new drives.

 

to recap, drive 6 was rebuilding itself when other drives dropped from array. Drive 5 redballed but was a dubious drive. I have 3  4 TB drives brand new (precleared) and tested that can be used to attempt corrective measures.

 

Drive 7 (8TB) but it should be a good drive. I do not have a spare 8TB drive to rebuilt it onto.

 

As instructed I have made no changes to the array and it remains stopped and spun down. Dockers are turned off.

 

Now trying to figure out how to revive array and minimize (or eliminate) data loss with 2 drives dropped, one in aborted rebuild with only dual parity.

 

I've rerun diagnostics after drive clearing done and attached here 

 

Awaiting further instructions and assistance (which I greatly appreciate)

 

 

mediawhore-diagnostics-20230222-1336.zip

Edited February 22, 2023 by Deryg
update

[trurl]

OK, I had to review the thread to understand where we are. Disk7 looks OK. Disk6 has definitely failed. The thing that was confusing me was the disk currently assigned as disk5 doesn't look that bad. Turns out, it isn't the same definitely failed disk5 we saw in your first diagnostics.

 

So, is the disk currently assigned as disk5 one of the new disks? Or what is the correct explanation for this discrepancy?

 

 

 

[Deryg]

hmmm, that is bizarre. I just noticed that. I did not think I messed with any assignments but I realize opening the diagnostic smart reports that they are indeed different drives...

 

I did pull a 3TB seagate which should have been an unassigned device.

 

Should I power down, swap it out with drive 5 and run diagnostics to see if it comes with the same ID which would mean I'd just mislabelled a cage? 

 

Update. I am holding that drive in my hand right now. Drive was in wrong cage. Should I swap it with the other drive and assign it to drive 5?

 

 

 

 

 

 

Edited February 22, 2023 by Deryg
Further Info

[Deryg]

or should I pop it into unassigned, look at the files on there and I will be able to tell if it is indeed the drive that should have been in array? 

 

[trurl]

18 minutes ago, Deryg said:

they are indeed different drives...

Did you ever start the array with that disk assigned? Probably not or it would have begun rebuilding to it.

 

If that isn't one of the new disks we don't need it. Or the bad disks 5, 6 either.

 

I assume the new disks are at least as large as the original disks 5, 6, but not larger than parity. Is that correct?

 

Let me explain briefly what we will do, and then in my next post I will explain how to do it.

 

We will get all disks accepted into the array, with the new disks assigned as disks 5, 6. Then we will disable the new disks so we can check the emulated disks, repair their filesystems if necessary, then rebuild onto the new disks.

 

Go to Disk Settings, disable Autostart. Shutdown and replace any disks as needed. We need both new disks connected, and all other currently assigned disks including disk7 connected. Doesn't matter if other disks are connected or removed.

 

Boot up and post new diagnostics.

 

[Deryg]

20 minutes ago, trurl said:

Did you ever start the array with that disk assigned? Probably not or it would have begun rebuilding to it.

 

If that isn't one of the new disks we don't need it. Or the bad disks 5, 6 either.

 

I assume the new disks are at least as large as the original disks 5, 6, but not larger than parity. Is that correct?

 

Let me explain briefly what we will do, and then in my next post I will explain how to do it.

 

We will get all disks accepted into the array, with the new disks assigned as disks 5, 6. Then we will disable the new disks so we can check the emulated disks, repair their filesystems if necessary, then rebuild onto the new disks.

 

Go to Disk Settings, disable Autostart. Shutdown and replace any disks as needed. We need both new disks connected, and all other currently assigned disks including disk7 connected. Doesn't matter if other disks are connected or removed.

 

Boot up and post new diagnostics.

 

1. Array has been stopped since issue came up.

2. yes, the new 4TB drives (one extra) are bigger or equal to the drives 5/6 that are existing but smaller than parity 8TB

3. Disabled autostart, shutdown and restarted. Diagnostics attached. Please confirm if I should have assigned the new drives to 5 and 6 at this point? If so I will repeat but assign new drives to 5 and 6

 

mediawhore-diagnostics-20230222-1545.zip

[trurl]

41 minutes ago, Deryg said:

Please confirm if I should have assigned the new drives to 5 and 6 at this point? If so I will repeat but assign new drives to 5 and 6

Doesn't really matter at this point what is assigned as disks 5, 6 since we aren't going to actually try to access those disks for a while

1. Go to Tools - New Config, Retain All, Apply
2. Might as well assign the new disks as disks 5, 6. You don't really have to, but if you go ahead and assign them now you won't have to later. Don't change any other assignments.
3. Very Important! Check BOTH Parity Valid and Maintenance Mode checkboxes, then start the array. This will accept all disks just as they are without accessing any disk and changing any contents.

Then, post a screenshot of Main - Array Devices and new diagnostics.

 

[Deryg]

New config, retain all done.

assigned new drives to drive 5 and 6.

started array in maintenance mode with parity marked valid

 

mediawhore-diagnostics-20230222-1657.zip

[trurl]

That looks good. Now

1. Stop the array, unassign disks 5, 6, keep all other assignments as is.
2. Start the array in Normal (not Maintenance) mode with nothing assigned as disks 5, 6.

Then post new diagnostics

 

[Deryg]

Here we go

mediawhore-diagnostics-20230222-2128.zip

[trurl]

Stop the array, click on each of disks 5, 6, set the filesystem to XFS.

Start the array with nothing assigned to disks 5, 6.

Then post new diagnostics.

[Deryg]

mediawhore-diagnostics-20230222-2142.zip