Disk Replacement Issue

[Goreliann]

Hi all.

 

I think I've managed to bugger up replacing a failed disk in my array. I use a single parity drive and Had a drive fail and its contents emulated. The documentation I found on replacing the drive was pretty varied so I followed as many common instructions as I could, but I think I've buggered it up and now I need to know how to fix it if you can help. Here are the steps I took.

 

1. Stop the array.

2. Shut it down.

3. Replaced the drive. I had multiple slots open so I just picked one and pulled the other.

4. Rebooted.

5. Started the array. This is where I think I screwed up because the disk number is different for the new disk and the old disk still still shows as emulated.

6. Formatted the new drive.

7. Preformed Read Check.

 

Now the new drive shows it has some data on it, but very little. The old drive still shows as emulated and no rebuild has taken place. I read later that the new drive should be assigned to the same disk number as the one replaced, but now I can't do that because if I stop the array, unassign the new drive and try to assign it to the other disk number slot the array won't start. It says there are too many missing disks.

 

So now I guess I just need to know how I can remove the new drive again so I can assign it to the emulated slot and start a rebuild. Like I said though, the documentation on this is pretty varied with different sites showing different processes. Even the unraid manual documentation didn't easily outline all steps. I tried following that, but it didn't say anything about assigning the new drive to the same slot. That or I missed it somehow. 

 

I'm an unraid newb so baby steps would be appreciated if possible.

 

TIA.

[itimpi]

the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page should be accurate.   The Unraid OS->Manual section in particular covers most features of the current Unraid release.

 

You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread. it is always a good idea when asking questions to supply your diagnostics so we can see details of your system, how you have things configured, and the current syslog.

[Goreliann]

Thanks. I'll do that. Do I just attach the whole zip file? That seems to make the most sense.

storage-diagnostics-20240126-1434.zip

[trurl]

The disk with nothing on it is disk1. Do you mean you didn't originally have anything assigned as disk1?

 

Nothing assigned as disk4, but it is emulated and 76% full.

 

Is that correct?

 

Safest thing would be to just accept the new disk1 as additional storage and rebuild disk4 to another disk.

 

Has anything been written to your server since disk4 became disabled?

[Goreliann]

Yes, you have that correct. Nothing was assigned as Disk 1, but that's where the new drive is. Disk 4 is the one that is currently emulated but has no physical disk installed and was approximately 76% full.

 

So yes, I'd like to just rebuild Disk 4 to another disk, like maybe Disk 1. There hasn't been much written to the array since doing this swap, but yes, a fair bit has been written since Disk 4 became disabled. I guess the other disks just took up the slack.

 

So that just leaves the question of how do I rebuild Disk 4 to another Disk? I don't mind losing the current data on Disk 1 as there isn't much of it anyway.

[trurl]

1 minute ago, Goreliann said:

So yes, I'd like to just rebuild Disk 4 to another disk, like maybe Disk 1

Disk1 is not another disk. It is a disk already assigned to the array.

 

2 minutes ago, Goreliann said:

a fair bit has been written since Disk 4 became disabled. I guess the other disks just took up the slack.

Not at all how things work. Emulated disk4 works just fine as far as Unraid is concerned, and can be read from and written to. Quite possible a lot has been written to the emulated disk and all that would be lost if you don't rebuild disk4 to another disk.

 

Perhaps more concerning, disk4 emulation would probably be broken if you now removed disk1 from the array, and so it wouldn't be possible to rebuild disk4.

 

When I say another disk, I mean a disk not currently assigned to the array.

 

 

[Goreliann]

So basically, it seems the best way to deal with this would be to add yet another disk to the array and assign it as Disk 4 and then restart the array to initiate the rebuild process? I was afraid you'd say that.

 

Sigh.

[trurl]

I prefer to say "replace" disk4 with another disk, and use the verb "add" to mean what you already did by mistake.

 

In what way did original disk4 fail? Possibly nothing at all wrong with it and it could be used for rebuilding. Do you still have original disk4? If so, attach it (but don't assign) then post new diagnostics so we can take a look.

[Goreliann]

I still have the original Disk 4. What happened is that one day Disk 4 just showed up as disabled, contents emulated. I tried doing some research and it seemed the best thing to do was just replace it, so I did. Unfortunately, I seem to have buggered up the replacement process and created this situation we're now discussing.

 

So you're saying just plug Disk 4 back in, start up the array and post new diagnostics? I can do that, but it may be a couple hours before I can post again.

[trurl]

11 minutes ago, Goreliann said:

So you're saying just plug Disk 4 back in, start up the array and post new diagnostics?

Yes, be sure you don't assign that disk to any slot in the array or pools. We need to take a look at it as an Unassigned Device. If you assign it anywhere Unraid is going to do something with it, not what we intend at this point.

[Goreliann]

OK. I've plugged the old Disk 4 back in, rebooted, and restarted the array. I did not assign Disk 4 so it still says contents emulated, but I can see the drive as a new device.

 

I've attached the new diagnostics.

storage-diagnostics-20240126-1728.zip

[trurl]

WD-WCC7K1CY2VSP

This disk? SMART looks fine for that one. Since we don't have logs from when it became disabled, just a guess it was a bad connection.

 

Were you careful with all connections when putting that disk back in? Might want to double check.

 

You can rebuild disk4 to that same disk.

[Goreliann]

Yes, that disk. They're all in hot swap bays so there's no messing around with cables. If it looks like I can just rebuild Disk 4, my understanding of how to do that is just stop the array, assign Disk 4 with the drive I just reinstalled and restart the array. From what I've read, that should start the rebuild process. Is that right?

[trurl]

Right!

[Goreliann]

Beautiful. It now says Data Rebuild in progress. I'm still nervous as hell about this drive going bad again and also when I did a read check, a different drive (Disk 6) showed read errors. It's not showing any now, but with only single parity, this whole process makes me nervous. Should I look to replacing these drives soon?

[trurl]

I don't think there is anything wrong with any disks. Connection problems are much more common than bad disks. Any power splitters?

[trurl]

Do any disks show SMART warning ( 👎 ) on the Dashboard page?

[Goreliann]

No warnings that I can see. I'll let you know if that changes. My rig is just a couple 4 drive cages running off a raid controller. I believe the cages are linked, if I remember correctly, but that would be the only splitting going on. No splitting cables other than the two fly away data cables connected to the raid card.

[Goreliann]

Well I still have no warnings, but now that the rebuild is completed, and successful it looks like, Disk 1 is showing disabled, contents emulated. This is a brand new drive and shouldn't be having any issues. I've stopped the array and rebooted, but it still shows disabled. How can I get it to start a rebuild onto itself? Do I need to reboot it with the disk not present? It seems that's what had to happen to the last one.

[trurl]

Post new diagnostics with the array started in normal mode and without rebooting.

[Goreliann]

I've followed instructions to initiate a rebuild onto itself. Let's hope that solves the issue. In the meantime, here's the latest diagnostics in case you want to have a look.

storage-diagnostics-20240127-1101.zip

[Goreliann]

Oh and FYI, I took those diagnostics before starting the rebuild for  the second time.

[trurl]

Looks like you rebooted before getting the diagnostics, so can't tell anything about why disk1 became disabled.

[Goreliann]

Ok here's a new diagnostic. Now when it tries rebuilding Disk 1, it keeps having errors and shuts down the rebuild. So now I have a completely different drive disabled with contents emulated. This is a brand new drive causing problems now, but it still doesn't show any SMART errors or even warnings. WTH?

storage-diagnostics-20240127-2126.zip

[trurl]

Disk1 has disconnected. Looks like there may be connection problems with others as well.

 

Check connections, all disks, both ends, SATA and power, including splitters. Then post new diagnostics