Installed new 10GB NIC - Lost 3 drives - SOLVED (bad power cable)

[geekdomo]

I have been running unraid for years with no problems at all. I am a photographer and wanted to transfer all of my photos (20TB or so) to a 5 disk array on my desk. It was obviously taking forever to move from my unraid server so I got a 10GB Nic card for it (my Mac has 10GBe). I shut down safely and then put the card in. Upon boot up I had 3 red drives one being the parity. I tried to troubleshoot it and tried other SAS cards, plugging my parity directly into the motherboard (I have 2 disks that way because my SAS was full). I removed the new Nic and some other troubleshooting techniques. I have tried multiple cables, swapping power plugs everything I can think of to get my basically brand new parity drive back. Not sure what happened to the 3TB Drive. All of the drives are in cages and sleds. 
Parity is a 3 month old drive, the 2nd one is quite old and I have a spare already in the mail. 

I managed through moving cables to get 1 drive back. I am still unable to get my fairly new 8TB parity drive to respond. If I can get it back I can just rebuild the 3TB one.  I am at a loss what to do. I have my entire career on these drives. Any way I can restore this somehow?

Edited January 13, 2022 by geekdomo
misspelled unpaid

[trurl]

You must always double check connections whenever you are mucking about inside.

 

3 minutes ago, geekdomo said:

I have my entire career on these drives.

 

Parity is not a substitute for backups.

 

Attach diagnostics to your NEXT post in this thread.

[geekdomo]

Just now, trurl said:

Parity is not a substitute for backups.

 

 

As you see I was attempting to transfer to another separate array. I shoot in RAW the files are pretty massive. I cannot upload them to a cloud based system due to bandwidth issues. 

tank-diagnostics-20220108-1929.zip

[geekdomo]

I have

5 minutes ago, trurl said:

You must always double check connections whenever you are mucking about inside.

 

See OP:I have tried power, SAS connections, SATA cables and anything else I can think of to figure this out. 

[trurl]

The disk assigned as disk8 does not appear in diagnostics smart folder so it isn't responding for some reason, possibly just a connection issue. Check connections, both ends, SAS, SATA, power, including splitters.

 

Since each Unraid data disk is independent, even without disk8 all your other disks should be fine.

 

Parity, though disabled, does appear to be connected, and its SMART attributes look OK. Are there any SMART warnings on the Dashboard page for any of your disks?

 

There is a way to get Unraid to rebuild disk8 instead of the disabled parity disk. How well that works depends on whether parity is mostly still in sync with the missing disk and all others.

[geekdomo]

Just now, trurl said:

The disk assigned as disk8 does not appear in diagnostics smart folder so it isn't responding for some reason, possibly just a connection issue. Check connections, both ends, SAS, SATA, power, including splitters.

 

Since each Unraid data disk is independent, even without disk8 all your other disks should be fine.

 

Parity, though disabled, does appear to be connected, and its SMART attributes look OK. Are there any SMART warnings on the Dashboard page for any of your disks?

 

There is a way to get Unraid to rebuild disk8 instead of the disabled parity disk. How well that works depends on whether parity is mostly still in sync with the missing disk and all others.

Ok I am going to try to connect disk 8 to the SAS card directly (It was SATA). I have not been able to write anything to any disk so I hope they are all synced. Ill let you know what happens with Disk 8 and the SAS connection

 

[geekdomo]

Ok I disconnected the parity drive and connected disk 8 up to the SAS cable that Parity was attached to. Its still showing as dead. Parity however is not showing at all. Was it just the wrong disk before? I have not changed the position of the SAS cable for the drives other than to trouble shoot them (see if the bad drive moved from parity to some other one thinking maybe the cable was bad). 

[trurl]

1 minute ago, geekdomo said:

Was it just the wrong disk before?

Not clear what you mean, Unraid doesn't care how a disk is connected, it keeps track of the assignments by serial number

[geekdomo]

3 minutes ago, trurl said:

Not clear what you mean, Unraid doesn't care how a disk is connected, it keeps track of the assignments by serial number

Sorry my brain is a mess right now trying to deal with this. I meant that its red, yet when I unplugged it it showed that its not there. So how does unraid see it when its plugged in yet won't accept that its the right one for parity? I just shut the server back down Ill plug it back in and see what is happening. I am also going to try a spare old 3TB drive I have in disk 8 spot because I genuinely think that one died. 

[geekdomo]

I am so confused right now. I reconnected my parity drive. Its still "Parity Device Disabled". The motherboard/sas recognizes it right? Now I somehow lost the 3rd disk again while reconnecting the parity back up. I swapped SATA cables and its still not coming back. I need to walk away from this for a bit. If you have any tips on how I can fix this I would really appreciate it because I am not doing too well right now.
 

[trurl]

Do any disks appear in the Unassigned Devices section on Main?

[geekdomo]

4 minutes ago, trurl said:

Do any disks appear in the Unassigned Devices section on Main?

They do not. How is it unraid/system see the disks but I cannot enable them? If I unplug one then reboot the drive shows up as not even installed. Plug it back in and it says missing. I’m at a loss 

[trurl]

2 minutes ago, geekdomo said:

How is it unraid/system see the disks but I cannot enable them? If I unplug one then reboot the drive shows up as not even installed. Plug it back in and it says missing. I’m at a loss 

The drive shows up when you unplug it? What does that mean? Where are you looking that makes you think it shows up?

[geekdomo]

18 minutes ago, trurl said:

The drive shows up when you unplug it? What does that mean? Where are you looking that makes you think it shows up?

Scroll up a few posts. Specifically the parity drive. When I have it plugged in it says “parity device disabled”. Then I shut the server down. Unplug that drive, start it back up and that drive is gone. It’s not even seeing anything there (I shared a screenshot )

[trurl]

32 minutes ago, geekdomo said:

When I have it plugged in it says “parity device disabled”. Then I shut the server down. Unplug that drive, start it back up and that drive is gone.

If the disk is unplugged of course it doesn't show up. If it is plugged in, it is disabled because it is out-of-sync and hasn't been rebuilt.

55 minutes ago, geekdomo said:

Plug it back in and it says missing.

Still a little confused though. Do you mean it says it is missing if it is plugged in, but if it isn't plugged in it doesn't say it is missing?

[trurl]

Just now, trurl said:

If the disk is unplugged of course it doesn't show up. If it is plugged in, it is disabled because it is out-of-sync and hasn't been rebuilt.

And even if the disk is unplugged, it is still disabled because it hasn't been rebuilt. A missing disk will often be disabled because the fact that it is missing makes it out-of-sync.

 

If the array is started with a missing disk then the missing disk is automatically disabled/emulated and will be rebuilt when replaced.

[geekdomo]

So the parity drive is there? How can I reenable it?

 

how can I fix this? We seem to be going around in circles. Maybe I’m not explaining it right. I have 3 drives with red astrix on them. 2 now say missing. The parity drive has as astrix and says “parity device disabled”. 
 

What is the procedure to bring this back online?

thanks for trying to help me it seems we are stuck on semantics. I am probably using wrong terms.

[trurl]

The parity disk is the disk Unraid considers disabled. None of the others are disabled because you can't have more disabled disks than you have parity. If all the other disks could be read you could rebuild parity, which is what Unraid wants to do.

 

There is a way to get parity reenabled and a different disk disabled by jumping through a few hoops, and then Unraid will want to rebuild that other disabled disk instead.

 

But with single parity, you can only rebuild a single disk, and all the other disks have to be read to rebuild that disk.

[trurl]

Unraid disables/emulates a disk when a write to it fails. Unraid has to disable the disk because the failed write means it is out-of-sync. The disk is emulated from the parity calculation by reading all the other disks. That initial failed write, and any subsequent writes to the emulated disk, can be recovered by rebuilding the disk.

 

We can get Unraid to consider parity enabled, and disk8 disabled/emulated and ready to be rebuilt instead, but all other disks must be readable.

[geekdomo]

I do have a functioning spare 3tb drive I can put into place of the failed drive 8. Let’s say that I can get drive 9 to work again ( it seems super intermittent probably a loose cable). What steps do I have to do to bring it all back online? I’ve had drives fail before but never saw parity disabled.

Please share the steps and I’ll try tomorrow to bring it back to life.

Edited January 9, 2022 by geekdomo

[trurl]

1. New Config, Retain All, Apply. Assign disks as needed.
2. Check BOTH Parity Valid and Maintenance Mode checkboxes and START the array. This will result in all assigned disks enabled and accepted into the array exactly as they are.
3. STOP the array, unassign the disk to be rebuilt, START the array with that disk unassigned. This will result in the unassigned disk being disabled and ready to be rebuilt when it is reassigned.
4. Reassign the disk and START the array to begin rebuild.

[geekdomo]

So the parity drive will function at that point? Thanks a ton I’ll give it a go tomorrow morning!

[trurl]

After step 3, the disabled disk will be emulated from the parity calculation. Hopefully, the emulated disk will not show as Unmountable on the Main page. The emulated disk is exactly what will be rebuilt when the disk is reassigned. If it is not Unmountable, you can even look at its contents with no disk assigned to that slot. In the more general situation, the emulated disk can be read and written even if there is no corresponding disk in the server, since only the emulation is used.

 

If the emulated disk is Unmountable, it may still be possible to repair the emulated filesystem before rebuilding, or repair the rebuilt filesystem afterwards. We usually recommend repairing before rebuild, but if you aren't rebuilding to the same disk it doesn't matter, since you won't be overwriting a disk that might have good data on it. And rebuild will be a good test that everything is working well. Only the rebuilding disk is written so nothing will change on all the other good disks.

 

[geekdomo]

6 hours ago, trurl said:

After step 3, the disabled disk will be emulated from the parity calculation. Hopefully, the emulated disk will not show as Unmountable on the Main page. The emulated disk is exactly what will be rebuilt when the disk is reassigned. If it is not Unmountable, you can even look at its contents with no disk assigned to that slot. In the more general situation, the emulated disk can be read and written even if there is no corresponding disk in the server, since only the emulation is used.

 

If the emulated disk is Unmountable, it may still be possible to repair the emulated filesystem before rebuilding, or repair the rebuilt filesystem afterwards. We usually recommend repairing before rebuild, but if you aren't rebuilding to the same disk it doesn't matter, since you won't be overwriting a disk that might have good data on it. And rebuild will be a good test that everything is working well. Only the rebuilding disk is written so nothing will change on all the other good disks.

 

I think we are good!

I had to sleep and quit panicking. So your solution DID fix the array. I was sure to follow it exactly and at the last step my server said "Unmounting disks... retry unmounting disk..." and it stuck there. I found on another thread to reboot the server. When it came back up It was unmounted and ready for rebuild. I am rebuilding now and I think its good to go. THANK YOU. 

 

Edited January 9, 2022 by geekdomo

[geekdomo]

Now I am going to add a 2nd Parity and get a few new drives.