One Failed disk, Rebooted, All disks show missing

[Krak]

Recently lost one of my drives and it showed that it had alot of "errors". I stopped the array and attempted to reboot to see if that would fix it or if we could remove it and replace it. However, when the array came back online it is now showing that all disks are missing. I have read a few posts from Jorge that mention creating a new configuration but I am hesitate in doing so until I understand that process and why. Does anyone have any insight into this and why all disks are now "missing"

 

[JorgeB]

Creating a new config won't help with missing disks, start by power cycling the server, not just rebooting, previous screenshot shows errors on all the disks, suggesting a controller or storage issue, how are the disks connected?

[trurl]

54 minutes ago, JorgeB said:

start by power cycling the server

and post diagnostics

[Krak]

4 hours ago, JorgeB said:

Creating a new config won't help with missing disks, start by power cycling the server, not just rebooting, previous screenshot shows errors on all the disks, suggesting a controller or storage issue, how are the disks connected?

3 hours ago, trurl said:

and post diagnostics

 

Thank you for the feedback so fast! Thanks for the clarification Jorge. Are those supposed to show 0 for the errors? I feel like they have spiked recently and were always sub 100 for the most part. I am using a the LSI raid controller with fiber cables connected to a BPN-SAS-846A backplane in IT mode. It has run flawlessly for 4years but here we are.

Trul, Thank you for that and I will post them. However, Is there a specific diag that would be beneficial that i could dig into as well instead of sharing it all ?

I have sense power cycled the server but performing a clean shutdown and a power on. I am still unable to see any disks but they show up in the bios.

Edited May 29, 2023 by Krak

[Krak]

@JorgeB -- I would like to say all of the drives were functioning just fine before I clicked on "Stop"Array. I did this because the Disk was bad and i wanted to take the array offline cleanly. Once i click stopped it loaded and attempted to spin down the disks and that's when all of the other disks went to missing and no longer show connected in unraid. I am hesitate to think the backplane went bad

[JorgeB]

9 hours ago, Krak said:

Are those supposed to show 0 for the errors?

Yes, post the diagnostics.

[Krak]

 

@JorgeB - which files are you going to look at ? I would like to help as much as possible. I have been trying to search the forums for when all disks go missing but im not finding anything. Like i said they show up in the bios and they check out ok, its only when unraid boots they reflect "missing" and that only happened after i "stopped" the array due to the bad disk.

Edited May 30, 2023 by Krak

[JorgeB]

May 29 10:45:43 Tower kernel: mpt2sas_cm0: port enable: FAILED with timeout (timeout=300s)

 

Controller is failing to initialize, make sure it's well seated or try a different PCIe slot if available, if the same issue it might be failing.

[Krak]

That is wild, why would it look fine in the bios ? I will try that though. Thanks for the fast reply !!

Edited May 30, 2023 by Krak

[Krak]

Same Issue as before "May 30 07:26:50 Tower kernel: mpt2sas_cm0: port enable: FAILED with timeout (timeout=300s)" Used another slot.

 

Any recommendations on a new controller? @JorgeB

[JorgeB]

 

[Krak]

@JorgeB - Side question, i have identified one of the drives when plugged in prevents the controller from intializing at startup before the unraid is loaded. When i unseat that hard-drive the controller initializes. I am now at this screen and see the drives. I now have the failed drive + the one that prevents the controller from initializing.

 

 

 

I am not sure what to do now, i dont think disk 9 is totally toast but it is definitely having issues.

[itimpi]

You could try plugging the drive that prevented the controller initialising into a motherboard port to see if it works there.   Since you only have a single parity drive you cannot recover 2 drives failing so you need to get that back to be able to recover the other one.

[Krak]

@itimpi - if i plug it in now it should be able to recognize it if its working correctly, right ? I totally agree that i cant have 2 failed drives with 1 parity. However, i am hesitate to suggest that both drives are truly failed but more in not great state lol.

[JorgeB]

24 minutes ago, Krak said:

if i plug it in now it should be able to recognize it if its working correctly, right ?

You can try.

[Krak]

@JorgeB - any other ideas on why a specific drive would prevent the card from initializing ? Also, assuming i cant is there a way to mark the disk 9 as "ok" and try to get data off of it?

[JorgeB]

It's the first time I see a controller failing to initialize because of a disk, but we are always learning, did it also not work on the onboard SATA?

[Krak]

I am going to try another port then i will try that too. Great idea. For context, i am using a supermicro 24bay

[Krak]

@JorgeB - i swapped the port for the drive and everything initialized!!! I do have the failed disk 9 - what is the best way to approach using the pairty? Do I set disk 9 to "no device" and then start the array? I have another drive on the way to replace it and plan on going to parity 2.

 

 

 

Also -- this is odd the cache shows no device now ?

 

Edited May 30, 2023 by Krak

[JorgeB]

Post new diags after array start to see SMART for disk9 and confirm the emulated disk is mounting.

[Krak]

Latest Diag.
 

Edited May 30, 2023 by Krak

[JorgeB]

Array is not started, but disk9 looks bad, just start array, having the disk assigned or not will have the same result since it's disabled, you can remove the disk now or later when you get the spare, just check that emulated disk9 is mounting and contents look correct.

[Krak]

@JorgeB - I appreciate your fast response and working with me through this. I started the array and the contents look like they are there. So to replace that drive, i just need to add a new drive and rebuild the parity? I was going to add another spare so I am parity 2 instead of 1 as well. Do i need to wait until i add one disk or could i add three new ones which would replace the failed drive, expand the array, and add an extra parity?

[JorgeB]

15 minutes ago, Krak said:

i just need to add a new drive and rebuild the parity?

You assign a new disk9 and rebuild that, not parity.

 

16 minutes ago, Krak said:

Do i need to wait until i add one disk or could i add three new ones which would replace the failed drive, expand the array, and add an extra parity?

You can add another parity and rebuild the disabled disk at the same time, only after that is done can you add a new disk.

[Krak]

2 hours ago, JorgeB said:

You assign a new disk9 and rebuild that, not parity.

 

You can add another parity and rebuild the disabled disk at the same time, only after that is done can you add a new disk.

Thanks, i ordered a new controller too just in case.