Array Won't Start After Server Reboot, One Disk Missing

[nicecube]

Hello everyone,

 

I'm experiencing a significant problem with my Unraid server. After a routine server reboot, my array refuses to start, and I have one disk that is not functioning as expected.

 

Before the reboot, everything seemed to be working normally and no error indications were present. However, now one of my disks is missing and I'm unable to get it working. Additionally, another disk displays the status "Device is disabled, content emulated".

I've tried to troubleshoot the issue myself but haven't been successful so far. I'm hoping someone from this community could possibly guide me through the steps to fix these issues.

 

Moreover, if there's a way to reactivate Disk 10 while replacing Disk 1, that would be the most optimal solution for me. Any guidance on how I can achieve this would be incredibly helpful. I am ready to provide any further information required to diagnose the issue and find a suitable solution.

 

I can remove Disk 1 and replacing it with a new one. I could attempt to recover the data from the removed disk and copy it back to the array later on. If anyone has any advice or suggestions regarding this approach, please let me know. Your input would be invaluable as I work to resolve these issues.

 

 

An update on Disk 1: I connected it to my personal computer to check it, and by the sounds it's making, I'm afraid it might be dead.

 

 

tower-diagnostics-20230516-1457.zip

Edited May 16, 2023 by nicecube

[JorgeB]

You have two invalid disks with single parity, so it's not possible to start the array, when did disk10 got disabled?

[nicecube]

After shutting down the server overnight, I restarted it the next morning and discovered that Disk 10 had been disabled. Despite leaving the server running all day, the issue persisted. When I powered it off in the evening and rebooted it the following morning, I noticed that two disks 1 were now missing.

 

Disk 1 is clearly defective, emitting unusual noises and displaying signs of damage. While Disk 10 doesn't appear to be faulty, Unraid continues to disable it.

 

Edited May 17, 2023 by nicecube

[JorgeB]

2 minutes ago, nicecube said:

Despite leaving the server running all day, the issue persisted.

Once a disk gets disabled it will remains disabled until rebuilt by the user, I can post instructions to force enable disk10, but it will only work if parity is still valid with it, i.e., if nothing was written to the array after that disk got disabled.

 

 

[itimpi]

1 minute ago, JorgeB said:

Once a disk gets disabled it will remains disabled until rebuilt by the user, I can post instructions to force enable disk10, but it will only work if parity is still valid with it, i.e., if nothing was written to the array after that disk got disabled.

 

 

Probably worth pointing out that the process will enable disk10, but it is the emulation of disk1 that depends on whether anything was written to the array after disk10 got disabled.  
 

Having said that sounds like it might be the only approach that might work for recovering the contents of disk1 if that physical drive has really completely died.

[JorgeB]

1 hour ago, itimpi said:

Probably worth pointing out that the process will enable disk10, but it is the emulation of disk1 that depends on whether anything was written to the array after disk10 got disabled.

Yes, and also anything written to the emulated disk10 would be lost.

[nicecube]

I can afford to lose data it's only movies and series, I can download them again. What would be the best approach can you give me indications that I can follow to restart my array. Is there a way to force disk 10 online and try to rebuild disk 1. My english is not very good sorry if i misunderstood

 

[JorgeB]

If you don't mind losing all data on disk1 and just want to get the array back up do a new config (Tools -> New config) without disk1 and re-sync parity, if you want to try and recover data on disk1 and replace it with a new one you can try this:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including the new disk1, replacement disk should be same size or larger than the old one
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk1
-Start array (in normal mode now) and post new diags

[nicecube]

I followed the instructions until the step to stop the array and unassign disk 1 (i added a new 6TB drive in the server (Disk 1)). To start the array again I have this message with a check box I imagine that I must proceed ?

 

 

Edited May 18, 2023 by nicecube

[itimpi]

 

4 minutes ago, nicecube said:

To start the array again I have this message with a check box I imagine that I must proceed ?

 

Yes.

 

The idea is to see whether the 'emulated' disk1 will mount and show data.

[nicecube]

I started the array and it look like this:

 

tower-diagnostics-20230518-0928.zip

[JorgeB]

Stop array, click on disk1 and set fs to xfs, then check filesystem on that disk.

[nicecube]

Do i need to add disk 1 to array to check the file system ? I replace that disk for a new one

 

 

 

 

 

 

[itimpi]

2 minutes ago, nicecube said:

Do i need to add disk 1 to array to check the file system ?

No.

 

the idea is to check the emulated disk1 to see if it can be repaired so that the data on disk1 can be recovered.    If that is successful that will be the time you put back a physical drive as disk1 so a rebuild can put the emulated contents onto the physical drive.

[nicecube]

 

Phase 1 - find and verify superblock...

superblock read failed, offset 1099511623680, size 131072, ag 1, rval -1

fatal error -- Input/output error

 

 

 

Edited May 18, 2023 by nicecube

[JorgeB]

Looks like disk2 is failing, post new diags.

[nicecube]

I only had 1 spare disk and I used it to replace disk 1, I will have to order u another for disk 2

 

I included the new diagnostic

tower-diagnostics-20230518-1158.zip

[JorgeB]

Problem is that with single parity you cannot lose two disks, run an extended SMART test on disk2 but looks like it's failing.

[nicecube]

I started the self test but after 10 minutes it is still at 10% is it normally long?

 

[JorgeB]

It will take a few hours, it does a complete surface scan.

[itimpi]

23 minutes ago, nicecube said:

I started the self test but after 10 minutes it is still at 10% is it normally long?

 

The progress only updates in 10% increments, and it will take something like 1-2 hours per TB.

[nicecube]

 

tower-smart-20230518-1416.zip

Edited May 18, 2023 by nicecube

[JorgeB]

Disk is bad, you can still try rebuilding disk1 unmountable and after it finishes run xfs_repair, depending on how many errors there are on disk2 it might still recover most data.

[nicecube]

If you don't mind, can you give me the steps to follow I don't want to make mistakes

 

Should I just select my new disk in disk1 and start the array?

 

edit: its rebuilding right now thank  for the help !

Edited May 18, 2023 by nicecube

[JorgeB]

👍