[SOLVED] Too many wrong or missing disks!

[MurrayW]

I have been running 4.7 for more than 6 months without any problems and I haven't added any disks since I started.  Last night I went to a folder on the unraid server to find a file and the folder structure was all there, but all the folders except the Movies folder were empty.  The Movies folder was never assigned to all the disks in the array.

 

I logged on via the web interface and all disks were flashing green except for disk 6 which was red.  We had a storm a few days ago when we lost power and the server rebooted (yeah I know I should have had it connected to a UPS).  When I saw this red disk, I rebooted the server in hopes that it would magically come back on line with a reboot.  After the reboot, disk 4,5 and 7 now show as missing and 6 (the original red disk) shows as not installed.  The other disks are blinking green.

 

What should I try to get the array back up and running?

thanks, Murray

[lionelhutz]

Look for something that is common between the disks that would have caused this failure (SATA card, power splitter, SATA breakout cable etc).

[Joe L.]

I have been running 4.7 for more than 6 months without any problems and I haven't added any disks since I started.  Last night I went to a folder on the unraid server to find a file and the folder structure was all there, but all the folders except the Movies folder were empty.  The Movies folder was never assigned to all the disks in the array.

 

I logged on via the web interface and all disks were flashing green except for disk 6 which was red.  We had a storm a few days ago when we lost power and the server rebooted (yeah I know I should have had it connected to a UPS).  When I saw this red disk, I rebooted the server in hopes that it would magically come back on line with a reboot.  After the reboot, disk 4,5 and 7 now show as missing and 6 (the original red disk) shows as not installed.  The other disks are blinking green.

 

What should I try to get the array back up and running?

thanks, Murray

Looks like an entire disk controller has stopped responding...  Three disks missing, and one more not installed?

 

Step 1.  Post a system log from before you do anything more.

Step 2. power cycle.  perhaps the disks/controller just need to be initialized and a simple reboot does not do that.

Step 3. post a second system log.

Step 4. consider the purchase of a UPS and configuring the server to shut down cleanly in the event of a power failure.

 

Joe L.

[MurrayW]

I had already rebooted the server (twice) last night before I got the initial log.  Attached is the log from a few minutes ago after the two reboots last night.  The reboots did not bring the missing drives back online.  After the first reboot was when disks 4,5, and 7 went from flashing green to red and "missing".  An yes, I know about the UPS...the most embarrassing thing is that I have one sitting right next to the server but had never shut down the server to connect it to the UPS!

thanks for your help,

Murray

syslog.txt

[JonathanM]

I had already rebooted the server (twice) last night before I got the initial log.

When you say reboot, it implies a powered restart. Have you totally shut down and powered off?

[Johnm]

I have one sitting right next to the server but had never shut down the server to connect it to the UPS!

I hate when deferred maintenance comes back to bite you. we have had things like this happen at work.

 

Great example is we lost a drive in a massive RAID6, I went to order a new drive and the boss said, we will be migrating that array to a new one at the end of the month. dont bother, we still have a safety net..  a week later a second drive dropped.. same thing. "we are migrating in 4 days, you cant a drive before then anyways"...

 

migration day... the array goes belly up..  what should have been a 6-10 hour migration took days while we restored from tapes.

 

PS assume your data is still there and it is just a controller error.

[MurrayW]

I had already rebooted the server (twice) last night before I got the initial log.

When you say reboot, it implies a powered restart. Have you totally shut down and powered off?

Actually I did it both ways.  The first time a reboot.  The second time I powered it off, reseated all my drives in the drive cages then powered it back on.

[Joe L.]

Your system log only shows 5 disks.  Can you see the missing 4 in the BIOS.

(If not, look for a common cable, drive rack, power connector...)

 

Joe L.

[MurrayW]

Your system log only shows 5 disks.  Can you see the missing 4 in the BIOS.

(If not, look for a common cable, drive rack, power connector...)

 

Joe L.

I'm going to open my server up and dig into it this weekend.  Can I cause any irreversible problems in troubleshooting by swapping out cables, disk controllers, drive bays among the disks that show up and those that are missing?

thanks,

Murray

[Joe L.]

Your system log only shows 5 disks.  Can you see the missing 4 in the BIOS.

(If not, look for a common cable, drive rack, power connector...)

 

Joe L.

I'm going to open my server up and dig into it this weekend.  Can I cause any irreversible problems in troubleshooting by swapping out cables, disk controllers, drive bays among the disks that show up and those that are missing?

thanks,

Murray

As long as you do not connect a data disk in place of your parity disk your data should be OK.

 

Just be careful... ground yourself so you don't zap any electronics.  (purchase an electrostatic wrist strap ... and use it)

Unplug the server so you don't electrocute yourself. (unlikely, but good advice regardless...)

 

Make a copy of your disk assignments page so you'll know how you had them connected.

 

Make a copy of your config folder, so we can help you regroup if needed.  To make it easier on yourself, make the copy with the array stopped.

 

Joe L.

[MurrayW]

Your system log only shows 5 disks.  Can you see the missing 4 in the BIOS.

(If not, look for a common cable, drive rack, power connector...)

 

Joe L.

I'm going to open my server up and dig into it this weekend.  Can I cause any irreversible problems in troubleshooting by swapping out cables, disk controllers, drive bays among the disks that show up and those that are missing?

thanks,

Murray

As long as you do not connect a data disk in place of your parity disk your data should be OK.

 

Just be careful... ground yourself so you don't zap any electronics.  (purchase an electrostatic wrist strap ... and use it)

Unplug the server so you don't electrocute yourself. (unlikely, but good advice regardless...)

 

Make a copy of your disk assignments page so you'll know how you had them connected.

 

Make a copy of your config folder, so we can help you regroup if needed.  To make it easier on yourself, make the copy with the array stopped.

 

Joe L.

Thanks Joe.  I thought I would start out with the unraid flash drive unplugged and just work in the bios to see if swapping cables/etc. made a difference on which disks were detected.  I'll make copies of the two items you suggested above.

 

So how exactly do I prevent connecting a data disk in place of my parity disk?  Does that mean if my parity disk is now connected to SATA controller 1, that I leave that connection alone when doing the troubleshooting?

[Joe L.]

Just do not assign a data disk in the parity slot on the disk assignment screen. 

 

Other than that you can swap out just about anything.

 

Joe L.

[MurrayW]

Your system log only shows 5 disks.  Can you see the missing 4 in the BIOS.

(If not, look for a common cable, drive rack, power connector...)

 

Joe L.

I'm going to open my server up and dig into it this weekend.  Can I cause any irreversible problems in troubleshooting by swapping out cables, disk controllers, drive bays among the disks that show up and those that are missing?

thanks,

Murray

As long as you do not connect a data disk in place of your parity disk your data should be OK.

 

Just be careful... ground yourself so you don't zap any electronics.  (purchase an electrostatic wrist strap ... and use it)

Unplug the server so you don't electrocute yourself. (unlikely, but good advice regardless...)

 

Make a copy of your disk assignments page so you'll know how you had them connected.

 

Make a copy of your config folder, so we can help you regroup if needed.  To make it easier on yourself, make the copy with the array stopped.

 

Joe L.

Sorry for asking so many questions.  I searched but probably did not have the correct search terms.  How do I make a copy of my config folder?  And is the first screen shot I posted in this thread the disk assignments page?

thanks,

Murray

[Joe L.]

Your system log only shows 5 disks.  Can you see the missing 4 in the BIOS.

(If not, look for a common cable, drive rack, power connector...)

 

Joe L.

I'm going to open my server up and dig into it this weekend.  Can I cause any irreversible problems in troubleshooting by swapping out cables, disk controllers, drive bays among the disks that show up and those that are missing?

thanks,

Murray

As long as you do not connect a data disk in place of your parity disk your data should be OK.

 

Just be careful... ground yourself so you don't zap any electronics.  (purchase an electrostatic wrist strap ... and use it)

Unplug the server so you don't electrocute yourself. (unlikely, but good advice regardless...)

 

Make a copy of your disk assignments page so you'll know how you had them connected.

 

Make a copy of your config folder, so we can help you regroup if needed.  To make it easier on yourself, make the copy with the array stopped.

 

Joe L.

Sorry for asking so many questions.  I searched but probably did not have the correct search terms.  How do I make a copy of my config folder?

The flash drive is acessible from windows as \\tower\flash

The config folder wold them be

\\tower\flash\config

Make a copy of it from windows file-explorer is the easiest way for some.

And is the first screen shot I posted in this thread the disk assignments page?

No, it is on the "Devices" page... but both have the model/serial numbers of the disks you would need.  The devices page also gives you the "sdX" device name assigned by linux.

[MurrayW]

I opened up my server and all the disks that are missing/not installed are on the same icy dock 5/3 tray.  These 5 disks are connected to 5 of the motherboard SATA connectors.  The other 5 disks that are showing green are in a second 5/3 tray with one SATA connection from the motherboard and the other 4 from two SATA pci cards.

 

I disconnected the drive that was originally the only red drive and booted the server and now all the drives that are connected show as green (just like I originally saw before my first reboot that made the others go red).  Would one bad disk when connected within a 5/3 tray make the other 4 disks connected in that 5/3 tray show as Missing?

 

So it appears to me that I have a bad disk and I need to follow the procedures for replacing a bad disk.  Does that seem logical?

[Joe L.]

So it appears to me that I have a bad disk and I need to follow the procedures for replacing a bad disk.  Does that seem logical?

You might have a loose power connection to that tray.  The replacement of the one drive may have re-seated it.

 

Yes, at this time you can go about the process of replacing the defective drive.

 

Joe L.

[MurrayW]

I ran pre-clear on a new 2TB drive 3x and it was successful (no errors).  I restarted my server with the new disk (same size as the old disk) in the same location (disk 6).  The main page shows this disk as not installed.  Where do I go from here to add this disk to the array?  Disk 6 is showing up as not installed.

thanks,

Murray

[Joe L.]

I restarted my server with the new disk (same size as the old disk) in the same location (disk 6).  The main page shows this disk as not installed.  Where do I go from here to add this disk to the array?  Disk 6 is showing up as not installed.

thanks,

Murray

Does it show in the BIOS?  Do you see it listed (by model/serial number) in the syslog?  If you stop the array, is it available in the drop-down list for assignment? 

 

If not, then you have exactly the same conditions that might have taken the original out-of-service.  (the disk is not responding, for whatever reason... defective/intermittent controller-port/drive-tray/cabling/power)

[MurrayW]

When I stop the array it is available for assignment and lists the model/serial numbers.  Do I just assign it and then the array will rebuild and incorporate it with the other disks?  Or is there another step to get it into the array? thanks,Murray

[Joe L.]

When I stop the array it is available for assignment and lists the model/serial numbers.  Do I just assign it and then the array will rebuild and incorporate it with the other disks?  Or is there another step to get it into the array? thanks,Murray

That's it.  Just assign it, then start the array.

[MurrayW]

When I stop the array it is available for assignment and lists the model/serial numbers.  Do I just assign it and then the array will rebuild and incorporate it with the other disks?  Or is there another step to get it into the array? thanks,Murray

That's it.  Just assign it, then start the array.

Thanks...everything is back to normal now with the new drive.