I removed a wd ears drive to jump pin 7-8 but ran into some problem so I took the jumper out and everything booted up fine and data rebuild started on the one disk I removed (disk14).

 

For some reason it was really slow so I stop data rebuild to reboot the system.  Now disk4 is also showing up unformatted and disk14.

 

What should I do now?  I tried to reboot but disk4 still shows unformatted. I attached the syslog.

syslog.txt

As you are running an older version of unRAID, I believe you may be running into a bug that has since been fixed in newer versions.  First, try just stopping the array and then starting it again (do not reboot).  This may make your unformatted drives appear formatted again.

i'm running version 4.4.2.  I was reading around and saw that some people were having problem like you said and upgrading might fix it.  The other fix I saw was to run some riser (something like that) command to fix something on the drive.

 

To upgrade all I have to do is download the new version and copy it over to the flash drive?  Can I do this over the network to the machine or should I remove the flash drive and do it on another computer.  THanks.

When a drive appears as unformatted it simply means that the drive can not be mounted. Version 4.4.2 was not affected by the bug that erroneously caused multiple drives to appear unformatted.

 

I suspect that the cause is that the filesystem has become damaged on disk4, thus the suggestion to run reiserfsck to repair the damaged disk. I would suggest you sit tight until Joe L. happens along. I suspect he will help you repair the damage to your disk4, you may loose some data on disk14. Depending on the value of the data, I would proceed very carefully at this point.

When a drive appears as unformatted it simply means that the drive can not be mounted. Version 4.4.2 was not affected by the bug that erroneously caused multiple drives to appear unformatted.

 

I suspect that the cause is that the filesystem has become damaged on disk4, thus the suggestion to run reiserfsck to repair the damaged disk. I would suggest you sit tight until Joe L. happens along. I suspect he will help you repair the damage to your disk4, you may loose some data on disk14. Depending on the value of the data, I would proceed very carefully at this point.

Everything ohwiler said is correct.

 

You version of unRAID did not have the bug that caused drives to appear un-formatted when there was a race condition in mounting them. 

 

Your disk4 has file-system corruption of some sort.

 

Your disk14 is missing.  (as in the OS does not detect it online)

 

The advice to proceed carefully is very good advice.

 

First.  Do NOT even think about pressing the button labeled "Restore" as it has nothing to do with restoring data.  It is an "Initialize Configuration" button, but poorly labeled.  It was removed and replaced with an "initconfig" command in recent versions of unRAID.  If you press that button at this time you WILL lose the contents of both drives.

 

Second, You probably dislodged the cable to disk14 when changing the jumper.  (This is assuming you did not put the jumper on the wrong pins, or zap it with static electricity due to improper ESD procedures)  So... power down.  Check the cables to the drive. (Both DATA and POWER)

 

Power up.  Hopefully you'll see the drive in the BIOS as it boots.  Hopefully it will show up in unRAID's syslog.

 

No point in doing anything else until you check the cable since if it was loose it suggests one course of action and if you killed the drive there will be another.

 

Neither drive will restore from "red" without specific action at this point so do not expect the array to come online.  You basically have a two drive failure.  The data on the remaining drives is OK, but the data on both disk4 and 14 might be lost if you cannot get one of them working.

 

Joe L.

Joe L.

Disk 14 was the disk that I jump the pin and was currently getting rebuilt.  Disk4 is the one that should be fine. 

 

So to make sure i got this right.  My next step would to run reiserfsck on disk4 to try and fix it?

Disk 14 was the disk that I jump the pin and was currently getting rebuilt.  Disk4 is the one that should be fine. 

 

So to make sure i got this right.  My next step would to run reiserfsck on disk4 to try and fix it?

No.

 

I said:

 

You probably dislodged the cable to disk14 when changing the jumper.  (This is assuming you did not put the jumper on the wrong pins, or zap it with static electricity due to improper ESD procedures)  So... power down.  Check the cables to the drive. (Both DATA and POWER)

 

Power up.  Hopefully you'll see the drive in the BIOS as it boots.  Hopefully it will show up in unRAID's syslog.

No point in doing anything else until you check the cable since if it was loose it suggests one course of action and if you killed the drive there will be another.

 

Joe L.

 

Ok I check the cables and booted up and it still shows disk4 and 14 as unformatted with the option to hit format.  I know not to hit format or the restore.  What would be the next step?  Should I grab another syslog?

Ok I check the cables and booted up and it still shows disk4 and 14 as unformatted with the option to hit format.  I know not to hit format or the restore.  What would be the next step?  Should I grab another syslog?

I had suggested you look to see if the drive was visible in the BIOS disk listing...  Was it? 

 

Yes post a fresh syslog.

 

What "device" is disk4?  Which is disk14?  I cannot tell from the prior syslog. 

 

basically you need to issue the following command on each of the two drives in turn:

smartctl -d ata -a /dev/sdX

where sdX = the correct device name for disk4 and for disk14. 

 

Ok I check the cables and booted up and it still shows disk4 and 14 as unformatted with the option to hit format.  I know not to hit format or the restore.  What would be the next step?  Should I grab another syslog?

I had suggested you look to see if the drive was visible in the BIOS disk listing...   Was it?  

 

Yes post a fresh syslog.

 

What "device" is disk4?  Which is disk14?   I cannot tell from the prior syslog.  

 

basically you need to issue the following command on each of the two drives in turn:

smartctl -d ata -a /dev/sdX

where sdX = the correct device name for disk4 and for disk14.  

 

 

disk4 is sdp and dis14 is sdj, you get this from the device listing right? here is the full list

 

Disk devices

parity device: pci-0000:00:1f.2-scsi-0:0:0:0 (sdl) ata-ST31500341AS_9VS1S2ZA

disk1 device: pci-0000:00:1f.2-scsi-1:0:0:0 (sdn) ata-WDC_WD10EAVS-32D7B1_WD-WCAU46698054

disk2 device: pci-0000:00:1f.2-scsi-0:0:1:0 (sdm) ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0449112

disk3 device: pci-0000:00:1f.2-scsi-1:0:1:0 (sdo) ata-WDC_WD10EADS-00M2B0_WD-WCAV50027751

disk4 device: pci-0000:00:1f.5-scsi-0:0:0:0 (sdp) ata-ST31500341AS_9VS0BQP8

disk5 device: pci-0000:00:1f.5-scsi-1:0:0:0 (sdq) ata-ST31500341AS_9VS1VQTD

disk6 device: pci-0000:02:00.0-scsi-3:0:0:0 (sdi) ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0898303

disk7 device: pci-0000:02:00.0-scsi-2:0:0:0 (sdh) ata-WDC_WD10EACS-00D6B0_WD-WCAU40205511

disk8 device: pci-0000:02:00.0-scsi-0:0:0:0 (sdf) ata-Hitachi_HDS721010KLA330_GTE005PBGZ083K

disk9 device: pci-0000:02:00.0-scsi-1:0:0:0 (sdg) ata-ST31500541AS_5XW05H4E

disk10 device: pci-0000:01:00.0-scsi-2:0:0:0 (sdc) ata-ST31500541AS_9XW00B0M

disk11 device: pci-0000:01:00.0-scsi-1:0:0:0 (sdb) ata-WDC_WD15EARS-00Z5B1_WD-WMAVU2248296

disk12 device: pci-0000:01:00.0-scsi-0:0:0:0 (sda) ata-ST31500341AS_9VS0G81J

disk13 device: pci-0000:03:00.0-scsi-1:0:0:0 (sdk) ata-ST31500341AS_9VS11SDR

disk14 device: pci-0000:03:00.0-scsi-0:0:0:0 (sdj) ata-WDC_WD15EARS-00Z5B1_WD-WMAVU2359800

disk15 device: unassigned

cache device: pci-0000:01:00.0-scsi-3:0:0:0 (sdd) ata-ST9160310AS_5SV4EG8E

Array must be Stopped to make changes.

 

 

i ran smartctl but found out that version 4.4.2 doesn't have it so i followed your instructions at this link http://lime-technology.com/forum/index.php?topic=2817.msg23548#msg23548

 

is the smartctl saved somewhere? i attached a photo of what came up on the screen also the new syslog

 

syslog.txt

this is the smart for disk14

To capture the output of a command re-direct the output into a file, then you can attach the files. 

 

So...

smartctl -d ata -a /dev/sdp > /boot/smart_sdp.txt

 

smartctl -d ata -a /dev/sdj > /boot/smart_sdj.txt

 

They'll be viewable on the \\tower\flash shared drive on the lan of you create them at /boot

 

The picture of the tail end of the smart output is not showing the section I need to see.

 

 

 

 

done here are the smart for the 2 drives.  Thanks

smart_sdj.txt

smart_sdp.txt

OK...

 

sdp is showing this:

 

 5 Reallocated_Sector_Ct   0x0033   001   001   036    Pre-fail  Always   FAILING_NOW 4095

 

It has basically crashed... it is failing the SMART tests.    We do NOT want it in your array.  

 

sdj is showing this:

 

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       23

 

It has 23 sectors that are pending re-allocation.    We'll have to accept the loss of whatever was in those sectors... and go from there.

 

So... Step 1.  We need to force sdp as the failed drive, force sdj as being "good" and then, once we can get to it, run reiserfsck on each in turn.

 

There is risk in all of this.   Just so you know.  If you feel uncomfortable in trying any of this, write to limetech and see if they have any better ideas.   I can see both drives are "responding" so I feel we can force the real failing drive as invalid and hope for the best with the other.   Worst case is it also fails when you attempt to write to it, and you end up in the situation you are now.  Two failed disks and the loss of what is on them.   I think we can fix the file-system damage on sdj (disk14)  easier than that on sdp (disk4).  

 

To proceed...

 

Step 1. Log in via telnet as root.

Step 2. Normally we would stop the array but, the array is already stopped, so no need to stop it.

Step 3. Copy the existing "config" folder to old_config by typing:

cp -r /boot/config /boot/old_config

Now, we want to set a new initial configuration AND also to force disk4 as invalid.  To do that you need to press the dreaded "Restore" button on the management web-page, but DO NOT start the array... not yet.  Pressing the "restore" buton sets a new initial configuration and immediately invalidates parity...  We need to force parity to be considered good but disk4 to be considered bad.  To do that we need another command typed at the command line.

 

After clicking on restore, but before doing anything else, type on the linux command line

/root/mdcmd set invalidslot 4

It will respond with

cmdOper=set

cmdResult=ok

 (or something like that)

 

Now, you can refresh the management web-page.  It should show disk4 as "red" and the others as green.  

If so, you can start the array.

 

Now, it might not mount the disks, but we'll see at that point how things go.  With luck you'll be able to follow the instructions as shown here in the wiki to fix the corruption on disk14.

http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

Note: if it does not mount the disks the odds are they will show as"unformatted" on the display.  DO NOT format them.  Repeat DO NOT press the format button.  (I can repeat myself... you really do not want to shoot yourself in the foot... again)

 

You need to run reiserfsck on /dev/md14

 

It will probably state it will need repair, but let's see what it says.  

 

In the interim, I'd purchase/order a new disk to replace the failed disk4.  

 

Joe L.

ok i did everything and it said

 

cmdOper=set

cmdResult=ok

 

but when I hit refresh it shows everything as blue except the cache drive.  Did I have to start then hit refresh? or should i go to the link to fix the disk now?

ok i did everything and it said

 

cmdOper=set

cmdResult=ok

 

but when I hit refresh it shows everything as blue except the cache drive.  Did I have to start then hit refresh? or should i go to the link to fix the disk now?

Everything blue is probably ok if it said ok when you typed

/root/mdcmd set invalidslot 4

 

The next thing to do is to press "Start" to start the array. (may need to check the box under it to enable it)

 

As I said, disk4 and 14 will probably not mount and show as unformatted.  Dis-regard that, it simply indicates they did not mount.  Post a new syslog.  DO NOT FORMAT anything.

 

You can proceed to check the file system on disk14.

 

Joe L.

ok I hit start and everything started up with disk4 ball being red and all other green and it started data-rebuilding.  So I should leave it alone for now until its done rebuilding?

the syslog was larger so I had to zip it to be able to upload.

syslog.zip

What power supply are you using?

What disk controller? 

What motherboard?

 

I ask because I now am seeing errors from disks sda and sdk in your syslog.  The disk controller keeps resetting them... so let the re-construction continue.

 

Joe L.

 

 

Actually, the re-construction is really not doing anything useful because disk4 has failed.

 

It appears as if disk14 was mounted, so go through the steps of checking its file-system.  (and order/purchase a replacement disk for the failed disk4.)

 

The issue you will face is that you were using the failed disk4 to re-construct disk14 when you changed the jumpers on it. 

It might have corrupted the re-construction.

 

We'll know when you perform a file-system check on disk14 (/dev/md14) how bad it is.

What power supply are you using?

What disk controller? 

What motherboard?

 

I ask because I now am seeing errors from disks sda and sdk in your syslog.  The disk controller keeps resetting them... so let the re-construction continue.

 

Joe L.

 

 

 

2x Adaptec 2241000-R PCI Express x4 SATA II (3.0Gb/s) 1430SA SGL

SUPERMICRO MBD-C2SEE-O LGA 775 Intel G43 ATX Intel Motherboard

OCZ ModXStream Pro OCZ500MXSP 500W

and some cheap 2 port sata card I picked up.

 

So I should let the re-construction go on.  Its running really slow, I have a 1.5tb drive right now so I should replace disk4 after it is done with re-construction? or should I run the command to fix disk14?

You do realize that supply is a dual rail supply with ALL the SATA and MOLEX connectors on the same 12volt rail limited to 18Amps capacity?

 

You have 15 drives on it, and it is WAY overloaded even if they were all "green" drives drawing only 2 Amps each.  Add in any non-green drives at 3 Amps and a few fans and it is easy to see why you are in the shape you are.

 

See here: http://www.ocztechnology.com/res_old/manuals/psu/OCZMXSP500-700.pdf

 

While you are ordering a new disk, consider ordering/purchasing a new power supply with a SINGLE 12Volt rail of 50Amps or more.

 

Joe L.

I have a 1.5tb drive right now so I should replace disk4 after it is done with re-construction? or should I run the command to fix disk14?

Stop the array, replace disk4 with the new.  It might work when you power up... At least the reconstruction may be helpful.

 

I do not have great hope until you replace the power supply too.  You are getting too many different disks acting up, since they are probably starved for 12Volt power.

hmm didn't know that.  I have this other ps http://www.newegg.com/Product/Product.aspx?Item=N82E16817341039 would this one work better?

It said it still has another 48300min before it finish.

hmm didn't know that.  I have this other ps http://www.newegg.com/Product/Product.aspx?Item=N82E16817341039 would this one work better?

It said it still has another 48300min before it finish.

 

No, probably not.  It has multiple rails in it also with them all split into 18amps.  Either of those supplies is severely lacking for using in an unRAID server with 15 drives.  Frankly it is kind of a miracle it is working at all and has not crashed and burned earlier.  The max number of drives I would have put on either PSU would have been 8 green drives, and probably really only 7 green drives.