One of my disk went bad (just clicking).  So disk3 is missing.  When I rebooted the server it also is saying that disk15 is wrong.  unmenu shows the drive model/serial shifted a little ?

 

DISK_WRONG /dev/md15  /dev/sdp SAMSUNG HD103UJ / S13PJ1NQ206647

                                                      SAMSUNG HD103UJ _S13PJ1NQ206647 <-- was old disk in this slot

 

I'm hoping this will go away once drive3 is back. But if not, what if anything to make drive15 green without altering or rebuilding my parity since I will need it to rebuild drive3 later?

syslog is attached.

You are in a delicate state.

 

Be very careful that you don't do anything hasty, like pressing restore or changing devices.

 

I'd recommend that you make a copy of the "config" directory from your flash.  If something goes wrong, it is likely you can restore this directory and be back to where you are now.

 

I have an idea of how you may be able to recover using the "set invalidslot" command, but think that you should wait for RobJ and/or Joe L. to review your syslog.  They may be able to suggest something a little less drastic than what I am thinking.  I will post back if no one has another suggestion with the specifics (DO NOT USE SET INVALIDSLOT 99!).

 

It is important that you have a drive ready to replace the one that actually failed.

You were wise to pause, for a full evaluation.  As Brian said, 'You are in a delicate state'.  The first thing I was going to suggest is make sure you have a backup of the config folder of the flash drive, but Brian has already said that too!  Also, make sure you preserve any backup you already have, from just prior to this issue, that is, don't overwrite it with a new backup.  These backups may prove to be vital.

 

The Device inventory shows that Disk 15 (SAMSUNG_HD103UJ_S13PJ1NQ206647) was identified correctly and identically, before and after this issue, but the super.dat does have a small conflict with its 'analysis'.  The first line below is from your previous syslog, and the second is from the new.

modprobe md-mod super=/boot/config/super.dat slots=8,144,8,160,8,192,8,176,8,208,8,224,8,80,8,96,8,32,8,240,8,128,8,0,8,16,8,112,8,48,65,0 >>/var/log/go 2>&1
modprobe md-mod super=/boot/config/super.dat slots=8,144,8,160,8,176,0,0,8,192,8,208,8,80,8,96,8,32,8,224,8,128,8,0,8,16,8,112,8,48,8,240 >>/var/log/go 2>&1

 

Disk 15 is the 65,0 pair in the previous syslog, but it comes up as 8,240 in the new.  I am in no way an expert here, so will welcome more expert opinion, but the new one actually looks correct, where as the previous pair looks wrong.  The 8,240 pair became available because the Disk 3 pair was empty, and this may be a real problem here.  Perhaps installing the new replacement drive, using the same port it was connected to, will shift the pair numbers back where they were.  The next pair after 8,240 would be 8,256, but that 256 would overflow its one byte, which may be the bug mentioned below.  Tom may have to get involved here, if he is available.

 

I would like to note that you are running unRAID v4.3-beta6, which is prior to an important and relevant bug fix in v4.3.2 and v4.3.3, which had to do with the mis-identification of the 16th drive, which yours is.  I would strongly recommend that anyone with 16 drives (or going to 16 drives) upgrade to v4.3.3 first.  It is too late here though, you don't want to make that kind of change now.

 

I note that this drive is one of the 2 drives on the JMB controller, common to the exception Emask errors previously reported, but I don't know of a connection to this issue.  I hope you were able to replace those 2 SATA cables.  And I hate to mention it, but the parity errors you were getting there, make this particular drive rebuild even more worrisome.  There is no guarantee here of a perfect drive rebuild.

 

After you have ensured your backups are made, try rebooting, perhaps several times, and check the status of Disk 15, and the syslog each time for that last pair.  If no change, un-assign Disk 15 and re-assign Disk 15, then reboot again and check.  If no change, restore the current backup to the config folder.  Before doing anything drastic, if you can, wait another day or 2 for others (hopefully Tom) to throw out some more ideas.  If you possibly have other copies of everything on Disk 3, then your final choice may be to un-assign the current failed Disk 3, click the Restore button to start fresh, let it build a new parity drive from the 14 data disks remaining, then add the new drive in the slot for Disk 3.  And update afterward to v4.3.3!

Have you ordered a replacement for the failed disk #3 ?

If so, it might be worthwhile to install that and see how the drives are allocated.

Have you ordered a replacement for the failed disk #3 ?

If so, it might be worthwhile to install that and see how the drives are allocated.

I would do this before anything else.  Odds are good the other disk would move back to its original slot in the array and no longer be considered in the wrong slot.  At that point, all you would need to to is press "Start" to rebuild the disk that failed.

 

Joe L.

BJP and Rob,

Yes, my unraid is at a crossroads in the intensive care unit - but hopefully in good hands     (I understand that I might not get a 100% data rebuild because of the parity issue from the JMB controller)  It's hard not to mess/try things!  But I know better.  Didn't know about the 16th drive bug - v4.3-beta6 was the latest when I started this around 4-5 months ago.  Once drives are stable will goto 4.3.3.  Can't wait for 4.4 to get out of beta!

 

People like you that know so much about unraid and test the beta's out and offer support are saints.

 

First, I don't understand the drive pair numbers in the slots= from the super.dat  Any reference to decode the numbers?

Interesting how it mumbo jumbled up the drive pair assignments.  Logic would be they just all wound shift over by two bytes.

 

Second, I might have an older super.dat backed up. Not sure if it matches the current drive set however.  Already have backed up the whole key.    Rebooted a number of times and drive15 always keep coming up 'wrong'

 

Third, no other copies of disk 3 is available.  :'(

 

Forth: Should I order a PCI sata card to bypass the JMB controller ports even though it slow things down?  Any JMB flash, abit bios, or unraid items that fix this or is getting worked on ?

 

So my homework list is:

1. I'll swap the cable to disk 15. 

2. Wait for a new drive to show up and put into drive3.  Hoping this will resolve disk15 being wrong.

3. Should I unassign disk 15, assign disk 15, and reboot now without drive3, or wait until I try a new drive first ?

4. Any other ideas from Tom or others?  I estimate my new drive won't show up until Monday

Thanks everyone

 

First, I don't understand the drive pair numbers in the slots= from the super.dat  Any reference to decode the numbers?

Interesting how it mumbo jumbled up the drive pair assignments.  Logic would be they just all wound shift over by two bytes.

Linux assigns each piece of hardware it finds a "Major" and "Minor" device number.  Those two are the internal identification of the "device" in the system.  The two number pairs in the "modprobe" command are the major/minor number pairs of the disks in the array.  The names you see in the directory listing are simply pointers to the actual device "nodes". 

   

Because of your hardware failure, the major and minor numbers were re-assigned to different hardware in your PC since it detected the available hardware and assigned device numbers as it scanned the PC during its boot up process..   

 

unRAID expects disks to be in the same slots and it even though it can actually deal with you moving disks around between the same set of hardware, apparently, the new assignment is to a major-minor pair not in your original configuration.  It cannot handle that and stops the array from starting.

 

Whatever you do, DO NOT PRESS THE RESTORE Button.  It immediately invalidates parity and eliminates almost any possibility of data from the failed drive.

 

When you replace drive 3, you should only need to press the "Start" button. (You might have to check an "I'm sure" checkbox beneath it to enable it)

So my homework list is:

1. I'll swap the cable to disk 15. 

Odds are it will do noting to fix the missing disk, but it might help with the intermittent errors you have in the syslog.

2. Wait for a new drive to show up and put into drive3.  Hoping this will resolve disk15 being wrong.

That is what I would do.

3. Should I unassign disk 15, assign disk 15, and reboot now without drive3, or wait until I try a new drive first ?

I would NOT do that, at least not now, as it might make things even worse to recover from.

4. Any other ideas from Tom or others?  I estimate my new drive won't show up until Monday

Thanks everyone

The beauty of all this is that when the new disk arrives and replaces the defective one it is very high odds you will be back up in a short time.  I'd shut the server down in the interim since it is not usable, no sense spinning drives waiting for a concurrent second disk failure.

 

You definitely want to update the version of unRAID you are using once things settle down.

 

Joe L.

Joe L, thanks for explaining the major.minor pairing.  Weird that it naming through a different node would make slight change to the model/serial naming.  The clicking drive 3 is truly missing since it's now heading back to newegg.  The server has been powered down now till my new drive arrives Monday.

 

It fix the random sync errors does anyone know of a fix to the JMB controller for sata8 and sata9 on the ab9 pro ?  Or if JMicron, abit, or any Linux solutions coming ?

 

There are a large number of ideas floating around the internet to try to get a few more hours of life out of a dead of dying disk drive.  I might have suggested the freezer method if you had something important on disk3 - before sending it back for exchange.

 

I agree with Joe L.  You have a good chance of getting back your data.  If, when you add your new disk back in you still can't get unRAID to recognize your disk 15, let me know.

I'll post what happens next week.  It's going to be a very loooong weekend. I miss the fan noise already 

Ok, new drive is installed into drive3. (sdl) ata-SAMSUNG_HD103UJ_S13PJ9AQ901501 Array started by itself.

GUI says drive3 is not installed (red dot).  But drive15 is back and status green as last drive pair 65,0

New syslog3.txt file attached.

 

Keep in mind that my parity drive is probably not 100% perfect thanks to the JMB controller issue.

But 30-100 bits being off is better than no data!

 

Note: I now own a PCI controller and can install it and move drive 9 and 10 off of the JMB controller on the MB.

 

What steps do you recommend?

1. Backup my virtual drive 3 data off of the array.

Then

A. Assign the new drive?  Will it then ask to rebuild drive 3?

B. Should I power down and get the two drives off of the JMB controller before the rebuild or after?

C. Backup my config from my key now?

D. By any chance could I compare the rebuild from my virtual drive 3 data to what's rebuilt to know what files are different because of never having a clean parity sync?

E. Will upgrading to 4.3.3 fix this issue of loosing drive15 when 3 went out?  Or is this something else because of my odd pairing number for the last drive?

F. Anything else?

 

Thanks a bunch

 

 

A little surprised it did not offer to begin rebuilding Disk 3 already, but I guess with all the testing, stopping and starting, it is waiting for the new drive to be assigned.  Here's a link to a good walk-through with screen shots similar to what you should see:  FAQ:How do I replace a hard disk? (credit to Joe!).

 

Now for the order of your steps and other questions:  I don't think your parity drive is perfect, but you don't have any choice here.  Once you rebuild, then run reiserfsck on it (Check Disk Filesystems), and when that finishes, examine the contents of Disk 3.  We can't really help you here, only you can determine what is damaged/missing or intact.  Usually compressed files like zips have testing functions.

 

Don't make any other hardware or software changes, until the drive is rebuilt.  Once the array is back up, then upgrade to v4.3.3, and then consider any hardware changes.

 

1. I like your idea of backing up anything important on the virtual Disk 3.  Although we hope everything will be restored successfully, it's always a good idea to make sure that anything important or that is the only copy, is safe elsewhere first.

 

A. Yes, assign the drive, and I *think* it should be ready to rebuild Disk 3.

B. No hardware changes yet.

C. Can never have too many backups!

D. What will be rebuilt IS what your virtual drive 3 is now.

E. Yes.

From Rob

A. Yes, assign the drive, and I *think* it should be ready to rebuild Disk 3.

B. No hardware changes yet.

 

Will rebuilding cause more bits to be written incorrect to drive 3?  Because some bits are off on the parity drive already.  Plus drive 9 and 10 under the JMB controller could introduce even some new ones?

 

Rob/Brian/Joe,

Ok. I have rebuilt drive 3.  Parity check showed:

Check will start a Parity-Check.

(Last checked on 10/22/2008 10:12:55 PM, finding 32 errors.)

 

reiserfsck info:

reiserfsck --check started at Thu Oct 23 17:27:35 2008

###########

Replaying journal..

Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed

Checking internal tree..finished

Comparing bitmaps..finished

Checking Semantic tree:

finished

No corruptions found

There are on the filesystem:

        Leaves 123602

        Internal nodes 738

        Directories 18

        Other files 298

        Data block pointers 125071451 (0 of them are zero)

        Safe links 0

###########

reiserfsck finished at Thu Oct 23 17:29:51 2008

 

Some more good news I think: I ran a MD5 hash on all the files from the virtual backup and the now rebuilt drive 3 and they all match! Was expecting some to be off since the rebuild had 32 sync errors. Hmmmm ?

 

Now to get the parity check stable to 0 sync errors.  Should I in order:

1. Before trying 2 and 3 should I upgrade to 4.3.3 from 4.3.beta6 ?

1A. Will that cause a parity rebuild or just abort it if starts?

2. Swap cables to drive 9 and 10 and rebuild parity to see if any sync's are corrected?

2A. Do it twice to make sure it's stable?

3. Add in my new PCI TX4 board to bypass the JMicron controller for drive 9 and 10 ?  Then rebuild parity?  Twice?

 

I've found that oft times you'll get less than a hundred or so sync errors resulting from the Reiser FS's "housekeeping" stuff at the beginning of the disk that has nada to do with your data.

 

2. Swap cables to drive 9 and 10 and rebuild parity to see if any sync's are corrected?

 

Sync errors are "corrected" immediately upon detection by unRaid. So if you reran your parity check (which I would suggest), you should get zero errors. After a 100% clean parity check, upgrade to 4.3.3.

Hello Brian,

So far I have never had a parity check go zero errors.  So do you want me to redo parity as is or swap out the cables first?  If that doesn't get a clean parity should I try a new controller from drive 9 and 10 that was getting the communication errors ?

 

Jim: Are you saying you always get a few sync errors on a parity check because of how Reiser file system works ?  This might make sense since my data does seem to be in tack.

 

no, I mean that when you move disks around or otherwise reconfigure the array, or boot a diagnostic linux cd for instance, the RFS "header" at the beginning of the disk gets updated and will often result in immediately seeing a hundred or so sync errors as parity check starts.... next parity check should be 100% clean...

 

Hello Brian,

So far I have never had a parity check go zero errors.  So do you want me to redo parity as is or swap out the cables first?  If that doesn't get a clean parity should I try a new controller from drive 9 and 10 that was getting the communication errors ?

 

Jim: Are you saying you always get a few sync errors on a parity check because of how Reiser file system works ?  This might make sense since my data does seem to be in tack.

 

I've had my unRAID array for several years and the only time I've ever had any parity errors detected is if the array was in the process of writing a file and it lost power.

You will, of course have errors if you boot your PC with a slackware live CD, as it does update the individual drive's superblock when it mounts a drive, and that change will not exist in your parity data when subsequently booted under unRAID.

 

You have a hardware issue... you know that... it could be as simple as a cable, or as complicated as the motherboard design itself.  I'd try the external SATA controller.  It might get you past the poorly performing SATA chipset on the motherboard.  (If you suspect a cheap cable is the issue, try it first)

 

 

I have had parity errors after a power outage (before buying UPS) on two separate occasions. All the drives were inactive and spun down at the times. Other than that, my parity checks have been 100% clean.

 

Since you are running an old beta that has known problems with a full set of disks, I'd upgrade to 4.3.3 and run your next parity check.

 

If you can't get a clean parity check on the 1st or 2nd try, then I agree with Joe L's comments and advice.

Hi Brian/Joe/Jim/Rob,

Haven't done any other work or power issues to be getting a few sync errors.  Joe is right, I have something a muck.  But I do have good news: I finally have a zero sync parity check!!!     

 

Just don't know which of the following fixed it. Since I did all of this at once:

A. I replaced both cables to drive 9 and 10.

B. Upgraded to unRaid 4.3.3 from 4.2.beta6

C. Moved drive 9 from the JMB controller to a spare port on my PCIe controller. So the JMB is only running 1 drive now.

 

Again - Thank you so much for all your help in getting me through all of this. 

Hard to argue with success. And since you have a full boat of drives I would stick with those cables and those ports and not tempt the fates.

 

I'd probably try running a parity check every few days until you are confident of the array integrity.