So I think I have a bad case of if it ain't broke don't fix it now...

 

Ok a little background first.  I have had UnRAID running on ESXi 5.1 for about 8-10 months now.  640GB datastore drive for ESXi.  Then I have a supermicro adapter that's being passed thru to the UnRAID array.  1 640GB cache drive, 3 3TB array drives (1 parity, 2 data).  I had picked up a new SSD to replace the cache drive in the array.  And I also picked up an LSI adapter to turn my datastore drive into a mirrored drive array since I didn't want to lose my ESXi datastore.

 

So I put in the new card, no problems.

Put in the SSD on the same controller I'm passing through to UnRAID so I can copy the files on the cache drive over to it.

 

I tell the SSD to be part of the array so I can format it and then get it all moved over.

 

The SSD became part of the array, but I have not formatted it yet.  At the same time one of my disks now shows up as Red and is a "disabled disk".  My Parity and my other disk are both green and fine.  So I decided to unplug my SSD just because it was getting late, figuring maybe it just didn't like that SSD being part of the array and it did something to that disk.

 

So now it just thinks the SSD is missing instead and that disk that was spun down and disabled is still spun down and disabled...

 

I am using UnRAID 5.0.  All my apps are on my cache drive, such as Plex.

 

What did I do wrong and how do I get my 'disabled disk' in the array back without losing it's data? 

 

your error was assigning the ssd to the array. now unraid thinks its part of the array and will throw fits when removed.

 

i cant help you with disk1 redball but the way i transitioned my cache was this:

 

unassign old cache, assign new drive to cache.

manually mount old cache as /mnt/oldcache

opened "midnight commander" "mc" and copied all data from /mnt/oldcache to /mnt/cache

Thanks, I found instructions on how to remove the drive from the array, so I'm less concerned about that now, especially since I never formatted it, than I am the redball on disk 1...

 

It looks as though my Parity and Disk 2 are in tact so even if I can't get my disk 1 back online the hope is I can at least recover it.

 

But any suggestions would be extremely helpful.

If you have altered the contents of the SSD at all since you removed it, you lost your ability to fully recover your failed disks data unless the drive itself is really ok. You will need to put the SSD back so unraid can use it to emulate the other failed disk. Parity can only reconstruct one drive at a time, and right now you have 2 missing drives. If you put the SSD back untouched, unraid should be able to reconstruct disk1. Without it, disk1 data is gone, unless the drive itself is actually ok. The fact that you never formatted the SSD is irrelevant, unraid still uses the raw drive to calculate parity for the missing disk.

 

You may want to physically remove the failed disk1 and see if it is readable in another box, or just restore from your backups.

 

 

I can always put the SSD back in, but I didn't see any way to restore Disk 1 at that point either.

I can always put the SSD back in, but I didn't see any way to restore Disk 1 at that point either.

If the SSD is no longer listed as missing in the screenshot you posted earlier, you should be able to start the array degraded, and access the files to copy them elsewhere, or put a new disk in slot 1 and let it rebuild the data onto that. As long as more than one drive is missing, it will not allow the array to start at all. If only one drive is missing, you should be able to start the array.

Yes that shouldn't be a problem I actually did that last night in the hopes that Disk 1 would come back to life.  But nothing seemed to get that drive back at that point in time.

 

So given that the array comes back online and that Disk 1 still shows the red ball, even though I'm fairly confident it's still a good drive.  What should my next steps be?

So given that the array comes back online and that Disk 1 still shows the red ball, even though I'm fairly confident it's still a good drive.  What should my next steps be?

First step would be to verify that your backups of the data contained by disk1 are good. Next would be to get your array healthy again, so a second failure doesn't cause data loss. Getting the array healthy means reconstructing the contents of disk1 that are now being emulated by the sum of all the remaining drives written onto another drive. If you are confident that disk1 really is ok, keep in mind that a write to it failed, as that is the only reason unraid red-balls a drive. Many things can cause a failed write, but I would be cautious about trusting the drive without at least getting a clean smart report. You can unassign disk1, start the array, stop the array, and reassign the drive, and it should allow you to rebuild the emulated data back onto the same drive. Better option would be to preclear a new drive to check for infant mortality failures, and rebuild onto it, and then preclear the failed drive to exercise it fully. If it passes preclear, then I would be confident about continuing to trust it.

Thanks.  The reason why I feel the drive is probably fine is that up until I tossed in the new drive, it was fine.  I think it must have been a write error during a reboot or array action that caused it to red ball.  Ultimately, if I get a clear smart report on the drive.  Would there be any harm in removing it and just readding it to the array or at this point will I be forced to preclear the drive and rebuild it?

 

Unfortunately I really don't have a spare drive I can toss in there at the moment.

if I get a clear smart report on the drive.  Would there be any harm in removing it and just readding it to the array or at this point will I be forced to preclear the drive and rebuild it?

If the drive tests good, you have two viable options. One, rebuild the drive onto itself, and no, you don't have to preclear it to rebuild on it. Two, invalidate parity and recalculate parity based on the current contents of all the array drives you want to keep. Keep in mind, that invalidating parity means that any writes to that drive after the slot was red-balled will be lost. That may mean file system corruption, which will have to be fixed. If the emulated red-balled drive is fully accessible and valid, then the rebuilt drive will be as well.

Ok the drive was confirmed ok, I removed it from the array and then readded it, it acted as though it was rebuilding the drive (seemed to be going faster than I would have thought).

Before I actually rebuilt it the drive was showing up as orange instead of red or green.  I could actually access the files on the disk at this point.  Shortly after this I just ran the rebuild just feeling it would be safer...

 

Via command line I was seeing an error while it was rebuilding 'REISERFS error (device sdb1): zam-7001 reiserfs_find_entry: io error' over and over again.

On the web gui, everything is green now, but if I attempt to view any of the contents of the disks nothing shows up.

 

When using the command line if I attempt to list files in /mnt/disk1 or /mnt/disk2 or even /mnt/cache I get the same error.  /bin/ls: reading directory .: Input/output error.

 

So I thought ok I'll stop the array.

The web interface went into a loop of "Retry unmounting user share(s)..."

So at this point I rebooted via command line.

 

And I was typing this all out while it was occurring.  Upon reboot it seems to be functioning correctly and I can see everything.

 

Now the only outstanding issues will be the removal of the SSD from the array as well as to Plex now for some reason it's not displaying certain thumbnails.

 

Thanks for your help jonathanm!

Did you stop the array, unassign the disk, start, stop, re-assign, start?

Yep

I may have spoke too soon.  I'm getting a lot of really strange stability issues now.  For instance sometimes the unraid system kind of stalls out.  Plex can still read data, but if I try to browse a directory via command line or network share it just stalls out.  On top of that the web gui won't load.  It's hard to explain.

Disk 2 now has a red ball...

Zip and post a full syslog, it may have clues as to what's causing this. Also, you may want to temporarily run baremetal unraid for a bit, or try putting things back the way they were before things started going nuts. You could have bumped some cables loose while you were mucking around inside the case, so I'd go back through and check all connections. In any case, I would be sure you have current backups, things may get hairy while you try to troubleshoot this.

Knowing those disks checked out ok and seeing the data still in them, I went the easy way out and did a new config.

Reassigned the drives without the SSD in there and let the parity check rerun.

 

I haven't had a chance to get the SSD switched over as the cache drive yet, but hopefully I get a chance to do that this week along with getting my apps going again.