I am running 6.4 rc9f and I had a disk go bad 2 weeks ago while I was out of town.  I have removed the bad disk from the array and it is currently being emulated.  I have formatted the new disk as XFS which is how the other disks in the array are formatted.  The new disk is showing up in unassigned devices prior to trying to add it to the array.  When I try to add the new disk to the array it does not add it then it disappears from unassigned devices.  What the heck is going on!

 

Last time I had to replace a disk in 6.3 it was simple.  I removed the bad disk, installed a new disk of the same size booted the array and it started rebuilding it. 

 

SCSI Devices

[0:0:0:0]    disk    SanDisk  Cruzer Fit       1.27  /dev/sda   15.6GB
[2:0:0:0]    disk    PNY      USB 2.0 FD       PMAP  /dev/sdb   15.6GB
[4:0:0:0]    disk    ATA      KINGSTON SV300S3 BBF0  /dev/sdc    240GB
[5:0:0:0]    disk    ATA      Samsung SSD 850  2B6Q  /dev/sdd    256GB
[8:0:0:0]    disk    ATA      SanDisk Ultra II 00RL  /dev/sde    480GB
[9:0:0:0]    disk    ATA      SanDisk Ultra II 00RL  /dev/sdf    480GB
[10:0:0:0]   disk    ATA      ST1000LX001-1EM1 SD02  /dev/sdg   1.00TB
[10:0:1:0]   disk    ATA      ST9750420AS      SDM5  /dev/sdh    750GB
[10:0:2:0]   disk    ATA      ST1000LM014-1EJ1 SM36  /dev/sdi   1.00TB
[10:0:3:0]   disk    ATA      ST9750420AS      SDM5  /dev/sdj    750GB
[10:0:4:0]   disk    ATA      ST1000LX015-1U71 SDM1  /dev/sdk   1.00TB  This is the new disk
[10:0:5:0]   disk    ATA      ST1000LX001-1EM1 SD02  /dev/sdl   1.00TB
[10:0:6:0]   disk    ATA      ST1000LX001-1EM1 SD02  /dev/sdm   1.00TB
[10:0:7:0]   disk    ATA      ST1000LX001-1EM1 SD02  /dev/sdn   1.00TB

I am freaking out here cause I do not know what to do!  Please help!

 

mccserverur01-diagnostics-20171106-1807.zip

Hi -

 

I think you have your steps mixed up but maybe just terminology?  You don't format the replacement disk, and you don't add it to the array unless you plan to copy the contents of the emulated disk to the new disk.  To rebuild, you stop the array, assign the new disk in the old slot, start the array and let the rebuild start... 

Edited November 6, 20178 yr by tdallen

18 minutes ago, csmccarron said:

I have formatted the new disk as XFS which is how the other disks in the array are formatted

 

You don't need to format a disk before rebuilding it, it's not a problem but it's a waste of time.

 

Replacement disk dropped offline:

 

Nov  6 17:50:20 mccserverur01 kernel: blk_update_request: I/O error, dev sdk, sector 0
Nov  6 17:50:20 mccserverur01 kernel: sd 10:0:4:0: Device offlined - not ready after error recovery

Since it dropped offline there's no SMART report, so powerdown, check/replace cables and power back on, try again but also check SMART.

Ok, thanks...it must have been either the cable or sata port, not sure yet.  I move the disk to a spare port and changed the cable and my array is currently rebuilding. 

 

Holy s%$t that scared me to death.  As soon as the rebuild finishes I will move it back to the original port.  Hopefully its a bad cable and not the port.  I also update to 6.4 rc10b.

Edited November 7, 20178 yr by csmccarron

So after the array finished rebuilding, I powered down and moved the new cable to the old port.  The system booted without issues and the array came back online.  Overnight it again reported errors.  This morning I moved the cable to the port I rebuilt the array on but I cannot get the disk to come back online.  I have attached the smart diagnostics for the new drive.

 

How do I get this disk back online?

mccserverur01-smart-20171107-0808.zip

6 minutes ago, csmccarron said:

So after the array finished rebuilding, I powered down and moved the new cable to the old port.  The system booted without issues and the array came back online.  Overnight it again reported errors.  This morning I moved the cable to the port I rebuilt the array on but I cannot get the disk to come back online.  I have attached the smart diagnostics for the new drive.

 

How do I get this disk back online?

mccserverur01-smart-20171107-0808.zip

The "reported errors" is too vague to know what you mean. And you only posted SMART. You should always post the complete diagnostics zip unless we specifically ask for something less.

 

Maybe you already know and aren't likely to make this mistake, but since you said something in your OP about formatting a replacement disk I will give you this additional warning.

 

You absolutely must NOT format a disk as part of the rebuild process, even if unRAID says it is unformatted. Formatting a disk (or even the emulated disk) writes an empty filesystem to the disk. unRAID treats this write just like any other, by updating parity. So after a format of a disk (or emulated disk) in the array, parity agrees it has an empty filesystem. Obviously a bad thing if you were planning to rebuild its data.

I formatted the disk prior to adding it to the array.  Once I changed the cable and move it to a new sata port I was able to add it to the array and the array rebuilt the disk.  After the rebuild was complete, I moved the disk back to the original port to see if I just had a bad cable.  Around 2am it started reporting errors again.  This morning I shutdown the system, moved the disk back to the port I used last night and booted the system but I cannot get the disk to come back online.  Smart is telling me there is nothing wrong with the disk.  Its a brand new disk, never been used until yesterday.

 

Diagnostics attached.

mccserverur01-diagnostics-20171107-0833.zip

Once a disk is disabled you need to rebuild again:

 

https://wiki.lime-technology.com/Troubleshooting#Re-enable_the_drive

 

 

Thanks for the information.

This morning I awoke to Reallocated sectors on my parity disk.  I ordered a new ASUS Pike 2008 card, 8 new WD Black 1TB hard drives and 3 ICY DOCK 6 x 2.5 SSD to 5.25 Drive Bay Hot Swap.  I also added an extra parity disk just to be safe.

 

When it rains it poors!

 

Edited November 12, 20178 yr by csmccarron

I hope those drives were free or cheaper than dirt. Seems like working with so many drives is more hassle than it's worth considering you could replace all your headaches with 2 8TB to 10TB drives, 1 for parity and 1 for data.

2 hours ago, BRiT said:

I hope those drives were free or cheaper than dirt. Seems like working with so many drives is more hassle than it's worth considering you could replace all your headaches with 2 8TB to 10TB drives, 1 for parity and 1 for data.

 

The drives were not free or cheap.  I also do not mind working with lots of drives which is why I went ahead an bought the hot swap bays to make it easier to replace a drive when it fails.  Currently the way they are mounted makes it a royal pain in the butt. 

 

I personally prefer lots of drives because it provides faster access to my movies and music at the expense of a little longer write time.  While I agree it would be easier to use fewer larger drives, at this point on this build I am not ready to completely redo the entire array.  I am just going to keep it healthy in its current configuration.  Maybe when I rebuild my system I will move to fewer larger drives or maybe I will just switch out to 1 TB SSD's when they come down in price.

 

What sucks is I think my PIKE card is failing and they are getting hard to find so that means a system upgrade/rebuild is in the not to distant future.

 

Array rebuilt and all 8 drives have been installed.

On 11/12/2017 at 4:35 PM, csmccarron said:

 

The drives were not free or cheap.  I also do not mind working with lots of drives which is why I went ahead an bought the hot swap bays to make it easier to replace a drive when it fails.  Currently the way they are mounted makes it a royal pain in the butt. 

 

I personally prefer lots of drives because it provides faster access to my movies and music at the expense of a little longer write time.  While I agree it would be easier to use fewer larger drives, at this point on this build I am not ready to completely redo the entire array.  I am just going to keep it healthy in its current configuration.  Maybe when I rebuild my system I will move to fewer larger drives or maybe I will just switch out to 1 TB SSD's when they come down in price.

 

What sucks is I think my PIKE card is failing and they are getting hard to find so that means a system upgrade/rebuild is in the not to distant future.

 

You are smart to incorporate drive cages to make swapping disks easier, but even more importantly, safer.

 

Have to agree with @BRiT about using larger drives though. Simpler config and 8T drive prices have recently been very economic. But ultimately depends on your array growth rate, and certainly nothing wrong with using the 1T drives. It does reduce the amount of potential data loss should multiple drives fail simultaneously. And the cost of parity can be substantially less with a smaller parity drive. If you said you were hard wiring each drive I'd be more concerned with the higher drive count, because with each drive you add you increase the chance of a cabling issue (the greatest risk to unRaid servers), but with hot-swap style cages, after initial burn in, those risks are quite minimal.

 

Enjoy your array! 

My original intent was to use 4 - 8 1TB SSD's but the price of them went in the wrong direction.  

 

Thanks for the feedback.