I'm hoping this is a bug, but I am running the latest version of unraid pro 6.3.5...had two drives drop from the array (slot 1=4TB and slot 3=3TB) because of a controller burp (5TB, dual parity array, running for months), stopped array, unassigned both slots, started array, stopped array, assigned same drives to same slots, started array...and it showed it was going to rebuild 4TB (immediately stopped array rebuild)! Right now, I'm running with both drives emulated...What the **** is happening? Can someone of extreme authority tell me that this is just an unraid bug saying 4TB because drive one is 4TB and not 5TB like both parity drives?

Edited August 28, 20178 yr by bsim

Post diagnostics.

unraid-diagnostics-20170827-2250.zip

I’ve never tried to rebuild 2 disks simultaneously, but when rebuilding a single disk the rebuild is reported as the size of the disk being rebuilt - not the size of the parity disk (if they are different).   It therefore sounds reasonable that when rebuilding 2 disks that the size of the larger one is used (as it will take the longest) as they are rebuilt in parallel.

1 hour ago, itimpi said:

I’ve never tried to rebuild 2 disks simultaneously, but when rebuilding a single disk the rebuild is reported as the size of the disk being rebuilt - not the size of the parity disk (if they are different).   It therefore sounds reasonable that when rebuilding 2 disks that the size of the larger one is used (as it will take the longest) as they are rebuilt in parallel.

 

This is correct. A rebuild takes the size of the disk being rebuilt, or in case of two disks (dual parity) the size of the largest disk.

Thanks...just freaked me out a bit with total size showing less than the actual total size of the entire array!

 

So, I started the rebuild, looked like it was going great...until I noticed one drive was building fine, then the second drive seemed to be showing an issue of a few writes too many. I stopped the rebuild, pulled out the drive, ran a smart check and a short smart check on it with success for both...Any idea why this would happen? Is this a possible cabling error that would show up after several months of working fine? The controller uses SFF-8087's to put multiple drives on one port, so I would probably lean away from the controller having issues. I am preclearing a drive just in case right now. Should this be my next direction? Does this diagnostics tell why?

 

 

unraid-diagnostics-20170829-1338.zip

Disk3 dropped offline, could be cable/backplane issue or a disk problem, if SMART is fine swap cables with another disk and try again.

When i pulled the smart stats from drive 3 on another system, it showed some bad numbers, but still said it passed, I tried Crystal (attached and it showed at least some warning)...The drive did make a strange noise, so I'm leaning towards a creepy bad drive. I have a fresh 5TB drive that's been cleared that i'm going to swap in and do a rebuild.

I would lean away from the controller because of being a high end controller and having cables that should show some sort of symptoms on other drives on that port.

Initially, the drop offline was caused during a parity check of a system that hasn't been touched for months through multiple parity checks. The controller took two drives offline for some reason, but there seemed to be a setting in the controller for keeping the drives online even through minor errors (read errors, no write errors)...I'll see if the rebuild works.

 

EDIT: I also just realized that the new drive will be in a different backplane/controller port anyway because I used the server to do the preclear during the attempted rebuilds...I'll have to wait to put the next 5TB drive in on the same port to see if the problem reoccurs. (It's a Supermicro 24 bay with an Areco 1280ML 24 Port 2GB)

 

Edited August 30, 20178 yr by bsim

WTF!....ok, so I installed the new 5TB pre-cleared drive, and passively started watching the rebuild to see if the new drive or the old drive go to pluto...I then witnessed the "writes" column of drive 1 go from about the equal size of the new 5TB drive, to the pluto (18 x 10 to the 18th!) instantly!

The controller is dumping the drive when this error occurs, so I would guess that it isn't an issue with a driver or unraid itself. The two drives are on different port of the controller card (and a different set of SFF-8087's). What's fishy is that the two drives are jumping to almost the same massive number + the number of actual writes it had when it went to pluto...This time, the areca isn't even reporting any read (or write) errors and just saying that the device failed (attached)!

 

At this point, I can only guess is that

1. Two non identical drives went bad in the same way after several months of operating/parity checks.

2. Two unrelated but specific ports on the controller board went bad at the same time,

3. Two unrelated but specific cables went bad on two separate SFF-8087's.

I can't really rule out a backplane issue, but I would have seen more issues from those slots for having it operating for multiple years 24x7 through an entire server gut/upgrade.

 

I'm preclearing the second drive now, will try swapping in the second drive for disk 1 to see if anything is nailed down.

 

Anybody have any thoughts? Anyone have any areca or supermicro backplane experience that would rival this really fluky problem? Why the gigantic number of writes reported to unraid? Why no errors in Areca the second time?

 

I do see that the Areca has support for NCQ, and I remember a long while back (unraid 5ish) that there were problems using NCQ drives and non-NCQ drives with unraid attempting the use NCQ....could older drives be causing the controller to go funky? I can only guess this because disk 1 and disk 3 are two of my older drives. (even though I have a few 1TB drives without any issues)

 

UPDATE: After pulling disk 1, and attempting to pull smart reports (from a different workstation), it is confirmed that the drive is dead (mechanical repetition and no drive detection).

 

So now I'm leaning towards two drives went bad mechanically in one month and appeared with a really odd controller report to unraid (huge writes).

 

I will probably wait for the second 5TB drive to preclear before rebuilding in the failed one's place (rebuild longer time than preclear + rebuild only once), but if I wanted to cut the risk of loosing even another in the preclear time, could I just keep one of the drives emulated while rebuilding on the previously inserted 5TB drive in waiting? I would essentially be rebuilding one of the two missing drives allowing me to at least recover from even one more additional failed drive during the second rebuild. If it is possible, would I just leave the bad drive assigned, and rebuild with the newly assigned 5TB?

 

 

unraid-diagnostics-20170829-2256 (SECOND ISSUE).zip

Edited August 30, 20178 yr by bsim

3 hours ago, bsim said:

Why the gigantic number of writes reported to unraid?

 

These gigantic write numbers are normal when a device drops offline, either when it's a bad device or a controller dropping disks, so these are normal in theses situations, you just need to find out why they're dropping.

Looks like I can confirm that I had two drives go bad mechanically in a month and a half! Thank goodness I'm double parity! Thanks for your help guys. I'm back up and running.