Hi -

 

After having trouble with beta12 dropping disks I eventually got things stabilized by moving all the disks onto the SAS2LP.  Since moving to beta14b, though, I've had two instances where unRAID has dropped a disk.  The first time it was a 3TB WD Red data disk, this time it was the  6TB WD Red parity.

 

Log attached, trouble starts on line 1724 in the syslog.  The smartctl report is also attached, though you can see it isn't talking to the disk.

 

I haven't rebooted the server since this failure, but I expect connectivity to the disk to be restored after a reboot - it has worked that way on all prior occasions.  I then have to do a new config to get the array back up and running.

 

Any ideas?  I'd rather not go back to beta12...

 

tower.zip.zip

Do you have a storage controller other than the SAS2LP to test with?  That SAS2LP controller has a Marvell 9480 chip on it, and I have seen a lot of troubles with Marvell sata/sas chips in Linux, especially with disks over 2TB, exhibiting the problems you are having.  It seems to be triggered by heavy utilization, such as when the mover is active or running a parity check.  Looks like you were running the mover right before the disks started to drop out.

FWIW, I've got a SAS2LP with 3TB drives on it, and have only had an issue with the same "dropping" with one particular WD drive which I RMA'ed and was replaced (even though the smart attributes on it showed zero problems) (running 14b)

 

That being said, the errors that start all of this:

ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Apr 11 16:29:48 Tower kernel: ata9.00: failed command: READ DMA
Apr 11 16:29:48 Tower kernel: ata9.00: cmd c8/00:00:f8:db:1e/00:00:00:00:00/e0 tag 5 dma 131072 in
Apr 11 16:29:48 Tower kernel:         res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Apr 11 16:29:48 Tower kernel: ata9.00: status: { DRDY }

 

is usually consistent with a bad / loose cable and/or power issue.

 

In my case, reseating the drive in its bay would solve it for a while and then the issues would hapen again.  I finally gave up and just rma'ed the drive.  Haven't looked back since

Do you have a storage controller other than the SAS2LP to test with?  That SAS2LP controller has a Marvell 9480 chip on it, and I have seen a lot of troubles with Marvell sata/sas chips in Linux, especially with disks over 2TB, exhibiting the problems you are having.  It seems to be triggered by heavy utilization, such as when the mover is active or running a parity check.  Looks like you were running the mover right before the disks started to drop out.

 

That's the only controller I have right now.  The issue was triggered by a manual Sickbeard post-processing, so yes - the disks were going from a sleep state to heavy activity.  I notice that you are using LSI based controllers on an older setup... makes me wonder if that's an alternative I should look at.

FWIW, I've got a SAS2LP with 3TB drives on it, and have only had an issue with the same "dropping" with one particular WD drive which I RMA'ed and was replaced (even though the smart attributes on it showed zero problems) (running 14b)

 

That being said, the errors that start all of this:

ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Apr 11 16:29:48 Tower kernel: ata9.00: failed command: READ DMA
Apr 11 16:29:48 Tower kernel: ata9.00: cmd c8/00:00:f8:db:1e/00:00:00:00:00/e0 tag 5 dma 131072 in
Apr 11 16:29:48 Tower kernel:         res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Apr 11 16:29:48 Tower kernel: ata9.00: status: { DRDY }

 

is usually consistent with a bad / loose cable and/or power issue.

 

In my case, reseating the drive in its bay would solve it for a while and then the issues would hapen again.  I finally gave up and just rma'ed the drive.  Haven't looked back since

 

I wish it were consistent, but it's been spread evenly across the drives - they've all dropped at one point or another now under beta12 or beta14b.  The cables are locking Startech breakouts.  I guess they could be part of the issue but I don't think so - based on prior experience I know this issue will clear by rebooting, no re-seating necessary.

 

Could the power supply be an issue?

FWIW, I've got a SAS2LP with 3TB drives on it, and have only had an issue with the same "dropping" with one particular WD drive which I RMA'ed and was replaced (even though the smart attributes on it showed zero problems) (running 14b)

 

That being said, the errors that start all of this:

ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Apr 11 16:29:48 Tower kernel: ata9.00: failed command: READ DMA
Apr 11 16:29:48 Tower kernel: ata9.00: cmd c8/00:00:f8:db:1e/00:00:00:00:00/e0 tag 5 dma 131072 in
Apr 11 16:29:48 Tower kernel:         res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Apr 11 16:29:48 Tower kernel: ata9.00: status: { DRDY }

 

is usually consistent with a bad / loose cable and/or power issue.

 

In my case, reseating the drive in its bay would solve it for a while and then the issues would hapen again.  I finally gave up and just rma'ed the drive.  Haven't looked back since

 

I wish it were consistent, but it's been spread evenly across the drives - they've all dropped at one point or another now under beta12 or beta14b.  The cables are locking Startech breakouts.  I guess they could be part of the issue but I don't think so - based on prior experience I know this issue will clear by rebooting, no re-seating necessary.

 

Could the power supply be an issue?

Could be quite probable.

 

You've got 6 hard drives, a fairly high wattage CPU, motherboard, and whatever fans etc you've got running through a dual rail supply.

 

Since the supply is ATV12V v2 compliant each rail is going to use at most 18A (I'm couldn't easily find a picture of the side of the power supply)

 

6 Drives at say 2A spin up current = 12A plus fans + motherboard + losses through splitters, etc and you're not leaving yourself much leeway (the cpu runs off the other rail)

 

One way of looking at it is that the problems happen during heavy I/O.  The other way of looking at it is that the problems happen during high current situations (ie: all drives spin up for mover, etc)

Check for BIOS and firmware updates. Try a single rail PSU.

Definitely all of the above.  First, make sure you don't have a power problem.  Add up the Amps and compare that to your PS.  I'm not sure if it's consistent across HDs, but mine all list the +5V and +12V Amps on the labels.  Then check firmware and BIOS - they should be updated if possible anyway.  If those efforts don't shake out, test with a different SATA/SAS controller.

 

I do like the LSI controllers best, mostly just as a personal preference.  And it only applies to home/desktop hardware. Office/enterprise servers are a different story.  But LSI has been super compatible with Linux and performs very well in my experience.  That being said, you can't beat direct experience, and I've seen posts from users here who have the same controller you have - Squid in particular has one and seems to like it. So I think you're more likely to have a power or firmware problem as Squid and dgaschk suggested, rather than a controller compatibility issue.

 

I also just realized I sort of mis-read your first post.  What controller were you using before you moved to the SAS2LP?

 

EDIT: It seems Marvell chips can be spotty, depending on which chip, exactly?  Not sure the details yet.

http://lime-technology.com/forum/index.php?topic=39196.0

Thanks for the help, all.

 

Firmware and BIOS are up to date.  I used the motherboard's 4 SATA slots before adding disks and the AOC-SAS2LP.  I never had any problem running under unRAID 5.05 with a similar configuration, but who knows? 

 

It seems like my next step is to look carefully at the power supply.  It's the oldest component besides the motherboard.  The Supermicro cards are widely used for unRAID, but if I continue to have problems after looking at the power supply I guess I'll have to test with different SATA controller after that.

I used to have issues with once in while a red balled drive and then I moved to a nice single rail 750 watt power supply (seasonic) and life has been better. I also moved from ReiserFS to XFS and then never got an unformatable (unmountable) drive again due to filesystem corruption. I use the SAS2LP with 6 drives and no issues. I use the breakaway cables from monoprice. Powersupply is platinum and costed 100 bucks. I figured 100 bucks to not have to worry anymore was worth it.

 

 

Thanks for the help, all.

 

Firmware and BIOS are up to date.  I used the motherboard's 4 SATA slots before adding disks and the AOC-SAS2LP.  I never had any problem running under unRAID 5.05 with a similar configuration, but who knows? 

 

It seems like my next step is to look carefully at the power supply.  It's the oldest component besides the motherboard.  The Supermicro cards are widely used for unRAID, but if I continue to have problems after looking at the power supply I guess I'll have to test with different SATA controller after that.