Hi guys,

 

I've been fighting with my unRAID array. Still running 5.0-rc12a, because the upgrade notes say that the array must be clean before I upgrade. A couple of weeks ago one of my harddrives failed, so I replaced it with a new Seagate NAS HDD 4TB. Preclear ran through fine (3 iterations). Also the Western Digital "Drive Fitness Test" reported the drive to be in good shape. But I keep having trouble getting the array repaired properly. I've tried with the Seagate connected to either a mainboard SATA port, or to my SAS controller. In the first case when trying to repair the array, the whole server freezes (tried 2-3 times). No ping, no telnet, nothing. In the second case until today the repair failed every time, with the Seagate harddisk ending up with a "red ball". The problem is unlikely to be the SATA controller, since problems occurred with two different controllers, and it's also unlikely to be the SATA cable, because it were also 2 different ones (one was even a SAS breakout cable).

 

Today finally the array repair succeeded, after another (successful) preclear. But looking at the log, there are problems reported that make me quite uneasy. Here's the log and the preclear report:

 

http://madshi.net/unraidLog.txt

http://madshi.net/preclear.txt

 

Here's an extract from the log, which shows what's going on:

 

Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command cfe679c0 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d297b180 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f75013c0 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command cfe67840 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d2979780 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d297ba80 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f7501b40 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f74d9840 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d29799c0 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d297b840 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f7501000 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f74d9a80 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d2979480 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d297b300 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f7501240 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f74d93c0 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d2979300 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d297b6c0 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f7501180 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command f74d9240 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d2979540 timed out
Oct 21 17:38:43 Tower kernel: sd 0:0:3:0: [sdj] command d297b000 timed out
Oct 21 17:38:53 Tower kernel: sd 0:0:3:0: [sdj] command f7501e40 timed out
Oct 21 17:38:53 Tower kernel: sas: Enter sas_scsi_recover_host busy: 23 failed: 23
Oct 21 17:38:53 Tower kernel: sas: trying to find task 0xc895ef00
Oct 21 17:38:53 Tower kernel: sas: sas_scsi_find_task: aborting task 0xc895ef00
Oct 21 17:38:53 Tower kernel: sas: sas_scsi_find_task: task 0xc895ef00 is aborted
Oct 21 17:38:53 Tower kernel: sas: sas_eh_handle_sas_errors: task 0xc895ef00 is aborted
Oct 21 17:38:53 Tower kernel: sas: trying to find task [...]
Oct 21 17:38:53 Tower kernel: sas: [...]
Oct 21 17:38:53 Tower kernel: sas: ata10: end_device-0:3: cmd error handler
Oct 21 17:38:53 Tower kernel: sas: ata7: end_device-0:0: dev error handler
Oct 21 17:38:53 Tower kernel: sas: ata8: end_device-0:1: dev error handler
Oct 21 17:38:53 Tower kernel: sas: ata9: end_device-0:2: dev error handler
Oct 21 17:38:53 Tower kernel: sas: ata10: end_device-0:3: dev error handler
Oct 21 17:38:53 Tower kernel: ata10.00: exception Emask 0x0 SAct 0xfffffd SErr 0x0 action 0x6 frozen
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:dc:15/02:00:03:00:00/40 tag 0 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:08:f8:58:8f/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:b0:15/02:00:03:00:00/40 tag 2 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:b2:15/02:00:03:00:00/40 tag 3 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:b4:15/02:00:03:00:00/40 tag 4 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: sas: ata11: end_device-0:4: dev error handler
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:b6:15/02:00:03:00:00/40 tag 5 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: sas: ata12: end_device-0:5: dev error handler
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: sas: ata13: end_device-0:6: dev error handler
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:b8:15/02:00:03:00:00/40 tag 6 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:ba:15/02:00:03:00:00/40 tag 7 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:bc:15/02:00:03:00:00/40 tag 8 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:be:15/02:00:03:00:00/40 tag 9 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:c0:15/02:00:03:00:00/40 tag 10 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:c2:15/02:00:03:00:00/40 tag 11 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:c4:15/02:00:03:00:00/40 tag 12 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:c6:15/02:00:03:00:00/40 tag 13 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:c8:15/02:00:03:00:00/40 tag 14 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:ca:15/02:00:03:00:00/40 tag 15 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:cc:15/02:00:03:00:00/40 tag 16 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:ce:15/02:00:03:00:00/40 tag 17 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:d0:15/02:00:03:00:00/40 tag 18 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:d2:15/02:00:03:00:00/40 tag 19 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:d4:15/02:00:03:00:00/40 tag 20 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:d6:15/02:00:03:00:00/40 tag 21 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:d8:15/02:00:03:00:00/40 tag 22 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10.00: failed command: WRITE FPDMA QUEUED
Oct 21 17:38:53 Tower kernel: ata10.00: cmd 61/00:00:f8:da:15/02:00:03:00:00/40 tag 23 ncq 262144 out
Oct 21 17:38:53 Tower kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Oct 21 17:38:53 Tower kernel: ata10.00: status: { DRDY }
Oct 21 17:38:53 Tower kernel: ata10: hard resetting link
Oct 21 17:38:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1521:mvs_I_T_nexus_reset for device[3]:rc= 0
Oct 21 17:38:55 Tower kernel: sas: sas_ata_task_done: SAS error 8a
Oct 21 17:38:55 Tower kernel: sas: sas_ata_task_done: SAS error 8a
Oct 21 17:38:55 Tower kernel: ata10.00: both IDENTIFYs aborted, assuming NODEV
Oct 21 17:38:55 Tower kernel: ata10.00: revalidation failed (errno=-2)
Oct 21 17:38:55 Tower kernel: mvsas 0000:01:00.0: Phy4 : No sig fis
Oct 21 17:38:59 Tower kernel: sas: sas_form_port: phy4 belongs to port3 already(1)!
Oct 21 17:39:00 Tower kernel: ata10: hard resetting link
Oct 21 17:39:00 Tower kernel: ata10.00: configured for UDMA/133
Oct 21 17:39:00 Tower kernel: ata10.00: device reported invalid CHS sector 0
Oct 21 17:39:00 Tower last message repeated 21 times
Oct 21 17:39:00 Tower kernel: ata10: EH complete
Oct 21 17:39:00 Tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0

 

The same problem repeats several times, every couple of minutes/hours.

 

How can the same drive breeze through multiple preclears and be reported to be ok by the Western Digital "Drive Fitness Test", but then during array repair bad things happen all the time?

 

I should also mention that I've had weird troubles with multiple different Toshiba 3TB harddisks, too. They seem to be all ok, but once in a while one of them ends up being reported with a "red ball" out of the blue. But then I can just force it back in, do a new parity check and everything works fine, and preclear and "Drive Fitness Test" find nothing bad to report, either. I'm not sure if this is a >= 2TB thing, because I've never ever had anything like this happening to any of those Samsung 1.5TB drives. It's only the Toshiba 3TB and Seagate 4TB drives that suffer from this problem. However, I've already two other Seagate 4TB drives in the array now which haven't shown any problems so far (but I hadn't checked the log when I had added them to the array). So this is all not very conclusive.

 

Since the array is now finally clean again (despite all those timeout problems reported in the log), I'll hurry to update to 5.0 final now. But I'm still left with a rather uneasy feeling. So any comments / suggestions are welcome - thanks!!

If the array is "clean" ==> i.e. no red-balled drives; and a clean (zero sync errors) parity check; then there's no reason not to go ahead and do the v5.0 upgrade.    That's not going to impact your parity, so the array will still be protected.

 

I've seen several cases where intermittent errors like this were due to the firmware on add-in controllers.    You didn't say which controller you're using; but whatever it is, check for any firmware updates that you may not have applied.    I know, for example, that a recent update to the Supermicro SAS2LP-MV8 controller made a major difference in its stability with 4TB drives.

 

Thanks for your reply.

 

Since the array repair seems to have succeeded, I've now gone ahead and upgraded to 5.0 final. Now I'm running another parity check. I'm at 10% now and the parity check already reports more than 3000 sync errors. I guess I'll have to consider the "repaired" disk to be broken, damaged and corrupted now...  :'( 

 

Fortunately I was clever enough to backup the contents of that disk to an external drive before doing the repair, so I hope to be able to manually repair it myself. But this is still a very frustrating experience.

 

I have a Supermicro AOC-SASLP-MV8 with the latest available firmware (3.1.0.21 from 2010). But the problem was not limited to the Supermicro. As I already mentioned, the problem also occurred (even worse) with the internal mainboard SATA controller. So how can it be the fault of the controller, if two different controllers both show problems?

Assuming the array was "clean" -- i.e. you ran a parity check and had zero sync errors before you did anything else [As I noted above you should do], then your disk should be fine.

 

Sync errors you're seeing now are apparently due to read issues with your disks.    They will, however, result in a corrupted parity drive (assuming you're doing a correcting check).

 

As you noted, there are no firmware updates for the AOC-SASLP-MV8 on SuperMicro's site, so that's not an issue [There's a March update for the newer SAS2LP, but not for your card].

 

Given your experience, I'd be suspicious of two areas:  power, and cabling.  It sounds like either you're having sporadic power issues (perhaps due to a failing power bus or a marginal PSU);  or cables that are loose or poorly seated -- or drives in a hot-swap cage that aren't firmly seated.

 

Well, the array repair was still done with 5.0 rc16 and ran through "successfully" (but with all the errors in the syslog, see first post). So I thought the array was clean, and upgraded to 5.0 final (without doing another parity check after the array repair). Now I'm running the parity check with 5.0 final and it's now at > 8000 parity errors at 27%. There is no indication that there are any read errors right now. The syslog is totally clean atm. So I believe the parity errors come from the array repair with 5.0 rc16 not working correctly. Which means that the array repair has probably failed, and as a result probably the repaired disk now has corrupted data on it.

 

Cabling sounds unlikely because (as described in the first post) similar problems with the same seagate 4tb disk occurred with 2 totally different controllers and 2 totally different cables. Power is a more likely cause, but the power supply is very capable (it's the well respected Corsair 650W). The hot-swap cage could be an issue, but I've also tested the same seagate 4tb disk in two different hot-swap cages (same model, though), powered by different power and SATA cables, and the problem occurred in both. And again: Any of these problems never occur with the Samsung 1.5TB disks. So I find it all very inconclusive...  :'(

Clearly it was a mistake to not do a parity check after your repair but before proceeding ... but nothing to do about that now.    The reality is that upgrading to v5.0 is unlikely to have made any difference anyway.    You'll simply need to restore the data for the failed disk from your backups.

 

All power supplies can fail, no matter how well respected they are ... and it I'd certainly put power at the top of the list of likely suspects here.    I'd try a new power supply -- depending on how many disks you have, you may also want to bump up the wattage.

 

Ok, I'll try to replace the power supply. FWIW, I've 11 hard disks currently. 650W should be plenty for that, no?