January 10, 200917 yr I just completed recovery from a nasty problem. I hope maybe my experience might be of value to others. About a month ago I updated 2 drives in my array due to two apparent disk failures. One was pretty old, but the other one was a relatively new 500G Seagate. I had to rebuild each disk – on two successive days – and then all appeared fine. I did a clean parity check. All good. On this month's parity check, my server crashed. Upon rebooting I found one drive didn't recognize, so I replaced its cable and booted again. unRAID recognized the drives, but when I started the array (and it tried to run a parity check) I got nasties in the syslog (ata6 = disk11): Jan 8 16:16:26 Tower kernel: ata6.00: exception Emask 0x0 SAct 0xfff SErr 0x0 action 0x6 Jan 8 16:16:26 Tower kernel: ata6.00: edma_err_cause=00000084 pp_flags=00000003, dev error, EDMA self-disable Jan 8 16:16:26 Tower kernel: ata6.00: cmd 61/08:00:bf:00:00/00:00:00:00:00/40 tag 0 ncq 4096 out Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:08:3f:00:6c/00:00:48:00:00/40 tag 1 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/00:10:1f:b7:00/04:00:00:00:00/40 tag 2 ncq 524288 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/00:18:1f:bb:00/04:00:00:00:00/40 tag 3 ncq 524288 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/00:20:1f:bf:00/01:00:00:00:00/40 tag 4 ncq 131072 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:28:4f:66:6d/00:00:48:00:00/40 tag 5 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:30:67:00:5c/00:00:62:00:00/40 tag 6 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:38:37:86:6d/00:00:48:00:00/40 tag 7 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:00:3a:86:6d/00:00:48:00:00/40 Emask 0x409 (media error) <F> Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:40:97:a5:6d/00:00:48:00:00/40 tag 8 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:48:7f:c5:6d/00:00:48:00:00/40 tag 9 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:50:df:e4:6d/00:00:48:00:00/40 tag 10 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6.00: cmd 60/08:58:c7:04:6e/00:00:48:00:00/40 tag 11 ncq 4096 in Jan 8 16:16:26 Tower kernel: res 41/40:24:1f:bf:00/40:00:00:00:00/40 Emask 0x9 (media error) Jan 8 16:16:26 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:26 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:26 Tower kernel: ata6: hard resetting link Jan 8 16:16:26 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 8 16:16:26 Tower kernel: ata6.00: configured for UDMA/133 Jan 8 16:16:26 Tower kernel: ata6: EH complete Jan 8 16:16:26 Tower kernel: sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors (1000205 MB) Jan 8 16:16:26 Tower kernel: sd 6:0:0:0: [sdg] Write Protect is off Jan 8 16:16:26 Tower kernel: sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00 Jan 8 16:16:26 Tower kernel: sd 6:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 8 16:16:30 Tower kernel: ata6.00: exception Emask 0x0 SAct 0x7ff SErr 0x0 action 0x6 Jan 8 16:16:30 Tower kernel: ata6.00: edma_err_cause=00000084 pp_flags=00000003, dev error, EDMA self-disable Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:00:c7:04:6e/00:00:48:00:00/40 tag 0 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 61/08:08:df:e4:6d/00:00:48:00:00/40 tag 1 ncq 4096 out Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:10:7f:c5:6d/00:00:48:00:00/40 tag 2 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:18:97:a5:6d/00:00:48:00:00/40 tag 3 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:20:37:86:6d/00:00:48:00:00/40 tag 4 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:00:3a:86:6d/00:00:48:00:00/40 Emask 0x409 (media error) <F> Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:28:67:00:5c/00:00:62:00:00/40 tag 5 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:30:4f:66:6d/00:00:48:00:00/40 tag 6 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/00:38:1f:c4:00/01:00:00:00:00/40 tag 7 ncq 131072 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/00:40:1f:c0:00/04:00:00:00:00/40 tag 8 ncq 524288 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/00:48:1f:c5:00/04:00:00:00:00/40 tag 9 ncq 524288 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6.00: cmd 60/08:50:3f:00:6c/00:00:48:00:00/40 tag 10 ncq 4096 in Jan 8 16:16:30 Tower kernel: res 41/40:0c:df:e4:6d/40:00:48:00:00/40 Emask 0x9 (media error) Jan 8 16:16:30 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:30 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:30 Tower kernel: ata6: hard resetting link Jan 8 16:16:30 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 8 16:16:30 Tower kernel: ata6.00: configured for UDMA/133 Jan 8 16:16:30 Tower kernel: ata6: EH complete Jan 8 16:16:30 Tower kernel: sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors (1000205 MB) Jan 8 16:16:30 Tower kernel: sd 6:0:0:0: [sdg] Write Protect is off Jan 8 16:16:30 Tower kernel: sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00 Jan 8 16:16:30 Tower kernel: sd 6:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 8 16:16:34 Tower kernel: ata6.00: exception Emask 0x0 SAct 0x40 SErr 0x0 action 0x6 Jan 8 16:16:34 Tower kernel: ata6.00: edma_err_cause=00000084 pp_flags=00000003, dev error, EDMA self-disable Jan 8 16:16:34 Tower kernel: ata6.00: cmd 60/08:30:37:86:6d/00:00:48:00:00/40 tag 6 ncq 4096 in Jan 8 16:16:34 Tower kernel: res 41/40:00:3a:86:6d/00:00:48:00:00/40 Emask 0x409 (media error) <F> Jan 8 16:16:34 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:34 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:34 Tower kernel: ata6: hard resetting link Jan 8 16:16:34 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 8 16:16:34 Tower kernel: ata6.00: configured for UDMA/133 Jan 8 16:16:34 Tower kernel: ata6: EH complete Jan 8 16:16:37 Tower kernel: ata6.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 Jan 8 16:16:37 Tower kernel: ata6.00: edma_err_cause=00000084 pp_flags=00000003, dev error, EDMA self-disable Jan 8 16:16:37 Tower kernel: ata6.00: cmd 60/08:00:37:86:6d/00:00:48:00:00/40 tag 0 ncq 4096 in Jan 8 16:16:37 Tower kernel: res 41/40:00:3a:86:6d/00:00:48:00:00/40 Emask 0x409 (media error) <F> Jan 8 16:16:37 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:37 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:37 Tower kernel: ata6: hard resetting link Jan 8 16:16:38 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 8 16:16:38 Tower kernel: ata6.00: configured for UDMA/133 Jan 8 16:16:38 Tower kernel: ata6: EH complete Jan 8 16:16:38 Tower kernel: sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors (1000205 MB) Jan 8 16:16:41 Tower kernel: ata6.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 Jan 8 16:16:41 Tower kernel: ata6.00: edma_err_cause=00000084 pp_flags=00000003, dev error, EDMA self-disable Jan 8 16:16:41 Tower kernel: ata6.00: cmd 60/08:00:37:86:6d/00:00:48:00:00/40 tag 0 ncq 4096 in Jan 8 16:16:41 Tower kernel: res 41/40:00:3a:86:6d/00:00:48:00:00/40 Emask 0x409 (media error) <F> Jan 8 16:16:41 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:41 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:41 Tower kernel: ata6: hard resetting link Jan 8 16:16:42 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 8 16:16:42 Tower kernel: ata6.00: configured for UDMA/133 Jan 8 16:16:42 Tower kernel: ata6: EH complete Jan 8 16:16:42 Tower kernel: sd 6:0:0:0: [sdg] Write Protect is off Jan 8 16:16:42 Tower kernel: sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00 Jan 8 16:16:45 Tower kernel: ata6.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 Jan 8 16:16:45 Tower kernel: ata6.00: edma_err_cause=00000084 pp_flags=00000003, dev error, EDMA self-disable Jan 8 16:16:45 Tower kernel: ata6.00: cmd 60/08:00:37:86:6d/00:00:48:00:00/40 tag 0 ncq 4096 in Jan 8 16:16:45 Tower kernel: res 41/40:00:3a:86:6d/00:00:48:00:00/40 Emask 0x409 (media error) <F> Jan 8 16:16:45 Tower kernel: ata6.00: status: { DRDY ERR } Jan 8 16:16:45 Tower kernel: ata6.00: error: { UNC } Jan 8 16:16:45 Tower kernel: ata6: hard resetting link Jan 8 16:16:46 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jan 8 16:16:46 Tower kernel: ata6.00: configured for UDMA/133 Jan 8 16:16:46 Tower kernel: sd 6:0:0:0: [sdg] Result: hostbyte=0x00 driverbyte=0x08 Jan 8 16:16:46 Tower kernel: sd 6:0:0:0: [sdg] Sense Key : 0x3 [current] [descriptor] Jan 8 16:16:46 Tower kernel: Descriptor sense data with sense descriptors (in hex): Jan 8 16:16:46 Tower kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 Jan 8 16:16:46 Tower kernel: 48 6d 86 3a Jan 8 16:16:46 Tower kernel: sd 6:0:0:0: [sdg] ASC=0x11 ASCQ=0x4 Jan 8 16:16:46 Tower kernel: end_request: I/O error, dev sdg, sector 1215137338 Jan 8 16:16:46 Tower kernel: ata6: EH complete Jan 8 16:16:46 Tower kernel: md: disk11 read error Jan 8 16:16:46 Tower kernel: handle_stripe read error: 1215137272/9, count: 1 A bunch of drives were stuck mounting, and there was some disk I/O activity that I didn't understand. (More on this at the end of this post). I tried to run WeeboTech’s powerdown script but could not get it to work. After 5 minutes of trying, I finally powered down the server using the big red switch method. With a rats nest of power splitters and molex converters, the inside of my box looked pretty gnarly. And with Joe L.'s recent power problems decided I would tackle that first. I created a custom “power rail” which eliminated countless splitters and adapters. (Note that 10 drives on one molex is NOT within spec, but it is working fine and unless someone scares the heck out of my based on this post, I'll likely leave it this way.) I setup the computer on a bench, hooked it back up, and booted. STILL having BIOS drive recognition issues. I tried replacing cables and then a different drive would stop being recognized. Each problem was 100% reproducible until I went in and changed something, and then the new problem was 100% reproducible. (Intermittence might have pointed to PSU problems). Crap. So I disconnected everything, removed my add-on controllers, and tested each drive with a known good port / cable, and then each port with a known good drive. Everything recognized. All my ports were good in isolation. Then I tried connecting all the motherboard drives. All good. Then I did the same basic thing with each of my 2 add-on controllers, and found that one of them had a port that didn't recognize. I grabbed a new cable and installed it, and it recognized (one bad cable). But then one of the drives on the MOTHERBOARD ports stopped recognizing. I was replacing that cable and noticed a bit of jiggling within the MB SATA connector. After a few minutes of testing, I found that if either the old or new cable jiggled to the left by just a tiny amount (1/32”) I'd lose connection. I have one locking cable from a gigabyte MB and installed it in that connector. Bingo – tight solid connection that allowed the drive to be recognized consistently. I put the thing back together and booted on the bench. unRAID recognized all the drives (just took a few changes on the device page to get back to a valid configuration). I started the array and got no syslog errors, but a bunch of drives were still stuck mounting (similar to screenshot above). It took me a minute to realize that it was ResiserFS replaying its journaled transactions (I wish unRaid would give ReiserFS a chance to finish this BEFORE beginning the parity check process!). Jan 8 17:00:24 Tower kernel: ReiserFS: md2: replayed 2 transactions in 1 seconds Jan 8 17:00:24 Tower kernel: ReiserFS: md8: replayed 2 transactions in 0 seconds Jan 8 17:00:24 Tower kernel: ReiserFS: md13: replayed 2 transactions in 0 seconds Jan 8 17:00:24 Tower kernel: ReiserFS: md5: replayed 2 transactions in 0 seconds Jan 8 17:00:24 Tower kernel: ReiserFS: md7: replayed 2 transactions in 0 seconds Jan 8 17:00:24 Tower kernel: ReiserFS: md1: replayed 2 transactions in 0 seconds Jan 8 17:00:24 Tower kernel: ReiserFS: md12: replayed 2 transactions in 0 seconds Jan 8 17:00:25 Tower kernel: ReiserFS: md6: replayed 2 transactions in 1 seconds ... Jan 8 17:11:39 Tower kernel: ReiserFS: md3: replayed 319 transactions in 675 seconds ... Jan 8 17:12:26 Tower kernel: ReiserFS: md14: replayed 317 transactions in 722 seconds ... Jan 8 17:13:24 Tower kernel: ReiserFS: md4: replayed 354 transactions in 780 seconds ... Jan 8 17:13:39 Tower kernel: ReiserFS: md9: replayed 398 transactions in 795 seconds ... Jan 8 17:13:45 Tower kernel: ReiserFS: md11: replayed 340 transactions in 801 seconds Almost 15 MINUTES to complete and mount. (This is what was going on during that first boot described earlier in the post. I probably should have let that finish before powering off). I wound up with 33 sync errors. As far as I can tell, 26 of the errors were in the data area of the drive in a few different sections (7 happened almost immediately so were in the “housekeeping” area). I know that ReiserFS is supposed to be good at recovering after a power failure, so am hopeful that there was no data damage. But really don't have any way to know for sure. I decided to buy a bunch of locking SATA connectors to (hopefully) prevent future problems. At least daily, it seems, someone asks a question about nasty errors in their syslog asking if their disk has failed. Frequently it is impossible to figure out where they are coming from by looking at a syslog – but generally if the driver's smart report is clean, the problem is related to cabling or backplane. (The other likely culprit is an unsupported motherboard). Sometimes getting to the bottom of a problem can be tough. It takes patience. My approach is usually to first try a few likely things to find a quick fix (e.g., change a cable, try a different port), but if that doesn't solve it to try to get the computer back to a point of working, and then incrementally add stuff until it stops working. This works better for me than starting for a non-working system, and removing stuff a little at a time to try to get to a working system. BTW, those 2 drives that appeared to have failed a month ago. They are both fine. In fact I replaced my last IDE drive with one of them as the cache disk. Now maybe I'll actually be able to watch a movie from my server!
Archived
This topic is now archived and is closed to further replies.