September 15, 201015 yr I had a drive die on me so I ordered a newer larger one to replace it. The system booted up and it took quite a while for the drives to mount. But after a while they did. Then the rebuild seemed to be moving very slowly and then come to a stop. So I rebooted the machine and the one of the drives came up in the BIOS with a SMART failure. This leaving me with 2 bad drives (since the rebuild did already start) any advice on what I can try here. In order to get the system to boot again I can disable the SMART in the bios and it will boot. But it will start the rebuild again and freeze afer a short amount of time. I do still have the initial bad drive I replace although it is a 1gb and the drive I replaced i with is a 2gb. I might be able to get the old drive to work again. It just has to be at a certain angle for it to spin. Is there any way to reinstate that drive (although it is smaller than the new one) just long enough for me to replace the dead SMART one? It doesn't look too great. Looking for any advice or options. Thanks. syslog at pastebin - http://pastebin.com/9QpNSckx It was too large to attach. Thanks everyone.
September 15, 201015 yr Hi SlaveUnit! Humm. If two drives have failed, then you will loose the data off both those drives, Sorry but no recovery here. That is the bad news. The good news is you will be able to recover the data off the remaining drives by mounting them separately and copying the data off to temporary storage. Then rebuilding the Array and copying the data back. Good Luck! Sideband Samurai
September 15, 201015 yr There are some advanced techniques that could be tried to recover your data. Suggest you power down your server and allow some dialog on the forum to determine a plan of action.
September 15, 201015 yr I just had this happen to me also. My parity drive decided to act up (not quite sure what caused it) but I got it back up and recognized and start a parity rebuild. While that was going One of my Seagate 1.5TB drives decided to make a serious clicking noise. As soon as i heard it i stopped the parity check/rebuild (whatever it was) and ordered an RMA for the drive. I think i can get the Seagate drive to be recognized long enough to to copy content onto another drive but I don't have the time until this weekend probably. Sorry to go a little off topic, i will post my own here in a second.
September 15, 201015 yr I had a drive die on me so I ordered a newer larger one to replace it. The system booted up and it took quite a while for the drives to mount. But after a while they did. Then the rebuild seemed to be moving very slowly and then come to a stop. So I rebooted the machine and the one of the drives came up in the BIOS with a SMART failure. This leaving me with 2 bad drives (since the rebuild did already start) any advice on what I can try here. In order to get the system to boot again I can disable the SMART in the bios and it will boot. But it will start the rebuild again and freeze afer a short amount of time. I do still have the initial bad drive I replace although it is a 1gb and the drive I replaced i with is a 2gb. I might be able to get the old drive to work again. It just has to be at a certain angle for it to spin. Is there any way to reinstate that drive (although it is smaller than the new one) just long enough for me to replace the dead SMART one? It doesn't look too great. Looking for any advice or options. Thanks. syslog at pastebin - http://pastebin.com/9QpNSckx It was too large to attach. Thanks everyone. Yes, you can go back to the old drive and then instruct the array on which disk to re-construct (the other failed disk) The first error I see is this: Sep 14 20:28:11 Tower kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 14 20:28:11 Tower kernel: ata4.00: edma_err_cause=10000080 pp_flags=00000001, EDMA self-disable Sep 14 20:28:11 Tower kernel: ata4.00: failed command: WRITE DMA EXT Sep 14 20:28:11 Tower kernel: ata4.00: cmd 35/00:f0:c7:1d:00/00:01:00:00:00/e0 tag 0 dma 253952 out Sep 14 20:28:11 Tower kernel: res d0/00:f0:c7:1d:00/00:01:00:00:00/e0 Emask 0x2 (HSM violation) ata4.0 is: ata4.00: ATA-8: WDC WD20EARS-00S8B1, 80.00A80, max UDMA/133 That seems to be assigned as disk5 Sep 14 20:26:54 Tower kernel: md: import disk5: [8,48] (sdd) WDC WD20EARS-00S WD-WCAVY4142983 offset: 63 size: 1953514552 According to this site: http://www3.big.or.jp/~sian/linux/DocBook/libata/ch07.html, an HSM error could be anything [color=brown]In these cases, HSM is violated and not much information regarding the error can be acquired from STATUS or ERROR register. IOW, this error can be anything - driver bug, faulty device, controller and/or cable. As HSM is violated, reset is necessary to restore known state. Reconfiguring transport for lower speed might be helpful too as transmission errors sometimes cause this kind of errors. [/color] The next errors (on another disk) are: Sep 14 20:32:33 Tower kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 14 20:32:33 Tower kernel: ata4.00: edma_err_cause=10000080 pp_flags=00000001, EDMA self-disable Sep 14 20:32:33 Tower kernel: ata4.00: failed command: WRITE DMA EXT Sep 14 20:32:33 Tower kernel: ata4.00: cmd 35/00:00:c7:0c:01/00:04:00:00:00/e0 tag 0 dma 524288 out Sep 14 20:32:33 Tower kernel: res d0/00:00:c7:0c:01/00:04:00:00:00/e0 Emask 0x2 (HSM violation) Sep 14 20:32:33 Tower kernel: ata4.00: status: { Busy } Sep 14 20:32:33 Tower kernel: ata4: hard resetting link Sep 14 20:32:33 Tower kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Sep 14 20:32:34 Tower kernel: ata4.00: configured for UDMA/133 Sep 14 20:32:34 Tower kernel: ata4: EH complete Sep 14 20:32:34 Tower kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x400000 action 0x6 frozen Sep 14 20:32:34 Tower kernel: ata4.00: edma_err_cause=04000000 pp_flags=00000001 Sep 14 20:32:34 Tower kernel: ata4: SError: { Handshk } Sep 14 20:32:34 Tower kernel: ata4.00: failed command: WRITE DMA EXT Sep 14 20:32:34 Tower kernel: ata4.00: cmd 35/00:00:c7:19:01/00:04:00:00:00/e0 tag 0 dma 524288 out Sep 14 20:32:34 Tower kernel: res d0/00:00:c7:19:01/00:04:00:00:00/e0 Emask 0x2 (HSM violation) Sep 14 20:32:34 Tower kernel: ata4.00: status: { Busy } Sep 14 20:32:34 Tower kernel: ata4: hard resetting link The next set of errors involve a different disk: Sep 14 20:29:52 Tower kernel: ata14.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 14 20:29:52 Tower kernel: ata14.01: failed command: READ DMA EXT Sep 14 20:29:52 Tower kernel: ata14.01: cmd 25/00:00:c7:43:00/00:04:00:00:00/f0 tag 0 dma 524288 in Sep 14 20:29:52 Tower kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/10 Emask 0x4 (timeout) Sep 14 20:29:52 Tower kernel: ata14.01: status: { DRDY } Sep 14 20:29:57 Tower kernel: ata14: link is slow to respond, please be patient (ready=0) Sep 14 20:30:02 Tower kernel: ata14: device not ready (errno=-16), forcing hardreset Sep 14 20:30:02 Tower kernel: ata14: soft resetting link Sep 14 20:30:03 Tower kernel: ata14.00: configured for UDMA/133 Sep 14 20:30:03 Tower kernel: ata14.01: configured for UDMA/133 Sep 14 20:30:03 Tower kernel: ata14.01: device reported invalid CHS sector 0 Sep 14 20:30:03 Tower kernel: ata14: EH complete here the device just stopped communicating with the controller. This seems to be scsi 14:0:1:0: Direct-Access ATA ST31500341AS CC1H PQ: 0 ANSI: 5 Followed eventually with all the controllers Sep 14 20:53:48 Tower kernel: ata2: failed to resume link (SControl FFFFFFFF) Sep 14 20:53:48 Tower kernel: ata1: failed to resume link (SControl FFFFFFFF) Sep 14 20:53:48 Tower kernel: ata4: failed to resume link (SControl FFFFFFFF) Sep 14 20:53:48 Tower kernel: ata3: failed to resume link (SControl FFFFFFFF) I'd actually suspect something common at this point.. a disk controller? or backplane?. but not knowing your equipment, can't say. It could even be an overloaded power supply. The basic instructions are: Stop the array power down physically connect the "old partially working drive" instead of the new drive you were trying to use to upgrade. Let's say it was assigned to slot 5 in the array. (I'm guessing it is the one you think you can replace with the old, but if I'm wrong use the correct "slot" number in the instructions that follow) Let's say you know that the disk in slot 10 is the drive with the SMART failure. It is the one you want to re-construct onto a replacement from parity and all the other data drives, including the one you must start at a specific angle to get it to spin. (you sure it is not an intermittent power or data cable to that drive? The symptoms match) Physically connect the drive you intend to use as the replacement for disk10 (replacement for the one with the SMART failure) Now, we need to force the array to think that the "failed" drive (disk10) the one you want to re-construct and that the old smaller drive in slot 5, now spinning is fine. To do that we need to force a new configuration AND then tell the array slot 10 is the one to re-construct and THEN start the array to let it perform the re-construction. To force a new configuration you need to Stop the array by pressing "Stop" on the managenent console. On the linux command line, after logging in as "root" type these three commands: cd initconfig mdcmd set invalidslot 10 Now, on the management web-interface refresh the browser. It should show disk10 as "red" and the others all green. Press "Start" to begin the re-construction of disk10 reading from all the other disks. If you are religious person, a prayer or two can't hurt. If you have any questions, ask before you proceed. You are basically forcing a disk to be marked as invalid and forcing the others to be trusted as good. Joe L.
September 15, 201015 yr Author I was wondering the same thing Joe L. All this did happen after I replaced a bad card (Rosewill 218). Before RMAing it it wouldn't get passed the scanning of the drives in the card's bios. I'm wondering if the MB is taking a dive, or maybe the new Rosewill card or the PSU?. The WDC WD20EARS-00S WD-WCAVY4142983 is the drive that is supposed to be rebuilding. It is new as of yesterday (although with HDDs these day that really means nothing). The other main one that seems to be causing issues might be my drive 15. Just a guess though. Any suggestions on what might be bad hardware wise? The PSU is a Corsair 650watt. Here is how my system is laid out, it may help somewhat: Slot 15 - disk15 ST31500341AS_9VS1R38D Motherboard SATA4 Slot 14 - disk12 ST31500341AS_9VS3W5RZ Motherboard SATA3 Slot 13 - disk11 ST31500541AS_6XW03TEQ Motherboard SATA2 Slot 12 - disk7 ST31500541AS_6XW0637J Motherboard SATA1 Slot 11 - disk13 ST3750640AS_5QD3J1PW PCIx4 Rosewill RC218 Card CON2 Slot 10 - disk5 WDC_WD10EACS-00D6B1_WD-WCAU45304446 PCIx4 Rosewill RC218 Card COM3 = DEAD RMA'D Replaced with new WD 2TB Slot 09 - disk3 ST31500341AS_9VS0G8YV PCIx4 Rosewill RC218 Card CON0 Slot 08 - disk4 ST31000340AS_9QJ1YEBR PCIx4 Rosewill RC218 Card COM1 Slot 07 - disk14 ST3750640AS_5QD46GV3 Upper PCI Silicon Image Card COM1 Slot 06 - disk10 ST3750640AS_5QD46PRL Upper PCI Silicon Image Card COM2 Slot 05 - disk6 ST3750330AS_5QK0295T Upper PCI Silicon Image Card COM4 Slot 04 - disk8 ST31500341AS_9VS03PAT Upper PCI Silicon Image Card COM3 Slot 03 - parity Hitachi_HDS72202_JK1130YAGKTRNT Lower PCI Silicon Image Card COM1 Slot 02 - disk2 ST31500341AS_9VS2MPRN Lower PCI Silicon Image Card COM2 Slot 01 - disk1 ST31500341AS_9VS0V4R5 Lower PCI Silicon Image Card COM4 Slot 00 - disk9 ST31500341AS_9VS0WVW0 Lower PCI Silicon Image Card COM3 Let me also note that while the system was trying to rebuild to the newer/bigger drive I would see/hear groups of 4-5 drives just shut off for a couple of seconds. Like they did lose power for some reason then spin back up. Edit: With the machine powered off all night I fired it up this morning and Drives 3, 4, 5 and 13 all show up as missing. This leads me to thing it's either the PSU not giving power to that cable that has those drives because it is the same line feeding all 4 of them. Or it's the Rosewill card since they are all connected to it. The syslog shows nothing of those drives existing or spinning up. So trying once more to boot up the machine, it seems like it will not even get to the POST anymore. It just keeps shutting off. That leads me to believe that the PUS can't handle the load or it is bad. After checking the power cables and moving some around I get the same result. I have opened an RMA with Corsair. I never thought a Corsair would die this quickly (less than 2 years), but I guess it happens. I would think a 650 should be plenty for this type of rig. The Corsair site even recommends that a 450W will work fine.
September 15, 201015 yr Given that ata4 and ata14 are located at different controllers, i would start with MB/Memory/controller, chance are both controllers have problem at same time is slim. As Joe L had pointed out your problem started with ata4 then eventually propagate to other disks even at different controller. From timestamp, you can tell the sequence is ata4 ---> ata14 ---> ata13 ----> ata1 ----> ata2 ---> all links at RC-218 are down. This looks to me more like issue in driver + HW especially this RC-218 controller. If you have a single-point failure on a controller what you will see is a set of disk are down that sometimes doesn't meant disks themselves have problems. and ata4 (md5) started to have problem when file system log replay is still going on which meant, i believed ,the data re-construction hasn't really started yet or just at very beginning. Sep 14 20:26:51 Tower kernel: scsi0 : sata_mv --------------> RC-218 Sep 14 20:26:51 Tower kernel: scsi1 : sata_mv Sep 14 20:26:51 Tower kernel: scsi2 : sata_mv Sep 14 20:26:51 Tower kernel: scsi3 : sata_mv Sep 14 20:26:51 Tower kernel: ata1: SATA max UDMA/133 mmio m1048576@0xfe700000 port 0xfe722000 irq 16 Sep 14 20:26:51 Tower kernel: ata2: SATA max UDMA/133 mmio m1048576@0xfe700000 port 0xfe724000 irq 16 Sep 14 20:26:51 Tower kernel: ata3: SATA max UDMA/133 mmio m1048576@0xfe700000 port 0xfe726000 irq 16 Sep 14 20:26:51 Tower kernel: ata4: SATA max UDMA/133 mmio m1048576@0xfe700000 port 0xfe728000 irq 16 Sep 14 20:26:51 Tower kernel: scsi13 : ata_piix Sep 14 20:26:51 Tower kernel: scsi14 : ata_piix Sep 14 20:26:51 Tower kernel: ata13: SATA max UDMA/133 cmd 0xa080 ctl 0xa000 bmdma 0x9800 irq 19 Sep 14 20:26:51 Tower kernel: ata14: SATA max UDMA/133 cmd 0x9c00 ctl 0x9880 bmdma 0x9808 irq 19 Sep 14 20:28:11 Tower kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 14 20:28:11 Tower kernel: ata4.00: edma_err_cause=10000080 pp_flags=00000001, EDMA self-disable Sep 14 20:28:11 Tower kernel: ata4.00: failed command: WRITE DMA EXT Sep 14 20:28:11 Tower kernel: ata4.00: cmd 35/00:f0:c7:1d:00/00:01:00:00:00/e0 tag 0 dma 253952 out Sep 14 20:28:11 Tower kernel: res d0/00:f0:c7:1d:00/00:01:00:00:00/e0 Emask 0x2 (HSM violation) Sep 14 20:28:11 Tower kernel: ata4.00: status: { Busy } Sep 14 20:28:11 Tower kernel: ata4: hard resetting link Sep 14 20:28:11 Tower kernel: REISERFS (device md5): replayed 2 transactions in 75 seconds
September 15, 201015 yr Author Thanks for the reply GK20. But also that all the drives on that card are powered by the same cable. And seeing that the machine is having power starting up at this point (which I added to the post as an update) leads me to think it might be the PSU. Although that is just a guess. My guess right now is that my PSU is dying. I have one drive that is in the rebuild process, and another drive that is dying. It feels like quite the mess right now.
September 15, 201015 yr My guess right now is that my PSU is dying. I have one drive that is in the rebuild process, and another drive that is dying. It feels like quite the mess right now. PSU is also a piece of HW, it could fail like all other component. However if your machine can not even pass POST, i will do following two things first. (a) Visually inspect all capacitors on motherboard make sure there is no deformities. The most common cause for MB failure is busted capacitor. (b) Pull out this PSU and connected Green and Black wires together by a metal paper clipper, this will ground PS_ON, then connect a spare case fan to 4-pin molex then turn on PSU through ON/OFF switch on this PSU to find out if there is any power goes out from this PSU.
September 15, 201015 yr Author My guess right now is that my PSU is dying. I have one drive that is in the rebuild process, and another drive that is dying. It feels like quite the mess right now. PSU is also a piece of HW, it could fail like all other component. However if your machine can not even pass POST, i will do following two things first. (a) Visually inspect all capacitors on motherboard make sure there is no deformities. The most common cause for MB failure is busted capacitor. [glow=red,2,300]All caps and general looks of the MB look ok.[/glow] (b) Pull out this PSU and connected Green and Black wires together by a metal paper clipper, this will ground PS_ON, then connect a spare case fan to 4-pin molex then turn on PSU through ON/OFF switch on this PSU to find out if there is any power goes out from this PSU. [glow=red,2,300]The PSU passes the test of running all the fans in the casing (6x120mm and 1x200mm)[/glow] Maybe I should try replacing the MB? It's the not best brand or quality. It was on of the cheaper basic ECS ones.
September 15, 201015 yr All caps and general looks of the MB look ok. The PSU passes the test of running all the fans in the casing (6x120mm and 1x200mm) Maybe I should try replacing the MB? It's the not best brand or quality. It was on of the cheaper basic ECS ones. In this situation, i will examine MB first then PSU later. Personally, i wouldn't use ECS MB for a 24x7 server.
September 15, 201015 yr Author So I went out and got another MB. GK20 I agree about I shouldn't have been using that ECS. I was one of those were I planned to replace it and never did. It seems to be rebuilding the drive now. But I did see a few of these in the syslog still: Sep 15 22:17:00 Tower kernel: ata14.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 15 22:17:00 Tower kernel: ata14.01: failed command: READ DMA EXT Sep 15 22:17:00 Tower kernel: ata14.01: cmd 25/00:00:c7:4c:00/00:04:00:00:00/f0 tag 0 dma 524288 in Sep 15 22:17:00 Tower kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/10 Emask 0x4 (timeout) Sep 15 22:17:00 Tower kernel: ata14.01: status: { DRDY } Sep 15 22:17:06 Tower kernel: ata14: link is slow to respond, please be patient (ready=0) Sep 15 22:17:11 Tower kernel: ata14: device not ready (errno=-16), forcing hardreset Sep 15 22:17:11 Tower kernel: ata14: soft resetting link Sep 15 22:17:11 Tower kernel: ata14.00: configured for UDMA/133 Sep 15 22:17:11 Tower kernel: ata14.01: configured for UDMA/133 Sep 15 22:17:11 Tower kernel: ata14.01: device reported invalid CHS sector 0 Sep 15 22:17:11 Tower kernel: ata14: EH complete Sep 15 22:17:17 Tower in.telnetd[2737]: connect from 192.168.1.13 (192.168.1.13) Sep 15 22:17:37 Tower login[2738]: ROOT LOGIN on `pts/0' from `192.168.1.13' Sep 15 22:17:42 Tower kernel: ata14.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen But they did seem to go away. Here is the current syslog while its building. http://pastebin.com./5N9DYUZ6 So I will let this run and hopefully I wont run into more issues. I Will report back either way though.
September 15, 201015 yr So I went out and got another MB. GK20 I agree about I shouldn't have been using that ECS. I was one of those were I planned to replace it and never did. It seems to be rebuilding the drive now. But I did see a few of these in the syslog still: Sep 15 22:17:00 Tower kernel: ata14.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 15 22:17:00 Tower kernel: ata14.01: failed command: READ DMA EXT Sep 15 22:17:00 Tower kernel: ata14.01: cmd 25/00:00:c7:4c:00/00:04:00:00:00/f0 tag 0 dma 524288 in Sep 15 22:17:00 Tower kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/10 Emask 0x4 (timeout) Sep 15 22:17:00 Tower kernel: ata14.01: status: { DRDY } Sep 15 22:17:06 Tower kernel: ata14: link is slow to respond, please be patient (ready=0) Sep 15 22:17:11 Tower kernel: ata14: device not ready (errno=-16), forcing hardreset Sep 15 22:17:11 Tower kernel: ata14: soft resetting link Sep 15 22:17:11 Tower kernel: ata14.00: configured for UDMA/133 Sep 15 22:17:11 Tower kernel: ata14.01: configured for UDMA/133 Sep 15 22:17:11 Tower kernel: ata14.01: device reported invalid CHS sector 0 Sep 15 22:17:11 Tower kernel: ata14: EH complete Sep 15 22:17:17 Tower in.telnetd[2737]: connect from 192.168.1.13 (192.168.1.13) Sep 15 22:17:37 Tower login[2738]: ROOT LOGIN on `pts/0' from `192.168.1.13' Sep 15 22:17:42 Tower kernel: ata14.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen But they did seem to go away. Here is the current syslog while its building. http://pastebin.com./5N9DYUZ6 So I will let this run and hopefully I wont run into more issues. I Will report back either way though. I am not SATA expert but based on my educated guess i will do (a) Replace SATA cable to this disk at ata14. Monitor situation through syslog. (b) Try different SATA port, prefer one on MB. Monitor situation through syslog © If both (a)(b) doesn't help, then i will start to worry about this disk. Your priority right now is trying to bring back your unRAID to fully functional mode by finishing data reconstruction first unless this disk at ata14 prevent you to do so. meanwhile if you are getting a new MB from Gigabyte, make sure to avoid HPA issue.
September 16, 201015 yr Author Hmm this one just popped up once: Sep 15 23:19:15 Tower kernel: ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Sep 15 23:19:15 Tower kernel: ata7.00: failed command: CHECK POWER MODE Sep 15 23:19:15 Tower kernel: ata7.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0 Sep 15 23:19:15 Tower kernel: res 40/00:ff:00:00:00/00:00:00:00:00/50 Emask 0x4 (timeout) Sep 15 23:19:15 Tower kernel: ata7.00: status: { DRDY } Sep 15 23:19:15 Tower kernel: ata7: hard resetting link Sep 15 23:19:15 Tower kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Sep 15 23:19:15 Tower kernel: ata7.00: configured for UDMA/100 Sep 15 23:19:15 Tower kernel: ata7: EH complete But yes, as you said right now Im just letting it rebuild, which is seems to be cruising along. I can try to replace cables. I have replaced a few already before I realized the MB issues. I can't try other parts really because I don't have any left. 16 drives tapped me out. I did get a Gigabyte MB and was aware about the HPA issue. If I did have the HPA issue I would see all the drive showing as too small from what I remember right? I wouldn't be able to start rebuilding I wouldn't think. Another question. Seeing these ata numbers...how can I link those to what disk number they are?
September 16, 201015 yr Another question. Seeing these ata numbers...how can I link those to what disk number they are? During bootup time, unRAID will sweep your controllers/devices and come up an inventory list Sep 14 20:26:51 Tower kernel: ata14.01: configured for UDMA/133 Sep 14 20:26:51 Tower kernel: scsi 14:0:0:0: Direct-Access ATA ST31500541AS CC32 PQ: 0 ANSI: 5 Sep 14 20:26:51 Tower kernel: sd 14:0:0:0: [sdp] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB) Sep 14 20:26:51 Tower kernel: sd 14:0:0:0: [sdp] Write Protect is off Sep 14 20:26:51 Tower kernel: sd 14:0:0:0: [sdp] Mode Sense: 00 3a 00 00 Sep 14 20:26:51 Tower kernel: sd 14:0:0:0: [sdp] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Sep 14 20:26:51 Tower kernel: scsi 14:0:1:0: Direct-Access ATA ST31500341AS CC1H PQ: 0 ANSI: 5 Sep 14 20:26:51 Tower kernel: sdp: sdn1 Sep 14 20:26:51 Tower emhttp: Device inventory: Sep 14 20:26:51 Tower emhttp: pci-0000:00:1f.2-scsi-0:0:0:0 host13 (sdn) ST31500541AS_6XW0637J Sep 14 20:26:51 Tower emhttp: pci-0000:00:1f.2-scsi-0:0:1:0 host13 (sdo) ST31500341AS_9VS3W5RZ Sep 14 20:26:51 Tower emhttp: pci-0000:00:1f.2-scsi-1:0:0:0 host14 (sdp) ST31500541AS_6XW03TEQ Sep 14 20:26:51 Tower emhttp: pci-0000:00:1f.2-scsi-1:0:1:0 host14 (sdq) ST31500341AS_9VS1R38D Sep 14 20:26:51 Tower emhttp: pci-0000:01:00.0-scsi-0:0:0:0 host0 (sda) ST31500341AS_9VS0G8YV Sep 14 20:26:51 Tower emhttp: pci-0000:01:00.0-scsi-1:0:0:0 host1 (sdb) ST31000340AS_9QJ1YEBR Sep 14 20:26:51 Tower emhttp: pci-0000:01:00.0-scsi-2:0:0:0 host2 (sdc) ST3750640AS_5QD3J1PW Sep 14 20:26:51 Tower emhttp: pci-0000:01:00.0-scsi-3:0:0:0 host3 (sdd) WDC_WD20EARS-00S8B1_WD-WCAVY4142983 Sep 14 20:26:51 Tower emhttp: pci-0000:04:01.0-scsi-0:0:0:0 host5 (sde) ST3750640AS_5QD46GV3 Sep 14 20:26:51 Tower emhttp: pci-0000:04:01.0-scsi-1:0:0:0 host6 (sdf) ST3750640AS_5QD46PRL Sep 14 20:26:51 Tower emhttp: pci-0000:04:01.0-scsi-2:0:0:0 host7 (sdg) ST31500341AS_9VS03PAT Sep 14 20:26:51 Tower emhttp: pci-0000:04:01.0-scsi-3:0:0:0 host8 (sdh) ST3750330AS_5QK0295T Sep 14 20:26:51 Tower emhttp: pci-0000:04:02.0-scsi-0:0:0:0 host9 (sdi) Hitachi_HDS722020ALA330_JK1130YAGKTRNT Sep 14 20:26:51 Tower emhttp: pci-0000:04:02.0-scsi-1:0:0:0 host10 (sdj) ST31500341AS_9VS2MPRN Sep 14 20:26:51 Tower emhttp: pci-0000:04:02.0-scsi-2:0:0:0 host11 (sdl) ST31500341AS_9VS0WVW0 Sep 14 20:26:51 Tower emhttp: pci-0000:04:02.0-scsi-3:0:0:0 host12 (sdm) ST31500341AS_9VS0V4R5
September 16, 201015 yr Author Good to know. Thank you. Its still cruisin along. Total size: 1,953,514,552 KB Current position: 80,288,848 (4.1%) Estimated speed: 9,998 KB/sec Estimated finish: 3122.3 minutes Pretty standard rebuild time for this system.
September 16, 201015 yr If users are really seeing multiple REAL drive failures (as opposed to PSU/cable problems that look like drive failures) occurring at the same time, make sure to send a note to Tom (LimeTech) to let him know. Many have been pushing for a second "parity" disk so unRAID could recover from a 2 drive failure. If Tom starts seeing real-world examples of this happening, it may accelerate this feature.
September 17, 201015 yr Author So an update to this about about 30 hours of rebuilding. I get home from work and the system is frozen. My telnet session to it is dead, my web page to the interface is dead. SO I go to the console login and try to login as root and right after I type my password it kicks me right back out to the login again. So I really have no idea what would cause that. Any ideas? So after a hard reboot everything looks normal again and the rebuild process starts again.
September 17, 201015 yr So an update to this about about 30 hours of rebuilding. I get home from work and the system is frozen. My telnet session to it is dead, my web page to the interface is dead. SO I go to the console login and try to login as root and right after I type my password it kicks me right back out to the login again. So I really have no idea what would cause that. Any ideas? So after a hard reboot everything looks normal again and the rebuild process starts again. Those symptoms indicate the out-of-memory process killer was killing off anything it could to make space. (including your login shell) Probably the syslog used up all available RAM. Joe L.
September 17, 201015 yr Author So you think that a syslog could get to 1.5gb that quickly? So far I have been rebuilding for about 10 hours and my "top" shows about 270M in RAM usage: top - 16:18:30 up 9:56, 1 user, load average: 1.01, 1.09, 1.26 Tasks: 85 total, 2 running, 83 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 5.4%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.7%hi, 0.7%si, 0.0%st Mem: 2039244k total, 270736k used, 1768508k free, 2620k buffers Swap: 0k total, 0k used, 0k free, 151256k cached The rebuild is still going but I get dives clicks a lot. If he rebuild does ever finish I ma still trying to figure out if I have a bad power cable somewhere or some type of bad hardware. During this latest rebuild I am now seeing error with ata13. current syslog during current rebuild: http://pastebin.com/6DGhd3M3 Once again, thanks everyone for reading these issues and taking time to respond.
September 17, 201015 yr So you think that a syslog could get to 1.5gb that quickly? So far I have been rebuilding for about 10 hours and my "top" shows about 270M in RAM usage: top - 16:18:30 up 9:56, 1 user, load average: 1.01, 1.09, 1.26 Tasks: 85 total, 2 running, 83 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 5.4%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.7%hi, 0.7%si, 0.0%st Mem: 2039244k total, 270736k used, 1768508k free, 2620k buffers Swap: 0k total, 0k used, 0k free, 151256k cached The rebuild is still going but I get dives clicks a lot. If he rebuild does ever finish I ma still trying to figure out if I have a bad power cable somewhere or some type of bad hardware. During this latest rebuild I am now seeing error with ata13. current syslog during current rebuild: http://pastebin.com/6DGhd3M3 Once again, thanks everyone for reading these issues and taking time to respond. If this time you still can not finish data reconstruction successfully, it is time for you to shutdown system and replace cables to ata13/ata14 as well as power cable to both disks and make sure keep the cabling order. depends on how long your data reconstruction take, if you get SATA link reset every now and then, your latest syslog show these two disks are getting link reset frequently, system will have to log an entry in syslog, your syslog could grow quite quick and large. Maybe unRAID should set a limit on syslog size and make it a circular log, move old one to flash drive as backup copy.
September 17, 201015 yr Author So right now its at about 35% and I hear clicks and see about 10 of the drives just drop and lose power for a second. Then they spin back up and continue with the rebuild. So with 10 drives dropping all at once I'm thinking it has to be more than the one cable on the PSU. Maybe the whole thing.
September 18, 201015 yr Author Well is finally finished but with: Parity-Check. (Last checked on 9/18/2010 1:30:42 PM, finding 437198 errors.) So I will run a parity check and see if I get more power issues.
September 18, 201015 yr So right now its at about 35% and I hear clicks and see about 10 of the drives just drop and lose power for a second. Then they spin back up and continue with the rebuild. So with 10 drives dropping all at once I'm thinking it has to be more than the one cable on the PSU. Maybe the whole thing. Definately sounds like a cabling issue, or an possibly an overloaded power supply. Many have overcurrent shutdown to protect themselves. What specific make/model power supply are you using? How many hard disks again? Still 16?
September 18, 201015 yr Author So I rebooted to see if any new errors would arise. The first time I rebooted disk15 was missing. Then I rebooted again and it said it was a new drive (blue dot). So Im lost. My PSU is a Corsair CMPSU-650TX. I still run 16 drives. On board video card. So nothing that should really be taxing the PSU except for the drives.
Archived
This topic is now archived and is closed to further replies.