August 28, 201114 yr Ok, here it is, I don't know how to proceed: I replaced a 1TB drive with a 3 TB drive (my second 3TB drive). I then started the "rebuild process" on this new disk (disk7). It runs super smooth to about 50-51%, then it slows to a crawl with a speed of only a few kbs, throwing errors from disk5 (a 2TB volume). It also keeps saying that a disk has been unplugged. So far I have: -checked all cables to make sure they are attached well - unplugged disk5 (the one with errors) from the controller card and attached it directly to the motherboard. - someone suggested to simply wait - I left if for over 1 week, and it only progressed about 50 MB. So it would clearly take years for a rebuild at that rate. What can I do? I would hate to lose any data. I am running 10 beta 11 here is an excerpt from the log with errors: Aug 28 08:49:02 Tower kernel: sas: Enter sas_scsi_recover_host Aug 28 08:49:02 Tower kernel: sas: trying to find task 0xf74a2f00 Aug 28 08:49:02 Tower kernel: sas: sas_scsi_find_task: aborting task 0xf74a2f00 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1703:<7>mv_abort_task() mvi=f7540000 task=f74a2f00 slot=f75515d4 slot_idx=x0 Aug 28 08:49:02 Tower kernel: sas: sas_scsi_find_task: querying task 0xf74a2f00 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1632:mvs_query_task:rc= 5 Aug 28 08:49:02 Tower kernel: sas: sas_scsi_find_task: task 0xf74a2f00 failed to abort Aug 28 08:49:02 Tower kernel: sas: task 0xf74a2f00 is not at LU: I_T recover Aug 28 08:49:02 Tower kernel: sas: I_T nexus reset for dev 0300000000000000 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x1081 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2111:phy3 Unplug Notice Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x1081 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x10000 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2138:notify plug in on phy[3] Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1224:port 3 attach dev info is 0 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1226:port 3 attach sas addr is 3 Aug 28 08:49:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 378:phy 3 byte dmaded. Aug 28 08:49:02 Tower kernel: sas: sas_form_port: phy3 belongs to port2 already(1)! Aug 28 08:49:04 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1586:mvs_I_T_nexus_reset for device[2]:rc= 0 Aug 28 08:49:04 Tower kernel: sas: I_T 0300000000000000 recovered Aug 28 08:49:04 Tower kernel: sas: sas_ata_task_done: SAS error 8d Aug 28 08:49:04 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 08:49:04 Tower kernel: ata11: status=0x01 { Error } Aug 28 08:49:04 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 08:49:04 Tower kernel: sas: --- Exit sas_scsi_recover_host Aug 28 08:49:04 Tower kernel: sas: sas_to_ata_err: Saw error 2. What to do? Aug 28 08:49:04 Tower kernel: sas: sas_ata_task_done: SAS error 2 Aug 28 08:49:04 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 08:49:04 Tower kernel: ata11: status=0x01 { Error } Aug 28 08:49:04 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 08:49:35 Tower kernel: sas: command 0xf07a9180, task 0xf74a2780, timed out: BLK_EH_NOT_HANDLED Aug 28 08:49:35 Tower kernel: sas: Enter sas_scsi_recover_host Aug 28 08:49:35 Tower kernel: sas: trying to find task 0xf74a2780 Aug 28 08:49:35 Tower kernel: sas: sas_scsi_find_task: aborting task 0xf74a2780 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1703:<7>mv_abort_task() mvi=f7540000 task=f74a2780 slot=f75515d4 slot_idx=x0 Aug 28 08:49:35 Tower kernel: sas: sas_scsi_find_task: querying task 0xf74a2780 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1632:mvs_query_task:rc= 5 Aug 28 08:49:35 Tower kernel: sas: sas_scsi_find_task: task 0xf74a2780 failed to abort Aug 28 08:49:35 Tower kernel: sas: task 0xf74a2780 is not at LU: I_T recover Aug 28 08:49:35 Tower kernel: sas: I_T nexus reset for dev 0300000000000000 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x89800. Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x1001 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2111:phy3 Unplug Notice Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x1081 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x10000 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2138:notify plug in on phy[3] Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1224:port 3 attach dev info is 0 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1226:port 3 attach sas addr is 3 Aug 28 08:49:35 Tower kernel: drivers/scsi/mvsas/mv_sas.c 378:phy 3 byte dmaded. Aug 28 08:49:35 Tower kernel: sas: sas_form_port: phy3 belongs to port2 already(1)! Aug 28 08:49:37 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1586:mvs_I_T_nexus_reset for device[2]:rc= 0 Aug 28 08:49:37 Tower kernel: sas: I_T 0300000000000000 recovered Aug 28 08:49:37 Tower kernel: sas: sas_ata_task_done: SAS error 8d Aug 28 08:49:37 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 08:49:37 Tower kernel: ata11: status=0x01 { Error } Aug 28 08:49:37 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 08:49:37 Tower kernel: sas: --- Exit sas_scsi_recover_host Aug 28 08:49:37 Tower kernel: sas: sas_to_ata_err: Saw error 2. What to do? Aug 28 08:49:37 Tower kernel: sas: sas_ata_task_done: SAS error 2 Aug 28 08:49:37 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 08:49:37 Tower kernel: ata11: status=0x01 { Error } Aug 28 08:49:37 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 08:50:08 Tower kernel: sas: command 0xf07a9180, task 0xf0726780, timed out: BLK_EH_NOT_HANDLED Aug 28 08:50:08 Tower kernel: sas: Enter sas_scsi_recover_host Aug 28 08:50:08 Tower kernel: sas: trying to find task 0xf0726780 Aug 28 08:50:08 Tower kernel: sas: sas_scsi_find_task: aborting task 0xf0726780 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1703:<7>mv_abort_task() mvi=f7540000 task=f0726780 slot=f75515d4 slot_idx=x0 Aug 28 08:50:08 Tower kernel: sas: sas_scsi_find_task: querying task 0xf0726780 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1632:mvs_query_task:rc= 5 Aug 28 08:50:08 Tower kernel: sas: sas_scsi_find_task: task 0xf0726780 failed to abort Aug 28 08:50:08 Tower kernel: sas: task 0xf0726780 is not at LU: I_T recover Aug 28 08:50:08 Tower kernel: sas: I_T nexus reset for dev 0300000000000000 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x89800. Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x1001 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2111:phy3 Unplug Notice Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x1081 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2083:port 3 ctrl sts=0x199800. Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2085:Port 3 irq sts = 0x10000 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2138:notify plug in on phy[3] Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1224:port 3 attach dev info is 0 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1226:port 3 attach sas addr is 3 Aug 28 08:50:08 Tower kernel: drivers/scsi/mvsas/mv_sas.c 378:phy 3 byte dmaded. Aug 28 08:50:08 Tower kernel: sas: sas_form_port: phy3 belongs to port2 already(1)! Aug 28 08:50:10 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1586:mvs_I_T_nexus_reset for device[2]:rc= 0 Aug 28 08:50:10 Tower kernel: sas: I_T 0300000000000000 recovered Aug 28 08:50:10 Tower kernel: sas: sas_ata_task_done: SAS error 8d Aug 28 08:50:10 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 08:50:10 Tower kernel: ata11: status=0x01 { Error } Aug 28 08:50:10 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 08:50:10 Tower kernel: sas: --- Exit sas_scsi_recover_host Aug 28 08:50:10 Tower kernel: sas: sas_to_ata_err: Saw error 2. What to do? Aug 28 08:50:10 Tower kernel: sas: sas_ata_task_done: SAS error 2 Aug 28 08:50:10 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 08:50:10 Tower kernel: ata11: status=0x01 { Error } Aug 28 08:50:10 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 08:50:10 Tower kernel: sd 0:0:2:0: [sdi] Result: hostbyte=0x00 driverbyte=0x08 Aug 28 08:50:10 Tower kernel: sd 0:0:2:0: [sdi] Sense Key : 0xb [current] [descriptor] Aug 28 08:50:10 Tower kernel: Descriptor sense data with sense descriptors (in hex): Aug 28 08:50:10 Tower kernel: 72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 Aug 28 08:50:10 Tower kernel: 00 00 00 bf Aug 28 08:50:10 Tower kernel: sd 0:0:2:0: [sdi] ASC=0x0 ASCQ=0x0 Aug 28 08:50:10 Tower kernel: sd 0:0:2:0: [sdi] CDB: cdb[0]=0x28: 28 00 ac d6 50 c0 00 02 00 00 Aug 28 08:50:10 Tower kernel: end_request: I/O error, dev sdi, sector 2899726528 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726464/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726472/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726480/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726488/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726496/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726504/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726512/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726520/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726528/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726536/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726544/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726552/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726560/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726568/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726576/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726584/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726592/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726600/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726608/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726616/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726624/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726632/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726640/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726648/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726656/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726664/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726672/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726680/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726688/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error Aug 28 08:50:10 Tower kernel: handle_stripe read error: 2899726696/1, count: 1 Aug 28 08:50:10 Tower kernel: md: disk5 read error
August 28, 201114 yr Author I also am continuously getting these errors for all ports that my controller card has HD's attached to. Aug 28 11:19:01 Tower kernel: sas: sas_form_port: phy3 belongs to port2 already(1)! Aug 28 11:19:02 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1586:mvs_I_T_nexus_reset for device[2]:rc= 0 Aug 28 11:19:02 Tower kernel: sas: I_T 0300000000000000 recovered Aug 28 11:19:02 Tower kernel: sas: sas_ata_task_done: SAS error 8d Aug 28 11:19:02 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 11:19:02 Tower kernel: ata11: status=0x01 { Error } Aug 28 11:19:02 Tower kernel: ata11: error=0x04 { DriveStatusError } Aug 28 11:19:02 Tower kernel: sas: --- Exit sas_scsi_recover_host Aug 28 11:19:02 Tower kernel: sas: sas_to_ata_err: Saw error 2. What to do? Aug 28 11:19:02 Tower kernel: sas: sas_ata_task_done: SAS error 2 Aug 28 11:19:02 Tower kernel: ata11: translated ATA stat/err 0x01/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Aug 28 11:19:02 Tower kernel: ata11: status=0x01 { Error } Aug 28 11:19:02 Tower kernel: ata11: error=0x04 { DriveStatusError } I think that whenever these errors occur on the port my disk5 is attached to, it then logs a parity error. So could the source of the problems be my controller card? It is the AOC-SASLP-MV8. I have read somewhere before that they sometimes produce errors when there are too many I/O's.
August 28, 201114 yr did you run a parity check before starting this process? It looks like disk5 is failing, or at least having some troubles right now.
August 28, 201114 yr Can I run a parity check without the rebuilt disk7? no. You would need all the original disks with their original data. You basically have a two disk failure... (one you removed and re-purposed, the other that is failing.) If you had performed a parity check before removing disk7 it might have uncovered the issue with disk5, when you could have recovered your data.
August 28, 201114 yr I was going to post about what Joe said. It sounds like you already had a bad disk. disk5. You then removed a disk yourself and reformated it before you rebuilt the data or performed a parity check You now have a 2 disk failure. pretty much unrecoverable. you might still be able to copy some data from disk 5 manually. how good the data is, is yet to be seen... i guess on the bright side... if you had a raid 5. all dirves would be lost now. PS the controller errors you are seeing are normal when the disk has a read errors. I'm not sure exacly what it is, but it looks like it cant read the drive so it drops the disk and reconnects it to see if that solves it. That was how i knew i had a bad disk a while back. the mv8 dropouts.
August 28, 201114 yr Author Thank you for your replies. I managed to copy all the data off of disk5, seemed to be without problems (I didn't do any md5 checksums). The data on the disk, as it turns out, is nothing I couldn't live without so it's not too bad. DVD's can be re-ripped. In terms of the mv8 dropouts - they occur on all ports. The "bad disk" (disk5) is no longer hooked up to the mv8 but rather straight to the motherboard. I am trying another rebuild now that the data from disk 5 is safe. We'll see how it pans out. I deleted all data off of disk5 manually.
August 29, 201114 yr Author A quick question regarding 2-disk failure: My problems are disk5 and disk7 (disk5 is failing disk, disk7 is new disk I wanted to expand the array with). If I copy the data off of disk7 and off of disk5, can I then simply remove both disks from the array and start it back up, or will I lose the entire array?
August 29, 201114 yr Unlike true RAID arrays, e.g., RAID-5, unRaid maintains data on separately formatted disks. Data is not striped. Therefore, data on an undamaged disk will not be lost, even if every other disk were to fail. If you have lost 1 disk, you can recover it's contents by rebuilding its data onto a new disk. If you lose 2 or more disks, unRaid cannot rebuild the disk. But as you've found out, an apparently failed disk may yet allow you to copy data off of it. Sounds like you have all of your data recovered and what you want to do is recreate your array with your existing disks. Reinitialing can be done with the initconfig command. Then assign the disks to the array, make doubly sure the correct disk is assigned as parity, and then Start the array. Parity will build and you'll have a protected array again.
August 29, 201114 yr A quick question regarding 2-disk failure: My problems are disk5 and disk7 (disk5 is failing disk, disk7 is new disk I wanted to expand the array with). If I copy the data off of disk7 and off of disk5, can I then simply remove both disks from the array and start it back up, or will I lose the entire array? With the array Stopped, use the devices assignment page to un-assign disk5 and disk7. Then log in via telnet or on the system console and type: initconfig answer "Yes" (Capital "Y", lower case "es") and then start the array. It will completely re-calculate parity without the two disks. Then, you can deal with the two un-assigned disks as desired. Joe L.
August 29, 201114 yr Author Guys, you are all legends! Everything worked, no data lost (at least upon first look). Parity is being recalculated as we speak. A close call! Thank you all again!
Archived
This topic is now archived and is closed to further replies.