May 3, 201214 yr Hi everyone, Server has thrown a new one at me this last couple of days. In running the monthly parity check with correction the web GUI seems to hang and become inaccessible about 4 hours into the parity check run. The server is still accessible as a network drive and via putty however the web based GUI no longer responds. To be able to re-access the web GUI the only remedy so far appears to be to reboot the server via the power button. I can access the server via putty but a shutdown command (assuming I am issuing the correct one) does not shut down the server. Putty tells me the server is going down for reboot but it does not actually happen on the machine. So I assume something is hanging up somewhere. Not sure what logs I can access since I have to restart the system to get back into the web-GUI. I cannot locate on the flash drive where any new logs are being written when all of this fun happens. The interesting thing is if I restart the server and cancel the automatic parity check it performs early (before the web GUI becomes inaccessible) and uncheck the "correct parity" option the check will run through to completion (each time finding 6 sync errors). However if I try to run the parity check and correct these 6 errors this is when the system seems to hang (at lest access to the web GUI and ability to shut down the server remotely hangs). Anyone have a direction to point me toward to try to troubleshoot? Thanks so much in advance.
May 3, 201214 yr Author Thanks for the reply. Sylogs attached. Working on the smart testing/reports now. I struggle to make sens of the syslogs so if anyone sees anything glaringly obvious please educate me. syslog-2012-05-03.txt syslogtail.txt
May 3, 201214 yr May have a problem with "ata7" May 3 00:01:55 Tower kernel: ata7.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 t0 May 3 00:01:55 Tower kernel: ata7.00: failed command: READ FPDMA QUEUED May 3 00:01:55 Tower kernel: ata7.00: cmd 60/00:00:c0:4d:fd/02:00:13:00:00/40 tag 0 ncq 262144 in May 3 00:01:55 Tower kernel: res 50/00:00:00:00:00/00:00:00:00:00/00 Emask 0x401 (device error) <F> Line 726: May 2 08:39:54 Tower kernel: ata7.00: ATA-8: SAMSUNG HD204UI, 1AQ10001, max UDMA/133
May 3, 201214 yr Author interesting. as for some reason I am unable to obtain a smart report nor run a smart test for my Disc 6 and Disc 7. I enter the command via putty and it just sits there forever with no reply. I am certain one of those is likely assigned to ata7. Probably via the one Supermicro AOC-SASLP-MV8 8-Port SAS SATA add-on Card i have in the server. I will need to open up the case to make certain.
May 3, 201214 yr Yeah..there were two drives throwing up an error on the MV8 Edit: Noticed you were running B12...check up on that version - those "BLK_EH_NOT_HANDLED" would lock up the array as well. http://lime-technology.com/forum/index.php?topic=15564.msg146338#msg146338 A later version eliminated the errors 13 or 14?..
May 3, 201214 yr Author Gotcha. I will check on the newer betas. SO I shut down the server and re checked the cabling to and from the hard drives and the SuperMicro card. Restarted and at that point was able to get the smart tests to run on those 3 drives (#6, 7, 8 attached to the SuperMicro card). I ran the smart test for those 3 without starting the array. So not sure if the smart tests ran this time due to that, due to my possibly re-seating a loose cable or just coincidence. Anyhoo, I will post the Smart reports for you to glance over if you have time. As I review them (if I am looking at numbers correctly which it is quite likely I am not) the reallocation sector #'s might be concerning? Again not sure. I go cross-eyed looking at these syslogs. Thanks again for your willingness to help. I really appreciate it. smartparity.txt smart1.txt smart2.txt smart3.txt
May 3, 201214 yr Everything looked clean, but run another one on Device Model: Hitachi HDS5C3020ALA632 Serial Number: ML0220F3136MRD SMART Error Log Version: 1 ATA Error Count: 1 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 1 occurred at disk power-on lifetime: 4 hours (0 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 71 3f d6 32 04 Error: ICRC, ABRT 113 sectors at LBA = 0x0432d63f = 70440511 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 80 30 d6 32 e0 00 04:02:57.077 READ DMA EXT 25 00 80 b0 d5 32 e0 00 04:02:57.075 READ DMA EXT 25 00 80 30 d5 32 e0 00 04:02:57.073 READ DMA EXT 25 00 80 b0 d4 32 e0 00 04:02:57.071 READ DMA EXT 25 00 80 30 d4 32 e0 00 04:02:57.070 READ DMA EXT
May 3, 201214 yr Author gotcha. unless you recommend otherwise... i think ill look into upgrading to beta 14 and see if that is helpful. thanks again
May 3, 201214 yr Author true. i will give it a try. upgrading from a beta to RC2 still the same process (backup current flash and then copying and replacing bzimage and bzroot). just want to make sure im not missing anything
May 3, 201214 yr Author thanks. upgrade completed. now running a parity check. fingers crossed. thanks again for your help!
May 4, 201214 yr Author well. it looks like we got about 2 hours from the parity check.correct completing and again lost web gui access and inability to shut server down remotely. Latest syslog attached. To my untrained eye... is ata7/8 an issue in some way again? Thanks syslogtail.txt
May 4, 201214 yr same problem <ata7>...cabling? Do you have another port to try? Before upgrade: May 3 00:01:52 Tower kernel: sas: command 0xf47dc9c0, task 0xf74dd040, timed out: BLK_EH_NOT_HANDLED May 3 00:01:52 Tower kernel: sas: Enter sas_scsi_recover_host May 3 00:01:52 Tower kernel: sas: trying to find task 0xf74dd040 May 3 00:01:53 Tower kernel: sas: sas_scsi_find_task: aborting task 0xf74dd040 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1818:<7>mv_abort_task() mvi=f75e0000 task=f74dd040 slot=f75f15d8 slot_idx=x0 May 3 00:01:53 Tower kernel: sas: sas_scsi_find_task: querying task 0xf74dd040 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1747:mvs_query_task:rc= 5 May 3 00:01:53 Tower kernel: sas: sas_scsi_find_task: task 0xf74dd040 failed to abort May 3 00:01:53 Tower kernel: sas: task 0xf74dd040 is not at LU: I_T recover May 3 00:01:53 Tower kernel: sas: I_T nexus reset for dev 0600000000000000 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x89800. May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x1001001 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2226:phy6 Unplug Notice May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x199800. May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x1001081 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x199800. May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x10000 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2253:notify plug in on phy[6] May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1338:port 6 attach dev info is 40002 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1340:port 6 attach sas addr is 6 May 3 00:01:53 Tower kernel: drivers/scsi/mvsas/mv_sas.c 379:phy 6 byte dmaded. May 3 00:01:53 Tower kernel: sas: sas_form_port: phy6 belongs to port2 already(1)! May 3 00:01:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1701:mvs_I_T_nexus_reset for device[2]:rc= 0 May 3 00:01:55 Tower kernel: sas: I_T 0600000000000000 recovered May 3 00:01:55 Tower kernel: sas: sas_ata_task_done: SAS error 8d May 3 00:01:55 Tower kernel: ata5: sas eh calling libata port error handler May 3 00:01:55 Tower kernel: ata6: sas eh calling libata port error handler May 3 00:01:55 Tower kernel: ata7: sas eh calling libata port error handler May 3 00:01:55 Tower kernel: ata7.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 t0 May 3 00:01:55 Tower kernel: ata7.00: failed command: READ FPDMA QUEUED May 3 00:01:55 Tower kernel: ata7.00: cmd 60/00:00:c0:4d:fd/02:00:13:00:00/40 tag 0 ncq 262144 in May 3 00:01:55 Tower kernel: res 50/00:00:00:00:00/00:00:00:00:00/00 Emask 0x401 (device error) <F> May 3 00:01:55 Tower kernel: ata7.00: status: { DRDY } May 3 00:01:55 Tower kernel: ata7.00: configured for UDMA/133 May 3 00:01:55 Tower kernel: ata7: EH complete May 3 00:01:55 Tower kernel: ata8: sas eh calling libata port error handler May 3 00:01:55 Tower kernel: sas: --- Exit sas_scsi_recover_host May 3 00:47:27 Tower sSMTP[6386]: Creating SSL connection to host May 3 00:47:27 Tower sSMTP[6386]: SSL connection using RC4-SHA May 3 00:47:30 Tower sSMTP[6386]: Sent mail for root@[email protected] (221 2.0.0 closing connection y28sm18182378yhi.16) uid=0 username=root outbytes=854 May 3 01:13:14 Tower kernel: sas: command 0xf5acbb40, task 0xf74e6a00, timed out: BLK_EH_NOT_HANDLED After upgrade: May 3 22:37:55 Tower kernel: sas: command 0xefdcb9c0, task 0xf0790140, timed out: BLK_EH_NOT_HANDLED May 3 22:37:55 Tower kernel: sas: Enter sas_scsi_recover_host May 3 22:37:55 Tower kernel: sas: trying to find task 0xf0790140 May 3 22:37:55 Tower kernel: sas: sas_scsi_find_task: aborting task 0xf0790140 May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1818:<7>mv_abort_task() mvi=f7560000 task=f0790140 slot=f75715d8 slot_idx=x0 May 3 22:37:55 Tower kernel: sas: sas_scsi_find_task: querying task 0xf0790140 May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1747:mvs_query_task:rc= 5 May 3 22:37:55 Tower kernel: sas: sas_scsi_find_task: task 0xf0790140 failed to abort May 3 22:37:55 Tower kernel: sas: task 0xf0790140 is not at LU: I_T recover May 3 22:37:55 Tower kernel: sas: I_T nexus reset for dev 0600000000000000 May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x99800. May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x1001 May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2226:phy6 Unplug Notice May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x199800. May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x11081 May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2253:notify plug in on phy[6] May 3 22:37:55 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2278:plugin interrupt but phy6 is gone May 3 22:37:57 Tower kernel: mvsas 0000:01:00.0: Phy6 : No sig fis May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2139:phy6 Attached Device May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x89800. May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x1001 May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2226:phy6 Unplug Notice May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x199800. May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x81 May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2198:port 6 ctrl sts=0x199800. May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2200:Port 6 irq sts = 0x10000 May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 2253:notify plug in on phy[6] May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1338:port 6 attach dev info is 40002 May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1340:port 6 attach sas addr is 6 May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 379:phy 6 byte dmaded. May 3 22:37:57 Tower kernel: sas: sas_form_port: phy6 belongs to port2 already(1)! May 3 22:37:57 Tower kernel: drivers/scsi/mvsas/mv_sas.c 1701:mvs_I_T_nexus_reset for device[2]:rc= 0 May 3 22:37:57 Tower kernel: sas: I_T 0600000000000000 recovered May 3 22:37:57 Tower kernel: sas: sas_ata_task_done: SAS error 8d May 3 22:37:57 Tower kernel: ata5: sas eh calling libata port error handler May 3 22:37:57 Tower kernel: ata6: sas eh calling libata port error handler May 3 22:37:57 Tower kernel: ata7: sas eh calling libata port error handler May 3 22:37:57 Tower kernel: ata7.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 t0 May 3 22:37:57 Tower kernel: ata7.00: failed command: READ FPDMA QUEUED May 3 22:37:57 Tower kernel: ata7.00: cmd 60/00:00:50:c0:be/02:00:7c:00:00/40 tag 0 ncq 262144 in May 3 22:37:57 Tower kernel: res 50/00:00:00:00:00/00:00:00:00:00/00 Emask 0x401 (device error) <F> May 3 22:37:57 Tower kernel: ata7.00: status: { DRDY } May 3 22:37:57 Tower kernel: ata7.00: configured for UDMA/133 May 3 22:37:57 Tower kernel: ata7: EH complete May 3 22:37:57 Tower kernel: ata8: sas eh calling libata port error handler
May 4, 201214 yr Author both ata 6, 7, and 8 are via a single breakout cable from the supermicro card. perhaps that one cable to that specific hard drive is bad. I will move things around and see if I can take that cable out of the equation.
May 4, 201214 yr Also see this with 5RC... http://lime-technology.com/forum/index.php?topic=19819.msg177387#msg177387
May 4, 201214 yr Author Well using a different cable as well as a different port on the SuperMicro card was not the solution. Thanks for the link. i will keep my eyes open for a possible non test RC2 to try. Latest sys log attached. syslogtail2.txt
May 6, 201214 yr Author Just to update for historical purposed i guess... Switching to 5.0 Beta 14 solved the issue apparently. I have run through to completion two parity checks so far (first one to correct the 6 errors found and then a second one that returned 0 errors (predictably and thankfully). I have yet to try the latest RC2 but will at some point in the next few days and hopefully have similar results. Thanks again to mbryanr for all of your help,
May 6, 201214 yr Looks like the timeout delay was increased in RC2 - so that should work as well. Thanks for being patient with me. I don't have a MV8, nor am I running unRAID 5 - soon though.
Archived
This topic is now archived and is closed to further replies.