May 30, 200719 yr My tower just siezed up again. The web interface is unresponsive, I can not telnet in, and the shares are inacessible) Luckily, I was monitoring the syslog (been suspecting it of late running tail-f). Can anyone (Tom?) Please help me interpret these results in the syslog which occured just prior to the sieze?: May 30 10:51:05 Tower dhcpcd[1167]: DHCP_ACK received from (192.168.1.1) May 30 12:59:29 Tower kernel: [180547.509641] hdj: lost interrupt May 30 12:59:29 Tower kernel: [180547.509650] hdj: task_in_intr: status=0x7f { D riveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 12:59:29 Tower kernel: [180547.509659] hdj: task_in_intr: error=0x7f { Dr iveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNot Found }, LBAsect=8830595334015, high=526344, low=8355711, sector=216461511 May 30 12:59:29 Tower kernel: [180547.509680] ide: failed opcode was: unknown May 30 12:59:29 Tower kernel: [180547.509711] pdc202xx_new: Primary channel rese t. May 30 12:59:29 Tower kernel: [180547.609584] ide4: reset: success May 30 12:59:39 Tower kernel: [180557.624210] hdj: lost interrupt May 30 12:59:39 Tower kernel: [180557.624219] hdj: task_in_intr: status=0x7f { D riveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 12:59:39 Tower kernel: [180557.624228] hdj: task_in_intr: error=0x7f { Dr iveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNot Found }, LBAsect=8830595334015, high=526344, low=8355711, sector=220214767 May 30 12:59:39 Tower kernel: [180557.624248] ide: failed opcode was: unknown May 30 12:59:39 Tower kernel: [180557.624279] pdc202xx_new: Primary channel rese t. May 30 12:59:39 Tower kernel: [180557.724154] ide4: reset: success May 30 12:59:48 Tower kernel: [180566.629433] hdj: status error: status=0x7f { D riveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 12:59:48 Tower kernel: [180566.629447] hdj: status error: error=0x7f { Dr iveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNot Found }, LBAsect=9343692930943, high=556927, low=8355711, sector=309053679 May 30 12:59:48 Tower kernel: [180566.629479] ide: failed opcode was: unknown May 30 12:59:48 Tower kernel: [180566.629515] pdc202xx_new: Primary channel rese t. May 30 12:59:48 Tower kernel: [180566.629518] hdj: drive not ready for command May 30 13:00:03 Tower kernel: [180581.371461] ide4: reset: success May 30 13:00:13 Tower kernel: [180591.366097] hdj: lost interrupt May 30 13:00:13 Tower kernel: [180591.366105] hdj: task_in_intr: status=0x7f { D riveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 13:00:13 Tower kernel: [180591.366114] hdj: task_in_intr: error=0x7f { Dr iveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNot Found }, LBAsect=8830595334015, high=526344, low=8355711, sector=309053679 May 30 13:00:13 Tower kernel: [180591.366134] ide: failed opcode was: unknown May 30 13:00:13 Tower kernel: [180591.366165] pdc202xx_new: Primary channel rese t. May 30 13:00:13 Tower kernel: [180591.466140] ide4: reset: success May 30 13:00:23 Tower kernel: [180601.460680] hdj: irq timeout: status=0x80 { Bu sy } May 30 13:00:23 Tower kernel: [180601.460686] ide: failed opcode was: unknown May 30 13:00:23 Tower kernel: [180601.460694] end_request: I/O error, dev hdj, s ector 309053679 May 30 13:00:23 Tower kernel: [180601.460934] md10: read error! May 30 13:00:23 Tower kernel: [180601.460939] handle_stripe read error: 30905361 6/9, count: 1 May 30 13:00:43 Tower kernel: [180621.489925] hdk: dma_timer_expiry: dma status == 0x21 ---------------------------------------------------------------------- That's it. What should I be looking for at this point to diagnose it?
May 30, 200719 yr Author After restarting (A couple of times as it hung on the first reboot), it started parity checking and failed at .2% with this in the log: May 30 14:09:09 Tower emhttp[1287]: shcmd (10): /usr/sbin/nmbd -D May 30 14:09:09 Tower emhttp[1287]: shcmd (11): /usr/sbin/smbd -D May 30 14:09:32 Tower in.telnetd[1434]: connect from 192.168.1.11 (192.168.1.11) May 30 14:09:34 Tower login[1435]: ROOT LOGIN on `pts/0' from `192.168.1.11' May 30 14:10:29 Tower kernel: [ 227.212858] md0: parity incorrect: 2156176 May 30 14:10:49 Tower kernel: [ 247.669963] hdj: dma_timer_expiry: dma status = = 0x61 May 30 14:10:59 Tower kernel: [ 257.664599] hdj: DMA timeout error May 30 14:10:59 Tower kernel: [ 257.664612] hdj: dma timeout error: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 14:10:59 Tower kernel: [ 257.664621] hdj: dma timeout error: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMar kNotFound }, LBAsect=8830595334015, high=526344, low=8355711, sector=2161711 May 30 14:10:59 Tower kernel: [ 257.664642] ide: failed opcode was: unknown May 30 14:10:59 Tower kernel: [ 257.664649] hdi: DMA disabled May 30 14:10:59 Tower kernel: [ 257.664654] hdj: DMA disabled May 30 14:10:59 Tower kernel: [ 257.664681] pdc202xx_new: Primary channel reset . May 30 14:10:59 Tower kernel: [ 257.764542] ide4: reset: success May 30 14:11:20 Tower kernel: [ 278.493416] hdj: dma_timer_expiry: dma status = = 0x41
May 30, 200719 yr Check the plugs going into the drives. SATA plugs, both power and data, get loose very easily. What I see indicates the power is not there. Bill
May 30, 200719 yr Check the plugs going into the drives. SATA plugs, both power and data, get loose very easily. What I see indicates the power is not there. Bill Specifically /dev/hdj looks like it is not responding as expected. Almost every error involves that drive. Good advise... check cables.
May 30, 200719 yr Author Additional errors logged on subsequent reboots: May 30 15:05:52 Tower kernel: [ 1142.430660] hdj: drive_cmd: status=0x7f { Drive Ready DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 15:05:52 Tower kernel: [ 1142.430672] hdj: drive_cmd: error=0x7f { DriveS tatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNotFoun d }, LBAsect=8830595334015, high=526344, low=8355711, sector=0 May 30 15:05:52 Tower kernel: [ 1142.430693] ide: failed opcode was: 0xb0 May 30 15:05:52 Tower emhttp[1618]: smart ioctl: Input/output error ---------------------------------------------------------------------------------------------- May 30 15:21:22 Tower kernel: [ 164.635425] hdj: dma timeout error: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error } May 30 15:21:22 Tower kernel: [ 164.635434] hdj: dma timeout error: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMar kNotFound }, LBAsect=8830595334015, high=526344, low=8355711, sector=3391 May 30 15:21:22 Tower kernel: [ 164.635455] ide: failed opcode was: unknown May 30 15:21:22 Tower kernel: [ 164.635463] hdi: DMA disabled May 30 15:21:22 Tower kernel: [ 164.635467] hdj: DMA disabled May 30 15:21:22 Tower kernel: [ 164.635494] pdc202xx_new: Primary channel reset . May 30 15:21:22 Tower kernel: [ 164.735356] ide4: reset: success May 30 15:21:23 Tower kernel: [ 165.516101] md0: parity incorrect: 3344 May 30 15:21:43 Tower kernel: [ 185.494215] hdj: dma_timer_expiry: dma status = = 0x41 ---------------------------------------------------------------------------------------------- hdj is not looking the happy camper right now. Unfortunately I'm remote right now so I'll recheck all cabling when I get home in a couple of hours.
May 30, 200719 yr IT life hasnt changed much the last 20 years. 1. Reboot, if not enough then.. 2. shut down, check and reseat cards, ram and reseat cabling, start up. If not enough.. 3. sleep on it and think during showering,, if no solution then.. 4. ask someone /Rene
May 30, 200719 yr Life has changed a bit, in between 3 and 4 ... 3a - google the problem and/or error message, if not enough ... Cheers, Bill
May 30, 200719 yr Author Well I reseated everything (Drive connectors, IDE cables, Power, Ram), and while I was in there I noticed I had changed the last three drives (9,10,11 and 10=hdj) the last time I was in there to all be driven off of 1 power supply output (with a 1:3 splitter). I don't know, but that may be a bad idea (especially with the POS leftover Dell 5 year old 200W power supply I use as the second [CoolerMaster Stacker case] for the last 4 drives in my 12 drive Array). I reconfigured those drives so there were only 2/output. I don't know if that's the case (I also have an IDE cable going to that drive and it's slave that is a real Streeeeetch to reach). After that she has booted straight up lickety split (as taking minutes this afternoon and hanging with lots in the syslog about hdj and IDE resets) and seems to be off to the races checking parity (fingers crossed). My parity checking has always been somewhat slow, starting at ~13,000 KB/Sec (projected 600 Mins) with 11 data drives and then cranking up to about 45,000KB/s when it drops down to 4 data drives at the end (7 of the drives are <=250GB, 1 data drive and parity are 500GB). Is this a normal data rate (at the beginning)? Thanks for the help and good intentions guys.
May 30, 200719 yr Author So out of curiosity, I ran a 'hdparm -tT' on each drive in the array to check performance (due to my 10 hour parity checks) results: ---------------------------------------------------------------------------------- /dev/hda: Timing cached reads: 2524 MB in 2.00 seconds = 1262.55 MB/sec Timing buffered disk reads: 178 MB in 3.00 seconds = 59.24 MB/sec /dev/hdc: Timing cached reads: 2524 MB in 2.00 seconds = 1262.07 MB/sec Timing buffered disk reads: 18 MB in 3.32 seconds = 5.43 MB/sec /dev/hdd: Timing cached reads: 2488 MB in 2.00 seconds = 1243.70 MB/sec Timing buffered disk reads: 140 MB in 3.03 seconds = 46.14 MB/sec /dev/hde: Timing cached reads: 2524 MB in 2.00 seconds = 1262.43 MB/sec Timing buffered disk reads: 202 MB in 3.02 seconds = 66.95 MB/sec /dev/hdf: Timing cached reads: 2500 MB in 2.00 seconds = 1250.09 MB/sec Timing buffered disk reads: 180 MB in 3.02 seconds = 59.55 MB/sec /dev/hdg: Timing cached reads: 2516 MB in 2.00 seconds = 1258.34 MB/sec Timing buffered disk reads: 190 MB in 3.03 seconds = 62.76 MB/sec /dev/hdh: Timing cached reads: 2508 MB in 2.00 seconds = 1253.16 MB/sec Timing buffered disk reads: 168 MB in 3.01 seconds = 55.90 MB/sec /dev/hdi: Timing cached reads: 2500 MB in 2.00 seconds = 1250.26 MB/sec Timing buffered disk reads: 176 MB in 3.01 seconds = 58.51 MB/sec /dev/hdj: Timing cached reads: 2512 MB in 2.00 seconds = 1256.04 MB/sec Timing buffered disk reads: 196 MB in 3.01 seconds = 65.12 MB/sec /dev/hdk: Timing cached reads: 2508 MB in 2.00 seconds = 1253.52 MB/sec Timing buffered disk reads: 224 MB in 3.00 seconds = 74.56 MB/sec /dev/sda: Timing cached reads: 2512 MB in 2.00 seconds = 1256.88 MB/sec Timing buffered disk reads: 164 MB in 3.02 seconds = 54.28 MB/sec /dev/sdb: Timing cached reads: 2508 MB in 2.00 seconds = 1254.40 MB/sec Timing buffered disk reads: 176 MB in 3.01 seconds = 58.51 MB/sec -------------------------------------------------------------- I checked hdc 3 more time, always ~5MB/s. I Imagine that is what's slowing my parity check down. Anyone know how I can further look into and test the performance on this drive (A Seagate ST3300831A)?
May 31, 200719 yr Author More info on my slow /dev/hdc: root@Tower:~# hdparm -i /dev/hdc /dev/hdc: Model=ST3300831A, FwRev=2.04, SerialNo=3NF00PH6 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4 BuffType=unknown, BuffSize=8192kB, MaxMultSect=16, MultSect=16 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 AdvancedPM=no WriteCache=enabled Drive conforms to: ATA/ATAPI-6 T13 1410D revision 2: ATA/ATAPI-1 ATA/ATAPI-2 TA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 * signifies the current active mode
May 31, 200719 yr Couple observations: Do not try to use a "POS leftover Dell 5 year old 200W power supply" for any of your drives. I'd say the #1 cause (by far) of server instability is caused by bad power supply. Also, try to use only SINGLE RAIL power supplies, since multi-rail will limit how much current is available to the peripheral +12V connectors. The /dev/hdc disk is running in non-DMA mode. This is not good & will definitely cause parity sync to crawl. Post back with a fresh syslog & I'll see if it gives any indication why this is happening.
May 31, 200719 yr Author OK, I'll splurge and send the Dell to the recycling Bin, I Promise (hey it worked for a year ;-) Although I have a quality PS for the Mobo and first 8 drives (Antec 450W) I suspect it is too much to run 12 drives off of it. Full syslog from boot attached any pointers on the non-DMA much appreciated ETA: syslog re-upped (via FTP using ASC to get rid of annoying CR/LF conversion)
May 31, 200719 yr Author Tom, if it helps, /dev/hdc is the Master drive on the secondary IDE port of the motherboard (our old friend Intel D865GLCLK, of course ) It's slave (/dev/hdd) is a WDC WD2500JB
May 31, 200719 yr Author Any tips on what to look for on that non-DMA drive would be appreciated. Are there Bios settings I should review that affect that (My unraid is headless so I have to cart it out and hook up a monitor/keyboard to look at those). I don't think Parity took this long under 3.X
June 1, 200719 yr Any tips on what to look for on that non-DMA drive would be appreciated. Are there Bios settings I should review that affect that (My unraid is headless so I have to cart it out and hook up a monitor/keyboard to look at those). I don't think Parity took this long under 3.X Most likely is the drive cable. Make sure it is a new style 80 conductor cable and not an old 40 conductor one. (80 conductor cables usually have a blue connector, a grey connector, and a black connector) Make sure the blue connector is the one connected to the disk controller card/motherboard. make sure the cable is as short as possible and is not a "round" cable. Joe L.
June 1, 200719 yr Author Thanks for the tips Joe. It is indeed a round cable (I know, I shouldn't, but they've been working fine). I'll swap it out with a quality flat cable (80-pin) and see if that makes a difference. On the topic, what's a good source of longer quality flat cables (would Fry's have them?). And wasn't there a photo of a very clear unRAID install with nicely routed 45 degree bent flast cables? The inside of my unRAID with 12 drives is starting to be quite the tangle.
June 1, 200719 yr Do not try to use a "POS leftover Dell 5 year old 200W power supply" for any of your drives.. 200W should be enough to run 4 drives. However I would also recommend a newer Active PFC power supply anyway. I'd say the #1 cause (by far) of server instability is caused by bad power supply. Also, try to use only SINGLE RAIL power supplies, since multi-rail will limit how much current is available to the peripheral +12V connectors. Yes, but cables play a very high role when people use P-ATA Drives. Especially with long cables and when they are mixed with MASTER/SLAVE of different vendors. I've always had difficulty with WD and Maxtor Drives when when the WD drivers were Slave or Cable select. On the machine I had issues with, I reversed the master/slave and the problem went away. This was with 2 different motherboards and Long Sheilded Round Cables. I've never had issues with the round cables, yet I've always purchased quality/brand name shielded cables for anything over 18".
June 1, 200719 yr Thanks for the tips Joe. It is indeed a round cable (I know, I shouldn't, but they've been working fine). I'll swap it out with a quality flat cable (80-pin) and see if that makes a difference. On the topic, what's a good source of longer quality flat cables (would Fry's have them?). And wasn't there a photo of a very clear unRAID install with nicely routed 45 degree bent flat cables? The inside of my unRAID with 12 drives is starting to be quite the tangle. You are probably thinking of the photo I took of the server Tom built for me in Oct, 2005. (excuse the quality, it was my cell phone camera) Construction is impressive. 12 drive trays and a very neat cabling job with flat cables. Joe L.
June 1, 200719 yr Nothing in the syslog looks out of the ordinary. There's an outside chance the slave drive on that cable (the Western Digital) could be causing problems as well. Make sure the drives are jumpered as "Cable Select". Maybe move one of the drives to a different cable, e.g., as the slave on your parity cable.
June 8, 200719 yr Couple observations: Do not try to use a "POS leftover Dell 5 year old 200W power supply" for any of your drives. I'd say the #1 cause (by far) of server instability is caused by bad power supply. Also, try to use only SINGLE RAIL power supplies, since multi-rail will limit how much current is available to the peripheral +12V connectors. Thought I would add some more info here which may help elaborate on the split vs single rail. http://www.pcpower.com/technology/myths/
Archived
This topic is now archived and is closed to further replies.