December 12, 200817 yr Just to make some of the new users of unRAID feel like they are not the only ones who occasionally run into hardware problems. I've been having problems with the second of my IDE controllers... Basically, accessing one of the disks causes errors in the system log. The problem is that depending on the drive and connector I use, various disks work, or not. Fortunately, this controller card is connected to unused drive trays. In troubleshooting I've managed to fit all my data on the other drives in my server. The last 4 drive trays in my server are either empty, or the disk in them not assigned to the unRAID array on the devices page. Currently I have one disk on each of the two cables on this second Promise controller. They are /dev/hdk, and /dev/hdi. Both are set as "cable-select." Both are on the "end" connector" on their respective IDE cable. I'm in the process of pre-clearing the disk on /dev/hdk, so it is getting a lot of very heavy I/O activity as it is being written to. The other disk, /dev/hdi is only being accessed when I issue an hdparm or smartctl command. Neither disk is assigned in my devices page to my array... (and boy, am I glad they are not) Every time I issue an hdparm command, or a smartctl command, the syslog errors occur. This same drive, in the same spot, worked earlier in the day... but apparently, once it goes off-line, it is gone. It would probably work again if I reboot, but I'll try that when I get a new IDE cable (my most likely suspect). The really good news is that this same error has caused my server to lock up when I was running the earlier version of unRAID. I upgraded to 4.4final today, and this error puts messages in the system log, but the server keeps running. I like that much better than a total system freeze. Early failures on unraid 4.3.3 looked like this, notice both disks on the same cable were affected: Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:44:39 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:44:39 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:44:39 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:44:39 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:44:39 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:44:39 Tower kernel: ide: failed opcode was: 0xea Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: md: disk8 replaced Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xea Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xea Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower emhttp: shcmd (12): killall -w smbd nmbd Dec 5 08:45:35 Tower emhttp: shcmd (13): /usr/sbin/nmbd -D Dec 5 08:45:35 Tower emhttp: shcmd (14): /usr/sbin/smbd -D Dec 5 08:45:36 Tower kernel: md: disk8 replaced Dec 5 08:45:55 Tower kernel: hdi: dma_timer_expiry: dma status == 0x61 <--- HARD FREEZE OCCURRED HERE A more recent failure on 4.3.3 was here (best I could do is capture the tail of the syslog, as it froze up hard too) Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:44:39 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:44:39 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:44:39 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:44:39 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:44:39 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:44:39 Tower kernel: ide: failed opcode was: 0xea Dec 5 08:44:39 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: md: disk8 replaced Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xea Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xb0 Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower kernel: hdj: status error: status=0x00 { } Dec 5 08:45:34 Tower kernel: ide: failed opcode was: 0xea Dec 5 08:45:34 Tower kernel: hdj: drive not ready for command Dec 5 08:45:34 Tower emhttp: shcmd (12): killall -w smbd nmbd Dec 5 08:45:35 Tower emhttp: shcmd (13): /usr/sbin/nmbd -D Dec 5 08:45:35 Tower emhttp: shcmd (14): /usr/sbin/smbd -D Dec 5 08:45:36 Tower kernel: md: disk8 replaced Dec 5 08:45:55 Tower kernel: hdi: dma_timer_expiry: dma status == 0x61 <---- HARD FREEZE OCCURRED HERE I've attached a zipped copy of my syslog for RobJ's enjoyment. Perhaps he'll have some ideas for me. So far, I've had these errors with different drives in different configurations (positions) on the cables. I wanted to get out to the store to get a pair of longer IDE cables to use as possible replacements, but did not get the chance today. That will be my next step. Oh yes, I don't suspect the power supply, at least not yet, as my server is an original MD-1200 built by Tom at Lime-Technology in October 2005, and at that time he supplied the server with two power supplies. Yup, each supply only has half of the disks. Edit: This problem was not the IDE cables, but instead turned out to be two "Y" power splitters, both have then same problem, both have poor quality "crimps" on their connectors causing them to open intermittently. Both were probably manufactured by the same company. One was on the lower drive trays I had not previously used. The other was on the drive trays I was using... Apparently it has worked for 3 years, but with all my moving of cables while diagnosing the intermittent problem on the lower trays, it showed itself. The reason this problem was so elusive was that after the brief power loss to a drive occurred, it then failed a subsequent I/O request, even if power was restored. There were no "clues" in the syslog, other than the error occurred when accessing one of the disks, and it appears as if that changed randomly based on which disk was accessed next and the type of access. One thing for sure... the "Drive not ready for command" error in the syslog was accurate. (hard to be ready if you have lost power ) This whole problem surfaced because I had just started to fill the drive trays in the lower part of my server and use them heavily while I developed the preclear_disk script. Up until then, I had not used those trays (or those "Y" power splitters feeding them) as nothing was ever plugged in them. Joe L.
December 12, 200817 yr It would probably work again if I reboot, but I'll try that when I get a new IDE cable (my most likely suspect). I might suggest disconnecting any unused segments of the IDE cable first. I.E. if you are using the END connector and the middle connector is connected to a tray, but is unused, I would disconnect it from the tray. I've had issues in the past whereby extraneous dangling connectors to unused trays caused drives to go offline like this. What happens if you put the two drives on the same cable on the controller?
December 12, 200817 yr Author It would probably work again if I reboot, but I'll try that when I get a new IDE cable (my most likely suspect). I might suggest disconnecting any unused segments of the IDE cable first. I.E. if you are using the END connector and the middle connector is connected to a tray, but is unused, I would disconnect it from the tray. I've had issues in the past whereby extraneous dangling connectors to unused trays caused drives to go offline like this. The middle connectors are indeed connected to their drive trays. I can try it with them unplugged and see if there is a difference. That's not a bad idea if I don't want to use the other trays, just a bit less convenient when I go to add a drive. What happens if you put the two drives on the same cable on the controller? I did that earlier, I put both drives one of the cables and left the other cable unused... pretty much the same errors occurred. In fact, at that point, if I put anything in the tray connected to the middle connector it managed to freeze the system with a DMA error. At least with unraid 4.4 and the newer OS and drivers it does not freeze, it just gives errors. Right now the small 8Gig drive is completely un-responsive... root@Tower:/boot# dd if=/dev/hdi count=1 | od -x -A d dd: reading `/dev/hdi': Input/output error 0+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000997385 s, 0.0 kB/s 0000000 root@Tower:/boot# tail /var/log/syslog Dec 12 12:01:29 Tower kernel: __ratelimit: 47 callbacks suppressed Dec 12 12:01:29 Tower kernel: Buffer I/O error on device hdi, logical block 0 Dec 12 12:01:29 Tower kernel: end_request: I/O error, dev hdi, sector 8 Dec 12 12:01:29 Tower kernel: Buffer I/O error on device hdi, logical block 1 Dec 12 12:01:29 Tower kernel: end_request: I/O error, dev hdi, sector 16 Dec 12 12:01:29 Tower kernel: Buffer I/O error on device hdi, logical block 2 Dec 12 12:01:29 Tower kernel: end_request: I/O error, dev hdi, sector 24 Dec 12 12:01:29 Tower kernel: Buffer I/O error on device hdi, logical block 3 Dec 12 12:01:29 Tower kernel: end_request: I/O error, dev hdi, sector 0 Dec 12 12:01:29 Tower kernel: Buffer I/O error on device hdi, logical block 0 The larger 750 Gig drive on the other cable seems to work fine. (although each time I go through a pre-read/zero/post-read preclearing process on it it reports two more "Hardware_ECC_Recovered" errors. It takes 10 hours to do a complete cycle of read/write/read again on it, during which I really don't want to stop it to try a different configuration.) Joe L.
December 12, 200817 yr Any DMA CRC errors in the SMARTLOG ? Have you done a complete and total power off, not just a soft off, but no power what so ever to the motherboard for a few moments?
December 12, 200817 yr Author Any DMA CRC errors in the SMARTLOG ? Smart report of the 750 Gig drive looks like this... (I just finished running a "long" test, it passed without errors.) [pre] Model Family: Seagate Barracuda 7200.10 family Device Model: ST3750640A Serial Number: 5QD29FXK Firmware Version: 3.AAE User Capacity: 750,156,374,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri Dec 12 15:02:06 2008 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 108 095 006 Pre-fail Always - 71187847 3 Spin_Up_Time 0x0003 096 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 485 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 078 060 030 Pre-fail Always - 60218236 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1294 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 163 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 065 031 045 Old_age Always In_the_past 35 (Lifetime Min/Max 34/38) 194 Temperature_Celsius 0x0022 035 069 000 Old_age Always - 35 (0 24 0 0) 195 Hardware_ECC_Recovered 0x001a 064 055 000 Old_age Always - 9701902 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 1293 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. [/pre] Have you done a complete and total power off, not just a soft off, but no power what so ever to the motherboard for a few moments? I always do a complete power down when I work on any hardware, I have no "power button" so I cannot start back up unless I completely power cycle. I had no luck in getting the longer IDE cables at the local computer store... he had 18" and 36"... I wanted 22 or 24"" I'm going to swap the cables on the controller if I cannot find any. In the mean-time, I'm going to unplug the one cable with the completely failing 8Gig drive, and physically move the drive to the other cable... I'm also going to replace the CMOS battery. Last Time I re-seated everything the %#@D%#R& motherboard bios reset itself to boot off of the first hard disk... In fact, I think it reset itself completely, as the USB option was also back to the slower legacy rate. Now, it might have been me, and I might have vibrated the motherboard enough to cause the battery to lose contact, or it might just be the battery getting week. It will be almost 3 1/2 years since I purchased the server, and who knows how old the battery was when it was manufactured. It sure can't hurt. Joe L.
December 12, 200817 yr I might have vibrated the motherboard enough to cause the battery to lose contact, or it might just be the battery getting week. It will be almost 3 1/2 years since I purchased the server, and who knows how old the battery was when it was manufactured. It sure can't hurt. I've never had a motherboard where the battery was not in place strong enough that a vibration lost CMOS.. Chances are it's the battery weakening, I've had this a number of times and 3 or so years is about right for the button batteries. The smartlog looks clean. I've seen issues where there is hardware problems with drives from multiple vendors on the same cable. Sometimes setting one to master and the other to slave alleviates this. Neither of these should lock up the system. Again, unterminated ends (Past the cable) can cause problems. I've seen this a number of times. (I've had lockups with it too).
December 12, 200817 yr Are you using cable select for master/slave? If so you might try setting the jumpers on the disks. I'd also recommend connecting your drives directly to the MB (bypassing the drive trays completely) and using the shorter ribbon cables that we all know and hate.
December 12, 200817 yr Author Are you using cable select for master/slave? If so you might try setting the jumpers on the disks. Yes, I am... it has not been an issue in the past, and it does make it way to move a tray from one slot to another. At least for now I'll leave them as is. I'd also recommend connecting your drives directly to the MB (bypassing the drive trays completely) and using the shorter ribbon cables that we all know and hate. One tiny step at a time... When I went to stop the array, it just froze up No errors, no nothing. I used the power switch to shut down. The "activity" lights were on on two of the drives on the OTHER PCI controller card, so odds are one of those could have been confused as they were being spun up... who knows. In any case, I left the larger 750Gig drive in place, on the end of its cable, in the drive tray. The CMOS battery is replaced. I also plugged in a new SATA controller into a spare PCI slot. I wanted to see if it would be recognized, and it is. There are no drives on it at this time, but I do have a 750Gig SATA drive I can try there once things are back stable. I powered up, and it went into a parity check. Since I know the array was completely idle, I stopped it, and reset the config to where it had safely been stopped. I could then re-start everything. I'm going to stress test the 750Gig drive again... If I start it now, and have it do 2 cycles, it should finish tomorrow about noon. To sum up, the suspected IDE cable is disconnected from the controller, and the port on the controller is not connected to anything. Joe L.
December 13, 200817 yr Author I'm about an hour into pre-reading the 750 Gig drive on /dev/hdk. It has read about one third of the drive (250 Gig) so far, and no errors of any kind in the syslog. One thing for sure... when I finally get this problem resolved, this drive will have been burnt-in. Joe L.
December 13, 200817 yr Are you using cable select for master/slave? If so you might try setting the jumpers on the disks. Yes, I am... it has not been an issue in the past, and it does make it way to move a tray from one slot to another. At least for now I'll leave them as is. FWIW, I've had issues in the past where it required the settings to be specific. Drives of different ages can cause this type of issue. However, you are locking up. This is not just a drive master/slave relationship problem. The hardware should not freeze. There's an IRQ issue, a DMA issue, Communication Issue (cable), or a driver issue.
December 13, 200817 yr Author FWIW, I've had issues in the past where it required the settings to be specific. Drives of different ages can cause this type of issue. I'll keep that in mind..., but the drives are being identified on the cable, and in my past experience, if the settings for master/slave were bad, one or both would not work at all, not freeze after a number of hours. However, you are locking up. This is not just a drive master/slave relationship problem. The hardware should not freeze. There's an IRQ issue, a DMA issue, Communication Issue (cable), or a driver issue. My thoughts exactly... Since the hardware has been the same, and the cables have worked in the past, it is just more of a mystery. Before all the errors started I had three of the four ports on that controller card hooked up to disks. I managed to migrate all my data to the other disks in my array (in between errors) and reset my configuration so I still had parity protection. Now, I just need to figure out what has failed, so I can start using those 4 drive trays as part of the array. If the pre-clear of /dev/hdk works, I'll populate the second tray on the same cable and see how it works. I've still got about 330Gig of free space on my protected data drives, so I can still add about 50 more DVD ISO images... That will certainly hold me for a few months, at least.... Joe L.
December 14, 200817 yr Author I still have the one IDE cable disconnected from the controller card. I've been using my preclear_disk script to read and write for hours to the one disk connected to the second cable. So far, no errors from it at all. However... my server did lock up once more the other night... here are the last three lines from the "tail -f syslog" I had running in a telnet session. Dec 13 11:08:29 Tower kernel: hdk: hdk1 Dec 13 11:08:39 Tower kernel: udev: starting version 130 Dec 13 11:11:31 Tower kernel: hdh: dma_timer_expiry: dma status == 0x61 Yes... you are reading it correctly... At 11:08 I completed pre-clearing /dev/hdk and restarted the "udev" daemon to create appropriate /dev/disk/by* entries for it. About three minutes later, I get a DMA timer error on /dev/hdh and the server freezes. (This is a disk on the other Promise IDE Controller card.) I did re-start the server, and a parity check worked just fine. I'm beginning to think this might not be as simple as a bad cable. Joe L.
December 14, 200817 yr Cable select needs a cable with one of the lines punched out between the 1st and second connector. Fraught with peril IMHO.
December 14, 200817 yr Author Cable select needs a cable with one of the lines punched out between the 1st and second connector. Fraught with peril IMHO. Interesting... I did not know that. Now I'll need to check how the other trays are configured. I have no issue with setting the jumpers specifically... It would be nice if it was as simple as that. Joe L.
December 14, 200817 yr Author Cable select needs a cable with one of the lines punched out between the 1st and second connector. Fraught with peril IMHO. Reading a tiny bit... Quote: Cable Select Support and Drive Assignment: All 80-conductor cables that meet the ATA specifications support the cable select feature automatically. This is accomplished by special connection of the CSEL signal on pin #28. The cable can still be used with drives that have been manually configured as master or slave, of course. See the discussion of cable select for more. Based on this information, the 80 conductor cables in my server should be ok. But I'll still take a look at how they are all configured the next time I power down. Joe L.
Archived
This topic is now archived and is closed to further replies.