September 29, 200619 yr I spoke too soon. The problems are back. Sigh. Interesting observation: If I stop the array, mount my external drive as /mnt/sdb1 and mount /dev/hde1 as /mnt/disk8, then everything appears fine. If I do this with the array started, I get the busy timeouts on hde and hdf (which are /dev/md8 and /dev/md9), even if I have no parity drive specified. I wonder what's up with that? Could this be a problem with the device assignment code or something?
September 29, 200619 yr Maybe interesting. Repeat for me this info: 1. unRAID Server beta currently running. 2. system config: m/b, controllers 3. device assignments 4. Given this config, sequence of steps you did to make the timeouts start happening. thanks!
September 29, 200619 yr Author Maybe interesting. Repeat for me this info: 1. unRAID Server beta currently running. 2. system config: m/b, controllers 3. device assignments 4. Given this config, sequence of steps you did to make the timeouts start happening. 1. Beta 4. 2. Your standard config - Intel mobo, promise controllers flashed to latest revision. 3. Parity unassigned, then hdb, hdc, hdd, hdi, hdj, hdk, hdl, hde, hdf, hdg, hdh. These are almost all Seagate 400gb drives - hdc and hdi are hitachi 400s, and hdl is a Seagate 300. 4. I'm trying to copy the contents of an old drive over to the new array. I attach it with an external USB dongle, so it shows up as /dev/sdb1. Then I mount that as /mnt/sdb1, cd to that directory, and do 'cp -ruv DVD/ /mnt/disk8' with the array started. I almost immediately get the busy error and DMA turns off. If I stop the array and manually mount /dev/hde1 /mnt/disk8, I can do the same command without error, and DMA stays on.
September 29, 200619 yr Ok thanks! I should be able to reproduce this - will have to wait until tomorrow.
September 29, 200619 yr Author Ok thanks! I should be able to reproduce this - will have to wait until tomorrow. I just hit the DMA errors on a manually mounted drive, so this may have been a red herring.
September 29, 200619 yr Author Ok thanks! I should be able to reproduce this - will have to wait until tomorrow. I just hit the DMA errors on a manually mounted drive, so this may have been a red herring. Just to further randomize things - when I'm doing an rsync to copy onto a drive that's empty, I get no errors. When I do an rsync to overwrite files on a drive that was already populated using cp, I get the DMA errors. After this current rsync completes, I'll clear the disk that's currently giving me problems and try again. This is just weird.
September 29, 200619 yr Ok, something I'd like you to try. First, capture some current information from the hard drives: hdparm -I /dev/hdb hdparm -I /dev/hdc hdparm -I /dev/hdd hdparm -I /dev/hde hdparm -I /dev/hdf hdparm -I /dev/hdg hdparm -I /dev/hdh hdparm -I /dev/hdi hdparm -I /dev/hdj hdparm -I /dev/hdk hdparm -I /dev/hdl This will produce quite a bit of output, also note the option is -I (uppercase), NOT -i (lowercase). Next, type this series of commands: hdparm -u1 /dev/hdb hdparm -u1 /dev/hdc hdparm -u1 /dev/hdd hdparm -u1 /dev/hde hdparm -u1 /dev/hdf hdparm -u1 /dev/hdg hdparm -u1 /dev/hdh hdparm -u1 /dev/hdi hdparm -u1 /dev/hdj hdparm -u1 /dev/hdk hdparm -u1 /dev/hdl Now do some transfers and tell me if the lost interrupt errors occur.
September 30, 200619 yr Author Same result, Tom, it almost immediately errors out, even after the -u1 thing. Here's the result of some of the hdparm -I's, I can't post the whole thing because the board has a 20K limit. /dev/hdb: ATA device, with non-removable media Model Number: ST3400832A Serial Number: 5NF0746G Firmware Revision: 3.03 Standards: Supported: 7 6 5 4 Likely used: 7 Configuration: Logical max current cylinders 16383 65535 heads 16 1 sectors/track 63 63 -- CHS current addressable sectors: 4128705 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 781422768 device size with M = 1024*1024: 381554 MBytes device size with M = 1000*1000: 400088 MBytes (400 GB) Capabilities: LBA, IORDY(can be disabled) Queue depth: 1 Standby timer values: spec'd by Standard, no device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Recommended acoustic management value: 128, current value: 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=240ns IORDY flow control=120ns Commands/features: Enabled Supported: * READ BUFFER cmd * WRITE BUFFER cmd * Host Protected Area feature set * Look-ahead * Write cache * Power Management feature set Security Mode feature set * SMART feature set * FLUSH CACHE EXT command * Mandatory FLUSH CACHE command * Device Configuration Overlay feature set * 48-bit Address feature set SET MAX security extension * DOWNLOAD MICROCODE cmd * General Purpose Logging feature set * SMART self-test * SMART error logging Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count not supported: enhanced erase HW reset results: CBLID- above Vih Device num = 1 determined by CSEL Checksum: correct /dev/hdc: ATA device, with non-removable media powers-up in standby; SET FEATURES subcmd spins-up. Model Number: HDS724040KLAT80 Serial Number: KRFA06RAH9E34D Firmware Revision: KFAOAC6A Standards: Supported: 7 6 5 4 Likely used: 7 Configuration: Logical max current cylinders 16383 65535 heads 16 1 sectors/track 63 63 -- CHS current addressable sectors: 4128705 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 781422768 device size with M = 1024*1024: 381554 MBytes device size with M = 1000*1000: 400088 MBytes (400 GB) Capabilities: LBA, IORDY(can be disabled) Queue depth: 32 Standby timer values: spec'd by Standard, no device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Advanced power management level: unknown setting (0x0000) Recommended acoustic management value: 128, current value: 254 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 udma6 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=240ns IORDY flow control=120ns Commands/features: Enabled Supported: * NOP cmd * READ BUFFER cmd * WRITE BUFFER cmd * Host Protected Area feature set * Release interrupt * Look-ahead * Write cache * Power Management feature set Security Mode feature set * SMART feature set * FLUSH CACHE EXT command * Mandatory FLUSH CACHE command * Device Configuration Overlay feature set * 48-bit Address feature set Automatic Acoustic Management feature set SET MAX security extension Address Offset Reserved Area Boot SET FEATURES subcommand required to spinup after power up Power-Up In Standby feature set Advanced Power Management feature set * READ/WRITE DMA QUEUED * DOWNLOAD MICROCODE cmd * General Purpose Logging feature set * SMART self-test * SMART error logging Security: Master password revision code = 65534 supported not enabled not locked frozen not expired: security count not supported: enhanced erase 220min for SECURITY ERASE UNIT. HW reset results: CBLID- above Vih Device num = 0 determined by CSEL Checksum: correct /dev/hdd: ATA device, with non-removable media Model Number: ST3400832A Serial Number: 5NF0SHY1 Firmware Revision: 3.03 Standards: Supported: 7 6 5 4 Likely used: 7 Configuration: Logical max current cylinders 16383 65535 heads 16 1 sectors/track 63 63 -- CHS current addressable sectors: 4128705 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 781422768 device size with M = 1024*1024: 381554 MBytes device size with M = 1000*1000: 400088 MBytes (400 GB) Capabilities: LBA, IORDY(can be disabled) Queue depth: 1 Standby timer values: spec'd by Standard, no device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Recommended acoustic management value: 128, current value: 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=240ns IORDY flow control=120ns Commands/features: Enabled Supported: * READ BUFFER cmd * WRITE BUFFER cmd * Host Protected Area feature set * Look-ahead * Write cache * Power Management feature set Security Mode feature set * SMART feature set * FLUSH CACHE EXT command * Mandatory FLUSH CACHE command * Device Configuration Overlay feature set * 48-bit Address feature set SET MAX security extension * DOWNLOAD MICROCODE cmd * General Purpose Logging feature set * SMART self-test * SMART error logging Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count not supported: enhanced erase HW reset results: CBLID- above Vih Device num = 1 determined by CSEL Checksum: correct /dev/hde: ATA device, with non-removable media Model Number: ST3400620A Serial Number: 5QHZ0HQH Firmware Revision: 3.AAC Standards: Supported: 7 6 5 4 Likely used: 7 Configuration: Logical max current cylinders 16383 65535 heads 16 1 sectors/track 63 63 -- CHS current addressable sectors: 4128705 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 781422768 device size with M = 1024*1024: 381554 MBytes device size with M = 1000*1000: 400088 MBytes (400 GB) Capabilities: LBA, IORDY(can be disabled) Queue depth: 1 Standby timer values: spec'd by Standard, no device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Recommended acoustic management value: 208, current value: 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=240ns IORDY flow control=120ns Commands/features: Enabled Supported: * READ BUFFER cmd * WRITE BUFFER cmd * Host Protected Area feature set * Look-ahead * Write cache * Power Management feature set Security Mode feature set * SMART feature set * FLUSH CACHE EXT command * Mandatory FLUSH CACHE command * Device Configuration Overlay feature set * 48-bit Address feature set SET MAX security extension * DOWNLOAD MICROCODE cmd * General Purpose Logging feature set * SMART self-test * SMART error logging Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count not supported: enhanced erase HW reset results: CBLID- above Vih Device num = 0 determined by CSEL Checksum: correct /dev/hdf: ATA device, with non-removable media Model Number: ST3400620A Serial Number: 5QH00CVJ Firmware Revision: 3.AAC Standards: Supported: 7 6 5 4 Likely used: 7 Configuration: Logical max current cylinders 16383 65535 heads 16 1 sectors/track 63 63 -- CHS current addressable sectors: 4128705 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 781422768 device size with M = 1024*1024: 381554 MBytes device size with M = 1000*1000: 400088 MBytes (400 GB) Capabilities: LBA, IORDY(can be disabled) Queue depth: 1 Standby timer values: spec'd by Standard, no device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Recommended acoustic management value: 208, current value: 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=240ns IORDY flow control=120ns Commands/features: Enabled Supported: * READ BUFFER cmd * WRITE BUFFER cmd * Host Protected Area feature set * Look-ahead * Write cache * Power Management feature set Security Mode feature set * SMART feature set * FLUSH CACHE EXT command * Mandatory FLUSH CACHE command * Device Configuration Overlay feature set * 48-bit Address feature set SET MAX security extension * DOWNLOAD MICROCODE cmd * General Purpose Logging feature set * SMART self-test * SMART error logging Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count not supported: enhanced erase HW reset results: CBLID- above Vih Device num = 1 determined by CSEL Checksum: correct
September 30, 200619 yr You can email the log to me at [email protected]. BTW should type these commands AFTER the dma error has occurred. To update you on this: the "hdparm -u1" commands were a bit of a "shot in the dark" because we have not been able to reproduce this problem. We have a couple full-up IDE test systems. Copying from a couple different external USB hard drives to any array disk, array stopped or not, parity valid or not, does not produce any DMA error - no errors of any kind. These systems use Promise Ultra100 controllers & have 2.0GHz Celerons (not CeleronD's). There's another system being built with Ultra133's and a 2.4GHz Pentium4 & I'll retest with it tomorrow or Monday. Also, external drive has reiserfs (linux) file system - didn't try any others (not that it should make a difference, but how is your usb drive formatted? fat or ntfs?). In your particular case, since you're in a mode where you're loading data and not worrying about parity, here is how I would approach this problem. It's especially "good" that you can get it to happen "almost immediately" because that means you should be able to isolate the cause. First, power off all the drives except disk8 & see if it still fails. If it does NOT fail, starting adding drives one-by-one until it does. If you get to a point where adding a particular drive makes it fail, then power off that drive and power on a different one to see it it fails. [Trying to see if this is a power issue]. If it does fail with having only disk8 powered on, then move disk8 to a different controller and see if still fails [Trying to isolate the controller]. I think you get the idea. Sorry, I know this is a P.I.T.A.
September 30, 200619 yr Author Is there any danger in going back to the released build from the beta? I thought I'd give that a shot and see if the error still repros.
September 30, 200619 yr Author Is there any danger in going back to the released build from the beta? I thought I'd give that a shot and see if the error still repros. Well, I tried it. Even with the array up and running, I get no DMA errors on the Unraid 2.060706. I also tried running with Beta 1, and I did get the errors there. So something's changed for the worse with the advent of 3.0.
September 30, 200619 yr The pre-3.0 code has these lines in the 'go' script: hdparm -c1d1a0m8A1W1u1 /dev/hda hdparm -c1d1a0m8A1W1u1 /dev/hdb hdparm -c1d1a0m8A1W1u1 /dev/hdc hdparm -c1d1a0m8A1W1u1 /dev/hdd hdparm -c1d1a0m8A1W1u1 /dev/hdi hdparm -c1d1a0m8A1W1u1 /dev/hdj hdparm -c1d1a0m8A1W1u1 /dev/hdk hdparm -c1d1a0m8A1W1u1 /dev/hdl hdparm -c1d1a0m8A1W1u1 /dev/hde hdparm -c1d1a0m8A1W1u1 /dev/hdf hdparm -c1d1a0m8A1W1u1 /dev/hdg hdparm -c1d1a0m8A1W1u1 /dev/hdh You can put them back in your 'go' script in the config directory of the 3.0-beta release like this: #!/bin/bash hdparm -c1d1a0m8A1W1u1 /dev/hda hdparm -c1d1a0m8A1W1u1 /dev/hdb hdparm -c1d1a0m8A1W1u1 /dev/hdc hdparm -c1d1a0m8A1W1u1 /dev/hdd hdparm -c1d1a0m8A1W1u1 /dev/hdi hdparm -c1d1a0m8A1W1u1 /dev/hdj hdparm -c1d1a0m8A1W1u1 /dev/hdk hdparm -c1d1a0m8A1W1u1 /dev/hdl hdparm -c1d1a0m8A1W1u1 /dev/hde hdparm -c1d1a0m8A1W1u1 /dev/hdf hdparm -c1d1a0m8A1W1u1 /dev/hdg hdparm -c1d1a0m8A1W1u1 /dev/hdh # # Start the Management Utility /usr/local/sbin/emhttp & Most of these settings are redundant, some, turns out don't have any effect, others don't seem to have any effect. If you could try this, I'd appreciate it. Thanks for your patience...
September 30, 200619 yr To pinpoint the config, m/b = D865GLCLK ? cpu = CeleronD ? what freq? mem = 512MB? pc2100? pci ctlrs = promise ultra100 or ultra133?
September 30, 200619 yr Author To pinpoint the config, m/b = D865GLCLK ? cpu = CeleronD ? what freq? mem = 512MB? pc2100? pci ctlrs = promise ultra100 or ultra133? d865GLCLK Celeron D 2.66 512MB PC 2100 I'm not sure if the promise controllers are 100 or 133, is there a way to tell without cracking the case?
September 30, 200619 yr Author The pre-3.0 code has these lines in the 'go' script: hdparm -c1d1a0m8A1W1u1 /dev/hda ... Most of these settings are redundant, some, turns out don't have any effect, others don't seem to have any effect. If you could try this, I'd appreciate it. Thanks for your patience... No errors with these settings in the go script, even with the array running. I checked my config and I have Promise TX133s in this machine.
September 30, 200619 yr No errors with these settings in the go script, even with the array running. So putting those hdparm commands in the 3.0-beta go script seems to have solved the timeouts?
September 30, 200619 yr Author No errors with these settings in the go script, even with the array running. So putting those hdparm commands in the 3.0-beta go script seems to have solved the timeouts? Yes, on 3.0 beta 4, everything looks fine with those in the go script. So far.
October 1, 200619 yr Author Yes, on 3.0 beta 4, everything looks fine with those in the go script. So far. After about 4 hours of rsync: Sep 30 16:50:46 media-two kernel: hde: dma_timer_expiry: dma status == 0x60 Sep 30 16:50:46 media-two kernel: hde: timeout waiting for DMA Sep 30 16:50:46 media-two kernel: PDC202XX: Primary channel reset. Sep 30 16:50:46 media-two kernel: hde: timeout waiting for DMA Sep 30 16:50:46 media-two kernel: hde: (__ide_dma_test_irq) called while not wai ting Sep 30 16:50:46 media-two kernel: hde: status error: status=0x58 { DriveReady Se ekComplete DataRequest } Sep 30 16:50:46 media-two kernel: Sep 30 16:50:46 media-two kernel: hde: drive not ready for command Sep 30 16:50:46 media-two kernel: hde: status timeout: status=0xd0 { Busy } Sep 30 16:50:46 media-two kernel: Sep 30 16:50:46 media-two kernel: hdf: DMA disabled Sep 30 16:50:46 media-two kernel: PDC202XX: Primary channel reset. Sep 30 16:50:46 media-two kernel: hde: drive not ready for command Sep 30 16:50:46 media-two kernel: ide2: reset: success
October 1, 200619 yr Author After about 4 hours of rsync: Sep 30 16:50:46 media-two kernel: hde: dma_timer_expiry: dma status == 0x60 Sep 30 16:50:46 media-two kernel: hde: timeout waiting for DMA Sep 30 16:50:46 media-two kernel: PDC202XX: Primary channel reset. Sep 30 16:50:46 media-two kernel: hde: timeout waiting for DMA Sep 30 16:50:46 media-two kernel: hde: (__ide_dma_test_irq) called while not wai ting Sep 30 16:50:46 media-two kernel: hde: status error: status=0x58 { DriveReady Se ekComplete DataRequest } Sep 30 16:50:46 media-two kernel: Sep 30 16:50:46 media-two kernel: hde: drive not ready for command Sep 30 16:50:46 media-two kernel: hde: status timeout: status=0xd0 { Busy } Sep 30 16:50:46 media-two kernel: Sep 30 16:50:46 media-two kernel: hdf: DMA disabled Sep 30 16:50:46 media-two kernel: PDC202XX: Primary channel reset. Sep 30 16:50:46 media-two kernel: hde: drive not ready for command Sep 30 16:50:46 media-two kernel: ide2: reset: success You know what's weird about this? I'm copying files to hde, but hdf is the drive that gets DMA disabled. Also if I manually reenable it with hdparm -d1, the error doesn't seem to repeat. At any rate, I'm done copying files and the parity check ran last night at the expected speed with no errors. For the time being, I'm going to assume that something about the TX133 requires those hdparm commands in the go script and call it good.
October 2, 200619 yr These DMA errors have been a pain for a long time now (ask rharvey). So, there is going to be a beta5 release posted soon which addresses this exact problem: There are certain sequences in the unRAID driver where disk interrupts are turned off for a relatively "long time". This part of the driver has been changed so that this doesn't happen. So, 'beta5' has been running so far all weekend. We don't see any DMA errors, though we didn't see DMA errors before. It simply must have to do with the right combination of hardware: m/b, timing, & hard drives.
October 3, 200619 yr These DMA errors have been a pain for a long time now (ask rharvey). So, there is going to be a beta5 release posted soon which addresses this exact problem: There are certain sequences in the unRAID driver where disk interrupts are turned off for a relatively "long time". This part of the driver has been changed so that this doesn't happen. So, 'beta5' has been running so far all weekend. We don't see any DMA errors, though we didn't see DMA errors before. It simply must have to do with the right combination of hardware: m/b, timing, & hard drives. Tom, I wonder if rharvey sent his hard drive out that is always getting the DMA errors if that might help you out?!?! Just a thought....
Archived
This topic is now archived and is closed to further replies.