Tezmo Posted January 11, 2012 Posted January 11, 2012 To max out my machine at 24 drives, I have installed a 4-port PCI SATA2 card. This one: It worked out of the box, and I added two WD20EARX drives to it, unjumpered, and started simultaneous MBR 4k-aligned pre-clears. At stage 2, each of them are going about 15MB/s - clearing using the onboard ports went at about 80MB/s. My load averages are showing as 3.49, 3.63, 3.67 - I interpret this to mean the CPU is getting hammered. My syslog is here: http://pastebin.com/du1nSh11 I expected that using these ports would be slow, but I did not expect it to be this slow. If this is just the way it is, then fine - but is there anything I can do to improve the situation? Thank you all
Tezmo Posted January 11, 2012 Author Posted January 11, 2012 pardon me, this is the SATA card: http://www.ebay.co.uk/itm/NEW-SATA-II-4-PORTS-PCI-RAID-Host-Control-Card-SIL3124-/280596439320#ht_5204wt_902
dgaschk Posted January 11, 2012 Posted January 11, 2012 That speed is expected using PCI. All of the PCI slots share bandwidth and can only support a single drive with good performance. A PCIe card will will support multiple drives at full speed. PCI - 1 drive (on all slots) PCIeX1 - 2 drives PCIeX4 - 8 drives PCIeX8 - 16 drives PCIeX16 - 32 drives Double these numbers for PCIe 2.0.
WeeboTech Posted January 11, 2012 Posted January 11, 2012 The speed should be faster then 15MB/s. I would expect 30-60MB/s. What could possibly help a little is stagger the preclear so that while one is reading the other is writing. I'm not very hopefull of that doing much. The chipset usually only supports SATA I (150MB/s) speed. The PCI bus is normally 133MB/s. (shared) Today's drives usually max out around 120-130MB/s on the outer tracks. Frankly at 15MB/s I would stop one and let the other finish, or I would move any of my smaller slower drives to the PCI controller. Also, just an FYI, that chipset is usually capable of PCI 2.3 66MHz if your motherboard supports it. If it did, I'm sure it would be faster. What motherboard do you have? What other controllers do you have?
Tezmo Posted January 11, 2012 Author Posted January 11, 2012 Thank you dgaschk & Weebotech. The motherboard in the system is an ASRock M3A770DE: http://www.asrock.com/mb/overview.asp?model=m3a770de It was selected because it has gone in a 24 bay norco case. Notionally, the four on board SATA ports, plus sixteen SATA ports from two Supermicro AOC-SASLP-MV8 cards in the PCI-E slots, and four other SATA ports on the PCI bus makes the case fully occupied. At the moment, the machine has 6 2tb WD20EARX drives - four are connected to the onboard (nforce) controller, one of which is configured as parity - they all work nice and fast. I added the above PCI card and plumbed in the two new WD drives to which I refer above with regard to pre-clearing. I appreciate I would probably solve the problem by using the PCI-E bus, but since I will have to deal with it at some point to hit 24 drives, I wanted to tackle it now. I had wondered whether PIO mode might be an issue - how can I tell from the syslog whether or not they have dropped into PIO mode? And of course, thank you
WeeboTech Posted January 12, 2012 Posted January 12, 2012 Thank you dgaschk & Weebotech. The motherboard in the system is an ASRock M3A770DE: http://www.asrock.com/mb/overview.asp?model=m3a770de It was selected because it has gone in a 24 bay norco case. Notionally, the four on board SATA ports, plus sixteen SATA ports from two Supermicro AOC-SASLP-MV8 cards in the PCI-E slots, and four other SATA ports on the PCI bus makes the case fully occupied. At the moment, the machine has 6 2tb WD20EARX drives - four are connected to the onboard (nforce) controller, one of which is configured as parity - they all work nice and fast. I added the above PCI card and plumbed in the two new WD drives to which I refer above with regard to pre-clearing. I appreciate I would probably solve the problem by using the PCI-E bus, but since I will have to deal with it at some point to hit 24 drives, I wanted to tackle it now. I had wondered whether PIO mode might be an issue - how can I tell from the syslog whether or not they have dropped into PIO mode? And of course, thank you I looked, I did not see they were in PIO mode. I saw them in UDMA/100 Mode. do a hdparm -iI /dev/sd? where /dev/sd? is the drive you want to check. you can also do hdparm -tT /dev/sd? to test the drive's MAX possible speed.
Tezmo Posted January 12, 2012 Author Posted January 12, 2012 for my two drives currently undergoing preclearing, I get: /dev/sdf: Model=WDC WD20EARX-00PASB0, FwRev=51.0AB51, SerialNo=WD-WCAZAC481555 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=3907029168 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma6 AdvancedPM=no WriteCache=enabled Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7 * signifies the current active mode ATA device, with non-removable media Model Number: WDC WD20EARX-00PASB0 Serial Number: WD-WCAZAC481555 Firmware Revision: 51.0AB51 Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0 Standards: Supported: 8 7 6 5 Likely used: 8 Configuration: Logical max current cylinders 16383 16383 heads 16 16 sectors/track 63 63 -- CHS current addressable sectors: 16514064 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 3907029168 Logical Sector size: 512 bytes Physical Sector size: 4096 bytes Logical Sector-0 offset: 0 bytes device size with M = 1024*1024: 1907729 MBytes device size with M = 1000*1000: 2000398 MBytes (2000 GB) cache/buffer size = unknown Capabilities: LBA, IORDY(can be disabled) Queue depth: 32 Standby timer values: spec'd by Standard, with device specific minimum R/W multiple sector transfer: Max = 16 Current = 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 udma6 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=120ns IORDY flow control=120ns Commands/features: Enabled Supported: * SMART feature set Security Mode feature set * Power Management feature set * Write cache * Look-ahead * Host Protected Area feature set * WRITE_BUFFER command * READ_BUFFER command * NOP cmd * DOWNLOAD_MICROCODE Power-Up In Standby feature set * SET_FEATURES required to spinup after power up SET_MAX security extension * 48-bit Address feature set * Device Configuration Overlay feature set * Mandatory FLUSH_CACHE * FLUSH_CACHE_EXT * SMART error logging * SMART self-test * General Purpose Logging feature set * 64-bit World wide name * WRITE_UNCORRECTABLE_EXT command * {READ,WRITE}_DMA_EXT_GPL commands * Segmented DOWNLOAD_MICROCODE * Gen1 signaling speed (1.5Gb/s) * Gen2 signaling speed (3.0Gb/s) * unknown 76[3] * Native Command Queueing (NCQ) * Host-initiated interface power management * Phy event counters * NCQ priority information DMA Setup Auto-Activate optimization * Software settings preservation * SMART Command Transport (SCT) feature set * SCT LBA Segment Access (AC2) * SCT Features Control (AC4) * SCT Data Tables (AC5) unknown 206[12] (vendor specific) unknown 206[13] (vendor specific) Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count supported: enhanced erase 388min for SECURITY ERASE UNIT. 388min for ENHANCED SECURITY ERASE UNIT. Logical Unit WWN Device Identifier: 50014ee25bbf4fde NAA : 5 IEEE OUI : 0014ee Unique ID : 25bbf4fde Checksum: correct and /dev/sdg: Model=WDC WD20EARX-00PASB0, FwRev=51.0AB51, SerialNo=WD-WCAZAC476527 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=3907029168 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma6 AdvancedPM=no WriteCache=enabled Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7 * signifies the current active mode ATA device, with non-removable media Model Number: WDC WD20EARX-00PASB0 Serial Number: WD-WCAZAC476527 Firmware Revision: 51.0AB51 Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0 Standards: Supported: 8 7 6 5 Likely used: 8 Configuration: Logical max current cylinders 16383 16383 heads 16 16 sectors/track 63 63 -- CHS current addressable sectors: 16514064 LBA user addressable sectors: 268435455 LBA48 user addressable sectors: 3907029168 Logical Sector size: 512 bytes Physical Sector size: 4096 bytes Logical Sector-0 offset: 0 bytes device size with M = 1024*1024: 1907729 MBytes device size with M = 1000*1000: 2000398 MBytes (2000 GB) cache/buffer size = unknown Capabilities: LBA, IORDY(can be disabled) Queue depth: 32 Standby timer values: spec'd by Standard, with device specific minimum R/W multiple sector transfer: Max = 16 Current = 0 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 udma6 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=120ns IORDY flow control=120ns Commands/features: Enabled Supported: * SMART feature set Security Mode feature set * Power Management feature set * Write cache * Look-ahead * Host Protected Area feature set * WRITE_BUFFER command * READ_BUFFER command * NOP cmd * DOWNLOAD_MICROCODE Power-Up In Standby feature set * SET_FEATURES required to spinup after power up SET_MAX security extension * 48-bit Address feature set * Device Configuration Overlay feature set * Mandatory FLUSH_CACHE * FLUSH_CACHE_EXT * SMART error logging * SMART self-test * General Purpose Logging feature set * 64-bit World wide name * WRITE_UNCORRECTABLE_EXT command * {READ,WRITE}_DMA_EXT_GPL commands * Segmented DOWNLOAD_MICROCODE * Gen1 signaling speed (1.5Gb/s) * Gen2 signaling speed (3.0Gb/s) * unknown 76[3] * Native Command Queueing (NCQ) * Host-initiated interface power management * Phy event counters * NCQ priority information DMA Setup Auto-Activate optimization * Software settings preservation * SMART Command Transport (SCT) feature set * SCT LBA Segment Access (AC2) * SCT Features Control (AC4) * SCT Data Tables (AC5) unknown 206[12] (vendor specific) unknown 206[13] (vendor specific) Security: Master password revision code = 65534 supported not enabled not locked not frozen not expired: security count supported: enhanced erase 396min for SECURITY ERASE UNIT. 396min for ENHANCED SECURITY ERASE UNIT. Logical Unit WWN Device Identifier: 50014ee25bbf4d50 NAA : 5 IEEE OUI : 0014ee Unique ID : 25bbf4d50 Checksum: correct Truth be told I am unsure how to interpret this. The speed check command I aborted because I am reluctant at this stage to abort the preclear process. It is good at least to have validation that its not going as fast as it should (total 30MB/s across the bus). Presumably the enablement of PCI 2.3 66MHz will be a BIOS setting? Am I right in my interpretation of the top output that the CPU is being worked very hard?
dgaschk Posted January 12, 2012 Posted January 12, 2012 How many cores does the CPU have? It is odd that it's working so hard.
WeeboTech Posted January 12, 2012 Posted January 12, 2012 The PCI bus can only handle 1 communication transaction at a time. It is usually 133MB/s. PCI 2.3 communicates at 266MB/s. You motherboard does not look like it supports PCI 2.3. Not many do and if they do it is published as such. There might be some PCI adjustments in tbe bios, however I would not put much weight into it. I think the Promise TX4's do a better job then the Sil3124's in PCI, However if you do not want to change out the motherboard, then you will have to live with the current speed if using multiple drives on the PCI bus. Once the preclear is done, single drive access to those drives will be at a reasonable speed. Parity checks may suffer. You may want to look at a port multiplier card on one of the PCIe Sata ports. Those should yield about 40MB/s to 60MB/s simultaneously. Otherwise you are where it's at. I would do a parity check after you add them to the array. Then you will know what your speeds will be. The preclear is an intense process for PCI. It's allot of data being shoved down the pipe. Ride it out, then do an hdparm -tT test, then add them to the array and do a parity check for a speed comparison. You might be OK. Also you may get better parity check speeds if you rotate the drives. I.E. do not put the drives on the PCI card next to one another. do some kind of round robin approach. Then while the CPU is waiting for a command to finish on the PCI bus a command can be issued to the PCIe bus. (It's all in theory, but there have been reports it works OK for port multipliers)
WeeboTech Posted January 12, 2012 Posted January 12, 2012 How many cores does the CPU have? It is odd that it's working so hard. Heavy interrupts from the PCI bus.
elkay14 Posted January 12, 2012 Posted January 12, 2012 How many cores does the CPU have? It is odd that it's working so hard. Heavy interrupts from the PCI bus. Yup. 'top' would show you the CPU is mostly in IOWait.
Tezmo Posted January 13, 2012 Author Posted January 13, 2012 Thank you all - I am indeed mostly in IOWait. I decided to allow the preclear to finish rather than potentially causing myself further problems by interrupting it, and in performing the Post-Read, the two discs combined are doing a lot better than before - more than 90MB/s combined. I appreciate the inherent slowness in using the PCI bus, and in general usage I doubt it'll be an issue, so I am very happy with how things are going for now - thanks again
Tezmo Posted January 13, 2012 Author Posted January 13, 2012 Finally, I am not able to edit the title of the first post to indicate [sOLVED]. Applying my edit gives the following: Forbidden You don't have permission to access /forum/index.php on this server. Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request. I got the same error when making my initial post - I had to use the iphone app to start this thread. Not really looking for help with this so much as making my excuses for not amending the title
Recommended Posts
Archived
This topic is now archived and is closed to further replies.