July 3, 201313 yr Currently I have 12 disks inc. parity and cache, 8 are on a LSI SAS1068E and the last 4 on-board sata. The parity and cache are on-board. I have a second LSI SAS1068E collecting dust, just wondering if adding it in and doing 4 + 4 + 4 instead of 8 + 4 would gain me anything? Or would I be better off getting an 8-port SAS2 card? Thanks.
July 3, 201313 yr Depends on the motherboard. Does the motherboard have a PDF with a Block layout of the chipset? Also depends on the drives, model and how they are sharing said block layout/chipset. When I had my 20 drive monster. I had 5 drives per controller. It was only PCI-X, but it worked out well for me. If I had another PCIe x8 slot and a spare controller. I might try putting 6 + 6 + 2 in keeping parity and cache onboard. Or you could try moving parity + cache to a separate controller in another slot. In my setup I had a separate controller for parity and cache under the premise of keeping the channel for those drives as free as possible... i.e. given that the sata speed matches the drive. Especially true if your cache drive is SATA III SSD.
July 3, 201313 yr Depends on the specific card you're using. The LSI SAS1068E chip supports 8 PCIe v2 lanes, so it can provide up to 4GB of bandwidth ... FAR more than 8 drives would use. So if it's on an x8 card, and plugged in to an x8 (or x16) slot, then the drives already have the maximum bandwidth they can use available. Even an x4 slot, as long as it's a v2 slot, would have plenty of bandwidth for your drives.
July 4, 201313 yr Author The chipset is H57(SATA 2), MSI doesn't have any detailed diagram but it's a consumer mobo so that's pretty standard. The card is an LSI SAS3081E-R (PCI-E x8 v1.0a).
July 4, 201313 yr The chipset is H57(SATA 2), MSI doesn't have any detailed diagram but it's a consumer mobo so that's pretty standard. The card is an LSI SAS3081E-R (PCI-E x8 v1.0a). Is the card plugged in to an x8 slot? You didn't give the motherboard model, so it's hard to know what slots are available. If it's only plugged in to an x4 slot, then you've only got 1GB of bandwidth ... a bit low for 8 drives (but not bad). If it's plugged in to an x1 slot then you've only got 250MB of bandwidth => WAY too low for 8 drives.
July 4, 201313 yr Gary, the card is an x8 card, so I'm sure it's plugged into an x8 slot. Dephcon, I tried to find some info about the chipset. I couldn't find what I was looking for. i.e. bandwidth to the chipset from DMI for sata utilization. I remember past chipsets having a maximum throughput of 384MB/s for the sata ports. In any case, the X8 slots are directly connected to the CPU according to block diagrams. Therefore I would put my fastest disks there. If it were me with the extra controller I would use it. As I mentioned earlier. I would still benchmark the isolated parity drive to see how it performs. I would try 5 + 5 + 2. Or maybe put the parity drive on the fastest controller or test it on the motherboard port. When it comes down to parity check speeds, you can only go as fast as your slowest drive. However if the parity drive has full bandwidth, it can sometimes reveal a benefit due to caching. Every nuance of tuning I've done with parity was small, 1-6MB/s, but after adding them all up, it started to provide more measurable results.
July 4, 201313 yr Gary, the card is an x8 card, so I'm sure it's plugged into an x8 slot. While it's true it SHOULD be if one is available, any open-ended x1 or x4 slot would work with the card ... it would just have much lower bandwidth. Since we don't know the motherboard involved, it's not clear that it has an available x8 (or x16) slot that would provide all 8 lanes the card is designed for. ... I've seen x16 video cards used in x1 slots (they work fine for 2D work ... but wouldn't want to try any gaming like that).
July 4, 201313 yr Author The board has two x16 physical ports, when both are used they're both x8. I'll get the other card flashed and slap it in, but I may also have a disk that's on its way out so i need to investigate that too. Most of the time my parity checks are fine, then sometimes they get stuck at like xxxKB/s and I just stop it as it causes buffering issues on XBMC. I also plan to replace the three Seagate drives has I've had them the longest from my HTPC w/JBOD days and the 7200.11 line was always shady. I'm just waiting out price drops on 3TB Red or the release of a 4TB Red. This is the mobo, btw http://msi.com/product/mb/H57M-ED65.html I should probably put my server specs in my sig.
July 4, 201313 yr sometimes they get stuck at like xxxKB/ Either there are collisions occurring with parity checks and active reads or a drive might be having issues. You may want to do a smart short test and/or a smart long test. If you do run a smart long test, you will want to disable access to the array and stop emhttp from spinning the drives down.
July 4, 201313 yr Author What are collisions all about? I quick scan of the forums seems to indicate network issues, I'm using a static IP well outside of my DHCP range. I'll need to wait for a maintenance window when the wife is out of town for a weekend to do any extended offline testing BUT i do really appreciate the help! I'll have to see what I can do in the meantime, maybe that diskspeed script can tell me something. *edit* diskspeed Version 1.8 <c> 2013 by Pourko Balkanski unRAID version=5.0-rc16b Mem: 3849MB Testing: /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 /mnt/disk10 /mnt/cache Waking up disks... Performing write tests... /mnt/disk1: 1048576000 bytes (1.0 GB) copied, 38.9687 s, 26.9 MB/s /mnt/disk2: 1048576000 bytes (1.0 GB) copied, 43.6692 s, 24.0 MB/s /mnt/disk3: 1048576000 bytes (1.0 GB) copied, 28.3262 s, 37.0 MB/s /mnt/disk4: 1048576000 bytes (1.0 GB) copied, 46.4661 s, 22.6 MB/s /mnt/disk5: 1048576000 bytes (1.0 GB) copied, 66.2269 s, 15.8 MB/s /mnt/disk6: 1048576000 bytes (1.0 GB) copied, 30.1024 s, 34.8 MB/s /mnt/disk7: 1048576000 bytes (1.0 GB) copied, 31.3963 s, 33.4 MB/s /mnt/disk8: 1048576000 bytes (1.0 GB) copied, 34.4939 s, 30.4 MB/s /mnt/disk9: 1048576000 bytes (1.0 GB) copied, 47.1292 s, 22.2 MB/s /mnt/disk10: 1048576000 bytes (1.0 GB) copied, 29.3194 s, 35.8 MB/s /mnt/cache: 1048576000 bytes (1.0 GB) copied, 10.0655 s, 104 MB/s Performing read tests... /mnt/disk2: 1048576000 bytes (1.0 GB) copied, 10.4571 s, 100 MB/s /mnt/disk1: 1048576000 bytes (1.0 GB) copied, 10.6507 s, 98.5 MB/s /mnt/disk6: 1048576000 bytes (1.0 GB) copied, 10.7808 s, 97.3 MB/s /mnt/disk7: 1048576000 bytes (1.0 GB) copied, 10.8673 s, 96.5 MB/s /mnt/disk3: 1048576000 bytes (1.0 GB) copied, 11.0171 s, 95.2 MB/s /mnt/disk9: 1048576000 bytes (1.0 GB) copied, 11.2641 s, 93.1 MB/s /mnt/disk4: 1048576000 bytes (1.0 GB) copied, 11.3808 s, 92.1 MB/s /mnt/disk8: 1048576000 bytes (1.0 GB) copied, 11.4623 s, 91.5 MB/s /mnt/disk10: 1048576000 bytes (1.0 GB) copied, 11.6301 s, 90.2 MB/s /mnt/cache: 1048576000 bytes (1.0 GB) copied, 11.9753 s, 87.6 MB/s /mnt/disk5: 1048576000 bytes (1.0 GB) copied, 12.012 s, 87.3 MB/s Disk 5 isn't running great, ran it twice, both times at 15MB/s write, tonight I'll map out where all the drives physically connect to. It's my newest EARX so it should be solid, but still has 2 yrs of warranty left so RMA isn't a problem. I've also got a feeler out to one of my vendors for some IBM M1015s, I'll probably buy some if I can get a good deal.
July 4, 201313 yr Disk 5 isn't running great, ran it twice, both times at 15MB/s write, tonight I'll map out where all the drives physically connect to. It's my newest EARX so it should be solid, but still has 2 yrs of warranty left so RMA isn't a problem. That's probably a big part of the issue, what does the smart report reveal? Do a -t long test on it. (make sure you turn off spin down) It will a very long time. I've also got a feeler out to one of my vendors for some IBM M1015s, I'll probably buy some if I can get a good deal. You may not need to if you get disk 5 sorted out. Use what you have first. Separating the faster drives on the x8 controller may reveal that you can wait. I.E. unless you want to run ESX.
July 4, 201313 yr Author That's probably a big part of the issue, what does the smart report reveal? Do a -t long test on it. (make sure you turn off spin down) It will a very long time. How long are we talking here? If it's less than 6 hours i can probably do it remotely from work and the wife will never know, muhahahahahaha...
July 4, 201313 yr That's probably a big part of the issue, what does the smart report reveal? Do a -t long test on it. (make sure you turn off spin down) It will a very long time. How long are we talking here? If it's less than 6 hours i can probably do it remotely from work and the wife will never know, muhahahahahaha... Last time I did it it was about 5 hours. You need to disable the spin down or it will interrupt the test. I would also suggest unmounting the drive.
July 8, 201313 yr Author I stopped the array and disabled spin-down. Running the long test doesn't seem to work: root@Vault13:~# smartctl -t long /dev/sdf smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Extended self-test routine immediately in off-line mode". Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 255 minutes for test to complete. Test will complete after Mon Jul 8 17:25:09 2013 Use smartctl -X to abort test. root@Vault13:~# smartctl -l selftest /dev/sdf smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Aborted by host 90% 11510 - # 2 Extended offline Aborted by host 30% 11510 - # 3 Short offline Completed without error 00% 11505 - # 4 Extended offline Aborted by host 90% 11505 -
July 8, 201313 yr I stopped the array and disabled spin-down. Running the long test doesn't seem to work: smartctl -t long /dev/sdf is all you need to do and you need to leave the machine alone and not touch it or do anything until it's done. it might be worthwhile to disable emhttp from running in the go script, then running this test. The Aborted by host means that some kind of message was sent to the drive and it interrupted the test. You can't do anything with 'some' drives when this test is run. Even refreshing the webgui, causes smartctl to go out and do things with the drive. It did get pretty far. # 2 Extended offline Aborted by host Remaining 30% 11510 - I wouldn't do the selftest while the extended test was running. I "think" you can do a smartctl -a /dev/sdf to get a log while it's running, but any other activity will certainly interrupt the test. Other choice is to remove it from the array and do a diagnostic with the manufacturers software. post a full smartctl -a of the drive in question. Maybe it will reveal something.
July 8, 201313 yr Author smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: WDC WD20EARX-00PASB0 Serial Number: WD-WCAZAD808720 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Mon Jul 8 14:21:45 2013 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 248) Self-test routine in progress... 80% of test remaining. Total time to complete Offline data collection: (41100) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 160 159 021 Pre-fail Always - 7000 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1939 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 11511 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 73 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 38 193 Load_Cycle_Count 0x0032 195 195 000 Old_age Always - 16815 194 Temperature_Celsius 0x0022 122 110 000 Old_age Always - 28 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Aborted by host 90% 11510 - # 2 Extended offline Aborted by host 30% 11510 - # 3 Short offline Completed without error 00% 11505 - # 4 Extended offline Aborted by host 90% 11505 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. I'll have to disable emhttp and re-run tonight before bed. I don't have enough time to start now before the wife is home.
July 8, 201313 yr Post a full syslog. I don't see anything that grabs me about the smart log. Also, how full is disk 5. Do a df -vH
July 8, 201313 yr Author It's pretty full, 223GB free. I'll just do the smart-long tonight or tomorrow and post that along with the syslog. If the long doesn't show anything I'll just RMA it with WD and say it's not preforming at the same speed as the rest of the same disk in a raid array. They're usually really good about RMAs.
July 9, 201313 yr Author Finished the long with emhttp disabled and I've attached my syslog: smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: WDC WD20EARX-00PASB0 Serial Number: WD-WCAZAD808720 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Tue Jul 9 15:37:33 2013 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (41100) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 160 159 021 Pre-fail Always - 7000 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1939 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 11536 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 73 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 38 193 Load_Cycle_Count 0x0032 195 195 000 Old_age Always - 16820 194 Temperature_Celsius 0x0022 124 110 000 Old_age Always - 26 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 11536 - # 2 Extended offline Aborted by host 90% 11529 - # 3 Extended offline Aborted by host 90% 11529 - # 4 Extended offline Completed without error 00% 11517 - # 5 Extended offline Aborted by host 90% 11510 - # 6 Extended offline Aborted by host 30% 11510 - # 7 Short offline Completed without error 00% 11505 - # 8 Extended offline Aborted by host 90% 11505 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Another lame diskspeed result: diskspeed Version 1.8 <c> 2013 by Pourko Balkanski unRAID version=5.0-rc16c Mem: 3849MB Testing: /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 /mnt/disk10 /mnt/cache Waking up disks... Performing write tests... /mnt/disk1: 1048576000 bytes (1.0 GB) copied, 59.7102 s, 17.6 MB/s /mnt/disk2: 1048576000 bytes (1.0 GB) copied, 57.4044 s, 18.3 MB/s /mnt/disk3: 1048576000 bytes (1.0 GB) copied, 57.4532 s, 18.3 MB/s /mnt/disk4: 1048576000 bytes (1.0 GB) copied, 60.759 s, 17.3 MB/s /mnt/disk5: 1048576000 bytes (1.0 GB) copied, 124.309 s, 8.4 MB/s /mnt/disk6: 1048576000 bytes (1.0 GB) copied, 50.6895 s, 20.7 MB/s /mnt/disk7: 1048576000 bytes (1.0 GB) copied, 52.5996 s, 19.9 MB/s /mnt/disk8: 1048576000 bytes (1.0 GB) copied, 56.8126 s, 18.5 MB/s /mnt/disk9: 1048576000 bytes (1.0 GB) copied, 78.6356 s, 13.3 MB/s /mnt/disk10: 1048576000 bytes (1.0 GB) copied, 44.1629 s, 23.7 MB/s /mnt/cache: 1048576000 bytes (1.0 GB) copied, 12.8535 s, 81.6 MB/s Waking up disks... Performing read tests... /mnt/disk3: 1048576000 bytes (1.0 GB) copied, 10.462 s, 100 MB/s /mnt/disk1: 1048576000 bytes (1.0 GB) copied, 10.6437 s, 98.5 MB/s /mnt/disk6: 1048576000 bytes (1.0 GB) copied, 10.7199 s, 97.8 MB/s /mnt/disk9: 1048576000 bytes (1.0 GB) copied, 10.7657 s, 97.4 MB/s /mnt/disk2: 1048576000 bytes (1.0 GB) copied, 10.7787 s, 97.3 MB/s /mnt/disk7: 1048576000 bytes (1.0 GB) copied, 10.8263 s, 96.9 MB/s /mnt/disk8: 1048576000 bytes (1.0 GB) copied, 11.3796 s, 92.1 MB/s /mnt/disk10: 1048576000 bytes (1.0 GB) copied, 11.4559 s, 91.5 MB/s /mnt/disk4: 1048576000 bytes (1.0 GB) copied, 11.664 s, 89.9 MB/s /mnt/disk5: 1048576000 bytes (1.0 GB) copied, 12.6061 s, 83.2 MB/s /mnt/cache: 1048576000 bytes (1.0 GB) copied, 76.5612 s, 13.7 MB/s I'm going to get that second LSI HBA in before I RMA anything. syslog.txt
July 12, 201313 yr Author Second HBA is in, performance is up. Waiting on more SAS cables from monoprice to get the other drives off the mobo to run 6+6.
July 25, 201312 yr Author Got the new breakout cables... now I know my LSI cards don't support 3TB drives lol. I guess I'll be needing that SAS2008 based card after all, which I might be getting tomorrow from my vendor on the free. On the plus side, I stripped everything down and dusted everything out. I also made a custom SATA power cable for one of my 4in3 bays and it plugs directly into my modular PSU.
Archived
This topic is now archived and is closed to further replies.