Drive performance testing (version 2.6.5) for UNRAID 5 thru 6.4

[jbartlett]

Tests run to completion but clearly the Areca card is causing some confusion... or I have invisible warp drives lol.

 

Odd number too for the Seagate 6TB. I wouldn't think it was so slow.

 

Can you try this test version and execute it with the -l / --log option and PM/post the generated diskspeed.log file

http://strangejourney.net/Temp/diskspeed.v2.4.zip

 

I added logic to resolve your invisible drive issue. The log file should allow me to investigate the abnormal graphs.

 

Can you post the exact command line I need to run? Reduces operator errors. ;P

 

No problem.

>diskspeed.sh -l

[interwebtech]

Tests run to completion but clearly the Areca card is causing some confusion... or I have invisible warp drives lol.

 

Odd number too for the Seagate 6TB. I wouldn't think it was so slow.

 

Can you try this test version and execute it with the -l / --log option and PM/post the generated diskspeed.log file

http://strangejourney.net/Temp/diskspeed.v2.4.zip

 

I added logic to resolve your invisible drive issue. The log file should allow me to investigate the abnormal graphs.

 

Can you post the exact command line I need to run? Reduces operator errors. ;P

 

No problem.

>diskspeed.sh -l

 

login as: root
root@tower's password:
Last login: Mon Aug 10 11:35:31 2015 from dell-i7.home
Linux 4.1.1-unRAID.
root@Tower:~#  cd /boot
root@Tower:/boot# diskspeed.sh -l

diskspeed.sh for UNRAID, version 2.4
By John Bartlett. Support board @ limetech: http://goo.gl/ysJeYV

/dev/sdb (Disk : 1103 MB/sec avg
/dev/sdc (Disk 7): 1114 MB/sec avg
/dev/sdd (Disk 6): 1105 MB/sec avg
/dev/sde (Disk 5): 1100 MB/sec avg
/dev/sdf (Disk 4): 1113 MB/sec avg
/dev/sdg (Disk 11): 1106 MB/sec avg
/dev/sdh (Disk 10): 1114 MB/sec avg
/dev/sdi (Disk 9): 1105 MB/sec avg
/dev/sdj (Disk 3): 114 MB/sec avg
/dev/sdk (Disk 2): 157 MB/sec avg
/dev/sdl (Disk 1): 48 MB/sec avg
/dev/sdm (Parity): 139 MB/sec avg
/dev/sdn (Cache): 372 MB/sec avg

To see a graph of the drive's speeds, please browse to the current
directory and open the file diskspeed.html in your Internet Browser
application.

root@Tower:/boot#

 

log attached.

diskspeed.log.txt

[jbartlett]

Thank you! I believe I see the issue - fdisk is also not returning the drive capacity which I missed when you first posted the results I ask before. When it tests the drive read, it's testing the same spot over and over (start of the drive) and the 2nd+ tests are utilizing the Areca cache even though the dd command is requesting the cache to be bypassed. I'll fix this and give you a new version to test, which I greatly appreciate your assistance with.

 

I don't have any explanation why your 6TB Drive 2 is testing fast now but the 5TB Drive 2 is now testing so slow. Did you rearrange anything inside, re-route SATA cables? Was your array being accessed? This script is best ran when the array is idle and can be ran when the array is stopped.

[interwebtech]

Thank you! I believe I see the issue - fdisk is also not returning the drive capacity which I missed when you first posted the results I ask before. When it tests the drive read, it's testing the same spot over and over (start of the drive) and the 2nd+ tests are utilizing the Areca cache even though the dd command is requesting the cache to be bypassed. I'll fix this and give you a new version to test, which I greatly appreciate your assistance with.

 

I don't have any explanation why your 6TB Drive 2 is testing fast now but the 5TB Drive 2 is now testing so slow. Did you rearrange anything inside, re-route SATA cables? Was your array being accessed? This script is best ran when the array is idle and can be ran when the array is stopped.

 

The array was idle/spun down when I ran the script. Took several minutes to wake up before it actually did any work. No changes made since last run(s).

[jbartlett]

The array was idle/spun down when I ran the script. Took several minutes to wake up before it actually did any work. No changes made since last run(s).

 

Yeah, the sync command only wakes up one drive at a time. Spinning up the drives is on my To-Do list as is adding a warning if there is activity on the array.

[jbartlett]

interwebtech: Please try this version. I believe I have resolved all your issues including the delay if the drives are spun down.

 

http://strangejourney.net/Temp/diskspeed.v2.4b2.zip

[interwebtech]

looks like its working well now. Disk 8 is the last remaining EARS 2TB so I expect that to be the slowest but see Disk 1? That is a 7200 RPM Tosiba 5TB that should be near the top speed-wise. Oddness.

 

diskspeed.sh for UNRAID, version 2.4
By John Bartlett. Support board @ limetech: http://goo.gl/ysJeYV

/dev/sdb (Disk : 97 MB/sec avg
/dev/sdc (Disk 7): 126 MB/sec avg
/dev/sdd (Disk 6): 125 MB/sec avg
/dev/sde (Disk 5): 127 MB/sec avg
/dev/sdf (Disk 4): 118 MB/sec avg
/dev/sdg (Disk 11): 153 MB/sec avg
/dev/sdh (Disk 10): 143 MB/sec avg
/dev/sdi (Disk 9): 141 MB/sec avg
/dev/sdj (Disk 3): 116 MB/sec avg
/dev/sdk (Disk 2): 157 MB/sec avg
/dev/sdl (Disk 1): 46 MB/sec avg
/dev/sdm (Parity): 139 MB/sec avg
/dev/sdn (Cache): 371 MB/sec avg

[jbartlett]

looks like its working well now. Disk 8 is the last remaining EARS 2TB so I expect that to be the slowest but see Disk 1? That is a 7200 RPM Tosiba 5TB that should be near the top speed-wise. Oddness.

 

Looking at the log you sent previously, that's what the system is reporting via the dd command and not an issue with the script itself. But that's one of the big reasons for this script, to look for invisible things like that.

 

I'd replace the SATA cable or swap the SATA plug with another device to see if the issue stays with the drive or migrates.

[jbartlett]

Version 2.4 released, download link in first post

 

Change Log

If the drive model is not able to be determined via fdisk, extract it form mdmcd

Add -l --log option to create the debug log file speeddisk.log

Modified to not display the MB sec in drive inventory report for excluded drives

Modified to compute the drive capacity from the number of bytes UNRAID reports to support external drive cards.

Added -g --graph option to display the drive by percentage comparison graph

Added warning if files on the array are open which could mean drives are active

Added drive spin up support by reading a random sector in the first quarter of the drive via background tasks and then performing a sync

 

Upcoming version 3.0

Benchmark all drives at the same time at every 2% of the drive's capacity. Overall test duration will be equal to one pass over the slowest drive.

[interwebtech]

swapped both port & cable and results were pretty spectacular. Now it shows as the fastest drive as it should. Will have to determine which is the cause of the initial slow results.

 

diskspeed.sh for UNRAID, version 2.4
By John Bartlett. Support board @ limetech: http://goo.gl/ysJeYV

/dev/sdb (Disk : 97 MB/sec avg
/dev/sdc (Disk 7): 123 MB/sec avg
/dev/sdd (Disk 6): 125 MB/sec avg
/dev/sde (Disk 5): 123 MB/sec avg
/dev/sdf (Disk 4): 118 MB/sec avg
/dev/sdg (Disk 11): 152 MB/sec avg
/dev/sdh (Disk 10): 143 MB/sec avg
/dev/sdi (Disk 9): 141 MB/sec avg
/dev/sdj (Disk 3): 115 MB/sec avg
/dev/sdk (Disk 2): 158 MB/sec avg
/dev/sdl (Disk 1): 177 MB/sec avg
/dev/sdm (Parity): 139 MB/sec avg
/dev/sdn (Cache): 369 MB/sec avg

[JorgeB]

Many thanks for this, very useful to find the slowest disk to replace next.

 

Upcoming version 3.0

Benchmark all drives at the same time at every 2% of the drive's capacity. Overall test duration will be equal to one pass over the slowest drive.

 

Won’t this cause false results on systems that are bus limited?

 

For example, in one of my servers I have 8 disks connect to a SASLP, it limits my parity check speed to around 75Mb/s, will I get the same result from these disks or it doesn’t matter because of the way the test runs?

 

[RobJ]

I agree with Johnnie.  This new simultaneous testing could be a special test mode, one that will be very interesting for exposing bottlenecks in the system.  I would expect the results to be dramatically different on some systems, using slow controllers or lane-limited ones (e.g. PCIe-x8 card in PCIe-x1 slot) or port multipliers or with multiple drives on the old PCI bus.

 

I'm wondering if you could even tell which drive is slowest in some cases.  As an example, take a mix of slow and fast drives on a port multiplier, won't they all show the same very slow speed?  It will be interesting to see the results.  This new mode is a great idea!

[JonathanM]

I agree with Johnnie.  This new simultaneous testing could be a special test mode, one that will be very interesting for exposing bottlenecks in the system.  I would expect the results to be dramatically different on some systems, using slow controllers or lane-limited ones (e.g. PCIe-x8 card in PCIe-x1 slot) or port multipliers or with multiple drives on the old PCI bus.

The new name for this should be drive and controller performance testing. It would be cool if you could identify the port and controller group so correlations would stand out better. You could even automatically make recommendations on possible drive rearranging to minimize parity check time, or maximize performance of a particular drive.

[jbartlett]

The new name for this should be drive and controller performance testing. It would be cool if you could identify the port and controller group so correlations would stand out better. You could even automatically make recommendations on possible drive rearranging to minimize parity check time, or maximize performance of a particular drive.

 

I can do this.

[JorgeB]

I’m trying to test a disk I know has some areas with slow sectors to see if I can see the problem in the graph, because this is usually a sign that it’s going to go bad soon and I’d like to use this test as a preventive measure on my servers, but if I try a sample value of 21 or above I get nonlinear results.

 

For example, for –s 21 first test is at the 1tb mark, then comes back to the 200mb mark, then 400mb and so on.

 

What are the max samples it should work? I’d like to use -s 101 to test by 1% increments, but when I use it the test starts at 0mb, then 200mb, 400mb and so on until 2tb, then restarts and starts again now in smaller steps so the graph is not linear.

[Fireball3]

I think the preclear "stress test" is the better choice for your drive.

[jbartlett]

I’m trying to test a disk I know has some areas with slow sectors to see if I can see the problem in the graph, because this is usually a sign that it’s going to go bad soon and I’d like to use this test as a preventive measure on my servers, but if I try a sample value of 21 or above I get nonlinear results.

 

Thank you for letting me know, I'll take a look.

[JorgeB]

I think the preclear "stress test" is the better choice for your drive.

 

Drives I want to test have data on them, can’t run preclear, and I’m not sure it would give me the info I want anyway.

 

In my experience once a drive starts getting this slower sectors it will have some bad sectors sooner rather than later, but until then S.M.A.R.T. looks fine without any errors, I’ve had a few of these older Samsung drives fail with read errors and when I use MHDD on them almost always have a lot of these slower areas.

 

My plan is to use diskspeed as an early warning as I still have about 20 of these drives on 3 different servers and can’t afford to replace them all preemptively.

 

[jbartlett]

I’m trying to test a disk I know has some areas with slow sectors to see if I can see the problem in the graph, because this is usually a sign that it’s going to go bad soon and I’d like to use this test as a preventive measure on my servers, but if I try a sample value of 21 or above I get nonlinear results.

 

Thank you for letting me know, I'll take a look.

 

This is fixed.

[jbartlett]

Version 2.5

Fixed computation for percentages less than 10%

Reverted to 1 GB scans for better results but slower

Added -f --fast to scan 200 MB instead of 1 GB, same as version 2.3 & 2.4

[interwebtech]

Thanks again for your great script. Any thoughts on making this a v6 GUI plug-in? (easy for me to say... I don't know how to do it lol)

[jbartlett]

Thanks again for your great script. Any thoughts on making this a v6 GUI plug-in? (easy for me to say... I don't know how to do it lol)

 

I plan to go that route, once I figure out how to do it. 

[jbartlett]

I think the preclear "stress test" is the better choice for your drive.

 

Drives I want to test have data on them, can’t run preclear, and I’m not sure it would give me the info I want anyway.

 

In my experience once a drive starts getting this slower sectors it will have some bad sectors sooner rather than later, but until then S.M.A.R.T. looks fine without any errors, I’ve had a few of these older Samsung drives fail with read errors and when I use MHDD on them almost always have a lot of these slower areas.

 

My plan is to use diskspeed as an early warning as I still have about 20 of these drives on 3 different servers and can’t afford to replace them all preemptively.

 

 

What program is this? It's possible I can duplicate this functionality and display the data in a browser using a heat map.

[JorgeB]

It's MHDD

 

http://hddguru.com/software/2005.10.02-MHDD/

 

It's a very useful diagnostic tool but it only runs from a boot cd or usb pen and it will only test HDDs connected to the onboard sata master ports set to IDE mode, it doesn't support AHCI.

 

I always run a pass in any new disk before using it.

 

It would be very cool if could create an option to do a full sector scan and show slow sectors.

[JorgeB]

Testing every 1% was enough to see the slow areas on this drive, of course this will never be the equivalent of a full surface scan but is a quick way of alerting to a possible problem, in which case I’ll do further tests.

 

Thanks again for the utility.