DiskSpeed, hdd/ssd benchmarking (unRAID 6+), version 2.10.7


Recommended Posts

7 hours ago, jademonkee said:

Upgraded to v2.5 this morning and thought I'd re-run the benchmarks. 

Was receiving 'speed gap' notification, so I aborted, disabled all dockers and re-ran. Still got the 'speed gap' notification, so I aborted, and re-ran with the 'disable speed gap detection' option checked.

The final results page looks vastly different to the last time I ran it, however, going back to the main page, the curves look just as before (almost as if the main page didn't update to the latest results - even after I Ctrl+R refreshed the page).

Please see attached. 

So: is it something I should be worried about? I haven't received any notification from Unraid that anything is wrong with SMART, but should I do something to confirm that my disks (or controller?) are ok? 

Thanks for your help.

I can duplicate the home page graph not showing the most recent which is odd since I didn't change that logic in a long time. I'll look into it.

 

I've got one drive that shows the same erratic benchmark, on my backup system which should have zero outside activity against it. I removed the cache popping logic and that changed the results though it still shows some erratic hills but didn't throw the speed gap warning.

 

Version 2.6 pushed to remove cache popping logic.

Link to comment
16 hours ago, jbartlett said:

I can duplicate the home page graph not showing the most recent which is odd since I didn't change that logic in a long time. I'll look into it.

 

I've got one drive that shows the same erratic benchmark, on my backup system which should have zero outside activity against it. I removed the cache popping logic and that changed the results though it still shows some erratic hills but didn't throw the speed gap warning.

 

Version 2.6 pushed to remove cache popping logic.

Just re-ran the tests on v2.6

Much better results! I didn't receive the speed gap warning, either.

Thanks for the help.

DiskSpeed_results_20200305.png

DiskSpeed_controller_20200305.png

Edited by jademonkee
Added controller benchmark results
Link to comment

[SOLVED] At least partially solved. After investigating a little further it appears the 1st thing I had to resolve was putting the SAS2008 controller into a PCIe x8 slot.... not sure why the previous owner had the card in a x4 slot. Needless to say that has improved things dramatically taking the link speed to 5GT/s vs 2.5GT/s. I'll do a re-run of the full disk check overnight and see if it still hangs on the 9th drive at 36%.

 

<ORIGINAL POST>

 

I'm not sure if I missed something while scanning through this forum topic but I'm having apparent lockups of the Diskspeed docker/webpage while testing. I just moved my media server setup into a Supermicro CSE-847 with a x8DTN+ motherboard, dual Xeon X5650 cpus, and at the moment only 16GB of RAM. The 24 bay front backplane and 12 bay rear backplane are connected to a Dell Perc (LSI2008) in IT mode.

 

Unfortunately I've noticed my disk I/O for both the SSDs and the array are almost half what they were in the old Norco enclosure (specs still in my signature below). That's why I wanted to re-run DiskSpeed now that I'm in the new enclosure and running on 12 physical cores vs the 4 core i7 in my old Norco setup.

 

I have 22 drives in this system currently:

 

Parity: 2 x Ironwolf 10TB

Array: 14 x 10TB + 4 x 8TB (mix of Seagate, WD and HGST)

Cache: 2TB WD Blue SSD

UD for Dockers/VM: 1TB Samsung 850 EVO

 

I've tried 2 runs to check all drives. It has stopped testing (or at least outputting to the web page) at 36% through the 9th drive on both runs. For some reason the drives were tested in a different order for each of the 2 runs, but not sure if that's related to the apparent lockup.

 

If I try and refresh the webpage while it appears hung, it resends the test request and starts the testing over again. Note that clicking on 'Abort benchmarking' while it seems to be hung does nothing. The webpage just sits there with no further updates. The only way to get it to respond is to refresh the page which restarts the testing.

 

I did a restart of the system with all Dockers and VMs disabled other than DiskSpeed. I tried to create a debug file from the link in the lower left of the DiskSpeed startup webpage, but the tar file created is empty (wanted to see what was being sent before emailing it). My next test is to purge the DiskSpeed config and try from scratch.

 

Note that I just attempted to run the test using Safari instead of my default of Firefox and got the same result. The testing appears to hang at 36% of what appears to be the 9th drive tested. Again, the drives were tested in a different order than the other 2 runs. Let me know if there are any suggestions as to the cause or anything else that I can provide to help troubleshoot.

 

Thanks!

Dale

 

Edited by AgentXXL
Found a partial solution....
Link to comment
2 hours ago, AgentXXL said:

Needless to say that has improved things dramatically taking the link speed to 5GT/s vs 2.5GT/s.

2.5GT/s means a PCIe 1.0 slot, if it was also a x4 link instead of x8 it would only have a quarter of normal bandwidth, so big impact is expected.

Link to comment
31 minutes ago, johnnie.black said:

2.5GT/s means a PCIe 1.0 slot, if it was also a x4 link instead of x8 it would only have a quarter of normal bandwidth, so big impact is expected.

Yes, I found that the HBA was indeed limited by the PCIe slot. I waded through some logs and saw the link speed at 2.5 and knew something wasn't right. Then I used the DiskSpeed docker to benchmark the controller and it also reported the link speed as 2.5 at the start of the test. Moved the HBA to a x8 slot and it's now running at 5. The Supermicro is certainly an older unit and the previous owner did say it wasn't as fast as they wanted - turned it into a half-decent upgrade for me. Rare to find them locally, especially for the price I paid.

 

I'm still planning to eventually replace the motherboard with something faster and more modern. I've read a bit about using more than a single cable to the SAS backplane so I'll keep my eyes open for a deal on a better HBA. DiskSpeed also mentioned in the post-controller benchmark that the controller was underpowered for the number of drives I'm using. That gives me more ammo to watch for a better HBA. 

 

For now I'm happier as my disk I/O speeds doubled. After running a full check with the DiskSpeed docker I'm hoping that the tunables can be tweaked to get me even a bit more performance.

 

Link to comment
13 hours ago, AgentXXL said:

[SOLVED] At least partially solved. After investigating a little further it appears the 1st thing I had to resolve was putting the SAS2008 controller into a PCIe x8 slot.... not sure why the previous owner had the card in a x4 slot. Needless to say that has improved things dramatically taking the link speed to 5GT/s vs 2.5GT/s. I'll do a re-run of the full disk check overnight and see if it still hangs on the 9th drive at 36%.

The controller benchmark tests in drive over, then sdx order. The all drive benchmark does it in port order which should also be sdx order.

 

If you click around on the period at the end of the "Click on a drive label to hide or show it", there's a hidden link on it that will unhide the iframes that are doing the actual work and you can see the debug messages it's printing out or error log if it seems to be hanging.

Link to comment
9 hours ago, AgentXXL said:

Yes, I found that the HBA was indeed limited by the PCIe slot. I waded through some logs and saw the link speed at 2.5 and knew something wasn't right. Then I used the DiskSpeed docker to benchmark the controller and it also reported the link speed as 2.5 at the start of the test. Moved the HBA to a x8 slot and it's now running at 5. The Supermicro is certainly an older unit and the previous owner did say it wasn't as fast as they wanted - turned it into a half-decent upgrade for me. Rare to find them locally, especially for the price I paid.

Check the solder pin layout on the back of the board (may need to look up the board online), that'll tell you how the slots are actually wired regardless of how the physical slot size is.

Link to comment
7 hours ago, JaseNZ said:

Your website does not work when looking for drive images.

Thank you for bringing it to my attention. I recently moved the backend of the DiskSpeed docker off my web server at home to "the cloud" and I missed that. It's fixed.

 

I'm also updating the HDDB to give better graphs where it shows only the last benchmark per person per drive. What's out there now can be messy. Also working on cleaning up the crap drive manufacturers use for their identifications and adding newer drives & their images to ones that people haven't already contributed to the HDDB.

  • Thanks 2
Link to comment

Version 2.7 pushed.

  • Open up the permission on the created debug file
  • Added SMART drive information on Drive data view (brief by default, per-drive toggle for displaying all data)
  • Reworked the logic on the Drive information benchmark graph to display the benchmarks in descending order
  • Sorted benchmark graphs on drive information by date descending
  • Reworked the logic on the main page for displaying the most recent benchmarks for each drive to actually display the most recent benchmarks
  • Some history files weren't being saved with open permissions, updated to open them up

 

A "Manage Bookmarks" button is now displayed on the drive info page showing the following. Existing saved Benchmarks are displayed first and visible. Any previous benchmarks that were submitted to the HDDB will be displayed as hidden. Click on the drive label to toggle visibility. Any that are visible when you save are kept, the hidden ones are not. This will allow you to restore benchmarks if you did a full purge of data or if you have a LOT of benchmarks, allows you to delete all but the newest and oldest for example.

 

The SMART data will display lifespan & critical information by default and the rest can be seen by clicking the toggle under. This is a per-drive persistent toggle. If there's values that aren't displayed in the short list that you think should be, open a telnet session to your unraid box and change to the directory "/var/local/emhttp/smart" and cat the file for the drive in question and send me the label and why it should be displayed. Get the label from the file because the web version is "cleansed" to remove underlines/etc.

 

 

image.png.5d24de7198c085f44d3fb34b889ad719.png

 

  • Thanks 2
Link to comment

@jbartlett

Any idea why my 4 Toshiba N300s get mapped together as far as benchmarks go?

My 4 Segate Archive Drives didn't get lumped together in the benchmarks

image.png.11905e08559052b4a39d37b9814d53fe.png

Each of the Seagate get their own benchmark speed graph

Each N300 gets all of the N300 benchmarks together in the graph (there's one extra because I got confused as to which one had been benched)

image.png.574eea60fed0d3a9803a569261c6a897.png

Link to comment
2 hours ago, ken-ji said:

Each N300 gets all of the N300 benchmarks together in the graph (there's one extra because I got confused as to which one had been benched)

First, have you updated the docker since last night? If not, please do so as I did a lot of rework around the logic that builds the history files for display.

 

The graph data on each drive is stored independently of each other. To verify, please navigate to appdata/DiskSpeed/Instances/local/driveinfo on your NAS. You should see four directories representing these Toshiba drives
hdwn180_gx2m_3942k0gjfavg_8tb
hdwn180_gx2m_3992k0nqfavg_8tb
hdwn180_gx2m_399fk0g3favg_8tb
hdwn180_gx2m_79iuk0vufavg_8tb

 

If that is the case, please kick off a benchmark on them (or benchmark all drives from the main menu) on version 2.7+ which will rebuild the history files and should fix your problem.

Link to comment

Using 2.7,

Tried running the benchmarks again, but only got one benchmark run and its still assigned to all the drives.

image.png.ca7b18ca2e569b8aca9475ad25066182.png

Looking inside the container appdata

root@MediaStore:/mnt/disks/SSD/appdata/DiskSpeed/Instances/local/driveinfo# tree hdwn180_gx2m_*
hdwn180_gx2m_3942k0gjfavg_8tb
├── SMARTDescList.txt
├── benchmark
│   ├── 1583361860341
│   │   ├── 0000000000000000.txt
│   │   ├── 0000800155238400.txt
│   │   ├── 0001600311787520.txt
│   │   ├── 0002400468336640.txt
│   │   ├── 0003200624885760.txt
│   │   ├── 0004000781434880.txt
│   │   ├── 0004800936673280.txt
│   │   ├── 0005601093222400.txt
│   │   ├── 0006401249771520.txt
│   │   ├── 0007201406320640.txt
│   │   ├── 8001563222016.txt
│   │   ├── DriveLatency.sh
│   │   ├── DriveLatency.txt
│   │   ├── RandomSeek.sh
│   │   ├── RandomSeek.txt
│   │   ├── SequentialSeek.sh
│   │   ├── SequentialSeek.txt
│   │   ├── datestamp.txt
│   │   ├── latency.txt
│   │   ├── speed.json
│   │   ├── valid.txt
│   │   └── version.txt
│   ├── 1583377616913
│   │   ├── 0000000000000000.txt
│   │   ├── 0000800155238400.txt
│   │   ├── 0001600311787520.txt
│   │   ├── 0002400468336640.txt
│   │   ├── 0003200624885760.txt
│   │   ├── 0004000781434880.txt
│   │   ├── 0004800936673280.txt
│   │   ├── 0005601093222400.txt
│   │   ├── 0006401249771520.txt
│   │   ├── 0007201406320640.txt
│   │   ├── 8001563222016.txt
│   │   ├── DriveLatency.sh
│   │   ├── DriveLatency.txt
│   │   ├── RandomSeek.sh
│   │   ├── RandomSeek.txt
│   │   ├── SequentialSeek.sh
│   │   ├── SequentialSeek.txt
│   │   ├── datestamp.txt
│   │   ├── latency.txt
│   │   ├── speed.json
│   │   ├── valid.txt
│   │   └── version.txt
│   ├── 1583378375557
│   │   ├── 0000000000000000.txt
│   │   ├── 0000800155238400.txt
│   │   ├── 0001600311787520.txt
│   │   ├── 0002400468336640.txt
│   │   ├── 0003200624885760.txt
│   │   ├── 0004000781434880.txt
│   │   ├── 0004800936673280.txt
│   │   ├── 0005601093222400.txt
│   │   ├── 0006401249771520.txt
│   │   ├── 0007201406320640.txt
│   │   ├── 8001563222016.txt
│   │   ├── DriveLatency.sh
│   │   ├── DriveLatency.txt
│   │   ├── RandomSeek.sh
│   │   ├── RandomSeek.txt
│   │   ├── SequentialSeek.sh
│   │   ├── SequentialSeek.txt
│   │   ├── datestamp.txt
│   │   ├── latency.txt
│   │   ├── speed.json
│   │   ├── valid.txt
│   │   └── version.txt
│   ├── 1583411489259
│   │   ├── 0000000000000000.txt
│   │   ├── 0000800155238400.txt
│   │   ├── 0001600311787520.txt
│   │   ├── 0002400468336640.txt
│   │   ├── 0003200624885760.txt
│   │   ├── 0004000781434880.txt
│   │   ├── 0004800936673280.txt
│   │   ├── 0005601093222400.txt
│   │   ├── 0006401249771520.txt
│   │   ├── 0007201406320640.txt
│   │   ├── 8001563222016.txt
│   │   ├── DriveLatency.sh
│   │   ├── DriveLatency.txt
│   │   ├── RandomSeek.sh
│   │   ├── RandomSeek.txt
│   │   ├── SequentialSeek.sh
│   │   ├── SequentialSeek.txt
│   │   ├── datestamp.txt
│   │   ├── latency.txt
│   │   ├── speed.json
│   │   ├── valid.txt
│   │   └── version.txt
│   ├── 1583412489111
│   │   ├── 0000000000000000.txt
│   │   ├── 0000800155238400.txt
│   │   ├── 0001600311787520.txt
│   │   ├── 0002400468336640.txt
│   │   ├── 0003200624885760.txt
│   │   ├── 0004000781434880.txt
│   │   ├── 0004800936673280.txt
│   │   ├── 0005601093222400.txt
│   │   ├── 0006401249771520.txt
│   │   ├── 0007201406320640.txt
│   │   ├── 8001563222016.txt
│   │   ├── DriveLatency.sh
│   │   ├── DriveLatency.txt
│   │   ├── RandomSeek.sh
│   │   ├── RandomSeek.txt
│   │   ├── SequentialSeek.sh
│   │   ├── SequentialSeek.txt
│   │   ├── datestamp.txt
│   │   ├── latency.txt
│   │   ├── speed.json
│   │   ├── valid.txt
│   │   └── version.txt
│   ├── 1584049865832
│   │   ├── 0000000000000000.txt
│   │   ├── 0000800155238400.txt
│   │   ├── 0001600311787520.txt
│   │   ├── 0002400468336640.txt
│   │   ├── 0003200624885760.txt
│   │   ├── 0004000781434880.txt
│   │   ├── 0004800936673280.txt
│   │   ├── 0005601093222400.txt
│   │   ├── 0006401249771520.txt
│   │   ├── 0007201406320640.txt
│   │   ├── 8001563222016.txt
│   │   ├── DriveLatency.sh
│   │   ├── DriveLatency.txt
│   │   ├── RandomSeek.sh
│   │   ├── RandomSeek.txt
│   │   ├── SequentialSeek.sh
│   │   ├── SequentialSeek.txt
│   │   ├── datestamp.txt
│   │   ├── latency.txt
│   │   ├── nosubmit.txt
│   │   ├── speed.json
│   │   ├── valid.txt
│   │   └── version.txt
│   ├── BenchInfo.wddx
│   ├── allspeed.json
│   └── avgspeed.json
├── config.json
├── image.png
└── smartreport.wddx
hdwn180_gx2m_3992k0nqfavg_8tb
├── config.json
└── image.png
hdwn180_gx2m_399fk0g3favg_8tb
├── config.json
└── image.png
hdwn180_gx2m_79iuk0vufavg_8tb
├── config.json
└── image.png

7 directories, 146 files

 

Link to comment
On 3/4/2020 at 12:31 PM, jademonkee said:

Upgraded to v2.5 this morning and thought I'd re-run the benchmarks. 

Was receiving 'speed gap' notification, so I aborted, disabled all dockers and re-ran. Still got the 'speed gap' notification, so I aborted, and re-ran with the 'disable speed gap detection' option checked.

The final results page looks vastly different to the last time I ran it, however, going back to the main page, the curves look just as before (almost as if the main page didn't update to the latest results - even after I Ctrl+R refreshed the page).

Please see attached. 

So: is it something I should be worried about? I haven't received any notification from Unraid that anything is wrong with SMART, but should I do something to confirm that my disks (or controller?) are ok? 

Thanks for your help.

 

EDIT: I just ran the controller benchmark, which showed erratic results, and as per the recommendation I restarted the Docker and ran it again: same erratic results (although I think the disks with low speeds were different this time). Attached is a screenshot. 

 

Now that this is fixed in v2.6+, how do I delete the erroneous test results from my result history?

Do I have to purge everything, or can I just delete the erroneous ones?

Thanks.

Link to comment
9 hours ago, jademonkee said:

Now that this is fixed in v2.6+, how do I delete the erroneous test results from my result history?

Do I have to purge everything, or can I just delete the erroneous ones?

Thanks.

View the drive(s) in question and click on the "Manage Benchmarks" button. Hide the benchmarks you want to get rid of (or make visible the ones you want to restore).

  • Thanks 1
Link to comment
On 3/12/2020 at 3:33 PM, ken-ji said:

Using 2.7,

Tried running the benchmarks again, but only got one benchmark run and its still assigned to all the drives.

Can you create a debug file by clicking on "Create Debug File" link on the bottom of the page and clicking "Create Debug file with Controller Info" and then email the file it creates in your appdata/DiskSpeed share to [email protected]. I need to take a deeper look into your system.

Link to comment
  • 1 month later...

Hey all, firstly I hope everything is staying healthy and well in this tough time!

 

So I just found out about this tool, and I want to say thanks so much to JB for creating this - very easy to use and good info :)

 

*NOTE: if this is not the appropriate forum for this question, please feel free to move it*

 

So I have been having speed issues with my server for a long time, where I can not get anymore that 30/35Mb/s file transfer on any device in my home network, even when only transferring to one device at a time. I have tried testing SMB, FTP, SCP, AFP and they are all the same - I tried rsync and that came in at a thundering 8-9Mb/s - YAY! NOT! :( 

 

I was lazy and just dealt with it (bigger fish to fry!), but now that everyone is at the house, it is creating some issues. So I used this tool to see what my drives are doing to try and see if there is any issues.

 

I have a very simple set up

24GB RAM

Intel® Core i7-3770 CPU @ 3.40GHz

24TB array with cache pool

A few dockers (Emby, sonarr, radarr, lidarr, NZBget, deluge etc), and couple of VMs I spin up when I want to use them.

Full gigabit network and all devices support 1gb .... all client devices have SSDs or flash based storage apart from my unraid box.

 

According to the benchmarks the drives average around 160Mb/s, so I should be able to saturate my gigabit network without issue, but thats never been the case - I have tried 3 different gigabit switches, with the same result. No drive errors are presenting either.

 

Here are some screens

 

UnRAID-ArrayDetails01.thumb.png.61f55fc506cf2d88ead76100bb1314e3.png

 

UnRAID-ArrayDetails02.thumb.png.45616a3ee7c51f1c55032265a51a53fd.png

 

 

DiskSpeed-DriveOverview.thumb.png.74c4de1a10c60a1f3c381b46b39c66ac.png

 

DiskSpeed-Controller-Benchmark.thumb.png.92c38c28dd2c47dd44bd80f1d610d7e6.png

 

DiskSpeed-Disk-Benchmark.thumb.png.0dc125f45e60fe4d814de73e94d1cb12.png

 

Diagnostics

unraid-diagnostics-20200417-2353.zip

 

Any help at all would be appreciated guys! :)

 

Link to comment
9 hours ago, blade316 said:

So I just found out about this tool, and I want to say thanks so much to JB for creating this - very easy to use and good info

Thanks!

9 hours ago, blade316 said:

Any help at all would be appreciated guys!

Have you tried swapping out your lan cables & switches? Try connecting with only one switch between a PC & the server?

Link to comment
1 hour ago, jbartlett said:

Thanks!

Have you tried swapping out your lan cables & switches? Try connecting with only one switch between a PC & the server?

Yeah I have tried 3 different gigabit switches, with the same result. There is only 1 switch between clients and server. I don’t think it’s the LAN cables as copying from client to client is yielding around 75MB/s.

 

I’ve attached diags I took while I was copying, not sure if it would contain anything different to diags when not copying.

unraid-diagnostics-20200418-0013 (during file copy).zip

Link to comment
  • 2 weeks later...

Currently in development - Drive Heat Maps. Each square = 1GB. Each line here is 100GB. Color represents MB/s read over that GB. And it looks like I'll be able to tell you what file is located where on the drive.

 

Initial prototype. I'm currently going through my old drives trying to find one with a speed dip.

image.thumb.png.5811ce75a913145e582c2d4eece3729a.png

Edited by jbartlett
  • Like 1
  • Thanks 1
Link to comment
12 hours ago, jbartlett said:

And it looks like I'll be able to tell you what file is located where on the drive.

That sounds like a recipe to enable parity based file corruption detection. Instead of just telling you that parity is wrong at a certain address, you could take that address and generate a list of potentially affected files, one on each drive, if a file exists at that address. Then before you correct parity, you could verify each file against backup or external checksum to determine whether the the data drive should be updated instead. With dual parity, maybe you could correct corruption automatically?

 

It's been too long since I worked through the scenarios with dual parity and data correction, but I have it in the back of my mind that one of the stumbling blocks was determining exactly which files corresponded to a specific address.

  • Thanks 1
Link to comment
  • jbartlett changed the title to DiskSpeed, hdd/ssd benchmarking (unRAID 6+), version 2.10.7

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.