Drive performance testing (version 2.6.5) for UNRAID 5 thru 6.4

[JorgeB]

Looking good ?

[pwm]

Magnificent work.

 

Would be lovely with integrated inventory support, to store purchase date, purchase location, warranty length and some free text for each drive.

[jbartlett]

8 hours ago, pwm said:

Magnificent work.

 

Would be lovely with integrated inventory support, to store purchase date, purchase location, warranty length and some free text for each drive.

 

Excellent idea. The plugin that I used to map out where drives are located in the case hasn't worked for me for awhile now.

 

Note that manufacture/purchase date & warranty period can be added to the drive information screen in UNRAID (click on "Drive 1" for example and scroll down).

[Squid]

3 minutes ago, jbartlett said:

The plugin that I used to map out where drives are located in the case hasn't worked for me for awhile now.

 

https://lime-technology.com/forums/topic/38543-server-layout-plugin-for-unraid-v6/?page=11&tab=comments#comment-611122

 

[jbartlett]

2 hours ago, Squid said:

 

https://lime-technology.com/forums/topic/38543-server-layout-plugin-for-unraid-v6/?page=11&tab=comments#comment-611122

 

 

Sweet! I'll give this a try when I reboot my main box this evening to install 6.4 stable.

[jbartlett]

Put together tonight the start of a website to function as a hard drive database which I have plans to eventually make into a replacement for the other hard drive database web site that went offline - with drive information pulled from the drives themselves and a world min-max-avg drive throughput from multiple users. This site will also be used by the DiskSpeed plugin to link people to for drives that they don't have images for.

 

http://strangejourney.net/hddb.cfm?View=images

[jbartlett]

Send me a PM with your email address if you would like to be part of the pre-alpha plugin installation validation. This is a sanity shakeout of installing the Java & Lucee app server and quick test of Other People's Hardware.

 

Tested against a few motherboard drive controllers (intel/amd/virtualbox), NVMe controllers, SAS controller (LSI 9211-8i), PCI-e SATA cards, & PCI IDE cards. Unix treated them all the same except for the SAS controller which had it's configuration stored differently on the OS. There is a potential that some odd-wall controller might not be detected properly but there's no risk to your hardware.

 

Debug/troubleshooting will be shared via email via the PM thread.

Edited January 29, 2018 by jbartlett

[jbartlett]

Pre-alpha testing will be done via PM thread now that I see the PM functionality is a lot more robust than it used to be. Thanks to anyone who wants to assist with this.

[NewDisplayName]

That would be cool as plugin/docker and maybe auto run once a month and report if speed is below X%, so you might catch a hdd befrore it dies.

[jbartlett]

I considered it as a docker thing at the start but a Docker app can't access tools for scanning hardware on the host. If that could be possible, I'd gun for that route.

Edited January 30, 2018 by jbartlett

[jbartlett]

Pondering.

 

I could have it as a Docker app but I'd still have to have a companion Plugin which will run certain functions called via HTTP from the Docker app. It can get messy though, both would have to be in sync version wise but logic can be added for that. It would have a smaller footprint even though I've moved most of the footprint off to the flash drive as-is. Memory footprint is around 250MB or under.

 

I'll brainstorm this.

[NewDisplayName]

Hey, you can add a docker as "privileged" which would give the same rights as host, as far as i know.

 

But i guess, its possible to do this without. 

[jbartlett]

On 1/30/2018 at 3:35 AM, nuhll said:

Hey, you can add a docker as "privileged" which would give the same rights as host, as far as i know.

 

But i guess, its possible to do this without. 

 

I spent yesterday revisiting the Docker approach vs a Plugin and SUCCESS!

 

The issue I had before was that the commands I needed to scan the hardware such as lspci or lshw didn't exist in the Docker environment but I discovered that if I map out the directories under UNRAID to the container ("/usr/sbin" > "/usr/sbin2" for example), I was able to execute the command from the mapped directory and scan the controller & drive information on the host environment with the Docker running in privileged mode.

 

I think this will provide a much more satisfactory experience and provide the ease of only having the memory resources required by the Lucee app server being utilized as needed.

[jbartlett]

Hrm. Hitch. The Plugin approach allowed me to install the nvme-cli utility on UNRAID to fetch information on nvme drives. Might have to have an accompany plugin to provide additional support to the Docker app. Ie: If you want nvme drive support, you have to also install the "DiskSpeed Sidekick" (or whatever) plugin.

Edited January 31, 2018 by jbartlett

[pwm]

Docker is great for designing a generic "plugin" that contains all that is required (runtime libraries, tools, ...) to run a specific service. And the docker may then be run on a large number of different platforms without having to care what OS the machine has.

 

But unRAID is a quite homogenous solution.

If your goal is to run your drive performance tester on a number of different targets besides unRAID, then Docker is a good choice.

But is that your goal? If unRAID is the intended target, then what would the actual advantages be of going the Docker route?

 

[sureguy]

3 hours ago, jbartlett said:

Hrm. Hitch. The Plugin approach allowed me to install the nvme-cli utility on UNRAID to fetch information on nvme drives. Might have to have an accompany plugin to provide additional support to the Docker app. Ie: If you want nvme drive support, you have to also install the "DiskSpeed Sidekick" (or whatever) plugin.

 

Is it possible that this should be part of unRAID?  I"ve not got any nvme drives, but I'm assuming more users will be moving to nvme drives in the future and other software will benefit from having it available.

[trurl]

3 hours ago, pwm said:

Docker is great for designing a generic "plugin" that contains all that is required (runtime libraries, tools, ...) to run a specific service. And the docker may then be run on a large number of different platforms without having to care what OS the machine has.

 

But unRAID is a quite homogenous solution.

If your goal is to run your drive performance tester on a number of different targets besides unRAID, then Docker is a good choice.

But is that your goal? If unRAID is the intended target, then what would the actual advantages be of going the Docker route?

 

 

This really seems like a good case for a plugin to me. unBalance started as a docker but was converted to a plugin since it really needed to work with all of the disks and their files. No point in making mappings when you are going to access everything anyway.

 

In the case of drive performance testing, since it is working with all drives, and not really even accessing any user files, not sure what making it a docker really does for us. And mapping the unRAID OS folders really seems like breaking the isolation docker is intended to provide.

[ljm42]

@jbartlett mentioned that this needs a "Java & Lucee app server". That puts it out of the realm of a normal plugin, I would prefer to have it isolated in a docker. 

[BobPhoenix]

4 hours ago, jbartlett said:

The issue I had before was that the commands I needed to scan the hardware such as lspci or lshw didn't exist in the Docker environment but I discovered that if I map out the directories under UNRAID to the container ("/usr/sbin" > "/usr/sbin2" for example), I was able to execute the command from the mapped directory and scan the controller & drive information on the host environment with the Docker running in privileged mode.

I did something similar so I could have code within a docker restart the docker.  If you map a new path with the following "/var/run/docker.sock" on the container and host paths and another new path of "/usr/bin/docker" on the container and host paths.  Then you can restart your docker with the "/usr/bin/docker restart <container name here>" command from within the docker.  Then you can have the docker do scheduled restarts of itself.

Edited January 31, 2018 by BobPhoenix

[jbartlett]

Yup, running a java/lucee app server as this tool is being written in ColdFusion which will consume around 256MB of RAM while running. I'm knowledgeable enough to google may way though PHP but honestly, I wouldn't be anywhere near where I am now if I coded it in PHP from the start. I've been actively programming in ColdFusion just shy of two decades and my PHP work so far is akin to poking & banging at it until I believed it was working.

 

I figured out a way around mapping the directories via Docker with the dockerfile "ADD" command to install the different packages directly into the Docker container but I kept running into dependency issues. Basing the Docker on "debian:9.3-slim" first allowed me to use apt-get to install all of the desired packages right into the docker at build time.

 

So far, I'm down to just three mappings: 1 port, one for local storage, and one for the unRAID ini files in /var/local/emhttp. Running the Docker in privileged mode lets the commands such as lspci, lshw, and dd (for read performance) run against the host's hardware.

 

Going with Docker is making things a lot cleaner for me and I believe will be easier for the end user. The end user will be able to start up the Docker when they want to run tests against their drives and stop it when done to free up the resources it used. After getting feedback from users with minimal RAM available, this sounds like a good compromise.

 

I'm still working on porting the code to work from inside a Docker container so I could still hit a deal-breaker wall and have to go back with a Plugin.

[jbartlett]

All of the controller & drive scanning is working under Docker but still have a few quirks to work out with the different host environment.

[jbartlett]

22 hours ago, pwm said:

If your goal is to run your drive performance tester on a number of different targets besides unRAID, then Docker is a good choice.

But is that your goal? If unRAID is the intended target, then what would the actual advantages be of going the Docker route?

 

My ultimate goal is to be able to branch it out past unRAID and onto Unix in general and hopefully Windows one day. My goal from the start was to minimize dependencies on unRAID and to derive all information from the OS itself. As things stand now, the only information it needs UNRAID for is drive slot assignment.

 

Another one of my goals is to use this tool to gather information for a Hard Drive database since the "The HDD Platter Capacity Database" has gone offline. People will be able to submit their drive scans to a global database and compare their drive's performance with others. With a full drive scan giving a heat map of the drive's performance, determining the platter setup should be possible for drives that manufacturers don't provide any such information.

 

 

[JorgeB]

7 minutes ago, jbartlett said:

Another one of my goals is to use this tool to gather information for a Hard Drive database since the "The HDD Platter Capacity Database" has gone offline. People will be able to submit their drive scans to a global database and compare their drive's performance with others. With a full drive scan giving a heat map of the drive's performance, determining the platter setup should be possible for drives that manufacturers don't provide any such information.

?

[CraziFuzzy]

I definitely don't see any reason why this would gain anything from the added complication of going the docker approach.  Plugin seems the obvious place for this functionality.

[SiNtEnEl]

Just tested the docker approach by running dd from a container and i'm getting same disk performance as the alpha version of diskspeed. Was concerned it may influence the benchmarks.