Squid Posted July 25, 2017 Share Posted July 25, 2017 49 minutes ago, FreeMan said: I think that if anyone has an issue with that memory footprint, they should drag their server into the 21st century. Sounds like a good plan! Looking forward to the updates, and now that I discovered there's a plugin (and installed it!) I'll be sure to get them! You may want to consider updating the OP to indicate (up front and in big red letters) that there's a plugin now. I scanned the OP a day or two ago (i.e. looked at the bottom where the attachments are) looking to see if there was an update to the version I'd DL'd a while back. It wasn't until I started browsing back through previous posts that I started seeing mentions of a plugin. The plugin is a semi-hacked version of the script posted in the OP done by myself and while an authorized fork is not the "official" script. From reading, it actually sounds like @jbartlett is working on a new script with a fresh plugin of his own design from the ground up. Quote Link to comment
FreeMan Posted July 25, 2017 Share Posted July 25, 2017 14 minutes ago, Squid said: The plugin is a semi-hacked version of the script posted in the OP done by myself and while an authorized fork is not the "official" script. From reading, it actually sounds like @jbartlett is working on a new script with a fresh plugin of his own design from the ground up. Well... thanks for the semi-hack! And now that I'm figuring out the new forums, I think I'll get a notification now of when the official plugin is posted. Quote Link to comment
jbartlett Posted July 25, 2017 Author Share Posted July 25, 2017 Do use Squid's plugin in the meantime, it's pretty awesome. Quote Link to comment
Squid Posted July 26, 2017 Share Posted July 26, 2017 46 minutes ago, FreeMan said: I think I'll get a notification now of when the official plugin is posted. If/when @jbartlett releases a homegrown (and probably far better version), that plugin will wind up getting deprecated which will trigger Fix Common Problems (if you have it installed) to let you know about it. Quote Link to comment
jbartlett Posted July 26, 2017 Author Share Posted July 26, 2017 11 hours ago, FreeMan said: I think that if anyone has an issue with that memory footprint, they should drag their server into the 21st century. It's more of me thinking about systems that may be maxed out with VM's. Quote Link to comment
jbartlett Posted July 26, 2017 Author Share Posted July 26, 2017 Question if anyone knows, does one of the tunable settings equate to something like the block size of a hard drive? I'm wondering because if I'm determining the fastest block size for each drive, I should be able to compute a value that would yield the best read speed over all the drives for parity checks. In theory anyways. Quote Link to comment
JorgeB Posted July 26, 2017 Share Posted July 26, 2017 1 hour ago, jbartlett said: Question if anyone knows, does one of the tunable settings equate to something like the block size of a hard drive? You can read Tom's explanation, num_stripes and sync_window: And the newer sync_thresh: Quote Link to comment
FreeMan Posted July 27, 2017 Share Posted July 27, 2017 Great info, @johnnie.black. Reading those 2 comments make the tunables make so much more sense than I remember last time I read about them. Now, I just need to find some time I can take the server off-line to run the tunable tester and see what a difference I can make. We now return you to your regularly scheduled thread. Quote Link to comment
stomp Posted September 2, 2017 Share Posted September 2, 2017 Very useful. Thanks. Quote Link to comment
jbartlett Posted September 12, 2017 Author Share Posted September 12, 2017 (edited) Development on the new version is plugging along, slowly but surely. I'm currently working on bench marking controller cards. Here's the controller bandwidth test results for one motherboard controller on my main server with 6 Red Pro drives attached - and capacity wasn't maxed. I think I'll need to hook up a bunch of SSD's to my LSI 8 port card. In theory, once capacity has been maxed on the controller, the amount of data being read by each drive would shrink while the total data being read from all drives combined remained the same. Edited September 12, 2017 by jbartlett 4 1 Quote Link to comment
JorgeB Posted September 12, 2017 Share Posted September 12, 2017 31 minutes ago, jbartlett said: Here's the controller bandwidth test results for one motherboard controller on my main server with 6 Red Pro drives attached - and capacity wasn't maxed Limit here is going to be the DMI bus, around 1500/1600MB/s. 32 minutes ago, jbartlett said: I think I'll need to hook up a bunch of SSD's to my LSI 8 port card You'll need SSDs, max bandwidth if it's a SAS2008 variant is around >2500MB/s. I have a SSD only test server, with various controllers, happy to help testing if needed. Quote Link to comment
Videodr0me Posted September 15, 2017 Share Posted September 15, 2017 (edited) This graph is fantastic and exactely what i am looking forward to. Can't wait for a prerelease of the plugin. Will it look as pretty with 16 drives? Also please make an option where the user can choose the drives and sequence for this test. Not all my drives are in the array and even those are on different controllers. Edited September 15, 2017 by Videodr0me Quote Link to comment
jbartlett Posted September 15, 2017 Author Share Posted September 15, 2017 1 hour ago, Videodr0me said: This graph is fantastic and exactely what i am looking forward to. Can't wait for a prerelease of the plugin. Will it look as pretty with 16 drives? Also please make an option where the user can choose the drives and sequence for this test. Not all my drives are in the array and even those are on different controllers. It'll currently test the controller with all drives attached, adding an additional drive each iteration until all are used. Exception being that there's no benchmark if only one drive is attached. This test kicks off x number of drives with a full unrestricted read of the hard drives using the optimal block size previously determined. After 5 seconds, it kills the PIDs that were launched and records the last logged transfer rate. I also figure that this is a good test to see how the controller is with handling multiple drives at the same time. This tool doesn't care if the drives are in the UNRAID array or not other than for display purposes. Testing a full system drive push with all drives on all controllers active. Interesting. I can also identify if multiple controllers are nested under a given internal PCI/bus so that may be a future test - or make it so you can specify which controller cards to simultaneously test. Note: Each drive's optimal transfer speed is a combination of the drive itself along with the controller & bus. If a hard drive switches controllers or the controller itself is moved to a different slot, the app will require the optimal block rate to be retested. However, if the drive is put back to where it once was, it will reuse the previously detected optimal block size. Here's a graph with the app running under VirtualBOX with each virtual drive running on a Samsung 960 EVO M.2 500GB NVMe. This is a good example of maximizing the controller that is not running optimally. 1 Quote Link to comment
jbartlett Posted September 15, 2017 Author Share Posted September 15, 2017 I'm working from the ground up, starting with the drive & controller identification, and testing the controller. The next test I plan to add is the one that does the same test as the original DiskSpeed test does. However, the next part I'll be working on is adding customization for the drive image/info. On initial first-time run, the app will fetch the model information from my server for each brand and will pre-set the drive image based on the first image found. If no information was found on the drive model, a generic drive image will be used instead. The goal is to let you upload a new image, specify if you want to overlay the drive capacity over the image, and if so, the size, position, and rotation of the text. You'll have the option to upload your drive image and overlay settings (if any) to my central repository. Another way to select drive images is to get a view of every available drive image and select the one you want to use. Examples are when a brand such as Western Digital uses one basic image for an entire model line (think WD Black) but then later comes out with a new style for it. You'll be able to select between them. Once that part is done, I'll put out a call for people to email me if they want to be on the pre-alpha test team. This team's purpose is to see if my controller & drive identification logic works on different systems, motherboards, controllers, etc. So far, I had to rewrite the logic which detected how many drive ports are on the controller and which specific drive was attached to it after I installed a LSI SAS card. Totally different layout on the OS than motherboard and SATA card controllers I've tested. It's possible there could be some other controller that I haven't experienced or coded for. 1 Quote Link to comment
Smitty2k1 Posted September 21, 2017 Share Posted September 21, 2017 Thanks for all the hard work! Looks like it's paying off! Quote Link to comment
bonienl Posted October 11, 2017 Share Posted October 11, 2017 (edited) @jbartlett in the upcoming unRAID version 6.4 status reading from /proc/mdcmd is deprecated. mdcmd is only available to execute commands. Status can be read using /proc/mdstat with the same output as before. To ensure compatibility between old and new unRAID versions, scripts need to be updated. Perhaps the easiest way to do this is by introducing a variable. For example [[ -e /proc/mdstat ]] && mdcmd=mdstat || mdcmd=mdcmd And use the variable to do status reading. For example grep "mdNumDisabled=" /proc/$mdcmd And thank you for your tool Recently I purchased new 10TB drives and speeds are impressive (they start at 270MB/s). This is my graph. Edited October 11, 2017 by bonienl 1 Quote Link to comment
jbartlett Posted October 11, 2017 Author Share Posted October 11, 2017 7 minutes ago, bonienl said: @jbartlett in the upcoming unRAID version 6.4 status reading from /proc/mdcmd is deprecated. mdcmd is only available to execute commands. Status can be read using /proc/mdstat with the same output as before. Thank you for the heads up. I'm developing under 6.4 RC so I already noticed that so I opted to parse out the file disks.ini to get the drive assignments & additional information. I didn't know about mdstat but I'll make a note of that if I need it in the future. Quote Link to comment
jbartlett Posted October 11, 2017 Author Share Posted October 11, 2017 That's some friggin' impressive speeds on those 10TB drives! Here's a screen shot of the current Drive Configuration screen. You will have the option uploading a new drive image (to do) and optionally overlaying the drive capacity over the image. The color, location, size, and rotation are each individually configurable and the text will dynamically update as the sliders move. 1 Quote Link to comment
JorgeB Posted October 11, 2017 Share Posted October 11, 2017 1 hour ago, bonienl said: Recently I purchased new 10TB drives and speeds are impressive Wow, 200MB/s average speed, they are impressive Quote Link to comment
JorgeB Posted October 11, 2017 Share Posted October 11, 2017 Also of note are the speed of those 4TB Reds, I also noticed that the newer 4TB Blues are the same and noticeably faster the the standard 4 x 1TB platter models (average speed ~145MB/s instead of ~125MB/s), I suspect these newer models are short stroked with the same 1.2TB platters as the 6TB version. There used to be a great blog with HDDs platter size database, but it appears to be gone for some time now. Quote Link to comment
bonienl Posted October 11, 2017 Share Posted October 11, 2017 7 hours ago, johnnie.black said: There used to be a great blog with HDDs platter size database, but it appears to be gone for some time now. That is too bad, because manufacturers hardly (never) put this information in their drive spec's. Quote Link to comment
jbartlett Posted January 4, 2018 Author Share Posted January 4, 2018 Status Report on version 3 Life got in the way for the past couple months and development slowed quite a bit ("Assassin's Creed: Origins" didn't help either) but I've put in some solid development time over the past week and have made great progress. I hope to have an alpha version ready inside a week. I've been quite satisfied at the progress I've made and getting excited. Installation will be via the Plugin interface and the GUI will be fully in the browser with no telnet sessions. While I say "alpha", it'll be a fairly tested product having been tested on two UNRAID environments plus a VirtualBox environment and against both intel & amd motherboards and different controller cards - but it hasn't been tested against other user's hardware. 4 Quote Link to comment
CraziFuzzy Posted January 10, 2018 Share Posted January 10, 2018 As I'll be throwing a couple new drives in the box in the next day or so - I am very much anticipating you getting this pushed out. Thank you for the time well spent! Quote Link to comment
jbartlett Posted January 20, 2018 Author Share Posted January 20, 2018 (edited) I know I said it would be around a week and I'm trying hard to get things situated & cleaned up enough to put out the first alpha which will test controller & drive detection. A subsequent release will give disk level testing the way the old bash script did it. I'm currently building up the hard drive database so user's drive populate with model specific images instead of a generic default drive image and setting up a site where people can download images to use for models that I (gasp) don't have. I currently have 414 drive models (spinners, SSD's, & m.2) over 12 vendors and currently adding more SSD drives. Stupid OCD is making me build as complete of a database as I can for user satisfaction. Here are current screen shots of my dev/backup UNRAID server where I tossed in every controller card I could find and attached drives to them. Not every controller had drives powered up when I took these screenshots so they weren't listed (only controllers with drives attached are shown). NVME controllers & drives will be fully supported. Controller & Drive detection Scanning Hardware 04:15:42 Spinning up hard drives 04:15:42 Scanning storage controllers 04:15:52 Found Controller FCH SATA Controller [AHCI mode] (8 ports) 04:15:52 Found Controller 88SE912x IDE Controller (2 ports) 04:15:52 Found Controller 88SX7042 PCI-e 4-port SATA-II (4 ports) 04:15:52 Found Controller SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (8 ports) 04:15:52 Found Controller 88SE9123 PCIe SATA 6.0 Gb/s controller (8 ports) 04:15:52 Scanning hard drives 04:15:53 Found drive Western Digital WD6002FFWX Rev: 83.H0A83 Serial: K1GWVY1D (sdb) 04:15:53 Found drive Seagate ST4000DM000 Rev: CC51 Serial: Z3004WXB (sdc) 04:15:53 Found drive Seagate ST8000AS0002 Rev: AR15 Serial: Z840AF1T (sdd) 04:15:53 Found drive Seagate ST4000DM000 Rev: CC52 Serial: Z300RKE4 (sde) 04:15:53 Found drive Seagate ST8000AS0002 Rev: AR15 Serial: Z840AMC7 (sdf) 04:15:53 Found drive Seagate ST4000DM000 Rev: CC54 Serial: S301BCMR (sdg) 04:15:53 Found drive Hitachi HDS5C4040ALE630 Rev: MPAOA3B0 Serial: PL2331LAG924HJ (sdh) 04:15:53 Found drive Western Digital WD2002FAEX Rev: 05.01D05 Serial: WMAY01934977 (sdi) 04:15:53 Found drive Western Digital WD30EZRX Rev: 80.00A80 Serial: WMC4N1095471 (sdj) 04:15:53 Found drive Seagate ST4000DM000 Rev: CC51 Serial: Z300937K (sdk) 04:15:53 Found drive Seagate ST3000DM001 Rev: CC4C Serial: S1F034G7 (sdl) 04:15:53 Fetching drive images 04:15:53 Drive image found for Western Digital WD6002FFWX 04:15:54 Drive image found for Seagate ST4000DM000 04:15:54 Drive image found for Seagate ST8000AS0002 04:15:54 Drive image found for Western Digital WD2002FAEX 04:15:54 Drive image found for Western Digital WD30EZRX 04:15:54 Drive image found for Seagate ST3000DM001 04:15:54 Configuration saved Controller & Drives attached Controller Information Drive Information Drive Configuration Drive image uploading will support drag-n-drop and in-browser rotation & cropping (that was a PITA to code!). If you come up with a new drive layout you prefer more, you'll be able to share it with other users and be able to select from all user submitted versions of a drive model. Edited January 20, 2018 by jbartlett 3 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.