Pauven

With a slight modification, that did the trick, thanks! I had to add a -n to sort numerically, so the final command was sort -n -t: -k3

They are sorted, but it is an alpha sort, so 1 & 10-19 all sort before 2. Here's the code that outputs those disks and sorts them: for Disk in ${Disks[@]} do echo "${DiskSCSI[$Disk]} ${DiskNamePretty[$Disk]} ${DiskName[$Disk]} ${DiskSizePretty[$Disk]} ${DiskID[$Disk]//_/ }" done | sort >> $ReportFile As you can see, I simply pipe all of the lines to the "sort" function. Does anyone know how I can make this sort numerical based upon the 3rd octet in [5:0:x:0]? The only idea I have is to prefix each line with the port numer, and to make it 2-digit with a leading zero, like this: [5] scsi5 mpt3sas - SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) 00 - [5:0:0:0] disk3 sde 8.00TB 02 - [5:0:2:0] disk1 sdf 8.00TB 03 - [5:0:3:0] disk9 sdg 8.00TB 04 - [5:0:4:0] disk10 sdh 8.00TB 05 - [5:0:5:0] parity sdi 8.00TB 06 - [5:0:6:0] disk2 sdj 8.00TB 07 - [5:0:7:0] disk11 sdk 8.00TB 08 - [5:0:8:0] disk5 sdl 8.00TB 09 - [5:0:9:0] parity2 sdm 8.00TB 10 - [5:0:10:0] disk8 sdn 8.00TB 11 - [5:0:11:0] disk7 sdo 8.00TB 12 - [5:0:12:0] disk6 sdp 8.00TB 13 - [5:0:13:0] disk4 sdq 8.00TB 14 - [5:0:14:0] disk12 sdr 8.00TB

Thanks @StevenD! I've fixed some things in the report, does this look right to you? SCSI Host Controllers and Connected Drives -------------------------------------------------- [0] scsi0 usb-storage - [0:0:0:0] flash sda 31.9GB [1] scsi1 ata_piix - [2] scsi2 ata_piix - [3] scsi3 vmw_pvscsi - PVSCSI SCSI Controller [4] scsi4 vmw_pvscsi - PVSCSI SCSI Controller [5] scsi5 mpt3sas - SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) [5:0:0:0] disk3 sde 8.00TB [5:0:10:0] disk8 sdn 8.00TB [5:0:11:0] disk7 sdo 8.00TB [5:0:12:0] disk6 sdp 8.00TB [5:0:13:0] disk4 sdq 8.00TB [5:0:14:0] disk12 sdr 8.00TB [5:0:2:0] disk1 sdf 8.00TB [5:0:3:0] disk9 sdg 8.00TB [5:0:4:0] disk10 sdh 8.00TB [5:0:5:0] parity sdi 8.00TB [5:0:6:0] disk2 sdj 8.00TB [5:0:7:0] disk11 sdk 8.00TB [5:0:8:0] disk5 sdl 8.00TB [5:0:9:0] parity2 sdm 8.00TB [N0] scsiN0 nvme0 - NVMe [N:0:4:1] cache nvme0n1 512GB

@StevenD, I'm working on using your values to plug into the report on my system, that way I should be able to 100% simulate your disc report output and get it fixed. I need a one more thing, if you can: egrep -i "\[|idx|name|type|device|color" /var/local/emhttp/disks.ini

Yeah, not everyone gets a super exiting report. Sorry. Fingers crossed! I feel your pain, been there. Thanks. Very disappointing I didn't get the report right. I wonder if your drives running 10 to 34 instead of 0 to 24 is having an impact on the logic. Multi-dimensional arrays sounds interesting, but I think the current flaw is very minor and just looks really bad. I'll give it one more go before trying a new approach. Actually, the NVMe disks did not show, just the NVMe controllers. Here's mine (NVMe way down at the bottom, which I got to show after adding the new lsscsi v0.30 upgrade function): SCSI Host Controllers and Connected Drives -------------------------------------------------- [0] scsi0 usb-storage - [0:0:0:0] flash sda 4.00GB Patriot Memory [1] scsi1 ahci - [2] scsi2 ahci - [3] scsi3 ahci - [4] scsi4 ahci - [5] scsi5 ahci - [6] scsi6 ahci - [7] scsi7 ahci - [8] scsi8 ahci - [9] scsi9 ahci - [10] scsi10 ahci - [11] scsi11 ahci - [12] scsi12 mvsas - HighPoint Technologies, Inc. [12:0:0:0] disk17 sdb 3.00TB WDC WD30EFRX-68A [12:0:1:0] disk18 sdc 3.00TB WDC WD30EFRX-68A [12:0:2:0] disk19 sdd 3.00TB WDC WD30EFRX-68E [12:0:3:0] disk20 sde 3.00TB WDC WD30EFRX-68E [12:0:4:0] parity2 sdf 8.00TB HGST HUH728080AL [12:0:5:0] parity sdg 8.00TB HGST HUH728080AL [13] scsi13 mvsas - HighPoint Technologies, Inc. [13:0:0:0] disk1 sdh 8.00TB HGST HUH728080AL [13:0:1:0] disk2 sdi 3.00TB WDC WD30EFRX-68A [13:0:2:0] disk3 sdj 3.00TB WDC WD30EFRX-68E [13:0:3:0] disk4 sdk 3.00TB WDC WD30EFRX-68A [13:0:4:0] disk5 sdl 3.00TB WDC WD30EFRX-68A [13:0:5:0] disk6 sdm 3.00TB WDC WD30EFRX-68A [13:0:6:0] disk7 sdn 3.00TB WDC WD30EFRX-68A [13:0:7:0] disk8 sdo 3.00TB WDC WD30EFRX-68A [14] scsi14 mvsas - HighPoint Technologies, Inc. [14:0:0:0] disk9 sdp 3.00TB WDC WD30EFRX-68A [14:0:1:0] disk10 sdq 3.00TB WDC WD30EFRX-68A [14:0:2:0] disk11 sdr 3.00TB WDC WD30EFRX-68A [14:0:3:0] disk12 sds 3.00TB WDC WD30EFRX-68A [14:0:4:0] disk13 sdt 3.00TB WDC WD30EFRX-68A [14:0:5:0] disk14 sdu 3.00TB WDC WD30EFRX-68E [14:0:6:0] disk15 sdv 4.00TB ST4000VN000-1H41 [14:0:7:0] disk16 sdw 4.00TB ST4000VN000-1H41 [N0] scsiN0 nvme0 - NVMe [N:0:2:1] cache nvme0n1 1.00TB Samsung SSD 960

UTT v4.1 BETA 1 is attached. I'm primarily concerned about the SCSI Hosts and Discs report, so if I could get a few users to run this with a Short test and post the reports that would be great. This does have the new logic to find the leading edge for Pass 2, rather than the peak, so feel free to run the longer tests if you desire, just run a Short first and share those results. Here's the changelog: # V4.1: Added a function to use the first result with 99.8% max speed for Pass 2 # Fixed Server Name in Notification messages (was hardcoded TOWER) # Many fixes to the SCSI Host Controllers and Connected Drives report # Added a function to check lsscsi version and optionally upgrade to v0.30 # Cosmetic menu tweaks - by Pauven 08/11/2019

So for users on older versions of Unraid 6.x, pre-6.7.0, would it be a good feature to have UTT offer to upgrade lsscsi to v0.30? If so, could someone help me out with the commands to do this? I'm a Windows guy, and I really get stumped when it comes to installing packages and updates unless there's a step by step guide. I looked at my own code from years ago to install lshw, and modified it to upgrade lsscsi to v0.30. Looks like it is working.

Thanks @jbartlett, that's exactly what I needed. I wanted to make sure that Cache2 was IDX 31. I'll post Beta 1 of UTT v4.1 here shortly for testing.

I finally figured out why my NVMe drives are not showing. On Unraid 6.6.6 (which is what I am running), the lsscsi version is 0.29, which doesn't have support for NVMe. Later versions of Unraid have lsscsi version 0.30, which is the latest and has NVMe support. Anyone know exactly what version of Unraid upgraded lsscsi to v0.30? Nevermind, I just read in the 6.7.0 release notes that lsscsi was upgraded to 0.30.

Much better. Looks like the accuracy is +/- 0.2 MB/s. The new logic in UTT v4.1 would have used md_sync_window 6144 (from TEST PASS 1_HIGH) for Pass 2, and test from 3072 - 9216. All things considered, I think the v.41 results would be identical to these results for you, as your server has a really flat curve that starts extremely low, and the new logic won't really affect those results.

Could anyone that has at least 2 Cache drives please run this command and provide the output: egrep "\[|idx" /var/local/emhttp/disks.ini

Not yet. I'm trying to get the disk report working correctly, and hope to have UTT 4.1 out soon, maybe even today...

Thanks for all the data @StevenD, you've been very helpful today.

Thanks @Xaero. I just added the -i to the egreps in the UTT script, just in case. Any idea why your nvme1n1 drive doesn't show up in your df -H results?

I'm thinking just array drives. This is for the SCSI Hot Controllers and Connected Drives report at the end of the UTT results. A lot of the report requires configuration data for array drives, and so far all these NVMe drives have been non-array Cache or Unassigned drives, so they don't fully make it into the report. I'm trying to connect data from various sources together, and so I need to see what NVMe array devices look like.

Wow!

If anyone has an NVMe drive as part of their array (not cache, but data or parity), please run the above.

@StevenD & @Xaero can you provide the output for: mdcmd status | grep "rdevStatus" and mdcmd status | grep "rdevName" and df -h

Thanks @StevenD & @wgstarks! I think I have what I need.

Thanks @StevenD! Can I bother you to also run: lsscsi -H and lsscsi -st

I like this sed fix, thanks! For those that have NVMe drives (and any others who want to share, the more samples the more I can program around variances), I need to see the output from this command: lshw -quiet -short -c storage

Currently, I see zero value in running a dirty pass with UTT. There's a reason we don't take pictures of the stars during the day, as the noise from the sun overpowers the faint starlight, and all you see is blue sky and a big ball of fire. The goal of UTT is to identify the right range of value combinations that works well on a server. If you are running a dirty pass, with events randomly running, then the fastest set of values might be tested during heaviest random load, and end up looking like the slowest combo, making the data worthless. I guarantee you that if you access your array for ANY reason, that the random reads and writes to random sectors on a hard drive, which force the heads to move out of position, will affect speeds. That's just simply physics. I think BRiT's results illustrate that perfectly. How much it affects the speeds is depending upon the nature of the reads/writes (big files, small files, 1 file, 1k files, position on disk platter, etc.) and your hardware/drives and which drive you're writing to, and is nearly impossible to replicate the exact same transactions pass after pass so that all UTT tests are under the exact same conditions. The only way for UTT to provide a comparative analysis of different combinations of values is to do so without any external influence. If you want to take pictures of stars, you want to do it on a dark, moonless night, with no clouds or light pollution. Safe mode is an easy way to turn off all the extraneous noise that is polluting the performance picture, which I why I recommended it as an option for BRiT. Any user that seems random highs/lows should consider running a UTT test in Safe Mode to see if that cleans up the results. "for sure", implying 100% certainty? No. But there are some manual tests you could do to get an idea. Essentially, run a clean (not dirty) UTT test. From that, take a few sets of values: Unraid stock values Thriftiest values Recommended values Fastest values And some values that consume even more memory, beyond the Fastest, that still provide fast speeds Apply each set of values, then start a Parity Check, then perform some read/write performance tests. I'll leave it up to you on how to do those performance tests, though there are plenty of ways to accomplish it. Whatever test you do, you should try to do it in a way that you can repeat it for each combination of Tunables above. Some ideas of tests: You could create a script to generate a random content file (Linux can do this easily), then make a copy of it, then delete it. You could read files from the array, or write files to the array over the network using a network benchmarking tool. You could do the network test from multiple client PC's concurrently for an even higher server load. You could stream a few movies to different PC's while running the parity check, seeing which Tunable settings provide a seamless viewing experience. Depending upon your own performance goals, you could be looking for which set of values provides the fastest parity checks, or which provides the fastest read/write speeds, or which has the best balance of both. In my small family, it's unlikely that I would ever be streaming more than 2 movies concurrently, but maybe you have a big family and lot's of TV's, and want to stream 6 movies concurrently. A real torture test would be to stream multiple movies from the exact same disk. Get 4 streams going, then start a parity check and see how far it progresses in 10 minutes, and monitor the streams for viewing glitches. Change your tunables, reset your streams, then start a new parity check and measure again. I think you could do all this in just a few hours. I would say that this approach is better than trying to run a dirty UTT test. UTT can ONLY tell you which values are good when run clean. But you can then take those values and do some manual load testing with them, to see which ones perform well when the server is being tasked with various concurrent loads. One last idea: I had started programming some read/write tests for UTT v3 which I didn't release. Unraid 6.x broke those tests (I don't remember why, but they somehow got broken), so I removed them from the new UTT v4, but I'm more than happy to post them if someone wants to take a stab at making them work under Unraid 6.x. It's been a while since I looked at them, but I believe that these read/write tests tried to automate some of what I discussed above - running repeatable read/write tests while running parity checks with different Tunable values. Essentially, you would pick a disk in your array to write the file to, pick a size for the file, and the script would write a file of that size using random data (isolating write speed from all other factors), then read it back to test read speed. But to be honest, in testing on my own server, I never found that a server under load responded differently to the Tunables than a parity check on a completely idle server - meaning that the fastest UTT values remained fastest even under load. But that was just on my hardware, and may not be an absolute for every server. For that reason, I don't find trying to tune for heavy server load worth my time, as in my experience tuning for Parity Check speeds works for all loads, on my hardware.

That checks out. Looks like the slowdowns were about every 5 tests. Since you were running 10-minute tests, and there's on average about 30 seconds of overhead on each of your tests (varies by array size and speed), every 5 tests is about 53 minutes. CacheDirs is less of a concern, since by rapidly scanning your directories, you keeping your disks from spinning up, so no disk access actually occurs.

I'm noticing a lot of variability in your results. While the fastest result from Pass 1 (Test 4b), which was retested in Pass 2 (Test 25) both produced 181.2 MB/s (100% consistent, nice), Pass 2 Test 48 (181.8 MB/s) was retested in Pass 3 Test 1q @ 171.2 MB/s. Also, Pass 1 Test 3b @ 153.4 MB/s was retested in Pass 2 Test 1 @ 181.2 MB/s. The way the #'s are jumping around in Pass 2, up/down/up/down, rather than a bell curve, suggests you are experiencing random slowdowns that are not related to the settings. I don't know if you have any Dockers, VM's, or other processes running, or file access occurring that is causing this randomness. You may want to try running this again in safe mode, with all plugins, dockers, VM's disabled, and no file access occurring. I would also imagine that if you ran the test again, as-is with no server changes, you would see different answers for the same tests, due to the randomness observed. You can also probably err on the lower side for values. It looks like your server can hit over 180 MB/s with md_sync_window in the 768 range (Pass 1 Tests 2a/b/c), which is 99% of the fastest and in the Recommended range.

I suppose someone could add an NVMe drive to their array, no? Seems like I remember @johnnie.black having an Unraid server using just SSD's. The --no-nvme option might be a good solution to at least fix this report. As Xaero mentioned, these error messages are harmless and informational only, but unwanted all the same.