jbartlett

You shouldn't be bottlenecking at 5 drives. My x4 card started to bottleneck at 7 drives at those same read speeds. The DiskSpeed docker has a controller benchmark test - click on the i icon next to it and benchmark it. It will read all drives at once to get their single active speeds and then all at the same time. If it's total shit at the same time, your controller might be failing. It can't hurt to try it in an x8 slot. But if things are behaving connected to your motherboard but not your HBA card, then the card is quite likely the issue.

Before running the tunables script, my parity check times were 11.5 hours. Ran a check last night with the new values and it ran for 9 hours & 49 minutes, shaving approx. 100 minutes.

That's correct, my utility only does non-destructive reads. If you want to identify if Disk 4 has issues and remove the Parity drive from the equation, carefully recreate your array without Disk 4. The parity rebuild speeds will tell you if the Parity drive is the issue. If it looks good, mount the old Drive 4 using the UD plugin and copy files to it to see if you can duplicate the slow writes. While the Parity is rebuilding, you can also kick off a long SMART test on the old Drive 4. If everything still looks good, take the array offline and kick off a long SMART test on the parity drive and all of the others for a shiz-n-grins sanity check.

That ... that is weird. I've never seen anything like that. Even if every other drive was being pounded at the same time, you wouldn't see something like this. Your Tunable's won't affect it because it doesn't go through the unraid driver to access, it's a straight dd read using a block size given by the drive as being the optimal size. The drive might not be bad. The only thing I can think of is to try pinning some CPU's to the docker to see if it cleans it up if it happens again. I'm also assuming that you didn't have any CPU intensive tasks going on or no VM's taking all the CPU's away.

Try creating a new VM with the same video/drive/etc assignments.

Did the dd command reveal anything?

Most assuredly, it impacted it. It might be a small spot, it could be huge. One of my future projects of this DiskSpeed app is to do a surface scan and create a heat map of the read speeds. This would show the impacted area of it. I would move everything off of the drive that you can move off, ASAP. Once you have your data moved off and the drive out, I'd like to acquire the drive if you're willing to part with it. I've got some bad drives I could test against but yours seems to be an ideal drive to develop the heat map against.

I'll need to dig into the code but I suspect it's just a formatting error. Telnet into your server and enter the following command, verify that Drive 1 is still sdj dd if=/dev/sdj of=/dev/null bs=1M skip=3000000 iflag=direct status=progress This will start a disk read of your drive starting at around the 3TB mark where you got a successful read and try to read to the end of the drive with progress updates roughly every second and giving how much data was read in the last second. You have some kind of media error on the drive in the middle where the read speeds are likely extremely variable. If you see this reproduced, run a long SMART test on it. I would avoid putting any data on this drive.

When you click on the Benchmark all drives button, uncheck "Check all drives" and only check "Disk 1". Set the checkbox for "Disable SpeedGap detection". SpeedGap is logic that checks the high & low speeds of each test iteration. If it's over a given threshold (starting at 45MB), it considers the drive having been accessed and retries with a slightly larger threshold - and it will repeat ad nauseum.

Install the DiskSpeed docker app in my sig and run a benchmark on all your spinners. It will help you identify if you have a drive performing slowly. It might be the drive you had to rebuild, it might be another.

The nVidia drivers can tell if a video card was already in use when it takes control. The drivers assume that it's running in a virtualized environment and refuse to initialize. So to pass a graphics card in a different slot than #1, you have to have a card in the #1 slot for the OS to grab - so it leaves the card you want to pass through alone.

For this system, can you tell me exactly how everything is set up? Card make & model, where the cables plug into, backplanes, etc.

This looks like it's going to be something I can't easily add support for without access to the card. So I ordered an SAS 9207-8i - same as yours but with internal ports. It'll be delivered tomorrow, I'll be able to plug it into my backup server sometime later this week.

Check out the spaceinvaderone videos on installing lstopo which will give you a visual representation of your numa nodes, the CPU's & PCIe slots tied to them. This will help you pick your CPU assignments.

Redid the 2 numa node test after properly configuring the XML so Windows 10 detected two numa nodes. Performance was the same <cpu mode='host-passthrough' check='none'> <topology sockets='2' cores='8' threads='2'/> <numa> <cell id='0' cpus='0-7' memory='4194304' unit='KiB'/> <cell id='1' cpus='8-15' memory='4194304' unit='KiB'/> </numa> </cpu>

Adding nohz_full & rcu_nocbs yielded no difference within a margin of error with a 0.23% decrease over the 2/8/2 topology settings.

I did some tests on my Threadripper 2990WX 32 core chip with the different topology settings and did 10 passes with PerformanceTest 9.0 and averaged the results. OS was Windows 10. <topology sockets='1' cores='32' threads='1'/> The default setting with the cores pinned to span two full numa nodes gave me an average score of 17910.3 <topology sockets='1' cores='16' threads='2'/> Setting it to match the 16 core HT scored 17867.8, 0.24% slower <topology sockets='2' cores='8' threads='2'/> Making it match the numa node with 8 HT cores scored 17955.5, 0.25% faster

Please execute the following command on the server, does it return the Serial Number? smartctl -i /dev/twa1

Can you create a debug file for me? Launch the app again and replace the URL after the port with the following when you get this error: http://[IP]:18888/isolated/CreateDebugInfo.cfm Select "Create debug file with controller info" This will create a zip file on your appdata share that contains log files from the app and a copy of the sys file tree of devices and email it to [email protected] or upload to a file share and send the link to it.

I added code to invalidate a test result if a drive gives a 10% or greater improvement in speeds over it's single-drive speed.

It's weird that you're seeing such a variance. It's the same command given to each drive - read balls-to-the-wall starting from the start of the drive and 15 seconds later a kill command is sent - regardless if it's running on it's own or all at once. Did you click off of it or otherwise leave the benchmarking page at any point prior? I added code to stop reads if such an abort takes place. If you have the "Dynamix System Statistics" plugin installed, check to see if there's constant drive activity such as one of those tasks are running wild. If stopping & starting the DiskSpeed docker clears it up, then I have more work to do on that front.

Perform a benchmark on it every week or so to see if it returns an identical test. If it does return an identical test over the span of a month, it may just be how that drive is.

@wgstarks - can you also include a screen shot of your System Bus Tree?

Can you share the graphs of the controller benchmarks? Drive 9 was found to have a steady read speed over a large portion of the drive when it should be declining for a spinner, this indicates that the drive can send data faster than the controller it's attached to can handle. You also have a drive that has a wave to it. Can you tell me what make/model drive that is? Waves can be a sign of degraded areas. At a minimum, perform a benchmark on it every month or less to see if it holds steady - if it does, it may be how that drive was designed to operate. I'm working on updating the Hard Drive Database web site so it displays the latest benchmark from everyone who has the same drive (instead of averaging all tests which can cause big spikes) to see if that's just how that drive operates.