Jump to content
Pauven

unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables

1039 posts in this topic Last Reply

Recommended Posts

Look in /mnt/user0

 

(BTW, if /mnt/user0 doesn't exist in the first place, then all checks for mover running is pointless)

Share this post


Link to post
Posted (edited)

Here's what I came up with.  Seems okay in my limited testing, but let me know if you see any issues.  I didn't bother checking available cache disk space for the "prefer" option, since worst case it shows a Mover warning that might be unnecessary.

 

CheckMover() {
	MoverWarning=false
	if [ -d "/mnt/user0" ]; then
		for f in /boot/config/shares/*.cfg
		do
			shareUseCache=""
			s=${f##*/}
			s=${s%.cfg}
			CacheDir="/mnt/cache/$s"
			ArrayDir="/mnt/user0/$s"
			eval $(egrep 'shareUseCache' "$f" | tr -d '\r')
			if [ $shareUseCache == "yes" ]; then
				if [ -d "$CacheDir" ]; then
					MoverWarning=true
				fi	
			elif [ $shareUseCache == "prefer" ]; then
				if [ -d "$ArrayDir" ]; then
					MoverWarning=true
				fi	
			fi
		done
	fi	
	if [ $MoverWarning == true ]; then
		ShowMoverWarning
	fi
}

 

Edited by Pauven
typo

Share this post


Link to post
8 minutes ago, Pauven said:

I didn't bother checking available cache disk space for the "prefer" option, since worst case it shows a Mover warning that might be unnecessary.

Now that you mention "available cache disk space", I wonder if even mentioning Mover to the users isn't asking for trouble.

 

Between the confusion of the Use cache settings, along with not even being aware of the Minimum Free setting for cache and what it does, and the fact that people often add cache AFTER they have already created docker image on the array...

 

I have dealt with a lot of users that have all of this misconfigured on their servers and it usually takes a half page of posts to get them straight.

Share this post


Link to post

I don't think the warning will cause too much trouble:

 

image.png.6a8663c9f4fafa53828f659a5f1aeda2.png

Share this post


Link to post

Does Unraid 6.x now ship with lshw installed?  I use it in the script, and back on Unraid 6.2 I had to download and install it.  Now I can't tell if my installation is from me installing it, or if it is now included in current Unraid versions.

 

I'm running 6.6.6.

 

You can run lshw -version from the command line to see if it is installed.

Share this post


Link to post

From 6.7.2, shows it in /usr/sbin/lswh, and /var/log/packages/ shows lshw-B.02.17-x86_64-r2588-1_SBo_LT so it seems like it's a Limetech package.

 

lshw -version

 

unknown

the latest version is B.02.18

Share this post


Link to post

Thanks.  That's the same that I get.  It didn't look like the script was installing it anymore, but I appreciate the confirmation.

 

I'm running a 15 hour test now.  If all goes well, I'll publish the new Unraid 6.x Tunables Tester v4 tomorrow.  🤞

Share this post


Link to post
21 hours ago, Pauven said:

I'm running a 15 hour test now.  If all goes well, I'll publish the new Unraid 6.x Tunables Tester v4 tomorrow.  🤞

 

I found a couple things I didn't like, so I tweaked the code and I am running another 16 hour test. 

Share this post


Link to post
Posted (edited)

Thought I would share something interesting.  In the screenshot below, the very first test is with my current values (which came from using UTT to find them), and that produces a peak speed of almost 140 MB/s, which is pretty much maxing out my 3TB WD Red drives.

 

The second set of results are using the Unraid stock settings.  Interestingly, they provide almost 135 MB/s peak speeds, which is really really good for stock values.  Had my system performed this way under Unraid 5.2, I never would have written this tool.  Unraid 6.6+ has come a long way.

 

But also notice 'Test 1a'.  This is using settings that are close to stock Unraid, but slightly different (smaller Stripes, larger Thresh) and the peak speeds tanked to less than 42 MB/s.  I think my stock speeds on Unraid 5.2 were a little better than this, but not by much. 

 

My point here is to show how the wrong combination of values can dramatically hurt performance.  Every server is different, and for some servers, the stock Unraid values are slow like Test 1a.  But also, stock Unraid 6.x is so much better than stock Unraid 5.x, at least on my hardware.

 

image.png.3736bc6ba6861711d84787117ca85981.png

 

Paul

Edited by Pauven
  • Like 2
  • Upvote 1

Share this post


Link to post

A quick usage question for you , Paul... when running the script through Putty, I assume the PC needs to be awake all the time the script runs as well as the server, yes?

Share this post


Link to post

It could be run through a "screen" session so only the server needs to be active the entire time.

Share this post


Link to post
8 hours ago, BRiT said:

It could be run through a "screen" session so only the server needs to be active the entire time.

 

Yes, you should be using Screen.  In fact, the script checks to see if you are running from inside Screen, and if not then it displays a warning.

Share this post


Link to post

OK, silly question but where can I read about Screen for Win10? Trying to google it doesn't work well as the word "screen" is too common when used with Win10. I have PuTTY already of course.

Share this post


Link to post
Posted (edited)
33 minutes ago, Pauven said:

 

Yes, you should be using Screen.  In fact, the script checks to see if you are running from inside Screen, and if not then it displays a warning.

you can make your script run inside screen automagically, replace shebang and line 2:

#!/bin/sh
if [ -z "$STY" ]; then exec screen -dm -S screenName /bin/bash "$0"; fi


Tweak screenName as appropriate.

Edited by Xaero

Share this post


Link to post
21 minutes ago, DanielCoffey said:

OK, silly question but where can I read about Screen for Win10?

 

You connect to your server via PuTTY (or whatever method you choose), then launch screen, which runs on your Unraid server.  Screen is great, because it allows you to initiate multiple virtual sessions, and these sessions survive even if you get disconnected, and you can reconnect to them after relaunching screen.

 

 

3 minutes ago, Xaero said:

you can make your script run inside screen automagically, replace shebang and line 2:

 

Interesting, but I'm not sure I want to force it.  I highly recommend using screen, especially when connected from another PC, but I think there are scenarios where some users might not want to use screen, so I'm good with just displaying a warning if you didn't start screen.

Share this post


Link to post

Ah, I understand. I wasn't familiar with Screen but now I remember using it once before when preparing the new drives for the array.

Share this post


Link to post
38 minutes ago, Pauven said:

Well, it's finally happened:  Unraid 6.x Tunables Tester v4.0

 

The first post has been updated with the release notes and download.

 

Paul

Sweet!

 

With the new Controller Benchmark added to my DiskSpeed app, I discovered that I'm saturating my motherboard 8 port SATA controller and getting less performance with all attached spinners being read simultaneously (similar to a parity scan). I'm going to run this test with the system as-is and then when I remove one of the drives out to be used as a hot spare which should take it under the threshold to see if there is a difference in recommended settings.

Share this post


Link to post
18 minutes ago, jbartlett said:

Sweet!

 

With the new Controller Benchmark added to my DiskSpeed app, I discovered that I'm saturating my motherboard 8 port SATA controller and getting less performance with all attached spinners being read simultaneously (similar to a parity scan). I'm going to run this test with the system as-is and then when I remove one of the drives out to be used as a hot spare which should take it under the threshold to see if there is a difference in recommended settings.

 

Very interesting!  I've wondered about the same.  I look forward to seeing the results.

 

Similarly, my peak speeds are constrained by my older 3TB 5400 RPM drives to about 140 MB/s, but my 8TB 7200 RPM drives can hit over 200 MB/s (and they do hit over 160 MB/s once I get through my 3 & 4 TB drives).  I've wondered if the recommended settings would change if I only had the 7200 RPM 8 TB drives.  My guess is that the minimum values would go up to reach the increased max possible speeds. 

 

But I can't really test this without spending $$$$ to buy replacement 8 TB drives.  Maybe another user will find themselves in this situation and can test it.

Share this post


Link to post
55 minutes ago, Pauven said:

I've wondered if the recommended settings would change if I only had the 7200 RPM 8 TB drives.

I assume it would. I've just started moving stuff off my youngest drive which will take a couple of days so I haven't run it yet. If it runs like the old version did, it starts a parity scan which starts at the start of each drive. The optimal settings would be with all existing drives and may not be optimized once you get past the slower drives. If there's a means of starting a sync at a given spot, you can start past the smaller drives to see how things would go.

 

58 minutes ago, Pauven said:

But I can't really test this without spending $$$$ to buy replacement 8 TB drives.

Took me over a year to replace nine 4TB Seagate consumer drives to 6TB WD Red Pros for that reason. 1 or 2 drives every month or two.

Share this post


Link to post

Right, I have installed Screen through the NerdPack and launched it. I assume that once I have started the tuneable script I am free to close the PuTTY window because Screen is keeping a session open on the server, yes? Then I would re-open the PuTTY session at a later time and be able to list (screen -ls) and attach to the Screen session? I am not familiar with Screen but does that sound correct?

Share this post


Link to post
3 minutes ago, jbartlett said:

If it runs like the old version did, it starts a parity scan which starts at the start of each drive.

 

Yes, no change there.

 

3 minutes ago, jbartlett said:

If there's a means of starting a sync at a given spot, you can start past the smaller drives to see how things would go.

 

I'm not aware of any way to start a parity check other than at the beginning.  Seems like a I read some discussion recently about Lime-Tech adding a way to pause and resume parity checks, which might offer a method to do this, but I don't think LT has implemented this functionality.  If they have, someone please clue me in.

 

5 minutes ago, jbartlett said:

The optimal settings would be with all existing drives and may not be optimized once you get past the slower drives.

 

Agreed, though I think this would be rare and would require a very specific combo of drives.  For example, on my server I only have 1 8TB drive, so after I finish processing the 3 & 4TB drives, my server isn't pushing much data around to read the 2 8TB parity and 1  8TB data drive.  At the beginning of the parity check, reading 22 drives at 140 MB/s, my server is pushing over 3 GB/s.  But at the 50% mark when it only has the 3 8TB drives, reading them at 160+ MB/s is only pushing 0.5 GB/s at most.

 

Now, if I had 19 8TB data drives and a single 3TB drive, that would paint a very different picture - I would expect the server to push 3.5 - 4.0 GB/s easily.

 

12 minutes ago, jbartlett said:

Took me over a year to replace nine 4TB Seagate consumer drives to 6TB WD Red Pros for that reason. 1 or 2 drives every month or two.

 

My problem is that I just don't need that much additional capacity anywhere near that fast.  I'm at 67 TB now, and it has taken me a decade to get to this point.  Upgrading to all 8TB drives would add another 93 TB on my server.

 

Beside, the longer I wait to buy each 8TB drive, the cheaper...  :)

 

Share this post


Link to post
11 minutes ago, DanielCoffey said:

Right, I have installed Screen through the NerdPack and launched it. I assume that once I have started the tuneable script I am free to close the PuTTY window because Screen is keeping a session open on the server, yes? Then I would re-open the PuTTY session at a later time and be able to list (screen -ls) and attach to the Screen session? I am not familiar with Screen but does that sound correct?

 

Bingo.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.