Jump to content
Pauven

unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables

1051 posts in this topic Last Reply

Recommended Posts

<whew>

 

Had a little scare after rebooting from safe mode.  None of my drives showed up.

 

PcCK4Mh.png

 

 

I reverted the disk settings to default before I rebooted again.  This time they came up.  not sure if they are related.  I will have to play around with it.

Share this post


Link to post
Posted (edited)
2 hours ago, Xaero said:

It's also entirely possible that the disk order not line up to the port numbers.

From my experience using my DiskSpeed app, the sd? drive assignment is done in the Port No order on the controller - so sdc is one port after sdb for example. All drives on a given controller are given sd? ID's and then all drives on the next controller and so on until all controllers have been processed. So if the unRAID disk order is done in sd? order, then it should match up barring the controllers initializing in a different order on boot.

Edited by jbartlett

Share this post


Link to post
49 minutes ago, StevenD said:

<whew>

 

Had a little scare after rebooting from safe mode.  None of my drives showed up.

 

PcCK4Mh.png

 

 

I reverted the disk settings to default before I rebooted again.  This time they came up.  not sure if they are related.  I will have to play around with it.

More likely that your "SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC)" didn't initialize correctly on boot. 

Share this post


Link to post
1 hour ago, StevenD said:

Interesting.

 

Man, you threw me for a loop there!  I thought you were posting the Short test from Beta 2, and I was frustrated that the disk report was still wrong.

 

I finally noticed it was from Beta 1 and a Long test.

 

Looks like you previous settings were close, just a little low to fully unleash all the performance.

Share this post


Link to post
2 hours ago, Pauven said:

 

Man, you threw me for a loop there!  I thought you were posting the Short test from Beta 2, and I was frustrated that the disk report was still wrong.

 

I finally noticed it was from Beta 1 and a Long test.

 

Looks like you previous settings were close, just a little low to fully unleash all the performance.

Don't worry, I'm sure some obnoxious niche thing with my server will cause a hiccup.

Share this post


Link to post
41 minutes ago, Xaero said:

Don't worry, I'm sure some obnoxious niche thing with my server will cause a hiccup.

Ah, a true Devotee of Mr. Murphy...

Share this post


Link to post

The megaraid disks don't show up still, not sure what's causing that.


SCSI Host Controllers and Connected Drives
--------------------------------------------------

[0] scsi0    usb-storage     
[0:0:0:0]    flash        sda    62.7GB    Extreme

[1] scsi1    megaraid_sas     MegaRAID SAS 2008 [Falcon]

[N0] scsiN0    nvme0     NVMe
[N:0:1:1]    cache        nvme0n1    1.02TB    INTEL SSDPEKNW01

[N1] scsiN1    nvme1     NVMe
[N:1:1:1]    cache2        nvme1n1    1.02TB    INTEL SSDPEKNW01


                      *** END OF REPORT ***



Everything else looks good.

Share this post


Link to post
7 hours ago, DanielCoffey said:

here is the Short v4.1 BETA 2 test result.

Looks good, thanks for sharing.

 

 

6 hours ago, jbartlett said:

My Beta2 short test.

I see a couple NVMe hosts listed at the bottom, but the NVMe drives are missing.  They also weren't listed in the data your shared with me, like the egrep of /var/local/emhttp/disks.ini.

 

The two NVMe hosts were listed in your lshw output you provided:

/0/100/1.2/0                             storage     NVMe SSD Controller SM961/PM961
/0/117/1.2/0                             storage     WD Black NVMe SSD

I've looked around, and I can't find where you've posted the output of lsscsi -st so can you do that for me?

Share this post


Link to post
18 hours ago, Xaero said:

Don't worry, I'm sure some obnoxious niche thing with my server will cause a hiccup.

 

13 hours ago, Xaero said:

The megaraid disks don't show up still, not sure what's causing that.

 

True to form!

 

I'm working on replicating your data this morning.  I have everything except:

egrep -i "\[|idx|name|type|device|color" /var/local/emhttp/disks.ini

 

Share this post


Link to post
Posted (edited)

Okay, I think I've found the issue with the report not showing certain disks. 

 

For reference, here's @StevenD's lsscsi -st output:

[0:0:0:0]    disk    usb:1-1.1:1.0                   /dev/sda   31.9GB
[0:0:0:1]    disk    usb:1-1.1:1.0                   /dev/sdb        -
[3:0:0:0]    disk                                    /dev/sdc   1.07GB
[4:0:0:0]    disk                                    /dev/sdd    960GB
[5:0:0:0]    disk    sas:0x300605b00e84f8bf          /dev/sde   8.00TB
[5:0:1:0]    enclosu sas:0x300705b00e84f8b0          -               -
[5:0:2:0]    disk    sas:0x300605b00e84f8bb          /dev/sdf   8.00TB
[5:0:3:0]    disk    sas:0x300605b00e84f8b3          /dev/sdg   8.00TB
[5:0:4:0]    disk    sas:0x300605b00e84f8b5          /dev/sdh   8.00TB
[5:0:5:0]    disk    sas:0x300605b00e84f8b9          /dev/sdi   8.00TB
[5:0:6:0]    disk    sas:0x300605b00e84f8bd          /dev/sdj   8.00TB
[5:0:7:0]    disk    sas:0x300605b00e84f8b7          /dev/sdk   8.00TB
[5:0:8:0]    disk    sas:0x300605b00e84f8ba          /dev/sdl   8.00TB
[5:0:9:0]    disk    sas:0x300605b00e84f8b4          /dev/sdm   8.00TB
[5:0:10:0]   disk    sas:0x300605b00e84f8b1          /dev/sdn   8.00TB
[5:0:11:0]   disk    sas:0x300605b00e84f8be          /dev/sdo   8.00TB
[5:0:12:0]   disk    sas:0x300605b00e84f8bc          /dev/sdp   8.00TB
[5:0:13:0]   disk    sas:0x300605b00e84f8b8          /dev/sdq   8.00TB
[5:0:14:0]   disk    sas:0x300605b00e84f8b0          /dev/sdr   8.00TB
[N:0:4:1]    disk    pcie 0x144d:0xa801                         /dev/nvme0n1   512GB

Notice that rows 3 & 4 (disks sdc & sdd) don't have the 3rd column populated. 

 

Now, here is StevenD's storage report:

SCSI Host Controllers and Connected Drives
--------------------------------------------------

[0] scsi0	usb-storage -	
[0:0:0:0]	flash		sda	31.9GB	

[1] scsi1	ata_piix -	

[2] scsi2	ata_piix -	

[3] scsi3	vmw_pvscsi -	PVSCSI SCSI Controller

[4] scsi4	vmw_pvscsi -	PVSCSI SCSI Controller

[5] scsi5	mpt3sas -	SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC)
[5:0:0:0]	disk3		sde	8.00TB	
[5:0:10:0]	disk8		sdn	8.00TB	
[5:0:11:0]	disk7		sdo	8.00TB	
[5:0:12:0]	disk6		sdp	8.00TB	
[5:0:13:0]	disk4		sdq	8.00TB	
[5:0:14:0]	disk12		sdr	8.00TB	
[5:0:2:0]	disk1		sdf	8.00TB	
[5:0:3:0]	disk9		sdg	8.00TB	
[5:0:4:0]	disk10		sdh	8.00TB	
[5:0:5:0]	parity		sdi	8.00TB	
[5:0:6:0]	disk2		sdj	8.00TB	
[5:0:7:0]	disk11		sdk	8.00TB	
[5:0:8:0]	disk5		sdl	8.00TB	
[5:0:9:0]	parity2		sdm	8.00TB	

[N0] scsiN0	nvme0 -	NVMe
[N:0:4:1]	cache		nvme0n1	512GB	

Notice that those two disks are missing.

 

Now let's look at @Xaero's lsscsi -st output:

[0:0:0:0]    disk    usb:3-9:1.0                     /dev/sda   62.7GB
[1:0:10:0]   enclosu                                 -               -
[1:0:11:0]   disk                                    /dev/sdb   8.00TB
[1:0:12:0]   disk                                    /dev/sdc   8.00TB
[1:0:13:0]   disk                                    /dev/sdd   8.00TB
[1:0:14:0]   disk                                    /dev/sde   8.00TB
[1:0:15:0]   disk                                    /dev/sdf   8.00TB
[1:0:16:0]   disk                                    /dev/sdg   8.00TB
[1:0:17:0]   disk                                    /dev/sdh   8.00TB
[1:0:18:0]   disk                                    /dev/sdi   8.00TB
[1:0:19:0]   disk                                    /dev/sdj   8.00TB
[1:0:20:0]   disk                                    /dev/sdk   8.00TB
[1:0:21:0]   disk                                    /dev/sdl   8.00TB
[1:0:22:0]   disk                                    /dev/sdm   8.00TB
[1:0:23:0]   disk                                    /dev/sdn   8.00TB
[1:0:24:0]   disk                                    /dev/sdo   8.00TB
[1:0:25:0]   disk                                    /dev/sdp   8.00TB
[1:0:26:0]   disk                                    /dev/sdq   8.00TB
[1:0:27:0]   disk                                    /dev/sdr   8.00TB
[1:0:28:0]   disk                                    /dev/sds   8.00TB
[1:0:29:0]   disk                                    /dev/sdt   8.00TB
[1:0:30:0]   disk                                    /dev/sdu   8.00TB
[1:0:31:0]   disk                                    /dev/sdv   8.00TB
[1:0:32:0]   disk                                    /dev/sdw   8.00TB
[1:0:33:0]   disk                                    /dev/sdx   8.00TB
[1:0:34:0]   disk                                    /dev/sdy   8.00TB
[N:0:1:1]    disk    pcie 0x8086:0x390d                         /dev/nvme0n1  1.02TB
[N:1:1:1]    disk    pcie 0x8086:0x390d                         /dev/nvme1n1  1.02TB

Notice that the disks missing the 3rd column are the sames ones missing from his storage report.

 

What's happening is that the code that extracts data from the 4th column (grabbing the /dev/sd? value) breaks because column 3 is missing, so instead of grabbing the disk's sd? value, it grabs the disk's size (8.00TB).

 

The solution is to make this code smarter to pull the correct column when column 3 is missing.  As a quick test, I hardcoded it to pull the previous column, and got good data for Xaero's MegaRAID drives:

[1] scsi1	megaraid_sas 	MegaRAID SAS 2008 [Falcon]
[1:0:11:0]	disk13		sdb	8.00TB	WDC WD30EFRX-68A
[1:0:12:0]	disk5		sdc	8.00TB	WDC WD30EFRX-68A
[1:0:13:0]	disk7		sdd	8.00TB	WDC WD30EFRX-68A
[1:0:14:0]	disk2		sde	8.00TB	WDC WD30EFRX-68A
[1:0:15:0]	disk3		sdf	8.00TB	WDC WD30EFRX-68E
[1:0:16:0]	disk4		sdg	8.00TB	WDC WD30EFRX-68A
[1:0:17:0]	disk10		sdh	8.00TB	WDC WD30EFRX-68A
[1:0:18:0]	disk21		sdi	8.00TB	
[1:0:19:0]	disk8		sdj	8.00TB	WDC WD30EFRX-68A
[1:0:20:0]	disk12		sdk	8.00TB	WDC WD30EFRX-68A
[1:0:21:0]	disk11		sdl	8.00TB	WDC WD30EFRX-68A
[1:0:22:0]	disk15		sdm	8.00TB	ST4000VN000-1H41
[1:0:23:0]	disk16		sdn	8.00TB	ST4000VN000-1H41
[1:0:24:0]	disk19		sdo	8.00TB	WDC WD30EFRX-68E
[1:0:25:0]	disk22		sdp	8.00TB	
[1:0:26:0]	disk17		sdq	8.00TB	WDC WD30EFRX-68A
[1:0:27:0]	disk18		sdr	8.00TB	WDC WD30EFRX-68A
[1:0:28:0]	disk20		sds	8.00TB	WDC WD30EFRX-68E
[1:0:29:0]	disk6		sdt	8.00TB	WDC WD30EFRX-68A
[1:0:30:0]	disk9		sdu	8.00TB	WDC WD30EFRX-68A
[1:0:31:0]	disk14		sdv	8.00TB	WDC WD30EFRX-68E
[1:0:32:0]	disk1		sdw	8.00TB	HGST HUH728080AL
[1:0:33:0]	parity2		sdx	8.00TB	HGST HUH728080AL
[1:0:34:0]	parity		sdy	8.00TB	HGST HUH728080AL

 

So how do I detect the missing column and auto adjust which field I'm extracting?  Here's the code in question:

#Get Drives linked to SCSI Hosts
echo " Querying lsscsi for the HDD's connected to each SCSI Host"
while read -r line 
do
	key=($line)
	scsistring=${key[0]//:/ }
	scsistring=`echo $scsistring | tr -d "[]"`
	scsistring=($scsistring)
	scsi=${scsistring[0]}       #${key[0]:1:1}
	if [ $scsi == "N" ]; then
		scsi=$scsi${scsistring[1]}
		x=${scsistring[3]} #${key[0]:5:1}
		DN=${key[4]/\/dev\//}  #path   #<--This would also be affected by a missing column 3
		DSP=${key[5]} #size            #<--This would also be affected by a missing column 3
	else
		x=${scsistring[2]} #${key[0]:5:1}
		DN=${key[3]/\/dev\//}  #path   #<--This is the line pulling the wrong column when col 3 is missing
		DSP=${key[4]} #size            #<--This ends up wrong too
	fi
	DK=${key[0]}
	Disk=${DiskName2Num[$DN]}          #<--This lookup fails when $DN has disk size instead of sd? drive letter
	if [ "$Disk" != "" ]; then         #<--Which in turn makes $Disk == "", so this section is skipped
		DiskSCSI[$Disk]="$DK"
		DiskSizePretty[$Disk]=$DSP #size
		eval scsi${scsi}disks["Z$x"]=$Disk  #"${key[0]:1:-1}" #scsi12[[2]=19
	fi
done < <( lsscsi -st; )

 

Edited by Pauven

Share this post


Link to post
Posted (edited)

I'm not sure if this will work;
awk '{ if ($3 == "") print "Column 3 is empty for " NR }'

 

This should print Colum 3 is empty for # where # is the row number. 

 

awk 'BEGIN { FS = OFS = "\t" } { for(i=1; i<=NF; i++) if($i ~ /^ *$/) $i = 0 }; 1'

 

This may also work to replace empty columns fields with "0" but is a copy paste from a different application

Edited by Xaero

Share this post


Link to post
27 minutes ago, Xaero said:

I'm not sure if this will work;
awk '{ if ($3 == "") print "Column 3 is empty for " NR }'

 

This should print Colum 3 is empty for # where # is the row number. 

 

awk 'BEGIN { FS = OFS = "\t" } { for(i=1; i<=NF; i++) if($i ~ /^ *$/) $i = 0 }; 1'

 

This may also work to replace empty columns fields with "0" but is a copy paste from a different application

Thanks for the suggestion.  That code is pretty complex, beyond my comfort level to use it.  Instead I went this path, which seems to do the trick:

if [[ ${key[3]} == *"dev"* ]]; then
	DN=${key[3]/\/dev\//}  #path
	DSP=${key[4]} #size
else
	DN=${key[2]/\/dev\//}  #path
	DSP=${key[3]} #size
fi

I should have BETA 3 ready to test in a few...

Share this post


Link to post
3 hours ago, Pauven said:

I've looked around, and I can't find where you've posted the output of lsscsi -st so can you do that for me?

I had but only the nvme drives so it likely was easy to miss. Here's the full report

[0:0:0:0]    disk    usb:1-4:1.0                     /dev/sda   32.0GB
[1:0:0:0]    disk    sata:5000cca255c167b1           /dev/sdh   6.00TB
[2:0:0:0]    disk    sata:5000cca24dce4c79           /dev/sdi   6.00TB
[3:0:0:0]    disk    sata:5002538e40270cde           /dev/sdj   1.00TB
[4:0:0:0]    disk    sata:5002538e40270f85           /dev/sdk   1.00TB
[5:0:0:0]    disk    sata:5000cca255dc42d7           /dev/sdl   6.00TB
[6:0:0:0]    disk    sata:5001b44ef905211e           /dev/sdm    240GB
[8:0:0:0]    disk    sata:5000cca24dcc471a           /dev/sdn   6.00TB
[11:0:0:0]   disk    sas:0x4433221100000000          /dev/sdb   6.00TB
[11:0:1:0]   disk    sas:0x4433221101000000          /dev/sdc   6.00TB
[11:0:2:0]   disk    sas:0x4433221102000000          /dev/sdd   6.00TB
[11:0:3:0]   disk    sas:0x4433221103000000          /dev/sde   6.00TB
[11:0:4:0]   disk    sas:0x4433221105000000          /dev/sdf   6.00TB
[11:0:5:0]   disk    sas:0x4433221107000000          /dev/sdg   6.00TB
[N:0:2:1]    disk    pcie 0x144d:0xa801                         /dev/nvme0n1   500GB
[N:1:0:1]    disk    pcie 0x1b4b:0x1093                         /dev/nvme1n1   256GB

 

Share this post


Link to post
56 minutes ago, jbartlett said:

Short run from Beta 3

Looks like your two NVMe drives still aren't showing.  I've got your lsscsi -st and lshw output, but I need more.  Please provide the output from the following:

 

lsscsi -H
 
mdcmd status | egrep -i "rdevStatus|rdevName|diskSize"

df -h

egrep -i "\[|idx|name|type|device|color" /var/local/emhttp/disks.ini

 

Share this post


Link to post
30 minutes ago, Pauven said:

Looks like your two NVMe drives still aren't showing.  I've got your lsscsi -st and lshw output, but I need more.  Please provide the output from the following:

 


lsscsi -H
 
mdcmd status | egrep -i "rdevStatus|rdevName|diskSize"

df -h

egrep -i "\[|idx|name|type|device|color" /var/local/emhttp/disks.ini

 

It looks like his nvme drives have a 5th column. Not sure if thats the cause or not

Share this post


Link to post
2 minutes ago, Xaero said:

It looks like his nvme drives have a 5th column. Not sure if thats the cause or not

I'm thinking no.  Here are his two NVMe drives next to your two:

[N:0:2:1]    disk    pcie 0x144d:0xa801                         /dev/nvme0n1   500GB
[N:1:0:1]    disk    pcie 0x1b4b:0x1093                         /dev/nvme1n1   256GB
[N:0:1:1]    disk    pcie 0x8086:0x390d                         /dev/nvme0n1  1.02TB
[N:1:1:1]    disk    pcie 0x8086:0x390d                         /dev/nvme1n1  1.02TB

Columns line up the same, and UTT 4.1 Beta 2+ already accounts for the extra column on NVMe drives.

Share this post


Link to post
4 hours ago, Pauven said:

Okay, I think I've found the issue with the report not showing certain disks. 

 

For reference, here's @StevenD's lsscsi -st output:


[0:0:0:0]    disk    usb:1-1.1:1.0                   /dev/sda   31.9GB
[0:0:0:1]    disk    usb:1-1.1:1.0                   /dev/sdb        -
[3:0:0:0]    disk                                    /dev/sdc   1.07GB
[4:0:0:0]    disk                                    /dev/sdd    960GB
[5:0:0:0]    disk    sas:0x300605b00e84f8bf          /dev/sde   8.00TB
[5:0:1:0]    enclosu sas:0x300705b00e84f8b0          -               -
[5:0:2:0]    disk    sas:0x300605b00e84f8bb          /dev/sdf   8.00TB
[5:0:3:0]    disk    sas:0x300605b00e84f8b3          /dev/sdg   8.00TB
[5:0:4:0]    disk    sas:0x300605b00e84f8b5          /dev/sdh   8.00TB
[5:0:5:0]    disk    sas:0x300605b00e84f8b9          /dev/sdi   8.00TB
[5:0:6:0]    disk    sas:0x300605b00e84f8bd          /dev/sdj   8.00TB
[5:0:7:0]    disk    sas:0x300605b00e84f8b7          /dev/sdk   8.00TB
[5:0:8:0]    disk    sas:0x300605b00e84f8ba          /dev/sdl   8.00TB
[5:0:9:0]    disk    sas:0x300605b00e84f8b4          /dev/sdm   8.00TB
[5:0:10:0]   disk    sas:0x300605b00e84f8b1          /dev/sdn   8.00TB
[5:0:11:0]   disk    sas:0x300605b00e84f8be          /dev/sdo   8.00TB
[5:0:12:0]   disk    sas:0x300605b00e84f8bc          /dev/sdp   8.00TB
[5:0:13:0]   disk    sas:0x300605b00e84f8b8          /dev/sdq   8.00TB
[5:0:14:0]   disk    sas:0x300605b00e84f8b0          /dev/sdr   8.00TB
[N:0:4:1]    disk    pcie 0x144d:0xa801                         /dev/nvme0n1   512GB

Notice that rows 3 & 4 (disks sdc & sdd) don't have the 3rd column populated. 

 

Now, here is StevenD's storage report:


SCSI Host Controllers and Connected Drives
--------------------------------------------------

[0] scsi0	usb-storage -	
[0:0:0:0]	flash		sda	31.9GB	

[1] scsi1	ata_piix -	

[2] scsi2	ata_piix -	

[3] scsi3	vmw_pvscsi -	PVSCSI SCSI Controller

[4] scsi4	vmw_pvscsi -	PVSCSI SCSI Controller

[5] scsi5	mpt3sas -	SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC)
[5:0:0:0]	disk3		sde	8.00TB	
[5:0:10:0]	disk8		sdn	8.00TB	
[5:0:11:0]	disk7		sdo	8.00TB	
[5:0:12:0]	disk6		sdp	8.00TB	
[5:0:13:0]	disk4		sdq	8.00TB	
[5:0:14:0]	disk12		sdr	8.00TB	
[5:0:2:0]	disk1		sdf	8.00TB	
[5:0:3:0]	disk9		sdg	8.00TB	
[5:0:4:0]	disk10		sdh	8.00TB	
[5:0:5:0]	parity		sdi	8.00TB	
[5:0:6:0]	disk2		sdj	8.00TB	
[5:0:7:0]	disk11		sdk	8.00TB	
[5:0:8:0]	disk5		sdl	8.00TB	
[5:0:9:0]	parity2		sdm	8.00TB	

[N0] scsiN0	nvme0 -	NVMe
[N:0:4:1]	cache		nvme0n1	512GB	

Notice that those two disks are missing.

 

 

If it matters, those two "missing" disks are mounted via Unassigned Devices and they are not part of the array.

 

Share this post


Link to post
1 minute ago, StevenD said:

 

If it matters, those two "missing" disks are mounted via Unassigned Devices and they are not part of the array.

 

Yours should be fixed in BETA 3 if you want to give the Short test a run.

Share this post


Link to post
7 minutes ago, BRiT said:

For completeness, I ran a Long test with 4.1 Beta 2.

Looks good.  I'd say your accuracy is +/- 0.2 MB/s.  So even though the new fastest recommendation is ever so slightly faster, it's within the error of margin so you likely won't see a difference.

Share this post


Link to post
2 hours ago, Pauven said:

Looks like your two NVMe drives still aren't showing.  I've got your lsscsi -st and lshw output, but I need more.  Please provide the output from the following:

 


lsscsi -H
 
mdcmd status | egrep -i "rdevStatus|rdevName|diskSize"

df -h

egrep -i "\[|idx|name|type|device|color" /var/local/emhttp/disks.ini

 

Information attached.

info.zip

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.