Still DMI 1st gen, but using mostly the nothrbridge pcie slots should perform well, better if you can use only 4 of the onboard sata ports, using all 6 will top at around 120MB/s, still much better than your current setup.

 

Can't help with the onboard SAS.

New CPU arrived yesterday and I proceeded to install it.  I got this on bootup:

 

 

 

Didn't have much time to troubleshoot and was not getting anywhere so I put the old chip back but got virtually the same thing. 

 

 

 

Looks like I've got a project this weekend.

Should have suggested this earlier -- but you may very well need a newer BIOS version to support the new chip ... if that's the case, you need to do the update with the old chip in;  THEN install the new chip.

 

The BIOS update must have done it.  No problems with the new CPU after that. 

 

It does look like this resolved the CPU issue.  I'm currently running a parity check and with all 24 disks it's running at 43.5MB/sec.  Using the htop command I can see that the CPU generally stays around 15-30% with occasional spikes to 45%.  I'm about 12.75 hours in an it's a hair away from 2tb where 10 disks will fall off and the speed with increase. 

Glad all's now working well => the speed should indeed jump notably after the 2TB point.

At the 2tb mark it went to approx 55MB/sec

 

 

Total time as definitely an improvement but based on the previous comments I was actually expecting better.

 

 

Still debating the MB upgrade but not really sure if it will make a big difference.

I was also expecting better, what is your unRAID version and disk tunnable settings?

 

Forgot to add that like I mentioned before the post 2TB speed can only improve significantly if most 2TB disks are on the DMI.

I suspect that the 14 remaining disks aren't optimally connected to maximize the speed at that point.  Once you've dropped to 14 disks, you have the same # of disks I have in my oldest media server (which uses the same board) ... so I'd expect the same (or better) performance.    You need to be sure no more than 10 of the drives are on the DMI channel at that point.

 

If you describe precisely how all of the drives are connected, we can suggest how you could better optimize them.

 

I suspect that the 14 remaining disks aren't optimally connected to maximize the speed at that point.  Once you've dropped to 14 disks, you have the same # of disks I have in my oldest media server (which uses the same board) ... so I'd expect the same (or better) performance.    You need to be sure no more than 10 of the drives are on the DMI channel at that point.

 

If you describe precisely how all of the drives are connected, we can suggest how you could better optimize them.

 

Can this be determined from the logs or do I need to crack open the case and trace the cables?

 

I'm guessing ideal situation is something like 2 of the 4tb drives on the DMI channel and the rest evenly split between the two supermicro cards?

I just reconfigured the drives

 

Old configuration:

 

2 2 4 2 - AOC-SASLP-MV8 #1

2 2 4 2 - AOC-SASLP-MV8 #1

4 4 4 4 - AOC-SASLP-MV8 #2

4 4 4 4 - AOC-SASLP-MV8 #2

4 4 4 4 - MB / sil3132

2 2 2 2 - MB

 

New Configuration:

 

2 4 4 4 - AOC-SASLP-MV8 #1

2 4 4 4 - AOC-SASLP-MV8 #1

4 4 4 2 - AOC-SASLP-MV8 #2

4 4 4 2 - AOC-SASLP-MV8 #2

2 2 4 4 - MB / sil3132 (I'm not sure if the 4's go into the MB or card)

2 2 2 2 - MB

 

I was also expecting better, what is your unRAID version and disk tunnable settings?

 

Forgot to add that like I mentioned before the post 2TB speed can only improve significantly if most 2TB disks are on the DMI.

 

Before I saw this I had already started tweaking the tunables so I don't know what they were when the parity ran.  Right now they are back to the defaults of 1800, 1280 and 384.

 

After moving disks around I figured it's probably best to give this a shot.

 

My unRAID version is currently 6.1.9.

 

 

2 2 4 4 - MB / sil3132 (I'm not sure if the 4's go into the MB or card)

 

On the MB has the potential to be a little better.

 

 

 

Before I saw this I had already started tweaking the tunables so I don't know what they were when the parity ran.  Right now they are back to the defaults of 1800, 1280 and 384.

 

After moving disks around I figured it's probably best to give this a shot.

 

My unRAID version is currently 6.1.9.

 

V6.2-beta has 2 more tunnables that can make a significant difference,  but try the new disk organization first.

 

 

...

New Configuration:

 

2 4 4 4 - AOC-SASLP-MV8 #1

2 4 4 4 - AOC-SASLP-MV8 #1

4 4 4 2 - AOC-SASLP-MV8 #2

4 4 4 2 - AOC-SASLP-MV8 #2

2 2 4 4 - MB / sil3132 (I'm not sure if the 4's go into the MB or card)

2 2 2 2 - MB

 

I'm interested in Johnnie's thoughts on this config; but my thoughts are ...

(a)  MV8 #1 connections are fine, as long as this is the card that's plugged in to the x16 slot (and thus bypasses the DMI channel)

(b)  I'd absolutely NOT plug any of the 4's into the sil3132 card => you need to confirm that's not the case

©  I'd plug 4's into all of the motherboard slots

 

... looked at your old config as well; and my guess is the primary reasons you didn't get as much of a bump as we'd hope for are (a) you only had 2 4TB drives on mv8 #1 ... so you still had 12 drives on the DMI channel and (b) you had 2 4TB drives on the sil3132.

 

 

 

2 2 4 4 - MB / sil3132 (I'm not sure if the 4's go into the MB or card)

 

On the MB has the potential to be a little better.

 

Went back and carefully traced the sata cables.  The two remaining 4's are plugged into the MB and not the Sil card.

Rerunning parity to test.

I used the tunables testing utility to determine the best values which turned out to be:

 

 

md_num_stripes="1096"

md_write_limit="768"

md_sync_window="224"

 

 

For the first 2tb I was getting just north of 49 MB/sec, and now for the final 2tb...

 

 

 

This is precisely the results I was hoping for.

 

 

That's a very nice speed bump considering the money spent, pre 2TB speed is the maximum you can get with your current board, post 2TB can possibly go a little higher with 6.2-beta and its adicional tunnables, but it can't go higher than ~100MB/s, so you're pretty close to optimal already.

 

That's a very nice speed bump considering the money spent, pre 2TB speed is the maximum you can get with your current board, post 2TB can possibly go a little higher with 6.2-beta and its adicional tunnables, but it can't go higher than ~100MB/s, so you're pretty close to optimal already.

 

Bang for the buck was excellent! 

$8

Thank you guys so much for all the help.  Still gonna keep my eye out for that MB/CPU/RAM combo you mentioned earlier.

 

Before:

 

 

 

After:

Looking good.    Certainly worth $8 :-)