Parity Checks Bus limited?

[bnevets27]

EDIT:
Turns out I am bus/chipset limited.

 

The max bandwidth of my northbridge is 10 Gbps. 

10 Gbps = 1.25 GB/s

10 Gigabit per second = 1.25 Gigabyte per second

 

1.25 GB/s /17 disks = 73.5 MB/s per disk

 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

I would like to get down to the bottom of why my parity checks seem to be capped at 1.0GB/s total for all my drives or an average of around 80 MB/s per disk for a parity sync. That's what unraid reports for my total read bandwidth during most of my parity checks.

 

Motherboard is: Supermicro: X7DWN+ Block diagram in the manual pdf here

CPU: L5420 x2

HBA: SuperMicro AOC-USAS2-L8I x2 Flashed to IT mode

HBA: Dell H310 flashed to IT mode x1

 

HBA1 = SuperMicro AOC-USAS2-L8I is in a pcie x8 gen2 slot

HBA2 = H310 is in a pcie x8 gen2 slot

HBA3 = SuperMicro AOC-USAS2-L8I is in either a pcie x8 gen1

All of those slots are connect directly to the northbridge. HBA3 ideally should be on a pcie x8 gen2 slot of course but that's currently not possible.

There are no other cards in the system (though there has been in the past a TV tuner)

 

HBA1 = 8 Drives

HBA2 = 8 drives (Parity 2)

HBA3 = 1 drives (Parity)

Onboard SATA = 2 x SSD Cache

 

Better yet, look at the attached pictures from the DiskSpeed docker.

 

This is my history of parity checks. I don't have any idea why some are so fast.  For example 282 MB/s happened once and I have no idea how. There has been some hardware changes over time. Replaced a failed disk or two, changed my parity 1 and 2, added another HBA. 

 

I was expecting to see maybe something in the diskspeed tests but unless I'm not reading them right I don't see any bottleneck.

 

I am aware and not surprised that my two 3TB drives are the slowest but until they die or I have a actual need to upgrade them they will stay.

2018-12-21, 07:56:45	1 day, 6 hr, 50 min	72.1 MB/s	OK	1
2018-12-19, 06:45:33	1 day, 9 hr, 27 min	66.4 MB/s	OK	0
2018-12-17, 00:11:47	17 hr, 59 min, 36 sec	123.5 MB/s	OK	0
2018-12-15, 23:03:58	1 day, 4 hr, 54 min	76.9 MB/s	OK	0
2018-12-04, 21:47:51	1 day, 51 min, 6 sec	89.4 MB/s	OK	0
2018-12-04, 21:47:51	1 day, 51 min, 6 sec	89.4 MB/s	OK	0
2018-12-01, 02:35:10	1 day, 6 hr, 2 min	74.0 MB/s	OK	549
2018-08-19, 11:57:07	12 hr, 56 min, 43 sec	171.7 MB/s	OK	0
2018-03-13, 23:50:01	7 hr, 52 min, 25 sec	282.3 MB/s	OK	0
2018-02-08, 17:10:59	21 hr, 32 min, 20 sec	103.2 MB/s	OK	0
2018-01-19, 12:09:34	13 hr, 31 min, 8 sec	82.2 MB/s	OK	0
2017-12-28, 17:16:15	16 hr, 20 min, 48 sec	68.0 MB/s	OK	0
2017-11-15, 04:54:55	11 hr, 44 min, 18 sec	94.7 MB/s	OK	0
2017-09-01, 19:52:04	14 hr, 16 min		77.9 MB/s	OK	0
2017-08-22, 13:25:43	14 hr, 32 min, 49 sec	76.4 MB/s	OK	0
2017-08-01, 15:20:19	14 hr, 26 min, 26 sec	77.0 MB/s	OK	0
2017-07-26, 16:31:32	14 hr, 17 min, 1 sec	77.8 MB/s	OK	0
2017-07-21, 17:01:41	16 hr, 13 min, 51 sec	68.5 MB/s	OK	0
2017-06-27, 12:27:54	17 hr, 2 min, 27 sec	65.2 MB/s	OK	0
2017-06-13, 00:50:53	23 hr, 52 min, 29 sec	46.5 MB/s	OK	0
2017-06-06, 13:59:25	19 hr, 18 min, 41 sec	57.5 MB/s	OK	0
2017-05-23, 05:13:46	16 hr, 3 min, 7 sec	69.2 MB/s	OK	0
2017-04-26, 17:03:55	18 hr, 17 min, 27 sec	60.8 MB/s	OK	0
2017-04-21, 04:11:16	21 hr, 22 min, 16 sec	52.0 MB/s	OK	11
2017-04-19, 05:14:44	16 hr, 32 min, 2 sec	67.2 MB/s	OK	10
2017-03-15, 20:43:38	16 hr, 6 min, 2 sec	69.0 MB/s	OK	
2017-03-13, 16:01:31	13 hr, 19 min, 26 sec	83.4 MB/s	OK	
2017-02-06, 00:42:16	11 hr, 45 min, 30 sec	94.5 MB/s	OK	
2016-11-10, 07:12:34	8 hr, 39 min, 34 sec	128.3 MB/s	OK	
2016-10-29, 09:57:38	9 hr, 11 min, 50 sec	120.8 MB/s	OK	
2016-10-28, 11:23:42	9 hr, 56 min, 58 sec	111.7 MB/s	OK	
Sep, 23, 11:18:22	11 hr, 20 min, 57 sec	97.9 MB/s	OK	
May, 13, 12:20:11	11 hr, 59 min, 48 sec	92.6 MB/s	OK	
Mar, 23, 11:35:34	11 hr, 6 min, 8 sec	100.1 MB/s	OK	
Feb, 21, 23:50:35	1 hr, 4 min, 48 sec	64.3 MB/s	OK	

 

Edited December 21, 2018 by bnevets27

[whipdancer]

Can't really comment on cause, but my parity checks seem to be in the range of 75 - 100 MB/s, with the occasional foray north of 120MB/s.

[trurl]

The fast ones are probably just a reporting error, such as an incomplete parity check cancelled by the user.

[UhClem]

On 12/5/2018 at 7:40 PM, bnevets27 said:

I would like to get down to the bottom of why my parity checks seem to be capped at 1.0GB/s total for all my drives or an average of around 80 MB/s per disk for a parity sync. That's what unraid reports for my total read bandwidth during most of my parity checks. 

It could be one drive, or one controller, or one/the bus, or ... Try the little script (dskt2.txt) at the bottom of the linked posting [link]. No GUI, no pretty pictures, just a simple and quick tool to get the info you need to answer your question for yourself.

 

--UhClem  "If you push something hard enough, it will fall over." -- Fudd's First Law

 

Edited December 8, 2018 by UhClem

[trurl]

On 12/5/2018 at 7:40 PM, bnevets27 said:

my parity checks seem to be capped at 1.0GB/s total for all my drives or an average of around 80 MB/s per disk for a parity sync.

Don't quite understand what you mean by this bit. Since the disks are read in parallel, parity check speed should be somewhat independent of the number of drives except for possible controller bottlenecks such as port multipliers.

[bnevets27]

I was just trying to report what I see. On the main GUI, when the parity sync is running, the total transfer speed, at the bottom of all the disk speeds doesn't get higher then 1.0 GB/s. Also from what I recall, the stats page doesn't show a speed slow down as the parity sync gets closer to finishing. The graph is flat.

 

According to johnnie.black that should indicate some sort of limit. Thing is I don't what's causing the limit. I would like to know so I can understand and hopefully remove the bottleneck. I have a new motherboard on its ways so there will be a hardware change but this current hardware will be going into another server.

 

I'll have to give that a try UhClem. Unless I missed something, it does pretty much the same thing as the diskspeed docker with the exception of testing all drive simultaneously?

 

What I'm struggling to understand is. If no drive is as slow as 80 MB/s (at the start/outside of the disk). And if non of the controllers are the bottleneck, as shown from the speed tests via diskspeed. The only thing left is the bus/motherboard which has more than enough bandwidth also.

Edited December 11, 2018 by bnevets27

[UhClem]

[Up-front disclaimer: I don't use unRAID, but I have a "feel" for its basic workings.]

Is it true that diskspeed now does simultaneous testing? (I recalled that it didn't (at some point), and proffered dskt2, to pursue that avenue.) If it does, and your (per controller) graphs with N-drive totals, represent  concurrent totals, then please excuse me; and I am inclined to agree with your conclusion:

19 hours ago, bnevets27 said:

The only thing left is the bus/motherboard which has more than enough bandwidth also.

Always remember: "Trust ... but verify." Can diskspeed test all 3 controllers (and their drives) concurrently? If not, you can use dskt2 (just specify the full list of drive-letters). In either case, if the ~1GB/s ceiling isn't replicated, a remaining candidate is the CPU, [if unRAID's 2-parity implementation is "uninspired" and single-threaded] -- check with mpstat (I think).

 

Good luck. (I love this type of puzzle--even moreso when it's my own system)

 

--UhClem  "It's bad enough that the boss sh*ts on our head, but we have to say 'Thanks for the hat.'"

 

[bnevets27]

On 12/8/2018 at 4:37 AM, UhClem said:

It could be one drive, or one controller, or one/the bus, or ... Try the little script (dskt2.txt) at the bottom of the linked posting [link]. No GUI, no pretty pictures, just a simple and quick tool to get the info you need to answer your question for yourself.

 

--UhClem  "If you push something hard enough, it will fall over." -- Fudd's First Law

 

Ran the following command
 

dskt2 s m r k h g f e l p n o d c b

This is the result

sds = 164.02 MiB/sec
sdm = 179.66 MiB/sec
sdr = 151.12 MiB/sec
sdk = 148.25 MiB/sec
sdh = 115.86 MiB/sec
sdg = 133.69 MiB/sec
sdf = 131.94 MiB/sec
sde = 140.57 MiB/sec
sdl = 109.92 MiB/sec
sdp = 164.80 MiB/sec
sdn = 100.93 MiB/sec
sdo = 148.13 MiB/sec
sdd = 143.46 MiB/sec
sdc = 126.61 MiB/sec
sdb = 175.05 MiB/sec

I'm running dual L5420's so while not modern, more than enough for parity sync.

[dukiethecorgi]

My last parity check shows as 136.9 MB/sec.  8 5400 data drives on a AOC-SAS2LP-MV8, 2 7200 rpm parity disks on builtin SATA2 controller.

 

I used the tunables script to tweak the disk settings, and also set Tunable (nr_requests): to 8.

 

 

[bnevets27]

Just completed another parity build. Updated the OP with the speeds of a few more parity checks. I get the feeling this has to do with the new 8TB seagate compute drives but that theory is hard to believe due to the fact that the first 8TB drive was installed on 2018-03-13 with a speed of 282.3 MB/s. Now I'm guessing that might be due to the fact it was the only 8TB and it wouldn't have been limited after 4TB or 50% of the parity check as all my other drives are smaller than 4TB. Furthermore the 8TB drives report a much higher speed in the diskspeed test compared to the parity check speed (74.7 MB/s).

 

This is from the start of the parity check, while all disks were being read simultaneously.

 

This is after the parity check finished.

 

This is the speeds I'm getting on my 8TB drives, after the parity check is done with the 4TB drives.

 

When all drives are being read for the parity check, all my drives report a max speed of 74.7 MB/s (with very small fluctuations from that speed)

 

So that would make it seem like the parity drives are the bottleneck but I don't understand why drives that are clearly capable of sustaining over 100 MB/s don't ever even get to the speed.

[Vr2Io]

On 12/6/2018 at 8:40 AM, bnevets27 said:

All of those slots are connect directly to the northbridge.

If all going thr north bridge, then bottleneck will exist, pls further check the northbridge/PCH spec. BTW modern Intel platform PCH should have more bandwidth.

 

Your graph was abnormal sharp cut after 4TB, the celling was 75MBx3 ~ 225MB, pls trace those 3 disk connect to which controller.

 

225MB bandwidth like a PCIe gen1 in x1 lane. May be focus more on HBA3. Anyway the celling 1.27G also match 75MBx17

 

On 12/6/2018 at 8:40 AM, bnevets27 said:

HBA3 = SuperMicro AOC-USAS2-L8I is in either a pcie x8 gen1

 

Edited December 21, 2018 by Benson

[bnevets27]

As I was just about to post the following I finally figured it out. The max bandwidth of my northbridge must be 10 Gbps. 

10 Gbps = 1.25 GB/s

10 Gigabit per second = 1.25 Gigabyte per second

 

I can't believe I messed up the Gigabit/Gigabyte thing (I always remember it when talking about networking/internet)

 

So it all makes sense now, why I'm limited to 1.25 GB/s overall. I am bus/chipset limited.

1.25 GB/s /17 = 73.5 MB/s

 

Thanks for kicking me into figuring this out. 

 

Except it does leave one question, why doesn't my speeds increase when the parity check is only accessing my 3 8TB drives. Shouldn't the speed of the 8TB drives go way up after its done with the 4TB drives?

 

 

 

 

 

I'm having a hard time finding a concrete answer to the the max bandwidth the northbridge/MCH has. This is the block diagram from intel:

 

Which makes is seem like the MCH should have 2.5 Gbps x 4 = 10 Gbps

 

Here's the supermicro block diagram. I have a HBA in J8, J5 and J9. The HBA in J9 should obviously be moved to a x8 slot, not sure why it is not but I do know at one point it was due it not fitting in a different slot due to physical interference with another card. Which also means I guess I was wrong to say all are attached to the MCH as J9 goes through the southbridge. I would have to confirm what's going on with HBA3 as I haven't been inside the server in a while.

 

Either way, HBA3 only has 1 disk on it currently, that being the one 8TB data drive.

 

The 3 8TB drives are on the following HBA

Parity 1 and 2 are both on HBA2

Disk 17 is the one and only disk on HBA3 

 

The disk speed docker is reporting
HBA1 = Current & Maximum Link Speed: 5GT/s width x8 (4 GB/s max throughput)
HBA2 = Current & Maximum Link Speed: 5GT/s width x8 (4 GB/s max throughput)
HBA3 = Current Link Speed: 2.5GT/s width x8 (2 GB/s max throughput)

 

In the coming weeks this server is getting a new motherboard, CPUs and memory. But only slightly more modern. New motherboard is the supermicro X8DAH+-F. Looking at the datasheet for the intel 5520 chipset, it shows pcie gen2 x16 x2 + pcie gen2 x4 = 18 GBps and this motherboard has two of these chipsets so a total of 36 Gbps. So it too should have enough bandwidth.

 

The current motherboard is going into another server so I did want to get to the bottom of this for a few reasons.

1) So the issue doesn't follow into the next server 
2) So I don't have the same issue with the new motherboard
3) So I can understand whats going on.

Edited December 21, 2018 by bnevets27

[Vr2Io]

13 minutes ago, bnevets27 said:

The current motherboard is going into another server so I did want to get to the bottom of this for a few reasons.

1) So the issue doesn't follow into the next server 
2) So I don't have the same issue with the new motherboard
3) So I can understand whats going on.

Suppose to be correct.

 

But an unknown still there, why sharp cut from ~1.27GB/s to ~225MB/s, those not come from disk ( according your disk speedtest ).

Edited December 21, 2018 by Benson

[bnevets27]

Yeah I had just thought of that after hitting submit. That part still doesn't make sense.

 

So the new question is why doesn't the speed of my 3 8TB disks increase after the bus is freed up by not reading any of the 4TB drives. 

 

It looks as if the 8 TB drives continue at the "capped" speed of 74 MB/s even after the bottleneck is removed.

 

It doesn't make sense. Because while all the drives are being read, HBA2 which contains both my parity (8TB) disks and 6 other drives can easily do ~ 592 MB/s (74 MB/s x 8). But then drops down to 148 MB/s (74 MB/s x 2) (on that HBA) when only 2 drives are being read from. It almost seems like a bug??

[Vr2Io]

3 minutes ago, bnevets27 said:

It almost seems like a bug??

I don't think so, but nice if you could found out. ( I haven't such speed issue ) 😁

Edited December 21, 2018 by Benson

[JorgeB]

You're likely CPU limited, parity checks are single threaded and your CPU has a rather low single thread rating, dual parity with that number of disks is likely maxing it out, and that explains why speed remains low after 4TB, since those disks are still included in the calculations until the end (as zeros obviously).

[bnevets27]

Well a bug might actually explain it. Unfortunately I don't have good record keeping of when I did updates to unraid. But if you look at my parity speeds, they took a hit beginning in december. I know I installed my first 8TB drive as a parity drive on 2018-03-13. The system had been running dual 4TB parity before that time. I know I had a 4TB WD black as parity 1 and either a blue or green WD 4TB as parity 2. So the for the most part my limit was due to the blue/green WD. When I added the 8TB, it replaced the blue/green. At that time I then had 8TB seagate compute and a 4TB WB black as parity drives. Both should have been decently fast. Add to that, that there was only the parity itself that was larger than 4TB, which means when it built the last 4TB of parity it wasn't limited to any other disk speed. Which would likely explain the very fast average speed of that parity sync (282 MB/s)

 

When I upgraded my second parity drive to 8TB I think it was then when I noticed my speeds were lower. I had tried a few different things at the time, though never let the parity sync complete as I was fixated on getting about the 74 MB/s which I now know would never happen.

 

 

Quote

You're likely CPU limited, parity checks are single threaded and your CPU has a rather low single thread rating, dual parity with that number of disks is likely maxing it out, and that explains why speed remains low after 4TB, since those disks are still included in the calculations until the end (as zeros obviously).

 

@johnnie.black would that explain what I saw then with running a single 8TB drive? Since the parity calculations were only having to be done on 1 of the 2 parity disks, after 4TB? What I'm trying to say is if I am CPU limited then I would be less limited with only one drive having parity calculated over 4TB. As soon as I added the the second 8TB it split my limit between the two drives and therefore lowered my speed after 4TB.

 

I didn't realize parity checks were single threaded, I had sworn I thought I saw all my cores active and not just a single one maxed out but I must have been mistaken. (I had nothing else running at the time)

 

Is there any information out there about the CPU specs needed to not slow down a parity check due to an underpowered CPU? Recommend single threaded passmark score?

 

2018-12-21, 07:56:45	1 day, 6 hr, 50 min	72.1 MB/s	OK	1
2018-12-19, 06:45:33	1 day, 9 hr, 27 min	66.4 MB/s	OK	0
2018-12-17, 00:11:47	17 hr, 59 min, 36 sec	123.5 MB/s	OK	0
2018-12-15, 23:03:58	1 day, 4 hr, 54 min	76.9 MB/s	OK	0
2018-12-04, 21:47:51	1 day, 51 min, 6 sec	89.4 MB/s	OK	0
2018-12-04, 21:47:51	1 day, 51 min, 6 sec	89.4 MB/s	OK	0
2018-12-01, 02:35:10	1 day, 6 hr, 2 min	74.0 MB/s	OK	549
2018-08-19, 11:57:07	12 hr, 56 min, 43 sec	171.7 MB/s	OK	0
2018-03-13, 23:50:01	7 hr, 52 min, 25 sec	282.3 MB/s	OK	0
2018-02-08, 17:10:59	21 hr, 32 min, 20 sec	103.2 MB/s	OK	0

 

Edited December 21, 2018 by bnevets27

[JorgeB]

1 minute ago, bnevets27 said:

Which would likely explain the very fast average speed of that parity sync (282 MB/s)

This was a rebuild or clear of a smaller disk, the parity size is wrongly used for the calculation, it's an old bug.

[JorgeB]

6 minutes ago, bnevets27 said:

would that explain what I saw then with running a single 8TB drive? Since the parity calculations were only having to be done on 1 of the 2 parity disks, after 4TB? What I'm trying to say is if I am CPU limited then I would be less limited with only one drive having parity calculated over 4TB. As soon as I added the the second 8TB it split my limit between the two drives and therefore lowered my speed after 4TB. 

Not understanding here, with dual parity, you just need to a have an 8TB data disk to affect the speed even when passing the remaining smaller disks, since these disks are still included in the parity2 calculations, if you only had an 8TB parity, then it would be faster after the smaller disks.

[bnevets27]

2 minutes ago, johnnie.black said:

Not understanding here, with dual parity, you just need to a have an 8TB data disk to affect the speed even when passing the remaining smaller disks, since these disks are still included in the parity2 calculations, if you only had an 8TB parity, then it would be faster after the smaller disks.

I had a feeling I was doing a terrible job explaining/forming that question.

 

Maybe this will explain the question better. Would the following statements be correct?

 

Scenario 1:

8TB Parity 1

4TB Parity 2

When CPU limited, after passing the 4TB mark, the CPU is free to do calculations solely for the 8TB drive.

 

Scenario 2:

8TB Parity 1

8TB Parity 2

When CPU limited, after passing the 4TB mark (of the data disks), the CPU has to split the single thread work load of calculating parity for both parity drives.

 

Scenario 1 would result in faster speeds for the last 4TB then scenario 2, This is because in scenario 2 the last 4TB being calculated will be slower as the CPU has to calculate parity for 2 drives.

 

[JorgeB]

18 minutes ago, bnevets27 said:

Scenario 1 would result in faster speeds for the last 4TB then scenario 2,

Correct.

[bnevets27]

Just now, johnnie.black said:

Correct.

Thanks Johnnie! I feel more sane now 😁 That explains why my parity speeds dropped when adding the second 8TB drive. I had thought that the speeds would increase as it was a faster drive but I obviously had a few bottlenecks. Glad I had already put an upgrade in motion. I'll be running dual Intel Xeon X5650 in the new board (not that dual will help as you mentioned parity calculations are single threaded which is disappointing) which has a single thread rating of 1231. Unfortunately not that much faster, will that CPU still be a bottleneck?

 

 
 

[Vr2Io]

2 hours ago, johnnie.black said:

You're likely CPU limited, parity checks are single threaded and your CPU has a rather low single thread rating, dual parity with that number of disks is likely maxing it out, and that explains why speed remains low after 4TB, since those disks are still included in the calculations until the end (as zeros obviously).

Awesome !!

 

Edit : But 17 disk nor 3 disk will limit to same 75MB each ?? This seems no sense.

 

That let me remember my Celeron J1800 ( retire ) got same problem if running 2 parity, both (J1800 or G3250) should be ~12 or 13 disks, but G3250 haven't got slow down.

 

J1800

Dec 22 03:58:09 Q851 kernel: smpboot: Total of 2 processors activated (9662.80 BogoMIPS)
Dec 22 03:58:09 Q851 kernel: devtmpfs: initialized
Dec 22 03:58:09 Q851 kernel: x86/mm: Memory block size: 128MB
Dec 22 03:58:09 Q851 kernel: PM: Registering ACPI NVS region [mem 0x0008f000-0x0008ffff] (4096 bytes)
Dec 22 03:58:09 Q851 kernel: PM: Registering ACPI NVS region [mem 0xb97b3000-0xb9911fff] (1437696 bytes)
Dec 22 03:58:09 Q851 kernel: clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
Dec 22 03:58:09 Q851 kernel: futex hash table entries: 512 (order: 3, 32768 bytes)
Dec 22 03:58:09 Q851 kernel: xor: measuring software checksum speed
Dec 22 03:58:09 Q851 kernel:   prefetch64-sse:  9668.000 MB/sec
Dec 22 03:58:09 Q851 kernel:   generic_sse:  8548.000 MB/sec
Dec 22 03:58:09 Q851 kernel: xor: using function: prefetch64-sse (9668.000 MB/sec)
Dec 22 03:58:09 Q851 kernel: pinctrl core: initialized pinctrl subsystem
Dec 22 03:58:09 Q851 kernel: NET: Registered protocol family 16
Dec 22 03:58:09 Q851 kernel: cpuidle: using governor ladder
Dec 22 03:58:09 Q851 kernel: cpuidle: using governor menu
Dec 22 03:58:09 Q851 kernel: ACPI: bus type PCI registered
Dec 22 03:58:09 Q851 kernel: PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
Dec 22 03:58:09 Q851 kernel: PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in E820
Dec 22 03:58:09 Q851 kernel: PCI: Using configuration type 1 for base access
Dec 22 03:58:09 Q851 kernel: raid6: sse2x1   gen()   773 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: sse2x1   xor()  2271 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: sse2x2   gen()  1554 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: sse2x2   xor()  2750 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: sse2x4   gen()  2257 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: sse2x4   xor()  2560 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: using algorithm sse2x4 gen() 2257 MB/s
Dec 22 03:58:09 Q851 kernel: raid6: .... xor() 2560 MB/s, rmw enabled
Dec 22 03:58:09 Q851 kernel: raid6: using ssse3x2 recovery algorithm

 

 

G3250

Dec 22 03:58:52 G3250 kernel: smpboot: Total of 2 processors activated (12770.46 BogoMIPS)
Dec 22 03:58:52 G3250 kernel: devtmpfs: initialized
Dec 22 03:58:52 G3250 kernel: x86/mm: Memory block size: 128MB
Dec 22 03:58:52 G3250 kernel: PM: Registering ACPI NVS region [mem 0xc24cf000-0xc24d5fff] (28672 bytes)
Dec 22 03:58:52 G3250 kernel: PM: Registering ACPI NVS region [mem 0xd8713000-0xd8c4efff] (5488640 bytes)
Dec 22 03:58:52 G3250 kernel: clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
Dec 22 03:58:52 G3250 kernel: futex hash table entries: 512 (order: 3, 32768 bytes)
Dec 22 03:58:52 G3250 kernel: xor: measuring software checksum speed
Dec 22 03:58:52 G3250 kernel:   prefetch64-sse:  6816.000 MB/sec
Dec 22 03:58:52 G3250 kernel:   generic_sse:  5980.000 MB/sec
Dec 22 03:58:52 G3250 kernel: xor: using function: prefetch64-sse (6816.000 MB/sec)
Dec 22 03:58:52 G3250 kernel: pinctrl core: initialized pinctrl subsystem
Dec 22 03:58:52 G3250 kernel: NET: Registered protocol family 16
Dec 22 03:58:52 G3250 kernel: cpuidle: using governor ladder
Dec 22 03:58:52 G3250 kernel: cpuidle: using governor menu
Dec 22 03:58:52 G3250 kernel: ACPI: bus type PCI registered
Dec 22 03:58:52 G3250 kernel: PCI: MMCONFIG for domain 0000 [bus 00-3f] at [mem 0xf8000000-0xfbffffff] (base 0xf8000000)
Dec 22 03:58:52 G3250 kernel: PCI: MMCONFIG at [mem 0xf8000000-0xfbffffff] reserved in E820
Dec 22 03:58:52 G3250 kernel: pmd_set_huge: Cannot satisfy [mem 0xf8000000-0xf8200000] with a huge-page mapping due to MTRR override.
Dec 22 03:58:52 G3250 kernel: PCI: Using configuration type 1 for base access
Dec 22 03:58:52 G3250 kernel: core: PMU erratum BJ122, BV98, HSD29 workaround disabled, HT off
Dec 22 03:58:52 G3250 kernel: raid6: sse2x1   gen()  2765 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: sse2x1   xor()  1873 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: sse2x2   gen()  3281 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: sse2x2   xor()  2148 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: sse2x4   gen()  3992 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: sse2x4   xor()  2519 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: using algorithm sse2x4 gen() 3992 MB/s
Dec 22 03:58:52 G3250 kernel: raid6: .... xor() 2519 MB/s, rmw enabled
Dec 22 03:58:52 G3250 kernel: raid6: using ssse3x2 recovery algorithm

Edited December 21, 2018 by Benson

[bnevets27]

35 minutes ago, Benson said:

Awesome !!

 

Edit : But 17 disk nor 3 disk will limit to same 75MB each ?? This seems no sense.

I had 2 problems. Chipset/northbridge/bus limit. The max my board will do is 10 Gbps = 1.25 GB/s /17 disks = 73.5 MB/s per disk

 

CPU limit. Since parity calculations are single threaded, calculating parity for both my parity disks maxed out the CPU and therefore it can't write any faster then 75MB/s, its waiting for the CPU. At least that's how I understood it.

 

It is odd that the bus limit and the CPU limit end up being the same number of 75 MB/s though.

 

New hardware will defiantly remove my bus/MCH/PCH/northbridge (whatever it's called) limit as the new board has a ton of bandwidth. Not sure the minimum requirements to insure the CPU isn't limiting the speeds though.  

[JorgeB]

3 hours ago, bnevets27 said:

I'll be running dual Intel Xeon X5650 in the new board (not that dual will help as you mentioned parity calculations are single threaded which is disappointing) which has a single thread rating of 1231. Unfortunately not that much faster, will that CPU still be a bottleneck?

That's a nice bump, about 20% better single threaded performance, probably enough for 17 disks, something around a 1500 rating would be better for the future, since newer kernels tend to slow down checks a little on lower end CPUs, I did some tests when DP was introduced, this was the max speed depending on the number of disks and without any controller bottlenecks:

 

                    			TOTAL DRIVES (Including both parity drives)                
		ST Rating	12		16		20		24		30
                                        
PENTIUM G2030	1604		272.5MB/s	222.5MB/s+	195MB/s		165MB/s    
PENTIUM G4400	1878										210MB/s
PENTIUM G4560	1987										245MB/s

Note that single threaded rating gives an idea but it's not the only factor, memory bandwidth is also very important, as is CPU architecture, AMD CPUs are usually slower compared to Intel with similar ratings.