Where is My Bottleneck?

[newoski]

I have an aging server (Xeon 1267v3, 32GB RAM) on a gigabit network. I've sorted out my Kodi cache settings and have things really stable, even with 4K Remuxes (finally). However, when I'm running parity check, I can see that my Kodi cache buffer never fills up -- It can't keep up. If I pause Parity Check, it fills instantly. Clearly, parity is impacting, however, is the decreased throughput during Parity Checks typically a result of CPU usage -- and a CPU upgrade would help -- or a result of disk performance and potentially unsolvable? 

What kind of logs would be helpful to diagnose this? Thanks so much!

[Frank1940]

Single or Dual parity?

 

EDIT:   I cannot find your processor on the Passmark site.  Please check that you have the complete  (and correct) part nnumber.

Edited October 7, 2020 by Frank1940

[newoski]

5 minutes ago, Frank1940 said:

Single or Dual parity?

 

EDIT:   I cannot find your processor on the Passmark site.  Please check that you have the complete  (and correct) part nnumber.

Dual 8TB

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1276+v3+%40+3.60GHz&id=2269

Edited October 7, 2020 by newoski

[itimpi]

I would expect that the problem is simply disk contention.

 

You might want to consider using the Parity Check Tuning plugin to offload parity checking to only run outside prime time.

[Frank1940]

4 minutes ago, itimpi said:

I would expect that the problem is simply disk contention.

 

You might want to consider using the Parity Check Tuning plugin to offload parity checking to only run outside prime time.

+1

36 minutes ago, newoski said:

Dual 8TB

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1276+v3+%40+3.60GHz&id=2269

In looking at your CPU, I would suspect that it is not a contributor to your problem.  You can read here for why:

 

      https://forums.unraid.net/topic/49598-unraid-server-release-620-rc4-available/#comment-487759

 

Plus, That CPU  has plenty of horsepower in any case to be able to do the brute-force 'Q' calculation without slowing down the parity calculation over what it takes to do the 'P' only calculation. (However, I suspect that it does have the AVX2 instruction...)  Normally, the parity check speed these days is almost always limited by the slowest disk in the system.  I am not sure that the process priority level is on the parity check is but I suspect that it is fairly high.  Are you actually experiencing a problem caused by the buffer not being full?  How often are you running a parity check? 

 

(My experience indicates that 3TB parity is about as large as it can be to have a parity check/resync complete in 8 hours.   I have been a bit surprised that there have not been more inquiries about this potential problem before.)

 

[newoski]

3 minutes ago, Frank1940 said:

+1

In looking at your CPU, I would suspect that it is not a contributor to your problem.  You can read here for why:

 

      https://forums.unraid.net/topic/49598-unraid-server-release-620-rc4-available/#comment-487759

 

Plus, That CPU  has plenty of horsepower in any case to be able to do the brute-force 'Q' calculation without slowing down the parity calculation over what it takes to do the 'P' only calculation. (However, I suspect that it does have the AVX2 instruction...)  Normally, the parity check speed these days is almost always limited by the slowest disk in the system.  I am not sure that the process priority level is on the parity check is but I suspect that it is fairly high.  Are you actually experiencing a problem caused by the buffer not being full?  How often are you running a parity check? 

 

(My experience indicates that 3TB parity is about as large as it can be to have a parity check/resync complete in 8 hours.   I have been a bit surprised that there have not been more inquiries about this potential problem before.)

 

OK, thanks both of you. With an 8TB parity, my checks usually run about 20ish hours. Any issues with checks running over like 4 nights?

[itimpi]

9 minutes ago, newoski said:

With an 8TB parity, my checks usually run about 20ish hours.

It was exactly the common availability of disks of this sort of size (and larger) leading to this sort of elapsed time (and longer) for parity checks that was the impetus for developing the plugin.

 

10 minutes ago, newoski said:

Any issues with checks running over like 4 nights?

No, other than that if you have to reboot for any reason the check would have to start again at the beginning.  If your server runs 24x7 this is unlikely to be an issue

 

I am hoping that at some point Limetech will provide a capability at the API level to start a parity check at a defined offset .  If this happens the plugin can easily be enhanced to exploit such a capability and restart from where it was after a reboot thus removing this restriction

[newoski]

Just now, itimpi said:

It was exactly the common availability of disks of this sort of size (and larger) leading to this sort of elapsed time (and longer) for parity checks that was the impetus for developing the plugin.

 

No, other than that if you have to reboot for any reason the check would have to start again at the beginning.  If your server runs 24x7 this is unlikely to be an issue

 

I am hoping that at some point Limetech will provide a capability at the API level to start a parity check at a defined offset .  If this happens the plugin can easily be enhanced to exploit such a capability and restart from where it was after a reboot thus removing this restriction

Thanks. And final question... there's no way to set Parity Checks to run at multiple windows, right? IE from 12AM - 7AM and from 3PM-5PM? 

[itimpi]

1 minute ago, newoski said:

there's no way to set Parity Checks to run at multiple windows, right? IE from 12AM - 7AM and from 3PM-5PM? 

Not that I have ever tested, and it is not something I remember anyone asking for before.   Thinking about it I think it should be easy enough to do using the "Custom" scheduling options - it is something I will do some experiments with.

[itimpi]

After some testing I can confirm that you can get multiple time slots using the Custom scheduling options.

 

2 hours ago, newoski said:

IE from 12AM - 7AM and from 3PM-5PM? 

When you switch on the custom option the field will be pre-filled with the options from the non-custom option.

 

To get the slots you want set the field for Resume to be 0 0,15 * * * * and the Pause field to be 0 7,17 * * * *

 

In other words use comma separated values in a field  (no spaces)  for multiple options for that field and a space between the fields. 

[newoski]

2 hours ago, itimpi said:

After some testing I can confirm that you can get multiple time slots using the Custom scheduling options.

 

When you switch on the custom option the field will be pre-filled with the options from the non-custom option.

 

To get the slots you want set the field for Resume to be 0 0,15 * * * * and the Pause field to be 0 7,17 * * * *

 

In other words use comma separated values in a field  (no spaces)  for multiple options for that field and a space between the fields. 

Thanks!

[newoski]

UPDATE: Happy to report that replacing my Linksys EA9500 router with an ASUS RT-AX88U AX6000, turning on Adaptive QOS and prioritizing Streaming Media fixes the problem so effectively that I am now able to watch two simultaneous NAS to Kodi box streams of Lucy (4K Remux), WITH 100MB/s download going on. In real-time, the router essentially deprioritizes the 100MB/s download to about 80MB/s and the streams both are 100% stable. I'm shocked. Not to mention, with WIFI 6 speeds, I'm getting over 500mbps from the router to my computer which is in the same room.