[SOLVED] Sudden horrible parity check performance after 6.11 upgrade

[weirdcrap]

Not sure if this related or not but I just upgrade from 6.11 RC5 to 6.11 stable. Upon rebooting the server never came back up so I had to perform a hard reboot which obviously initiated a parity check.

 

Well the server got .4% into the check and the speed dropped like a rock from 160MB/s~ (which is normal) down to 2.1MB/s and it has stayed there ever since. I've never had a speed issue like this before.

 

Stopping and starting the array didn't change anything. Rebooting didn't change anything. Upon starting another check it starts out fine, hits 0.4% and just drops.

 

It doesn't seem to recover once the speed drops, it just stays there until I cancel it.

 

 

What gives?

node-diagnostics-20220927-0718.zip

Edited September 27, 2022 by weirdcrap

[itimpi]

it looks like those diagnostics are taken shortly after booting.  If this is correct, can you post new diagnostics after the speed issue has manifested itself so e can see if anything relevant is being logged?

[weirdcrap]

1 minute ago, itimpi said:

it looks like those diagnostics are taken shortly after booting.  If this is correct, can you post new diagnostics after the speed issue has manifested itself so e can see if anything relevant is being logged?

I thought I grabbed them after I noticed the speed dropped in the parity check. I'm running Jbartlett's DiskSpeed docker right now to make sure none of my drives are the culprit then I'll start another parity check and grab new logs.

[JorgeB]

Something was reading from disk10 at 100MB/s+ when the diags were saved, that will slow down the parity check.

[weirdcrap]

Yeah it's definitely a docker. When the speed tanks stopping all dockers fixes it. I haven't added a new docker in months and this is the first time I'm aware of this ever happening so I'm at a bit of a loss.

 

Is there a way to see disk activity on a per docker basis somehow? I'm having trouble narrowing it down by just starting and stopping containers. I guess I need to wait longer before each container start to better identify my culprit.

 

EDIT: It appears to be Plex, its in the maintenance window and doing library tasks. I've never in my life seen a plex library scan cripple unraid's performance so much. It's like any disk activity during the parity check just completely cripples it's speed. This was not the case in previous versions (at least not that I've ever noticed). I've had parity checks running in the past with every single docker running and multiple streams going and it's never been like this. What gives?

 

EDIT2: Could the performance hit invoked by plex be 4k related? Just last night I added my first few pieces of 4k content to my server. This morning would have been when they got scanned into the library and the shows would have had intro detection run on them.

 

Maybe I've just never noticed the performance impact from non-UHD content before because plex chews through them much quicker?

 

It took 20 minutes to run intro detection on 6 episodes of Planet Earth II

 

node-diagnostics-20220927-0839.zip

Edited September 27, 2022 by weirdcrap

[JorgeB]

Yep, looks like it's Plex scanning for media:

 

nobody   26599 36.3  0.0 117064 59060 ?        DNl  08:38   0:17              \_ /usr/lib/plexmediaserver/Plex Media Scanner --

 

It's now reading disk10 @ 120MB/s

[weirdcrap]

14 minutes ago, JorgeB said:

Yep, looks like it's Plex scanning for media:

 

nobody   26599 36.3  0.0 117064 59060 ?        DNl  08:38   0:17              \_ /usr/lib/plexmediaserver/Plex Media Scanner --

 

It's now reading disk10 @ 120MB/s

4K content does have some other downsides besides the space it takes up apparently. I didn't even consider the prolonged CPU impact of scanning 4K content into the library vs 1080p content. The 1080p stuff scanned quick enough it couldn't drag down the array performance very far.

 

Now that it's scanned in everything it seems to be back to normal. I've got all m dockers turned back on and it's checking at around 180MB/s.

 

Is there a disk activity read out in there? is that just top? How did you find this out for my future reference?

Edited September 27, 2022 by weirdcrap

[JorgeB]

19 minutes ago, weirdcrap said:

Is there a disk activity read out in there?

In the diags, more than one place, loads.txt it's the easiest to check.

[weirdcrap]

2 minutes ago, JorgeB said:

In the diags, more than one place, loads.txt it's the easiest to check.

I saw plex in top and ps but I didn't know about loads, thanks.