Please help CPU spiked to 100% plex unusable


maxse

Recommended Posts

Hi everyone,

I need some help please. So over the past few weeks I upgraded to v6.6.6, learned and installed dockers, and upgraded my CPU, Mobo, and RAM to an i5-8400 to support hardware transcoding. I used to have everything running smoothly with unraid 5 on an AMD Athlon processor and Plex on a mac mini with no issues...

-------------

Dockers: sab, sonarr, radarr, nzbget

 

I would have stuttering and freezing on plex local direct streams even. Could not find the issue, thought it was a network switch, nope. Finally figured out that my CPU was being pegged at 100% pretty much on all 6 cores. I then pinned all the dockers (in the advanced docker menu) to 1 core, nope, still had the issue. Figured it was Sab, so I stoped Sab, learned how to use and configured nzbget. Got Nzbget going and NOPE same issue!  I even checked to pause downloads when unrarring, etc (which I didnt even have to do on an old AMD Athlon). But anyway, still having an issue.  It was freeze and buffer my plex sttreams (max of 2-3) and make it unwatchable.

 

I then narrowed it down to happening when the files are unpacking, however, I just had an issue where FIVE of the 6 cores were maxed out, and no unpacking was taking place! Can you guys PLEASE help me figure out what's going on, it's so frustrating and Plex is basically unusable since I upgraded from my 7+ years old AMD Athlon. I'm attaching screen shots here to show you guys. Screen shots were all pretty much around the moment this was happening. I did my best to capture. The last screen shot of system stats is from earlier in the day, not the exact time all the CPU cores were pinned. Screen shot 1 and 2 happened around the same time few seconds apart

Screen Shot 2019-01-04 at 5.09.27 PM.png

Screen Shot 2019-01-04 at 5.11.28 PM.png

Screen Shot 2019-01-04 at 4.32.49 PM.png

Edited by maxse
Link to comment
1 minute ago, maxse said:

I then pinned all the dockers (in the advanced docker menu) to 1 core, nope, still had the issue. Figured it was Sab, so I stoped Sab, learned how to use and configured nzbget. Got Nzbget going and NOPE same issue! 

So all 6 cores were @ 100% with the apps pinned to a single core?

 

You should grab diagnostics when this is happening (sooner rather than later -> while its happening is ideal, but after 15 minutes it would probably be useless)

Link to comment

YES! All except Plex, Plex was 0, 1, 2, 3, 4

all others to core 5.

 

I selected that core in the docker menu after clicking on the advanced tab. Not sure if that's the right way to do it but seems about right

 

Also my RAM shows to only have 400MB free, the rest pretty much cached, not sure if this is normal.

 

Here are the diagnostics, I know it's about 15 minutes later not sure if helpful

sabertooth-diagnostics-20190104-1730.zip

Edited by maxse
Link to comment
load average: 0.46, 0.95, 2.44

The 15 minute thing relates to the above.  First # is the average load over the last minute, 2nd is over 5 minutes, and last is over 15 minutes.  All three at this point indicate no problems for a 6-core machine.

 

5 minutes ago, maxse said:

Also my RAM shows to only have 400MB free, the rest pretty much cached, not sure if this is normal.

unused RAM is wasted ram.  What you're seeing is what you want to see  https://www.linuxatemyram.com/

Link to comment

Done, now what? Just saw CPU netdata go to 42 during unpacking with Plex docker disabled. 

I don't know what's considered normal but I've been working on this the entire day already and can't even enjoy and watch anything on plex because of this.

 

*EDIT* So I watched the CPU just now get to 54 (in netdata) while nzbget was unpacking... NZB get is pinned to Core5 only. I quickly went back to the unraid menu and saw CPU core 3 get to 70%.

This is all with plex docker still disabled

 

Not sure if this helps you guys narrow it down or help me?

sabertooth-diagnostics-20190104-1801.zip

Edited by maxse
Link to comment
11 minutes ago, maxse said:

Done, now what? Just saw CPU netdata go to 42 during unpacking with Plex docker disabled. 

I don't know what's considered normal but I've been working on this the entire day already and can't even enjoy and watch anything on plex because of this.

 

*EDIT* So I watched the CPU just now get to 54 (in netdata) while nzbget was unpacking... NZB get is pinned to Core5 only. I quickly went back to the unraid menu and saw CPU core 3 get to 70%.

This is all with plex docker still disabled

The overall loads are still fine.   Once its unpacking and reading / writing all the files, you can expect other cores to also go up as unRaid, the drivers etc still have to handle the actual reads / writes.

load average: 1.07, 0.92, 1.05  Cores: 6


 

Its what is going on when everything seems to be stuck at nearly 100% that we're interested in seeing what's going on.

Link to comment

It's hard to catch that moment. Something is definitely wrong here though, right? I recently changed my downloads share for Cache-prefer to Cache only and some of the shares have a mark next to them showing that some files are unencrypted even though I have all of the drives encrypted. Not sure if this has something to do with this?

*EDIT* BOOM GOT IT. But THREE cores pinned. Everything set to use Core 5 with the exception of plex.

 

sabertooth-diagnostics-20190104-1813.zip

Edited by maxse
Link to comment
load average: 12.78, 8.63, 4.32  Cores: 6

NZBGet is doing something, and Plex is also doing its thing.  Right now, the processes are being starved for CPU (load avg of 12), and if you grabbed the diagnostics while watching the CPU utilization, it is consuming the bulk at the moment.

 

This isn't about the odd spike here and there.  You want to know whats going on when every core is hitting 100% and sits there.  At that point you want to grab diagnostics, or start scrolling through the various charts in NetData so you can easily see what's going on.

Link to comment

I've never seem them ALL hit 100% and sit there, maybe 10-15 seconds at the most. I've only seen 5 out of 6 hit it once... Usually it's 4 cores like from the last diagnostic that I posted. 

 

That stops ALL plex streams! I wasn't able to watch a single stream of plex at the time of the last diagnostic I posted of the 4 cores. Plex streams just freeze and buffer when that happens. This is with hardware transcoding enabled and that was only ONE stream I had running. I tried to replicate the issue so that I could grab the diagnostic for your when the CPU levels spike and cause issues with plex streams like that

 

How come the processes are being starved for CPU on an i5-8400? And how can I resolve this guys, I can't watch plex. I noticed this mostly happens when there is unpacking going first it was with sab, and now with nzbget, same thing.

Never had an issue like this before on MUCH slower processors, and I'm sure others have those basic dockers running on weaker hardware without any issues. I guess Im also frustrated that I spent to much money upgrading it and now I can't get it to run, but something is not right like you said, the processes are being starved for some reason. 

 

Just to be clear, the last diagnostic I posted is EXACTLY the issue that I am describing taken at the exact time. All dockers pinned to core 5, with plex the only docker not pinned to a core, 1 stream (Attemping to go) with HW acceleration enabled at the exact time of that diagnostic

Edited by maxse
Link to comment
16 minutes ago, maxse said:

when there is unpacking going first it was with sab, and now with nzbget, same thing.

Unpacking and repairing if necessary are also hugely cpu intensive tasks.  By default, if 2 (or more) docker containers each require 100% of the CPUs available, they will split the resources.  You may be able to get better performance from Plex in this circumstance by not having it be able to use Core 5.  Beyond that, I'd look at prioritizing Plex over NZBGet https://forums.unraid.net/topic/57181-real-docker-faq/?page=2#comment-566087.  Don't run Plex, so can't really comment upon its CPU requirements.

 

It might also help to have an unassigned drive storing the intermediate files instead of the cache drive (where the media may be located also) to remove bottlenecks on the drives from the equation.

 

Link to comment

Yes I tried that also, not allowing plex to use core 5 and same thing. Just to make sure I do that by pinning 0,1,2,3,4 under the plex docker correct?

 

I just updated the BIOS on the motherboard also, just in case (you can tell Im desperate). Same result but I think I was able to catch the diagnostics when 5 cores were maxed out. Only lasted 3-4 seconds and seems to start when either nzbget or SAB start to unpack and then either sonarr/radarr move the files onto the array... It seems that this unpacking and moving runs up all of my cores

 

Plex should not require much, about 2000 passmark per stream to transcode without hardware transcoding. I have hardware transcoding enabled and working so it should barely take any CPU... Like I mentioned I had it running butter smooth on a much older and significantly less powerful CPU without any issues like this.

 

I don't have any media stored on my cache drive except when transcoding. Plex streams files directly from the array, the only time the SSD Cache gets used is when it's transcoding.I can look into transcoding to RAM but I use a Samsung SSD should be plenty of speed even with unpacking/transcoding?

 

I'll look up your link on prioritization, but I'm a newbie, I haven't seen many threads on this. I would imagine if people had an issue like this with my caliber processor everyone would be complaining left and right.

 

*EDIT*

So I'm positive, whenever there is unpacking going on and there is movement of files, 4-5 cores get pegged at 100%. It just happened again and froze 1 remote stream and 1 local stream! This is unacceptable :(

 

I have plex pinned to cores 0,1,2,3,4 and dockers to core 5 when it just happened again and froze everything. Somehow still 4 cores are getting pegged, I don't understand why, and it froze both plex streams :( Help

 

 

sabertooth-diagnostics-20190104-1948.zip

Screen Shot 2019-01-04 at 8.12.08 PM.png

Edited by maxse
Link to comment

Okay so I added --cpu-shares=2 to the advanced parameter for nzbget. Should I do the same for the other dockers as well? or would I need to change the number from '2'? Should I keep them pinned to core 5 or if I add this I can unpin them?

 

*EDIT*

Uhmm, so I entered the same thing for radarr "--cpu-shares=2" and I got an error message when I clicked apply where it said COMMAND FAILED at the end and now it's showing up as "orphan image" and the docker is shown as unavailable and when I click it only give me the option to remove it! I can still see the directory for the docker in appdata though. What did I do? :( 

Oh my goodness, I'm getting a headache from all this

 

*EDIT 2* 

Okay I got it back lol. Re-added the container. I guess you can't have both dockers with '2' so I assigned a higher number. So I guess the higher the number the lower the priority right? 

 

So after doing that I still saw 4 cores go to 100% but plex didn't freeze. So I guess it's working? I don't get how this could happen on such a processor. I haven't seen a single complaint from anyone about this

Edited by maxse
Link to comment

From your diags you are I/O bound not CPU bound.  Moving core assignments around will not remedy that.

top - 18:16:32 up 2 days, 18:02,  0 users,  load average: 12.78, 8.63, 4.32
%Cpu(s):  2.1 us,  5.3 sy,  0.0 ni, 28.7 id, 61.7 wa,  0.0 hi,  2.1 si,  0.0 st

At the time of your diagnostics you are 61.7% IO wait, 28.7% CPU idle.  The I/O wait is why your load is 12.78.  Everything that reads/writes from the disks will stall while it waits a turn in the I/O queue.

Link to comment
On 1/4/2019 at 10:32 PM, unevent said:

From your diags you are I/O bound not CPU bound.  Moving core assignments around will not remedy that.


top - 18:16:32 up 2 days, 18:02,  0 users,  load average: 12.78, 8.63, 4.32
%Cpu(s):  2.1 us,  5.3 sy,  0.0 ni, 28.7 id, 61.7 wa,  0.0 hi,  2.1 si,  0.0 st

At the time of your diagnostics you are 61.7% IO wait, 28.7% CPU idle.  The I/O wait is why your load is 12.78.  Everything that reads/writes from the disks will stall while it waits a turn in the I/O queue.

Thank you so much for helping me!

Wow, so what can I do to fix this? This has been so frustrating. I keep reading posts on here and reddit of people having 2 times as many dockers, and many more users streaming plex from an i3-8100 processor, and my plex is basically unsuable with the i5 :(

 

I figured sonarr and radarr move the files to the array because for my media shares I have "use cache disk=no" It's a 1tb SSD, and the sabnzbd folder is set to use cache only...

 

How can I fix these things?

 

 

Edited by maxse
Link to comment

Assuming if you disable your downloads Plex works fine.  Six cores to work with.  If you're doing CPU transcoding (no-HEVC/h265) then assign Plex three to four cores depending on how you have your transcoding quality configured and number of simultaneous streams you often see.  Don't assign anything else to those cores.  Assign two cores to sab/nzbget, sonarr, and radarr (the same two cores).  Within sab under 'config-switches-post-processing' utilize the 'nice' and 'ionice' (see sab wiki).  If using nzbget add 'nice -n 10' to the unrar config in post processing (see nzbget help).  Download to cache only and and post-process to the cache drive (enable cache on the necessary shares) so that the mover can do the bulk disk transfers during off-hours so not to bottleneck the I/O during prime time.

Link to comment

Thanks unevent. Yes, turning off all dockers except for plex makes plex run perfect. I have HW transcoding enabled so it should barely be using any CPU... I  have usually around 4 streams max, with 2- transcoding. Transcoder Quality=prefer higher speed encoding, background transcoding=super fast.

 

I set cache-yes, on all of my shares, that seemed to have helped, you're right. This would post-process on the cache before also so I left it that way. I went back to sabznbd as I am more familiar with that and switching to nzbget didn't help my original issue... I have never heard of 'nice' 'ionice' mentioned before but I will look into the wiki and do a search for it to see what it is and how to set it.

 

I will assign the cores like you said and will report back. These things should fix my issue right? I mean I have seen people have a dozen or so dockers with an i3 and plex and they say it runs without issues. But for me even my direct stream would buffer....

 

-So do I need to remove the "--cpu-shares=2" that I placed for the sabznbd docker with this set up?  added 2, 3, 4.. to sab, radarr, sonarr respectively

 

Also, does this mean that my IO stuff will be overloaded if I add more dockers and the problem will return? I plan to add a docker to have some kind of backup program, nextcloud, and also plan to run a constant VPN for PIA and perhaps another vpn to give me remote access. How would I assign cores for those dockers? It seems like my server may not even be able to handle that?

Edited by maxse
Link to comment

https://sabnzbd.org/wiki/configuration/2.3/switches#ionice  The defaults listed usually work fairly well.

 

More Dockers should not be a problem as most are not as disk intensive like sab/sonarr/radarr and those you can tame by restricting resources and using nice/ionice and tweaking like pause download during processing.  Which LSI HBA card do you have?  Noticed from the logs the firmware is P11 on the one you have.

 

Not familiar with the '--cpu-shares' setting, never used it.  You would want to use '--cpuset-cpus=' to assign a Docker to specific cpu core(s).  The '--cpu-sharing' not really needed I don't think in this case with sab/sonarr/radarr limited to the same two cores.  The apps will battle it out for time on the two cores thereby limiting themselves while the ionice will lower the disk priority of rar/par2 and nice will lower the cpu priority of rar.  Will take longer to process, but everything else will keep running.

Edited by unevent
Link to comment

Thahnk you SOOOO much! I will look into this tomorrow and modify what you said. I was starting to get really discouraged, spent all this money to get hardware transcoding and had my CPU max out and direct streams buffering. Thank you!

 

I also just set a buffer in plex transcoding to 300 seconds, and followed the guide here to transcode to RAM (I have 16gigs). Seems like my CPU isn't hitting 100 on 4 cores anymore, just the 2 that I assigned to sab, sonarr, radarr but the streams aren't buffering

 

I am using an IBM m1015 that I flashed to LSI IT mode about 5 years ago. Is this something that I should update? And also just got an IO Crest 2-port SATA III PCIe 2.0 x2 Controller Card for a total of 16 drives, including the cache...

 

So you're saying that this is def an IO problem? So everyone that has these dockers goes through this? Even those people with dual XEON CPUs still have this issue without tweaking the setting like I have to do? 

I'm just so surprised because I kept reading how these 3 dockers are considered relatively light wait, especially when plex uses HW transcoding in the 8th intel processors.

 

*EDIT* So I didn't do the ionice stuff yet (will read about it tomorrow) but I did set pause on downloading, set all shares to use my cache drive, and pinned sa, radarr, sonarr to 2 cores, and plex to the other 4 remaining cores. I have 1 stream transcoding right now and had 4 cores all almost reach 100%! I know plex had just finished unpacking and sonarr was probably doing its thing, but still if a stream started up during this time it would probably freeze. I managed to grab the diagnostics during this attaching below. Again, this just happened, and I didn't have a chance to do the Ionice stuff yet. Also, I have an option checked in plex Library to "Scan library automatically when changes to library folders are detected." Not sure if this contribute to the issue? Also Not sure if this diagnostics will confirm that IO is the issue or give any further insight?

 

Thank you again SO much!

 

 

sabertooth-diagnostics-20190108-2325.zip

Edited by maxse
Link to comment

Guys thank you so much! After doing what unevent said seems to have fixed it! Occasionally the cores still spike especially the 2 assigned to sab, radarr, sonarr to 100% but the other cores don't go to 100% and more importantly plex doesn't buffer any more! Thank you so much!

 

Is there a way to set sab to have even LOWER priority and use even less IO bandwidth? I don't care if it takes longer to move or unpack files. Everything now happens on the cache, and cache is set to enabled for all shares. I'm not sure why I had that disabled before, probably didn't want to leave a my data in limbo on the cache, but no worries, small price to pay.

 

Is there any way to make it use even less IO and CPU? How about for Radarr and Sonarr to prevent even the 2 cores from going to 100%

 

Lastly, just to make sure, when I add more dockers, such as to run a virtual vpn, nextcloud, backup dockers, I can don't have to assign any cores right, I can just install those dockers as normal?

Link to comment

In sab you can change nice up to 19 (-n 10 up to -n 19) in sab-config-switches-Post_processing to lower the CPU priority.  If you go to 19 on nice you can try leaving ionice at the defaults in the sab wiki (-c2 -n4) and see how the IO during unrar and par2 perform.  The man page for ionice will explain why.  There are also some switches for par2 in the sab wiki where can limit cores and other priorities depending on which version of par2 the Docker you have is using.

 

For radarr/sonarr you can limit them to a single (same) core as an option (ex: sab is core 1, 2 and sonarr and radarr are core 1).  Their bulk disk usage are copy/move and there are no exposed options to add nice/ionice parameters (that I know of).

 

Usually only need or want to limit cores when you need something to not affect something else and vice versa.  If running a CPU intensive Docker and want to maintain quality of Plex don't assign that Docker to the Plex cores, or configure core priorities. High disk IO affects everything regardless of what core it is running on.

 

I don't think everyone has this issue due to server vs consumer hardware and other factors.  Not sure what the latest firmware is for the 1015 or if upgrading will improve things regarding IO.  You can configure radarr and sonarr to notify Plex on new stuff and then have Plex trigger an update only at that time.  Not sure what method Plex uses when set to constantly watch for changes so up to you.

Edited by unevent
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.