Mover making Plex server unresponsive?


Recommended Posts

I've been noticing that if I kick off a manual mover process that my Plex server becomes unresponsive until the mover is complete.  Now normally this doesn't affect me because I have the mover scheduled to run once a day at 8am when I never use the server.  However, I started to investigate why this was happening and it's not adding up.  First off, none of my plex (and other dockers) appdata is on cache or my array drives (all appdata is on an SSD set as an unassigned disk).  Also, the data that was being moved to the array was not being moved to the disk where the current media I was streaming was located.  I saw nothing in the syslog other than the mover activity.  My system setup is as follows:

 

Array:  8x10TB WD Gold drives (dual parity)

Cache:  1TB Micron 5100 PRO Enterprise SSD

Appdata:  256GB Samsung NVMe SSD

 

CPU:  Xeon D-1541

RAM:  64GB

 

I'm kind of at a loss as to why the mover would be having any type of performance impact on my Plex server.

 

 

 

Edited by IamSpartacus
Link to comment

I just installed unraid few days ago and am running into the same issue. I don't use the appdata share, though, but all my Plex configs and databases are indeed in the share with Cache: Prefer setting. I have no idea why Plex stops working when there is a lot of HDD activity in the array. 

Link to comment
51 minutes ago, toddas said:

I just installed unraid few days ago and am running into the same issue. I don't use the appdata share, though, but all my Plex configs and databases are indeed in the share with Cache: Prefer setting. I have no idea why Plex stops working when there is a lot of HDD activity in the array. 

What do you mean you don't use the appdata share? So what cache-prefer share are you using then?

 

While it is possible to do things differently, I have to ask why you would want to go against the conventions. It is just going to make it harder for you to use the community dockers, and harder for people to support you.

 

Are you sure you know what you're doing?

Link to comment
58 minutes ago, toddas said:

I just installed unraid few days ago and am running into the same issue. I don't use the appdata share, though, but all my Plex configs and databases are indeed in the share with Cache: Prefer setting. I have no idea why Plex stops working when there is a lot of HDD activity in the array. 

 

3 minutes ago, trurl said:

What do you mean you don't use the appdata share? So what cache-prefer share are you using then?

 

While it is possible to do things differently, I have to ask why you would want to go against the conventions. It is just going to make it harder for you to use the community dockers, and harder for people to support you.

 

Are you sure you know what you're doing?

 

Yea as @trurl said I'm not sure what you mean you don't use the appdata share.  I use it i just purposely placed it on an unassigned disk and not my cache drive.  This is why I'm quite confused about what's causing my issue.

Link to comment
On 1/4/2019 at 6:57 AM, IamSpartacus said:

my Plex server becomes unresponsive until the mover is complete

Please elaborate.  Does this mean your video stream completely freezes, or bad stutter, or webui unresponsive? And you notice this precisely when mover starts and then frees up precisely when mover finishes?  What is being moved?  The 'mover' is very aggressive at copying files.  It uses 'sendfile()' kernel interface that can result in copying a large file in a single system call.  If cache is on SSD and moving to a HDD could be RAM is rapidly and totally filling up with buffered data.  If this is the case, will require some thought about how to fix.

Link to comment
8 hours ago, limetech said:

Please elaborate.  Does this mean your video stream completely freezes, or bad stutter, or webui unresponsive? And you notice this precisely when mover starts and then frees up precisely when mover finishes?  What is being moved?  The 'mover' is very aggressive at copying files.  It uses 'sendfile()' kernel interface that can result in copying a large file in a single system call.  If cache is on SSD and moving to a HDD could be RAM is rapidly and totally filling up with buffered data.  If this is the case, will require some thought about how to fix.

 

@limetech Ok I tested this again last night and got similar yet slightly different results.  My CPU was basically idle (less than 10% usage as I had just 2 direct plays going in Plex).  As soon as I kicked off the mover, within a few seconds the CPU usage jumped up over 70% as you can see from Screenshot #1.  This stayed the case for about 4-5 minutes and during this time my Plex Direct plays stopped as soon as the buffers on my Shield TV's ran out.  Screenshot #2 shows that virtually nothing is going on in the system that would explain the CPU usage showing as being so high at this time.  The CPU usage (or lack there of) in htop not matching up what's being shown in the webUI is very strange to me.

 

After 4-5 minutes, the CPU returned to a normal state for a few minutes or so and I was able to resume my Plex direct plays.  The mover was still going at this point and would continue for another 10-15 minutes.  Then, the CPU usage jumped up again (Screenshot#3).  This again made Plex unresponsive.  However, this time when I opened htop (sorry I didn't get a screenshot of this) I saw two Plex processes consuming a lot of the CPU and this continued for 10+ minutes.  I had to actually restart the Plex docker to regain usage of the server again.  I copied my Plex logs folder at this time so I could go through them and determine what was going on in Plex at that time.  I'm not really sure what I'm looking for but I will try to narrow it down.

 

I don't know why the mover starting to move files initially would be causing the CPU usage to be so high.  I've got an 8C/16t CPU (Xeon D-1541) so the mover should not be taxing it so much.  It then appears that the moving of the files is kicking some kind of process off in Plex that causes Plex to freak out.  The Plex piece I'll have to investigate on my own because none of this is really adding up.  I don't see why simply moving files around in Unraid would cause Plex to do anything at all.

 

The WebGUI never becomes unresponsive during any of this and the syslog shows nothing other than the mover logs.

 

kDTDmjG.jpg

MnESBgB.jpg

J44qGIp.jpg

Edited by IamSpartacus
  • Like 1
Link to comment
  • 2 months later...

I am having the same issue.  When mover runs all trans coded Plex streams buffer.   Has not been this way in the past, it could have started when i installed my P2000 and started trans-coding with the Nvidia version of Unraid.  But i am not certain of that.

 

I see no noticeable higher CPU or Memory usage when this happens.

 

Thanks

Link to comment
  • 1 month later...

I think I'm seeing this same issue.  When mover is running, Plex is for the most part frozen.

 

Happened both before and after installing an old Geforce 1050 (using Unraid-Nvidia) for transcoding, so it seems unrelated to Unraid Nvidia.

 

I'm not certain if this happened before I upgraded to 6.7.0 as I seldom manually ran the mover when I was on 6.5.x -- though I don't recall it happening before the upgrade to 6.7.0.

 

Edited by brainbone
Link to comment

I have same issue with mover, makes plex unusable. I have two unassigned disks. 1 unassigned disks runs. My docker apps and dockapps only. The other unassigned disk is where my downloads from nzbget download to and where they get unpacked. I also run the Nvidia version of unraid with a gtx 1070 doing my hw transcodes and transcode folder mapped to ram

Link to comment

I am experiencing this issue as well. All Plex streams essentially lockup while the mover service runs. The streams are stuck on buffering, it may play for a second or two and then buffers again. I am using a GTX 1050 with HW transcoding enabled, however I was experiencing this issue prior to installing the GPU. My issue started maybe a month ago or so. Currently running Unraid v. 6.7.1-rc1. Plex is mapped to RAM for transcoding. 

 

I may try downgrading to a stable version prior to 6.7.0 to see if that helps. 

Link to comment
3 hours ago, chad4800 said:

I am experiencing this issue as well. All Plex streams essentially lockup while the mover service runs. The streams are stuck on buffering, it may play for a second or two and then buffers again. I am using a GTX 1050 with HW transcoding enabled, however I was experiencing this issue prior to installing the GPU. My issue started maybe a month ago or so. Currently running Unraid v. 6.7.1-rc1. Plex is mapped to RAM for transcoding. 

 

I may try downgrading to a stable version prior to 6.7.0 to see if that helps. 

Just a thought - what is the reason that mover is trying to run at the same time as Plex?   An easy avoidance action would be to simply schedule mover to run at a time when you know Plex would not be active. 

Link to comment
On 1/9/2019 at 2:58 PM, IamSpartacus said:

 

@limetech Ok I tested this again last night and got similar yet slightly different results.  My CPU was basically idle (less than 10% usage as I had just 2 direct plays going in Plex).  As soon as I kicked off the mover, within a few seconds the CPU usage jumped up over 70% as you can see from Screenshot #1.  This stayed the case for about 4-5 minutes and during this time my Plex Direct plays stopped as soon as the buffers on my Shield TV's ran out.  Screenshot #2 shows that virtually nothing is going on in the system that would explain the CPU usage showing as being so high at this time.  The CPU usage (or lack there of) in htop not matching up what's being shown in the webUI is very strange to me.

 

After 4-5 minutes, the CPU returned to a normal state for a few minutes or so and I was able to resume my Plex direct plays.  The mover was still going at this point and would continue for another 10-15 minutes.  Then, the CPU usage jumped up again (Screenshot#3).  This again made Plex unresponsive.  However, this time when I opened htop (sorry I didn't get a screenshot of this) I saw two Plex processes consuming a lot of the CPU and this continued for 10+ minutes.  I had to actually restart the Plex docker to regain usage of the server again.  I copied my Plex logs folder at this time so I could go through them and determine what was going on in Plex at that time.  I'm not really sure what I'm looking for but I will try to narrow it down.

 

I don't know why the mover starting to move files initially would be causing the CPU usage to be so high.  I've got an 8C/16t CPU (Xeon D-1541) so the mover should not be taxing it so much.  It then appears that the moving of the files is kicking some kind of process off in Plex that causes Plex to freak out.  The Plex piece I'll have to investigate on my own because none of this is really adding up.  I don't see why simply moving files around in Unraid would cause Plex to do anything at all.

 

The WebGUI never becomes unresponsive during any of this and the syslog shows nothing other than the mover logs.

 

kDTDmjG.jpg

MnESBgB.jpg

J44qGIp.jpg

Just a quick sidenote, not sure if you have looked into this, what is your plex set to do when files in the database is changed? I assume you have set plex to scan it's library? Might it be that plex is starting to scan the files as this can take alot of resource and it might be plex itself that stalls and not the server?

Link to comment
7 hours ago, itimpi said:

Just a thought - what is the reason that mover is trying to run at the same time as Plex?   An easy avoidance action would be to simply schedule mover to run at a time when you know Plex would not be active. 

I do have mover scheduled to run very early in the morning, however I did a lot of downloading recently and manually kicked of the process in the early evening to free up space on my cache drive.  

Link to comment
2 hours ago, ProZac said:

Just a quick sidenote, not sure if you have looked into this, what is your plex set to do when files in the database is changed? I assume you have set plex to scan it's library? Might it be that plex is starting to scan the files as this can take alot of resource and it might be plex itself that stalls and not the server?

 

Good thought but in my testing this was not the case.  Yes Plex scans on library changes but with all the resources I have at my disposal the scan is done in less than 30 seconds.  The issue seems to be CPU IOWait.  Watching netdata during the mover process IOwait jumps up significantly and I don't understand why.

  • Like 1
Link to comment
1 minute ago, IamSpartacus said:

 

Good thought but in my testing this was not the case.  Yes Plex scans on library changes but with all the resources I have at my disposal the scan is done in less than 30 seconds.  The issue seems to be CPU IOWait.  Watching netdata during the mover process IOwait jumps up significantly and I don't understand why.

I typically don't have any issues streaming while Plex is scanning/updating my library either. I haven't noticed any high CPU, RAM, or Disk I/O utilization. I'll have to look more closely at the CPU I/O wait, and see if that's happening with me as well. 

Link to comment
5 minutes ago, chad4800 said:

I typically don't have any issues streaming while Plex is scanning/updating my library either. I haven't noticed any high CPU, RAM, or Disk I/O utilization. I'll have to look more closely at the CPU I/O wait, and see if that's happening with me as well. 

 

It's not Plex that causes high CPU, RAM, or Disk I/O.  It's the mover process.  I have mitigated this for the time being by scheduling my mover to only run once a day at 5am.  But now that I've moved into doing a lot of 4K, sometimes i need to manually run the mover during the day and I have to hope no one is using Plex.

  • Like 1
Link to comment
1 hour ago, IamSpartacus said:

 

It's not Plex that causes high CPU, RAM, or Disk I/O.  It's the mover process.  I have mitigated this for the time being by scheduling my mover to only run once a day at 5am.  But now that I've moved into doing a lot of 4K, sometimes i need to manually run the mover during the day and I have to hope no one is using Plex.

Same here man, I have a ryzen 1950x so I'm Quite sure I have enough cpu power. All My Dockers run off their own specific unassigned ssd. All my Downloads go to their own specific ssd so I should have zero to little io wait for any disk related things 

Edited by bradtn
Link to comment

It seems that if the video file still resides on the cache drive it streams fine, however if the file has been moved to the array, then I have the lockup/buffering issue while the mover service is running. That would seem to point to Disk I/O contention on the array, not sure. 

Link to comment

What is your core pinning for Plex? Try not using core 0.

I noticed Unraid mover tends to use core 0 almost exclusively so if Plex (or any other CPU-intensive docker) uses it (e.g. streaming), it would lag both regardless of CPU.

 

It has nothing to do with how powerful your CPU is. When it's stuck waiting for a response from a drive (which is slow) due to the mover, it will not be doing anything else and report 100% usage.

Link to comment

I am having a similar issue. I don't think it's related to the mover since my 'media' share doesn't use a cache drive. I use a seperate HDD outside the array for SABnzbd and when Radarr/Sonarr tries to import a file, everything becomes unresponsive (yes, not only plex but all the containers). Same thing happens when i copy a file from my PC to the server. I am aware that writing data to a parity protected array requires CPU power, but not 100% of my 2700X.

 

I thought this was a problem with Radarr / Sonarr (Mono), but then i pinned them both to only 1 core and gave them each 500MB of RAM and this still happened. 

Edited by Teixeira
Words
Link to comment
2 minutes ago, testdasi said:

What is your core pinning for Plex? Try not using core 0.

I noticed Unraid mover tends to use core 0 almost exclusively so if Plex (or any other CPU-intensive docker) uses it (e.g. streaming), it would lag both regardless of CPU.

 

It has nothing to do with how powerful your CPU is. When it's stuck waiting for a response from a drive (which is slow) due to the mover, it will not be doing anything else and report 100% usage.

In my case my Plex container is using 6/18, 7/19, 8/20, 9/21, 10/22, and 11/23, which I do not have pinned to any other containers or VMs. I just don't recall this happening a couple of versions ago. 

Link to comment
19 minutes ago, Teixeira said:

I am having a similar issue. I don't think it's related to the mover since my 'media' share doesn't use a cache drive. I use a seperate HDD outside the array for SABnzbd and when Radarr/Sonarr tries to import a file, everything becomes unresponsive (yes, not only plex but all the containers). Same thing happens when i copy a file from my PC to the server. I am aware that writing data to a parity protected array requires CPU power, but not 100% of my 2700X.

 

I thought this was a problem with Radarr / Sonarr (Mono), but then i pinned them both to only 1 core and gave them each 500MB of RAM and this still happened. 

ALL containers becoming unresponsive is a different problem. As I mentioned, when the CPU is waiting for the drive to respond, it will report 100% usage, regardless of how powerful your CPU is. A slow-responding drive will cause what you are seeing. In terms of why a drive responds slowly, it may be due to high IO (e.g. repairing + extracting + copying strain a HDD a lot due to repeated seek) or even a failing drive (that was what happened when my old Hitachi 3TB was about to kick the dust).

Link to comment
On 6/5/2019 at 6:22 PM, testdasi said:

ALL containers becoming unresponsive is a different problem. As I mentioned, when the CPU is waiting for the drive to respond, it will report 100% usage, regardless of how powerful your CPU is. A slow-responding drive will cause what you are seeing. In terms of why a drive responds slowly, it may be due to high IO (e.g. repairing + extracting + copying strain a HDD a lot due to repeated seek) or even a failing drive (that was what happened when my old Hitachi 3TB was about to kick the dust).

I found the problem to this issue. I have my 'system' share set to cache only, but some data was still on the array. I moved it all to the cache with the 'unbalance' plug in. Now the containers don't become unresponsive anymore when the array goes to shit. The reason i'm still here talking about something that has nothing to do with the mover is that my problem is exactly the same as the original post. 

 

The first 5 mins of the import makes the whole array unresponsive, meaning that nothing can be read from it. After 5 mins I gain access to it again but it is 

noticeably slower (kinda expected). After some time it goes back up to 100% CPU and nothing can be done, until it is done.

 

You said this is probably because of slow drives. I've excluded that one drive that contains all the data, forcing unRAID to write to the next one. Plex still can't play the media that is on the "idle" drive. 

Link to comment
58 minutes ago, Teixeira said:

I found the problem to this issue. I have my 'system' share set to cache only, but some data was still on the array. I moved it all to the cache with the 'unbalance' plug in. Now the containers don't become unresponsive anymore when the array goes to shit. The reason i'm still here talking about something that has nothing to do with the mover is that my problem is exactly the same as the original post. 

 

The first 5 mins of the import makes the whole array unresponsive, meaning that nothing can be read from it. After 5 mins I gain access to it again but it is 

noticeably slower (kinda expected). After some time it goes back up to 100% CPU and nothing can be done, until it is done.

 

You said this is probably because of slow drives. I've excluded that one drive that contains all the data, forcing unRAID to write to the next one. Plex still can't play the media that is on the "idle" drive. 

You misunderstood my point. It doesn't matter if Plex is accessing the idle drive or an active drive, the fact that the CPU is waiting for the drive to respond will exclude other activities. Think of it like this:

  1. Mover tells CPU: write file 1 to disk A
  2. CPU: ok, writing
  3. Plex tells CPU: read file 2 from disk B
  4. CPU: nope, writing
  5. Plex: are you done yet?
  6. CPU: nope, still writing
  7. Plex: are you done yet?
  8. CPU: nope, still writing
  9. Plex: are you done yet?
  10. CPU: nope, still writing
  11. Plex: are you done yet?
  12. CPU: nope, still writing
  13. Plex: are you done yet?
  14. CPU tells Mover: ok, file 1 written to disk A
  15. CPU tells Plex: ok, here is file 2 from disk B

The entire time from step 4 to step 13, CPU will report 100% usage because it is fully occupied with writing.

 

Yes, we can talk all days about how it is supposed to be multi-tasking in 2019 but the truth is some action requires the CPU (core) full attention and until it's done, it won't be able to do anything else. This is the same reason that if you plug a failing drive or SD card to Windows, it will hang the entire system while trying in vain to read the data.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.