Plex: Guide to Moving Transcoding to RAM

[imyourdaddy]

@munit85 Did you ever find a fix for this?  I have the same issue. I also have my plex appdata on an unassigned SSD, so I assume that's where things are breaking. 

 

I get the following error when attempting to play a movie/TV show:

 

"Please check that the file exists and the necessary drive is mounted."

 

I'm not really in the mood to move everything back to the array to test it out right now, but maybe soon just to see if that's the underlying issue.
 

Edited June 27, 2019 by imyourdaddy

[Spies]

My Plex appdata is temporarily on an unassigned disk and it's working just fine.

[imyourdaddy]

@Spies Was your Plex appdata there before you made the suggested changes in this thread, or after? Also, does all your unraid appdata reside on the unassigned disk, or solely Plex appdata? 

 

Thanks for the help!

[Spies]

1 minute ago, imyourdaddy said:

@Spies Was your Plex appdata there before you made the suggested changes in this thread, or after? Also, does all your unraid appdata reside on the unassigned disk, or solely Plex appdata? 

 

Thanks for the help!

I was already transcoding to RAM when the data was temporarily moved (its back on the cache drive now and still works fine).

[Marshalleq]

Thanks everyone for this fine thread.  I've been considering doing this for a while due to the series of unfortunate events below and hadn't clicked it was so easy.

 

My two issues were:

 

1:  If any automation or VM's etc decided to run, Plex couldn't transcode to disk fast enough in order to keep up with real time video (on a single stream even) and thus, a media item would 'pause' in the middle while watching it.  So, an I/O issue.

 

2. I had two SSD disks die (an Enterprise Samsung SM863 960GB (new, died in under a year and Samsung's not covering it due to being OEM) and an Intel SSD330 60GB).  The latter is obviously older, but being MLC based figured it would be fine as a transcode device, but no.

 

Lessons learned:

 

1 - Even though SSD's are rated for something, I don't really want to have to keep buying them at their current prices, so I should make their lives easy if I can.

2 - By using /tmp it reduces I/O on the PCI bus which helps other activities

3 - Don't buy OEM drives unless you're 100% sure the warranty will match the original manufacturers warranty

4 - At the moment buy Intel SSD's - The price for an Intel branded enterprise 1TB drive rated at 1 drive write per day for 5 years (the warranty is 5 years) is about the same as a Samsung good 1TB consumer drive which doesn't have the IOPS or endurance.  I paid about $300 for it in NZ dollars (compared to thousands for other brand Enterprise SSD's like the one that died above).  It's not the fastest drive, but you don't need that, you need reliability and IOPS in the server space.  And as it is, the Intel still does 95,000 IOPS READ / 36,000 IOPS writes.  If you think the write IOPS are low have a look at HDD's - a new Seagate Barracuda 12TB HDD does about 215 IOPS.  So who's going to complain about 36,000?  Not me.

[1812]

13 minutes ago, Marshalleq said:

Thanks everyone for this fine thread.  I've been considering doing this for a while due to the series of unfortunate events below and hadn't clicked it was so easy.

 

My two issues were:

 

1:  If any automation or VM's etc decided to run, Plex couldn't transcode to disk fast enough in order to keep up with real time video (on a single stream even) and thus, a media item would 'pause' in the middle while watching it.  So, an I/O issue.

 

2. I had two SSD disks die (an Enterprise Samsung SM863 960GB (new, died in under a year and Samsung's not covering it due to being OEM) and an Intel SSD330 60GB).  The latter is obviously older, but being MLC based figured it would be fine as a transcode device, but no.

 

Lessons learned:

 

1 - Even though SSD's are rated for something, I don't really want to have to keep buying them at their current prices, so I should make their lives easy if I can.

2 - By using /tmp it reduces I/O on the PCI bus which helps other activities

3 - Don't buy OEM drives unless you're 100% sure the warranty will match the original manufacturers warranty

4 - At the moment buy Intel SSD's - The price for an Intel branded enterprise 1TB drive rated at 1 drive write per day for 5 years (the warranty is 5 years) is about the same as a Samsung good 1TB consumer drive which doesn't have the IOPS or endurance.  I paid about $300 for it in NZ dollars (compared to thousands for other brand Enterprise SSD's like the one that died above).  It's not the fastest drive, but you don't need that, you need reliability and IOPS in the server space.  And as it is, the Intel still does 95,000 IOPS READ / 36,000 IOPS writes.  If you think the write IOPS are low have a look at HDD's - a new Seagate Barracuda 12TB HDD does about 215 IOPS.  So who's going to complain about 36,000?  Not me.

 

or, transcode to ram. I use the following in user scripts (which I found on here)

 

#!/bin/bash
mkdir /tmp/PlexRamScratch
mount -t tmpfs -o size=4g tmpfs /tmp/PlexRamScratch

 

Whenever I transcoded to just tmp or any already created subfolder it would fill and lock up the system. With a folder of a specified size like this, I'm my experience, It may fill up and halt the transcode the first time it fills the ram, but after (even after reboot) that I've never had any issue with plex doing it's own garbage service and deleting older transcode files (knock on wood). This costs me nothing and doesn't cause any appreciable wear and tear on physical hardware. 

 

 

 

 

[Marshalleq]

Unless I missed something in the 9 pages of comments (only skim read the first part and the last part - /tmp is ram.  So you're saying /tmp still has some limit on it?  I liked the idea of /tmp because it's pre-existing and I didn't want to faf about with scripts and such.

[wgstarks]

4 hours ago, 1812 said:

I use the following in user scripts (which I found on here)

 

#!/bin/bash
mkdir /tmp/PlexRamScratch
mount -t tmpfs -o size=4g tmpfs /tmp/PlexRamScratch

 

Are you running this at first array start?

[1812]

8 hours ago, wgstarks said:

Are you running this at first array start?

yes

[riprod]

Is it necessary to run the script at first array start or could you run it any time?  Also do you make your transcode directory in Plex GUI /PlexRamScratch or /tmp/PlexRamScratch?

[1812]

5 hours ago, riprod said:

Is it necessary to run the script at first array start or could you run it any time?  Also do you make your transcode directory in Plex GUI /PlexRamScratch or /tmp/PlexRamScratch?

It’s better to do it at the first array start before the docker starts, otherwise the path is invalid. If you’re just trying it out, shut down plex, make the directory with the script, edit plex settings mapping the transcode folder to the tmp directory you created, then start plex 

 

mine is in tmp/plexram... 

[Marshalleq]

I still don't understand why this is better than /tmp.

[1812]

I still don't understand why this is better than /tmp.

Because plex can fill /temp, using all system memory, causing system hangs, stalls, and lock ups.

[riprod]

17 hours ago, 1812 said:

It’s better to do it at the first array start before the docker starts, otherwise the path is invalid. If you’re just trying it out, shut down plex, make the directory with the script, edit plex settings mapping the transcode folder to the tmp directory you created, then start plex 

 

mine is in tmp/plexram... 

Thanks, Exactly what I had in mind.  Will give it a try.

[Marshalleq]

10 hours ago, 1812 said:

 

Because plex can fill /temp, using all system memory, causing system hangs, stalls, and lock ups.

 

Apologies if this is obvious, but as a hint - is the explanation of this buried somewhere in this thread that I've missed, or just personal experience?  I'm probably not likely to be affected by memory issues with the amount of RAM I have, nevertheless I do like to understand what's going on and why a Plex docker would fill /tmp differently than a RAM disk on some other mount point.  Perhaps some other process is getting in the way trying to clean it up or something?

[1812]

42 minutes ago, Marshalleq said:

Apologies if this is obvious, but as a hint - is the explanation of this buried somewhere in this thread that I've missed, or just personal experience?  I'm probably not likely to be affected by memory issues with the amount of RAM I have, nevertheless I do like to understand what's going on and why a Plex docker would fill /tmp differently than a RAM disk on some other mount point.  Perhaps some other process is getting in the way trying to clean it up or something?

 

No worries. Part personal experience, part understanding of what plex does.

 

When plex starts to use /tmp as a transcode directory, it either it doesn't seem aware of the size it can write into or uses the total size of ram as its "available space" regardless of system utilization. So, it fills until it can no longer fill anymore. If you fill /tmp with plex transcode data, which the system also uses but can't directly manage or flush because plex is in control of the transcode files, then the system no longer has the ability to use /tmp, causing system lockups. And it seems like when this happens, plex still tries to continuously use the /tmp even as the system is working to purge cached files to maintain operational status as it sees /tmp filling. Eventually the system is left with only its initially occupied ram space and no free space to operate in, causing hangs in operations. At the same time, the plex transcode crashes and in doing so, leaves the transcode files in place (vs removing when you exit a transcode stream), most likely due to plex not having enough ram itself to execute the procedure (because, it's full of transcoded files.) The last time I tested and experienced this was about 6 months ago. 

 

When you specify the a folder of fixed size, one of two things happens: 1. plex writes until it is full, stops the transcode, and then every stream thereafter has functional garbage collection of the oldest transcode streams utilizing the space up to the size of the folder or 2. plex is aware of the size of the folder before it reaches capacity and executes functional garbage collection of the oldest transcode streams up to the size of the folder. 

 

 

I once had a server with 64GB of ram and had no issues transcoding to /tmp. But when I moved my plex docker to a more power efficient server with only 8gb of ram, this was a continual issue transcoding to /tmp. I could watch df -h and see it happening in realtime.  I then move to a 1tb ssd for about 4 months without issue. I then discovered the ability to make a specified plex "scratch disk". This reduced wear/usage on the ssd (which was also doing other jobs) and I was able to have steady, smooth streams with plex.

 

 

 

[Marshalleq]

Fantastic, thanks for putting in the time to explain for me.  So I think I should work out a number of transcodes I'm not likely to have e.g. 10, figure out how much data each one would have at the worst resolution for say 2 minutes per stream (mostly 1080p for now), and that gives me an idea how much RAM I would need.  I agree the transcode directory on SSD is a concern and the advantage of /tmp is that it should give the ram back to the system as needed - maybe not an issue with 128GB though.  I have a high endurance SSD, but I still don't wish to put transcode directory on it because it seems pointless and wasteful.  And finally monitoring of what is going on may be more challenging on /tmp.

 

Time for some experiments.

 

Thanks again.

 

Marshalleq

[david279]

I will say I once had my ram fill up while watching and recording a football game. Half way thru the 3 quarter the system crashed. I have 32 gb of RAM...if I was just watching movies and TV shos etc I would just transcode to ram but I do a lot of watching/recording live TV. So I had to back off it. 

[Surgikill]

I'm having an issue with this. I am using linuxserver plex and when I map it to /tmp it ends up occupying space in the plex container. Any ideas on this?

[Marshalleq]

Seems like you've not mapped your transcode folder in your docker to /tmp file in the host?  I am assuming that because you're in an unraid forum you're using docker....  However, I agree with above, definately do a ramdisk.  It does work better.

Edited September 4, 2019 by Marshalleq

[rxnelson]

So if I am reading this correctly I can install CA User scripts and follow the directions to put this in?

 

#!/bin/bash
mkdir /tmp/PlexRamScratch
mount -t tmpfs -o size=4g tmpfs /tmp/PlexRamScratch

This will run at boot before the docker starts or do I have to do something else?  I saw some old posts about setting permissions for the ramdisk as well.  It isn't necessary in this case?  I have 128 gigs of memory and am dabbling in with the live TV stuff in Plex.  Any recommendations on how much to allocate?  

[Marshalleq]

Yes, and it's awesome.  Worked quite well for me.

[testdasi]

5 hours ago, rxnelson said:

This will run at boot before the docker starts or do I have to do something else?  I saw some old posts about setting permissions for the ramdisk as well.  It isn't necessary in this case?  I have 128 gigs of memory and am dabbling in with the live TV stuff in Plex.  Any recommendations on how much to allocate?  

CA User Script schedule has an option to "Run at first array start". Schedule the script with that and it should be fine.

4GB is more than enough for most users so it's a good starting point. If you start getting issues then increase it but it's unlikely. It should be enough for at least 5 1080p streams (a 4k stream is about 4x1080p streams).

[Simon]

On 9/6/2019 at 4:39 AM, testdasi said:

4GB is more than enough for most users so it's a good starting point. If you start getting issues then increase it but it's unlikely. It should be enough for at least 5 1080p streams (a 4k stream is about 4x1080p streams).

I'm playing around with this on my setup. I notice the Plex support site says:

Quote

Transcoder temporary directory

Directory to use when transcoding temporary files for streaming. This is useful if your primary drive has limited space. The directory used (whether default or not) needs sufficient free space, roughly equal to the size of the source file of the transcode plus 100MB.

Wouldn't that be way more than 4GB?

Edited December 23, 2019 by Simon

[Marshalleq]

I've actually stopped doing this.  The problem was multiple streams and it seemed to occasionally randomly fail playing mid stream, frustrating the Mrs and causing issues for the other streams too, I assume due to lack of disk space.  Sometimes I had to reboot the server to resolve - I assume it wasn't cleaning up the files properly after it dumped me out of the stream.  I was only using a 4G ramdisk so I could have made more, but then found out it was the reason my Live TV recording was failing and I then started wondering about things like creating thumbnails and other stuff.  I do have quite a bit of RAM in this machine, but ultimately thought the amount of RAM required to fix all of this was wasteful, so I turned it off nearly a week ago, and put it back on my Enterprise SSD.  Hopefully 1DWPD is enough.  Work in progress!