Plex: Guide to Moving Transcoding to RAM


Recommended Posts

A data point: Plex transcoding 3:10 Yuma full 1080 blu-ray to a single 3mb 720 stream and at 50% the transcode folder size is 1GB.  I expect ~2GB final size for just this one stream at movie completion.  Transcoder is set for speed over quality.

 

edit: movie is at 1:33 out of 2:02:

 

root@Tower:/tmp# du -ch
1.8G	./plex-transcode-386zepytk18-2c3445d1-ffcf-497e-9101-09b180aea3f8
0	./mc-root
4.0K	./.vbox-root-ipc
32K	./hsperfdata_nobody
0	./notifications/archive
0	./notifications/unread
0	./notifications
12K	./openvpn
4.0K	./web
0	./.X11-unix
0	./.ICE-unix
1.8G	.
1.8G	total

 

root@Tower:/temp# free -l
             total       used       free     shared    buffers     cached
Mem:       8277284    8011088     266196          0     110356    7032472
Low:        836380     582744     253636
High:      7440904    7428344      12560
-/+ buffers/cache:     868260    7409024
Swap:      8008396          4    8008392

Link to comment

Another data point, playing a movie with a Fire TV.  The Fire TV is setup to Direct Stream the video while transcoding the audio to AAC.  The 4.3Gig directory is the movie in question and is ~ halfway through.  It appears PMC is transcoding the audio, combining it with the video and chopping it into blocks of about ~15MB for streaming to the player.

 

Since PMC does not delete the transcoded files until the show is stopped, I am on track to have ~8GB of files for this one stream which is correct as the source file is ~8GB.  When I go to R6, I could probably get away with one stream, but not two going. 

 

This tells me that if you are not transcoding, i.e. direct streaming the video, the space needed in your /tmp directory, ie free ram is approximately the same as your source file size.  I will have to keep using my cache drive as I prefer the higher quality.

 

root@Tower:/mnt/cache/apps/tmp/plex# du -ch
63M     ./plex-transcode-316y29v9633anhfr-a465c997-0568-4b23-8d75-db045b2ba3cc
4.3G    ./plex-transcode-ee2df7a09472f15f-com-plexapp-android-8c544d57-f210-4b06-8910-88fd64654c94
4.4G    .
4.4G    total
root@Tower:/mnt/cache/apps/tmp/plex#

Link to comment

Some feedback from my side:

 

The switch to /tmp & /transcode was easy as jonp's guide is pretty straight forward. Here is my RAM allocation as shown in stats:

used: 4.5GB

cached: 3.4GB

free: 300MB

 

I'm running 4 Dockers: DUC, MariaDB, ownCloud and PMS. Additionally I gave OS X Yosemite 4GB of RAM (which is the large amount of RAM being "used")

 

When I started 2 streams, the movies were running for about half an hour .... then my OS X VM was shut down  >:(

 

I guess that I need more RAM. Syslog attached

Link to comment
  • 6 months later...

I've done as the instructions instructed, but i'm still noticing my CPU doing all the work.

What am I missing????

Your CPU still have to do all the work. RAM is only storage, no processing going on there.

 

Right, but to expand on this further....

 

Plex tries to stay ahead when it transcodes... normally, it stores the transcoded file on the disk, adding some lag. By using this setup we can store it in RAM to speed up things.

Link to comment

I've gotta say ... the "lag" seen from storing to disk vice RAM is, or should be, inconsequential. Most spinners can write a transcoded stream to platter as fast as the CPU can generate it.  That is, unless you're server is on the Top-100 list of world super computers ;-) And if not "as fast" as the CPU can push it, then at least "fast enough" so it wouldn't cause stuttering or overly prolong the time till Plex throttles back after its buffered what it considers enough. I mean really, in most any modern hardware combination I can imagine, the transcode stream storage location speed is not the bottle neck, it is the cpu doing the transcoding.

 

I use my SSD for tmp storage and when transcoding a full-rate blueray rip my 8-core AMD is pegged (until it throttles) and certainly isn't waiting for the stream to write to disk.

Link to comment

I've gotta say ... the "lag" seen from storing to disk vice RAM is, or should be, inconsequential.

 

It depends, on what else the drive is doing.  For example if you are using the cache drive as your temp storage and you have files being downloaded, multiple plex streams, mover running etch then you can get stuttering.  If nothing else is going then you are correct that you want notice the difference.

 

I would like to use ram but unfortunately do not have enough memory.

Link to comment

I do ot to save wear and tear on my ssd.  Might be a ridiculous reason, but, ive got enough ram and it works great.

 

You're right ... it is rediculous ;-)

 

OK I mean i understand the thinking, but seriously, there is enough data out there to show it would take you YEARS to even come close to wearing out the drive.  You'll want to upgrade before then.

Link to comment

I've gotta say ... the "lag" seen from storing to disk vice RAM is, or should be, inconsequential.

 

It depends, on what else the drive is doing.  For example if you are using the cache drive as your temp storage and you have files being downloaded, multiple plex streams, mover running etch then you can get stuttering.  If nothing else is going then you are correct that you want notice the difference.

 

I would like to use ram but unfortunately do not have enough memory.

 

That is indeed a very valid point. I do forget my server is fairly single function at a time use. But I hope it would take a lot to really cause tmp-storage related studdering.

Link to comment
  • 2 months later...

I have been transcoding to /tmp for quite some time and never had issues.  Recently (past 2 weeks or so) I have started getting errors where Plex suddenly stops playing and states the file is unavailable.  I have traced this down to /tmp running out of free space.  I can watch it edge up to 100% full while a transcode is happening and then the transcode dies.  I have looked everything over but nothing makes sense, so I thought I'd post up to see if anyone can offer guidance and/or point out what I'm not seeing.

 

System has 8GB RAM, so as expected /tmp is roughly 4GB

root@Landfill:/tmp# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
-               3.8G  3.5G  290M  93% /

 

But why is /tmp 93% full?  It is approx 93% full even after a reboot.  SO let's look at what's in /tmp

root@Landfill:/tmp# du -h
0       ./mc-root
24K     ./community.applications/tempFiles
24K     ./community.applications
28K     ./notifications/archive
0       ./notifications/unread
28K     ./notifications
32K     ./plugins
0       ./.X11-unix
0       ./.ICE-unix
32M     .
root@Landfill:/tmp# ls
community.applications/  tmp-1661007850.url  tmp-399711444.url
mc-root/                 tmp-168813488.url   tmp-449460109.url
notifications/           tmp-1719190793.url  tmp-456017590.url
plugins/                 tmp-1876646817.url  tmp-477880073.url
tmp-1075403738.url       tmp-1892929066.url  tmp-505641119.url
tmp-1094819657.url       tmp-1895527982.url  tmp-524278372.url
tmp-1100506559.url       tmp-1952698197.url  tmp-558278473.url
tmp-1206292614.url       tmp-201109261.url   tmp-597112372.url
tmp-1248792920.url       tmp-2011622529.url  tmp-602322518.url
tmp-1275068659.url       tmp-2067235427.url  tmp-656886916.url
tmp-1280569778.url       tmp-207572655.url   tmp-682545064.url
tmp-1290055322.url       tmp-209482678.url   tmp-714381734.url
tmp-1350278630.url       tmp-21930234.url    tmp-770145398.url
tmp-1547960682.url       tmp-350663172.url   tmp-875515522.url
tmp-1567121976.url       tmp-354557422.url   tmp-912727871.url
tmp-1622958749.url       tmp-384040099.url   tmp-924626499.url
tmp-1648043674.url       tmp-386724013.url   tmp-991523516.url
root@Landfill:/tmp#

 

So according to du, there's approximately 32M of files in /tmp, but according to df -h there's approximately 3.5G used.  There's also a lot of tmp-xxxxxxxxxxxxx.url files.  Anyone know what these are (or does it even matter)?

 

I know the issue here is there's something I don't understand about how Linux works.  I'm assuming most of the space in /tmp is cached for other reasons?  What am I missing?

 

In the mean time I moved transcoding back to the cache drive and it's working fine, but I'd like to keep it in RAM if possible.

Link to comment

/tmp is RAM so 93% used is expected (Linux caches all free RAM for other needs and freely gives it up to programs that require it).

 

It IS entirely possible that /tmp uses nearly the full amount of RAM until it stops, how large are the source video files you are transcoding? I think Plex saves the full transcoded file until the viewer stops watching the video.

Link to comment

/tmp is RAM so 93% used is expected (Linux caches all free RAM for other needs and freely gives it up to programs that require it).

 

It IS entirely possible that /tmp uses nearly the full amount of RAM until it stops, how large are the source video files you are transcoding? I think Plex saves the full transcoded file until the viewer stops watching the video.

 

hex, you are correct about Plex's behavior and it is why I advocated against moving transcoding to RAM in the past. I had the same problem as dirtysanchez, but it was only on long high bit rate videos (movie BD rips). All other stuff ran fine. When I dug into it I was basically juuuuuust not hitting the wall with the shorter lower bit rate stuff but the BD movie rips, at full bit rate, would fill /tmp in about 30-45 min. Crash. flush. restart. rinse. repeat.

 

Frankly the concern over SSD wear is misplaced given all current evidence and most plex / unraid implementations I can guess at. Even with 24/7 streaming of a reasonable number of 20mbit streams the SSD will outlive most people, no less its likely useful life.

Link to comment

Can you inspect the contents of those .url files?

 

I did.  It looks like they are related to the Community Applications plugin.  Here's a snip from one of the files

{
    "apps": 230,
    "requests": 72,
    "last_updated": "25th September 2015 at 17:42:37",
    "last_updated_timestamp": "1443199357",
    "applist": [
        {
            "Beta": "False",
            "Category": "Network:Voip",
            "Name": "binhex-teamspeak",
            "Description": "\n    TeamSpeak is proprietary voice-over-Internet Protocol (VoIP) software that allows computer users to speak on a chat channel with fellow computer users, much like a telephone conference call. A TeamSpeak user will often wear a headset with an integrated microphone. Users use the TeamSpeak client software to connect to a TeamSpeak server of their choice, from there they can join chat channels and discuss things.[br][br]\n    [b][u][span style='color: #E80000;']Configuration[/span][/u][/b][br]\n    [b]/config[/b] This is where teamspeak will store it's configuration file, database and logs.[br][br]\n    [b][u][span style='color: #E80000;']Notes[/span][/u][/b][br]\n    Connect to the server using the TeamSpeak client with the host IP address and port 9987.[br]\n    To authenticate use the privilege key shown in the supervisord.log file in the host mapped /config folder.\n  ",
            "Overview": "\n    TeamSpeak is proprietary voice-over-Internet Protocol (VoIP) software that allows computer users to speak on a chat channel with fellow computer users, much like a telephone conference call. A TeamSpeak user will often wear a headset with an integrated microphone. Users use the TeamSpeak client software to connect to a TeamSpeak server of their choice, from there they can join chat channels and discuss things.\n  ",
            "Support": "http://lime-technology.com/forum/index.php?topic=38055.0",
            "Registry": "https://registry.hub.docker.com/u/binhex/arch-teamspeak/",
            "GitHub": "https://github.com/binhex/arch-teamspeak",
            "Repository": "binhex/arch-teamspeak",
            "BindTime": "true",
            "Privileged": "false",
            "Networking": {
                "Mode": "host",
                "Publish": "\n    "
            },
            "Environment": {
                "Variable": [
                    {
                        "Name": "",
                        "Value": ""
                    }
                ]
            },
            "Data": {
                "Volume": [
                    {
                        "HostDir": "path to config",
                        "ContainerDir": "/config",
                        "Mode": "rw"

Link to comment

/tmp is RAM so 93% used is expected (Linux caches all free RAM for other needs and freely gives it up to programs that require it).

 

It IS entirely possible that /tmp uses nearly the full amount of RAM until it stops, how large are the source video files you are transcoding? I think Plex saves the full transcoded file until the viewer stops watching the video.

 

hex, you are correct about Plex's behavior and it is why I advocated against moving transcoding to RAM in the past. I had the same problem as dirtysanchez, but it was only on long high bit rate videos (movie BD rips). All other stuff ran fine. When I dug into it I was basically juuuuuust not hitting the wall with the shorter lower bit rate stuff but the BD movie rips, at full bit rate, would fill /tmp in about 30-45 min. Crash. flush. restart. rinse. repeat.

 

Frankly the concern over SSD wear is misplaced given all current evidence and most plex / unraid implementations I can guess at. Even with 24/7 streaming of a reasonable number of 20mbit streams the SSD will outlive most people, no less its likely useful life.

 

Hex, they are approx 1.5GB 720p mkv files.  It is palying on a Roku, and so is resulting in only the audio being remuxed, therefore the transcoded file size should be approx 1.5GB.  It's certainly filling up /tmp before the play completes.

 

I'm going to have to agree with jumperalex here.  While transcoding to RAM can save a bit of wear and tear on the cache drive (assuming SSD) it's likely not something that is going to make a significant difference in the life of the SSD, assuming newer-gen SSD's.  They have proven to have significantly longer endurance than even the manufacturer ratings in most cases, and even with the added writes of transcoding to the SSD will likely outlast their useful life.

 

I have moved transcoding back to the SSD and will leave it there.  If you can transcode to RAM without issues, then I see no point in not doing so.  But if you have issues with /tmp filling up, it's probably best to just move transcoding back to cache and be done with it.

Link to comment

Can you inspect the contents of those .url files?

 

I did.  It looks like they are related to the Community Applications plugin.  Here's a snip from one of the files

{
    "apps": 230,
    "requests": 72,
    "last_updated": "25th September 2015 at 17:42:37",
    "last_updated_timestamp": "1443199357",
    "applist": [
        {
            "Beta": "False",
            "Category": "Network:Voip",
            "Name": "binhex-teamspeak",
            "Description": "\n    TeamSpeak is proprietary voice-over-Internet Protocol (VoIP) software that allows computer users to speak on a chat channel with fellow computer users, much like a telephone conference call. A TeamSpeak user will often wear a headset with an integrated microphone. Users use the TeamSpeak client software to connect to a TeamSpeak server of their choice, from there they can join chat channels and discuss things.[br][br]\n    [b][u][span style='color: #E80000;']Configuration[/span][/u][/b][br]\n    [b]/config[/b] This is where teamspeak will store it's configuration file, database and logs.[br][br]\n    [b][u][span style='color: #E80000;']Notes[/span][/u][/b][br]\n    Connect to the server using the TeamSpeak client with the host IP address and port 9987.[br]\n    To authenticate use the privilege key shown in the supervisord.log file in the host mapped /config folder.\n  ",
            "Overview": "\n    TeamSpeak is proprietary voice-over-Internet Protocol (VoIP) software that allows computer users to speak on a chat channel with fellow computer users, much like a telephone conference call. A TeamSpeak user will often wear a headset with an integrated microphone. Users use the TeamSpeak client software to connect to a TeamSpeak server of their choice, from there they can join chat channels and discuss things.\n  ",
            "Support": "http://lime-technology.com/forum/index.php?topic=38055.0",
            "Registry": "https://registry.hub.docker.com/u/binhex/arch-teamspeak/",
            "GitHub": "https://github.com/binhex/arch-teamspeak",
            "Repository": "binhex/arch-teamspeak",
            "BindTime": "true",
            "Privileged": "false",
            "Networking": {
                "Mode": "host",
                "Publish": "\n    "
            },
            "Environment": {
                "Variable": [
                    {
                        "Name": "",
                        "Value": ""
                    }
                ]
            },
            "Data": {
                "Volume": [
                    {
                        "HostDir": "path to config",
                        "ContainerDir": "/config",
                        "Mode": "rw"

CA updated to fix this...  My bad -> typo
Link to comment

/tmp is RAM so 93% used is expected (Linux caches all free RAM for other needs and freely gives it up to programs that require it).

 

It IS entirely possible that /tmp uses nearly the full amount of RAM until it stops, how large are the source video files you are transcoding? I think Plex saves the full transcoded file until the viewer stops watching the video.

 

hex, you are correct about Plex's behavior and it is why I advocated against moving transcoding to RAM in the past. I had the same problem as dirtysanchez, but it was only on long high bit rate videos (movie BD rips). All other stuff ran fine. When I dug into it I was basically juuuuuust not hitting the wall with the shorter lower bit rate stuff but the BD movie rips, at full bit rate, would fill /tmp in about 30-45 min. Crash. flush. restart. rinse. repeat.

 

Frankly the concern over SSD wear is misplaced given all current evidence and most plex / unraid implementations I can guess at. Even with 24/7 streaming of a reasonable number of 20mbit streams the SSD will outlive most people, no less its likely useful life.

I have moved transcoding back to the SSD and will leave it there.  If you can transcode to RAM without issues, then I see no point in not doing so.  But if you have issues with /tmp filling up, it's probably best to just move transcoding back to cache and be done with it.

 

How did you get it back to the SSD? did you map it there or did you just remove the /transcode directory. I mention this as a caution that it does not transcode to the docker.img as it might fill it up!

Link to comment
  • 1 month later...

I am curious if this happened to anyone else.  I just updated my server to 6.1.4 and plex quit "working".  The plex interface worked fine, and I could play music but video would not play over any device under any circumstances.  I spent about 10 hours trouble shooting and pouring over my logs, and it looks like transcoding to /tmp was the problem.  I moved the transcode directory to my ssd in docker and it started working again (it breaks if I try to move it back).  Just in case anyone else is experiencing the same issue, that seems to be a work around, but did we lose access to /tmp in 6.1.4? 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.