Script to keep Emby's transcoding directory from filling up (useful when transcoding to RAM)


Recommended Posts

I have emby transcode to RAM (/dev/shm/) which works well except for garbage collection – temporary transcode files accumulate.

 

To solve that I wrote a script (run by the container) to delete temporary files and a user-script to launch it when the container's restarted or if it's not running for any other reason.

 

Assumptions:

 

/transcode

is the container's path to /dev/shm/ (or wherever you're transcoding to)

 

/transcode/transcoding-temp

is the container's path to the directory holding temp transcoding files (emby creates this subdirectory)

 

/system-share/transcoding-temp-fix.sh

is the container's path to the following script (make the script executable)

 

transcoding-temp-fix.sh

#!/bin/sh

TRANSCODE_DIR="/transcode/transcoding-temp"
# Delete old files when used space is above this %
PERCENT_LIMIT=50
# Delete this many files at a time
BATCH_SIZE=10

if [ -d "${TRANSCODE_DIR}" ]; then
    percent_full=$(df "${TRANSCODE_DIR}" | awk '{print $5}' | tail -1 | tr -dc '0-9')
    printf "Directory size: \t %3s%%\n" ${percent_full}
    printf "Directory limit:\t %3s%%\n" ${PERCENT_LIMIT}
    echo ""
    while [ $percent_full -gt $PERCENT_LIMIT ]; do
        if [ $(find ${TRANSCODE_DIR} -type f -name "*.ts" | wc -l) -gt 0 ]; then
            echo "(${percent_full}%) exceeds limit (${PERCENT_LIMIT}%), deleting oldest (${BATCH_SIZE}) files"
            find ${TRANSCODE_DIR} -type f -name "*.ts" -exec ls -1t "{}" + | tail -${BATCH_SIZE} | xargs rm
        else
            echo "*WARNING* (${percent_full}%) exceeds limit (${PERCENT_LIMIT}%) but files are not transcoding fragments"
            exit 1
        fi
        percent_full=$(df "${TRANSCODE_DIR}" | awk '{print $5}' | tail -1 | tr -dc '0-9')
    done
else
    echo "${TRANSCODE_DIR} (TRANSCODE_DIR): directory doesn't exist"
fi

 

Now the user script to launch it, set to run every 10 minutes (*/10 * * * *)

NOTE: Update EmbyServer name and system-share (if necessary) to match your system

#!/bin/bash
#arrayStarted=true
#clearLog=true

# Verify EmbyServer's running
running=$(docker container ls | grep EmbyServer | wc -l)
if [ "${running}" != "0" ]; then
  # verify watch command that calls clearing script is running
  watch_running=$(docker exec -i EmbyServer ps | grep 'watch ' | wc -l)

  # make sure the detection command ran properly otherwise
  # we might end up running multiple instances of the script
  if [ $? -eq 0 ]; then
    if [ "${watch_running}" == "0" ]; then
      echo "Clearing script is not running. Re-starting..."
      docker exec EmbyServer sh -c 'watch -n30 "/system-share/transcoding-temp-fix.sh 2>&1" > /transcode/transcoding-temp-fix.log &'
    fi
  else
    echo "ERROR: Command to detect script run status failed"
    /usr/local/emhttp/webGui/scripts/notify -e "emby-ClearTranscodingTmp" -s "Command to detect script status failed" -d "" -i "alert"
  fi
fi

 

Monitor the script's activity:

tail -f /dev/shm/transcoding-temp-fix.log

 

Sample output:

Every 30.0s: /system-share/transcoding-temp-fix.sh 2>&1     2022-10-10 14:45:19

Directory size: 	   5%
Directory limit:	   50%

 

NOTES:

  • Script can probably be tweaked to work for Plex
  • If a better solution exists let me know, this was quick and dirty.
Edited by CS01-HS
Link to comment
  • 8 months later...
On 10/4/2022 at 1:01 AM, Andrea3000 said:

Hi @CS01-HS, thanks for this script.

 

Have you found a better solution or are you still using it?

 

Still using them.

I haven't found a better solution but I haven't looked, this "just works."

It's possible Emby fixed their garbage collection problem so it's no longer necessary.

 

I've tweaked the scripts slightly for better error-handling (initial post edited with updated scripts)

Edited by CS01-HS
Link to comment

You're right, mine too. They must have changed  it.

(My punishment for saying "just works")

 

I've replaced the simple ls:

ls ${TRANSCODE_DIR}/*.ts -1t | tail -${BATCH_SIZE} | xargs rm

 

with a find:

find ${TRANSCODE_DIR} -type f -name "*.ts" -exec ls -1t "{}" + | tail -${BATCH_SIZE} | xargs rm

to work around it.

 

I'll update my initial post unless you have something cleaner?

Link to comment
  • 2 weeks later...
On 10/11/2022 at 6:03 AM, CS01-HS said:

That definitely works as long as they don't decide to change the directory structure again.

The find should be structure-agnostic (unless I screwed up.)

Have you tested this with live tv? Mine seems to transcode tv into one massive file, Ive had sporting events become 140gb on my ssd.

Link to comment
17 hours ago, lukeoslavia said:

Have you tested this with live tv? Mine seems to transcode tv into one massive file, Ive had sporting events become 140gb on my ssd.

 

No sorry, I don't use live TV so I've never tested it.

I'm guessing they do that for DVR functionality (serves as a reference file which in the standard case is the media itself.)

 

In the best case my script won't have any effect, in the worst case it'll delete it which could break things.

If active live TV files can be differentiated by name the script could be tweaked to ignore them, or if not we can infer by size (greater than X.)

 

What are you trying to solve exactly, because my guess is creation of these large files with live TV is unavoidable.

Link to comment
11 hours ago, CS01-HS said:

 

No sorry, I don't use live TV so I've never tested it.

I'm guessing they do that for DVR functionality (serves as a reference file which in the standard case is the media itself.)

 

In the best case my script won't have any effect, in the worst case it'll delete it which could break things.

If active live TV files can be differentiated by name the script could be tweaked to ignore them, or if not we can infer by size (greater than X.)

 

What are you trying to solve exactly, because my guess is creation of these large files with live TV is unavoidable.

I would love to have these large files eliminated so I can use ram as the transcode disk, but like you said, they seem to be intentionally created. I requested a feature to turn off "time-shift" on their forums to which they replied, we have part of the functionality now, and then closed it lol.

 

My main problem with the large files is that they never get deleted unless I restart the emby server (even when properly closed), which doesn't happen in windows.

Link to comment

In Live TV -> Advanced I see Default recording path

Does setting that change where emby stores tmp live TV files?

 

2 hours ago, lukeoslavia said:

My main problem with the large files is that they never get deleted unless I restart the emby server

 

Well that's easy enough to fix even with an unraid user script.

Something like this that runs every hour, deleting ts files more than an hour old:

find /host/path/to/transcode/dir/ -type f -name '*.ts' -mindepth 1 -mmin +60 -delete

 

You could even restrict it to files above a certain size with -size +100M

Edited by CS01-HS
Link to comment
On 10/22/2022 at 9:04 PM, CS01-HS said:

In Live TV -> Advanced I see Default recording path

Does setting that change where emby stores tmp live TV files?

 

it does not seem to, I have paths set for movies and tv separately, but the transcoding files are on a separate ssd.

 

On 10/22/2022 at 9:04 PM, CS01-HS said:

Well that's easy enough to fix even with an unraid user script.

Something like this that runs every hour, deleting ts files more than an hour old:

find /host/path/to/transcode/dir/ -type f -name '*.ts' -mindepth 1 -mmin +60 -delete

 

You could even restrict it to files above a certain size with -size +100M

 

I honestly never think to use a script for pretty much anything, but I will give this a shot. I will probably have to modify it a bit to accommodate sporting events.

Thanks for the tip! 

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.