rbroberts

So, there's a workaround. Each instance has it's own /watch and /output, but they don't do any moving of files; instead if it's not for them, they just delete the input. Yes, they do. The trick is in the setup. I'm going to script this, but roughly, under the assumption that my directory structure is no more than one level, i.e., /watch/MyMovie/MyMovie.mkv, and other files in the MyMovie directory, I do the following. First, my watch directory on the server is not where the containers looks. I have /mnt/user/Videos/rip/watch/{1080,0720,0480}, three directories. But I drop the MyMovie directory in .../watch. Then I run a script that does this: for X in [A-Z]*; do for D in 0480 0720 1080; do mkdir -p $D/"$X"; ln -t $D/"$X" "$X"/*; done; done for X in [A-Z]*; do for D in 0480 0720 1080; do mkdir -p $D/"$X" ln -t $D/"$X" "$X"/* done done Since I have hard links, they've visible in the container mount points. And since they're hard links, the setup is fast, no data is copied, and deleting doesn't really remove the original. And I can easily tell when things are really done.

And...a veto would be much nicer that the move stuff anyway. I really do like that one watch directory for them all. The 480p and 720p configs convert much faster than the 1080p (naturally), so stacking them behind the 1080p partially defeats the parallel setup.

Hmm. I've set this up with a single /watch all mapped the same for each converter and, for blurays, this seems like it will work well. In fact, I've set up three instances and assigned different CPU sets to each instance, so they can run in parallel and not fight each other. But.... Only my 480p config makes sense for DVDs. All three instances will try to rip them, and I have a post conversion script that will rename them so they don't conflict when I eventually merge the outputs, but the 1080p config's output won't be any different than the 480p config's output. With the old single instance, the pre conversion script would detect that and move the input to the next queue. Looking at autovideoconverter/run, there doesn't seem to be any way to veto a conversion except to move the input file.

I don't really know what's "normal" for /storage, I'm not completely sure why it's there at all. But I generally make it a parent directory of all the video-related things I want to operate on. Then I have subdirectories that map one-to-one with the container. My original config looked like this: /storage => /mnt/user/Videos/rip /watch => /mnt/user/Videos/rip/watch /output => /mnt/user/Videos/rip/output I use MakeMKV to rip the original media into /mnt/user/Videos/rip/staging. Then once that's done and I've renamed my files to something meaningful, I move the whole directory into "watch". Once Handbrake is finished, I move the watch subdirectory into /mnt/user/Videos/rip/done, and the output subdirectory into my Plex tree. You could, of course, just have Handbrake convert directly into your Plex output directory, which seems to be what you were aiming for, but I prefer to have Plex be ignorant of what's in process until I'm done.

That looks really strange to me. One of my instances looks like this: What looks weird in yours is those extra mounts that seem to be mapping a path on the host to the same path in the container.

I assume you meant /storage => /mnt/user/Plex Media/Converted Videos/ /watch => /mnt/user/Plex Media/Converted Videos/ /output => /mnt/user/Plex Media/Converted Videos/ /config => /mnt/user/appdata/Handbrake If not, then you'll have to explain a bit more. I'm not sure docker will let you munt the same directory three times with different permissions. /storage is normally read-only. /watch normally is, but doesn't have to be. /output is obviously read-write. Also, if your input and output containers are the same type, you're going to shoot yourself in the foot.

Bummer, and I thought I would have to give this a try 😁

Actually, that's pretty clever. I was actually thinking of trying to install a second instance of the container to see if is possible to rip the different versions in parallel, and that would very neatly work for that, too.

Yeah, nvm. I had autoupdate turned on. I'm not sure if it really pulled an update last night, still looking, but that seems the most likely explanation.

Hmm, this is not exactly a HandBrake question. This looks like my container restarted last night, which orphaned a nearly completed transcode. Any ideas on how to track down why it restarted? Encoding /watch/Avatar/Avatar.mkv: 85.38 % (2.66 fps, avg 4.06 fps, ETA 02h19m45s) [services.d] stopping services [services.d] stopping app... [services.d] stopping openbox... [services.d] stopping statusmonitor... [services.d] stopping logmonitor... [services.d] stopping autovideoconverter... Encoding /watch/Avatar/Avatar.mkv: 85.39 % (2.72 fps, avg 4.06 fps, ETA 02h19m40s) Encoding /watch/Avatar/Avatar.mkv: 85.39 % (2.72 fps, avg 4.06 fps, ETA 02h19m40s) Encoding /watch/Avatar/Avatar.mkv: 85.39 % (2.72 fps, avg 4.06 fps, ETA 02h19m40s) s6-svlisten1: fatal: timed out [services.d] stopping x11vnc... caught signal: 15 24/03/2020 03:12:45 deleted 40 tile_row polling images. [services.d] stopping xvfb... The XKEYBOARD keymap compiler (xkbcomp) reports: > Warning: Unsupported high keycode 372 for name <I372> ignored > X11 cannot support keycodes above 255. > This warning only shows for the first high keycode. > Internal error: Could not resolve keysym XF86MonBrightnessCycle > Internal error: Could not resolve keysym XF86RotationLockToggle Errors from xkbcomp are not fatal to the X server [services.d] stopping nginx... [services.d] stopping certsmonitor... [services.d] stopping s6-fdholderd... [cont-finish.d] executing container finish scripts... [cont-finish.d] done. [s6-finish] syncing disks. [s6-finish] sending all processes the TERM signal. [s6-finish] sending all processes the KILL signal and exiting. [s6-init] making user provided files available at /var/run/s6/etc...exited 0. [s6-init] ensuring user provided files have correct perms...exited 0. [fix-attrs.d] applying ownership & permissions fixes... [fix-attrs.d] done. [cont-init.d] executing container initialization scripts...

That doesn't look like a permission problem, but maybe I'm missing something. I assume you tried touching the file to make it appear new? Did you also look for dot files in the output directory to see if something was left lying about from a previous failed attempt? I think that touching the file is sufficient to invalidate any dot files. What am I talking about...? Right now my encoder is running and I have this: root@Tower:/mnt/user/Video/rip# find output/* -type f | grep "Yesterday (2019).mkv" output/1080/.RsjOZG/Yesterday (2019).mkv I've see those files get left if the encoding failed, or (naturally) I restarted the container.

I keep looking at the existing scripts, and since I don't really understand how docker works, I'm pretty sure I'm looking in the wrong place. The watch directories get mounted in /watch, /watch2, etc., but I'm wondering what it would take to do this differently. If on the unraid server, I've got them as some-path/watch/1080, some-path/watch/0720, etc, there's nothing to prevent mounting some-path/watch and symlinking from /watch -> mount-point-for-some-path/1080, etc. Well, nothing except figuring out how to manage the configuration, though I would just throw that at the user. Maybe a flag that says use symlinks. The main point of this being that the pre- and post-conversion scripts don't have to change, and everything that assumes files in /watch, /watch2, etc., still works. Admittedly, in the overall scheme of how long it takes to do multiple conversions, the copy time is small, but still, for a single bluray video it's a few minutes.

Well, my message was a bit confusing. I really want to get all the watch directories in the same tree. And I don't see a variable for that; those appear to be mount points at /watch2, /watch3, etc. So while I can make the output directories all part of the same tree, I can't seem to do that for the inputs, at least not that I can tell.

Well, there was no message until I edited the run script. I've saved aside my copy, so I'll see if there is a newer version of the contain, though unraid doesn't seem to think so. The version-looking part on the screen is the repository which say jlesage/handbrake:v1.11.2....which, now that I'm looking at the support info, looks like it's really old. So I guess the mystery of why it's not working is solved, now I have to figure out why unraid thinks I'm up-to-date.

And...okay, once I start tinkering, it's hard to stop 🙂 That run script works, which is the main thing. It's not ideal, only because of how the watch directories get mounted as separate devices. My pre_conversion.sh script runs HandBrakeCLI to find the dimensions of the video and, if it's DVD quality, it moves it straight to where I've configured a conversion for that input size/type. On the unraid server, that's nearly instantaneous since it's the same device. But on the unraid, mv become cp. It's still faster than doing the conversion. I realize that having a max size won't really take any longer, but the whole point (for me) is to have a set of conversions at different resolutions and let Plex figure out which one to stream, so I need to tack that that suffix on the name. And that's easier to manage on a per-watch-directory basis since conversion presets are tied to that, too. I don't want to start whacking that run script too much, and at this point, to get all the watch directories on the same mount point would require that AUTO_CONVERSION_OUTPUT_DIR* be something other that a top-level mount point which is a much bigger change. Even what I already did may risk breaking some peoples configurations....

And...because I can't leave it alone, here's a modified autovideoconvert/run that runs pre_conversion.sh and honors AUTO_CONVERSION_OUTPUT_DIR eliminating that extra /2, /3, etc from the output subdirectory. I'm running this on my instance now. I'd love to see or something equivalent incorporated into your distribution. run.new

And this bit of autovideoconverter/run doesn't look right either: while true; do for DIR in /watch /watch2 /watch3 /watch4 /watch5; do # Set default settings. AC_PRESET="${AUTOMATED_CONVERSION_PRESET:-Very Fast 1080p30}" AC_FORMAT="${AUTOMATED_CONVERSION_FORMAT:-mp4}" AC_SOURCE_STABLE_TIME="${AUTOMATED_CONVERSION_SOURCE_STABLE_TIME:-5}" AC_SOURCE_MIN_DURATION="${AUTOMATED_CONVERSION_SOURCE_MIN_DURATION:-10}" AC_OUTPUT_SUBDIR="${AUTOMATED_CONVERSION_OUTPUT_SUBDIR:-UNSET}" AC_KEEP_SOURCE="${AUTOMATED_CONVERSION_KEEP_SOURCE:-1}" # Apply per-watch folder settings. if [ -n "${DIR#*/watch}" ]; then for VAR in PRESET FORMAT SOURCE_STABLE_TIME SOURCE_MIN_DURATION OUTPUT_SUBDIR KEEP_SOURCE do eval "AC_$VAR=\"\${AUTOMATED_CONVERSION_${VAR}_${DIR#*/watch}:-\$AC_$VAR}\"" done fi # Process watch folder. process_watch_folder "$DIR" done sleep "${AUTOMATED_CONVERSION_CHECK_INTERVAL:-5}" done Note that OUTPUT_DIR is not part of the set that can be set, though that's what the documents say. It looks like the names have to be output/2, output/3, etc, but in that case, I really don't want watch -> output. I really want the default to be treated symmetrically with the others, but there doesn't seem to be any way to do that.

I think I've found the reason it's not working.... /etc/services.d # find . -type f | xargs grep _conversion.sh ./autovideoconverter/run: if [ -f /config/hooks/post_conversion.sh ]; then ./autovideoconverter/run: /usr/bin/with-contenv sh /config/hooks/post_conversion.sh $hb_rc "$OUTPUT_FILE" "$video" "$AC_PRESET" I'm assuming that someplace in that tree I should be finding the other hooks. That assumption might be wrong, but I don't have anything else to go on right now. Grepping for "hooks" shows the same matches... I'd really, really love it if the hooks could modify the output file name. That would avoid the issue of handbrake thinking a file wasn't converted when I want to name it with some suffix-part in the name to play nice with Plex.

Okay, yet another issue with hooks. My pre_conversion.sh hook wants to move the input to a different watch directory if if decides it shouldn't encode it. DVD rips from MakeMKV appear to always have the string "720x480" near the top, so I've passing the file through head/strings/grep/awk to find this (or not). If it sees the line and the current encoder is not my DVD quality encoder, it moves the input to the correct watch directory. Except I can't see any sign that pre_conversion is running. This hook should be putting something in the log, but while I see output in the log from my post_conversion.sh script, I'm not seeing anything from this. CONVERTED_FILE="$1" SOURCE_FILE="$2" PRESET="$3" echo "pre-conversion: Output File = $CONVERTED_FILE" echo "pre-conversion: Source File = $SOURCE_FILE" echo "pre-conversion: Preset = $PRESET" # We use a preset with the name of the max resolution, e.g., 480 for # DVD quality, 1080 for bluray quality. We refuse run a conversion for # something with DVD quality with a higher quality transcoder. This # makes our life a little easier for figuring out what we have. export T1080=$(echo "$PRESET" | grep -E '\b1080') export T0720=$(echo "$PRESET" | grep -E '\b0?720') export T0480=$(echo "$PRESET" | grep -E '\b0?480') export DVD=$(head -n 50 "$SOURCE_FILE" | strings | grep '^size:.*x480') if [ -z "$DVD" ]; then echo "Not a 480 video, proceed as usual for \"$SOURCE_FILE\"" else if [ -n "$T0480" ]; then # Let the conversion go as normal echo "Proceed as usual for \"$SOURCE_FILE\"" : else O=$(echo "$SOURCE_FILE" | sed 's=/watch/=/watch2/=') echo mv "\"$SOURCE_FILE\"" "\"$O\"" mkdir -p $(dirname "$O") mv "$SOURCE_FILE" "$O" fi fi env | sort | while read L; do echo "$L"; done Edited: the original script had a syntactic error, so i thought that was the issue. But after fixing it to the above, my pre_conversion.sh script still isn't being run. I can run it manually on the handbrake console and see it works...

Ugh. And I just discovered another issue with this approach. Handbrake can't tell that it has already transcoded the file after a restart if I've renamed the output. Is there a way to defeat the attempt to reencode everything?

I'm trying to set up a pipeline where movies are encoded at more than one resolution for streaming via Plex. My idea was to simply move the input to the next watch directory when finished, along with doing a rename of the output file to include something like "1080", "480", etc so the files could b dropping in the same Plex directory and let it handle the selection. For Handbrake, I figured I'd just setup multiple watch and output directories, each watch directory having it's own preset. It kind of works. But I'm getting a slight surprise in that the output directory isn't what I seem to think it is. Here's part of the output of my post-conversion hook: [autovideoconverter] Executing post-conversion hook... post-conversion: Status = 0 post-conversion: Output File = /output/2/t1/Gag Reel-deleted.mkv post-conversion: Source File = /watch2/t1/Gag Reel-deleted.mkv post-conversion: Preset = Custom H.265 MKV 480p conversion was successful moving to 480 APP_NAME=HandBrake APP_NICE_CMD= APP_USER=app AUTOMATED_CONVERSION_CHECK_INTERVAL=5 AUTOMATED_CONVERSION_FORMAT=mkv AUTOMATED_CONVERSION_KEEP_SOURCE=1 AUTOMATED_CONVERSION_MAX_WATCH_FOLDERS=5 AUTOMATED_CONVERSION_NON_VIDEO_FILE_ACTION=ignore AUTOMATED_CONVERSION_NON_VIDEO_FILE_EXTENSIONS=jpg jpeg bmp png gif txt nfo AUTOMATED_CONVERSION_NO_GUI_PROGRESS=0 AUTOMATED_CONVERSION_OUTPUT_DIR=/output AUTOMATED_CONVERSION_OUTPUT_DIR_2=/output-0480 AUTOMATED_CONVERSION_OUTPUT_SUBDIR=SAME_AS_SRC AUTOMATED_CONVERSION_PRESET=Custom H.265 MKV 1080p AUTOMATED_CONVERSION_PRESET_2=Custom H.265 MKV 480p AUTOMATED_CONVERSION_SOURCE_MIN_DURATION=10 AUTOMATED_CONVERSION_SOURCE_STABLE_TIME=5 and here's my hook CONVERSION_STATUS=$1 CONVERTED_FILE="$2" SOURCE_FILE="$3" PRESET="$4" echo "post-conversion: Status = $CONVERSION_STATUS" echo "post-conversion: Output File = $CONVERTED_FILE" echo "post-conversion: Source File = $SOURCE_FILE" echo "post-conversion: Preset = $PRESET" export T1080=$(echo "$PRESET" | grep -E '\b1080') export T0720=$(echo "$PRESET" | grep -E '\b0?720') export T0480=$(echo "$PRESET" | grep -E '\b0?480') if [ $CONVERSION_STATUS -eq 0 ]; then echo "conversion was successful" if [ -n "$T1080" ]; then echo "moving to 1080" eval mv "\"$CONVERTED_FILE\"" "\"$(echo "$CONVERTED_FILE" | sed 's/[.]\([^.]*\)$/ - (1080).\1/')\"" elif [ -n "$T0720" ]; then echo "moving to 720" eval mv "\"$CONVERTED_FILE\"" "\"$(echo "$CONVERTED_FILE" | sed 's/[.]\([^.]*\)$/ - (720).\1/')\"" elif [ -n "$T0480" ]; then echo "moving to 480" eval mv "\"$CONVERTED_FILE\"" "\"$(echo "$CONVERTED_FILE" | sed 's/[.]\([^.]*\)$/ - (480).\1/')\"" fi : else echo "conversion failed" : fi env | sort | while read L; do echo "$L"; done I don't understand the path of the output file. I expected that to be in the directory of AUTOMATED_CONVERSION_OUTPUT_DIR_2. That directory does exist and is mounted in the container; I added a path to the container for that. It seems I'm not really understanding something about the override variables.

Any idea how to get the background task to use more CPU/threads? When I build a queue using the GUI and submit it, Handbrake takes over the machine. That's actually good as far as I'm concerned. When I throw jobs into the watch directory, the CPU seems to plug along at about 30-40%. I have a server with dual Xeon E5-2695 CPUs, so there's plenty of horsepower, but it's not using it. I'm re-encoding my entire library after realizing that my original encoding had dropped the audio to stereo. I never noticed because I only recently got a surround system. Fortunately, I kept the original rips, so it's just a matter of dropping them into the watch directory and waiting. And waiting....

Thanks! It took me a couple of tries to get it right for both audio and subtitles, but it's working now.

I've been trying to figure out the magic settings to get the audio and subtitles to pass through into my mkv container. So far, nothing is really working.... I can sort-of get there by opening the original rip and looking at all the audio and subtitle stream and setting it up manually, for that particular movie. But that really doesn't work for running things via the watch queue in the background; if I save that setting, it has particular streams selected that might not make sense for the next video. My starting point was H.264 MKV 1080p30 preset followed by clearing the audio settings and adding them from scratch. When playing the original MKV rip, it shows has having a 7.1 (3F2M2R/LFE) English track with dts encoding Yet, if select DTS-HD Passthru 384kpbs, 7.1, same-as-source, as the first encoder, on the audio list, clicking select all doesn't pull in a 7.1 track. It does seem to pull in all of the audio tracks. But I'm clear how that manual setting is going translate to the next video which may not have the same set of audio tracks. I have a similar issue with the subtitles; I can manually add all the subtitle tracks, but don't see how this translates to a configuration I can save for the automatic transcoding. Is what I'm trying to do even possible, or do I have to do it one video at at time? That's not really going to work for me since I have quite a few to transcode....

Thanks, that worked perfectly and was relatively painless.