GJones Posted March 3, 2011 Author Share Posted March 3, 2011 At this point, testers are needed most. I am about ready to do this, but since I have never used Handbrake, I have to ask an ignorant question. All my files are BluRay ISO with lossless audio tracks, so do I need to convert them to MKV first to go through your procedure, or what? And what CLI options do you guys use? The Handbrake forums are all over the place with an infinite number of configurations. I just want a high quality setting that reduces file size by 50 - 66%. It would be best to build them as mkv files first. My preferred tool for accomplishing this is makemkv or makemkvcon (console version). If I could locate a few dev libraries for unraid's version of slackware, I could get both working on unraid itself. Keep in mind that Handbrake does not like some high-end audio tracks and will not even pass them through to the output file. I am fine with core DTS so it does not present a problem for me. If you can get by with core DTS, makemkv can help you separate them rather well. I am a fan of the High Profile default in Handbrake. It tends to strike a good balance for me for most content. There are two main areas of interest in this default: audio processing and quality setting. The audio processing (-a track1,track2 -E codec1,codec2) determine which audio tracks from the source to process and how to encode them. I pull all of the audio tracks in the source and copy them as-is to the output. At this level of file size, I feel it is unnecessary to save the small percentage on recompressing audio. The quality setting is a little more complex. The default quality for "High Profile" is 20, with lower values meaning less compression and higher values meaning more compression. An increase of 1 in quality means a potential increase of file size of roughly 6%. For DVDs, I want the quality to be slightly higher than the default at 18. This means, at a very high level, that the picture quality is 82% (100% - 18%) of the decoded stream. In other words, Handbrake paints the picture as decoded from MPEG2 and then saves it at 82% quality into the output stream. Keep in mind that saving at 100% quality from the decoded MPEG2 stream would actually result in a file size much larger than the input file, since the input file itself is compressed. For a DVD with all english AC3 audio tracks encoded in this way, I generally end up with a file of around 1.5GB. HD content has a different quality setting by default in my code. Since it is much higher resolution, I am willing to use a slightly higher q value. I use a value of 21 for most HD content, resulting in what appears to be consistent with a DirecTV feed of a good HD channel. The file size is generally between 7 and 8GB. Compared to sources at around 35 GB, I find this very reasonable. I use an ION-based XBMC Live implementation for playback and find these values very satisfying, balancing between file size and image quality. If you have other requirements or need to accommodate other playback requirements, you can make changes in tcOPTIONS accordingly. Link to comment
GJones Posted March 4, 2011 Author Share Posted March 4, 2011 Updated version posted. Fixed issue with the PATH available in cron. Link to comment
christuf Posted March 16, 2011 Share Posted March 16, 2011 How do you handle spaces in the path within tcOPTIONS? I'm trying to use a search folder of "/mnt/user/tv/This\ Series\ Name/", but it tries to find ".mnt/user/tv/This\", "Series\" and "Name\", getting a "No such file or directory" warning for each. Tried it without the forward slashes to no avail... What is the format I should use? Link to comment
kizer Posted March 16, 2011 Share Posted March 16, 2011 Normally in Linux I would recommend changing "This Series Name" to "This_Series_Name". I really don't want to convert all my spaces to _'s because I have a ton of files, but luckly all my episodes that have spaces in them are already done. I'm guessing for future MKV's and ISO's I should use _'s I've seen various ways of naming things like BattleStar_Galactica_-_S01E01.avi BattleStar.Galactica-_-S01E01.avi BattleStar-Galactica-S01E01.avi BattleStar-_-Galactica.S01E01.avi I honestly prefer to use _ for spaces. BattleStar_Galactica I haven't tried this script, but I think I might give it a spin today to see how it does. I have a buddy with a ton of DVD's and I'm struggling with the idea of figuring out how to use MKV's or ISO's, but if this works I wont have to. Link to comment
kizer Posted March 16, 2011 Share Posted March 16, 2011 @GJones What would you recommend for those of us that don't have a SNAP drive or a Cache drive? Link to comment
aiden Posted March 17, 2011 Share Posted March 17, 2011 At this point, testers are needed most. I am about ready to do this, but since I have never used Handbrake, I have to ask an ignorant question. All my files are BluRay ISO with lossless audio tracks, so do I need to convert them to MKV first to go through your procedure, or what? And what CLI options do you guys use? The Handbrake forums are all over the place with an infinite number of configurations. I just want a high quality setting that reduces file size by 50 - 66%. It would be best to build them as mkv files first. My preferred tool for accomplishing this is makemkv or makemkvcon (console version). If I could locate a few dev libraries for unraid's version of slackware, I could get both working on unraid itself. Okay, so before I travel down the long and arduous path of re-containing my ISO files as MKV, is there not a clever way to mount the ISO is a virual drive, thus allowing Handbrake to parse the folder structure? I know this works in Windows using Virtual Clone Drive. I was thinking of something along the unMENU Mount ISO script. Obviously it's much more cumbersome, requiring mount and umount commands, but it would be an interesting alternative to redoing all my movies. Any thoughts to this? Keep in mind that Handbrake does not like some high-end audio tracks and will not even pass them through to the output file. I am fine with core DTS so it does not present a problem for me. If you can get by with core DTS, makemkv can help you separate them rather well. Since ALL of my movies are backed up with lossless audio tracks only (LPCM, TrueHD, DTS-MA), does this mean handbrake will not correctly output the audio? Obviously if I'm compressing the video I'm happy to compress the audio as well, so changing the format to DTS or DD is no big deal to me. Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 HandBrakeCLI will read an ISO input file just like an MKV. You just need to tweak the line that begins with LIST in tcrepack and/or tcrepackblu. The existing line for tcrepack does a number of things: List all files and directories in the search variable from TCOPTIONS Filter out anything that is not an mkv file (this would be the part to change Skip files with MPEG2 in the filename (I use this to designate really old films in MPEG2 that would not translate well) Skip files with BLURAY in the filename (because this is the SD finder and I don't want to use the wrong settings) Skip files with H264 in the filename (I use this to tag movies that already got transcoded) Skip files with trailer in the filename (Please don't make me explain that) Use sed to tack on the command (ts tcremkv) and the quotation marks (DO NOT CHANGE) When you use ISO files, be careful. I specifically use MKV files because they have only one title in them at most times. ISO files do not. If you only change it to look for .iso files, it will always default to the initial title in the iso file. Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 How do you handle spaces in the path within tcOPTIONS? I'm trying to use a search folder of "/mnt/user/tv/This\ Series\ Name/", but it tries to find ".mnt/user/tv/This\", "Series\" and "Name\", getting a "No such file or directory" warning for each. Tried it without the forward slashes to no avail... What is the format I should use? If your tv shows are in subdirectories under /mnt/user/tv/Some Show Name, it will handle this by default. That happens to be the way that mine are organized. The path in tcOPTIONS should point to /mnt/user/tv. You don't have to go down any further due to the way that find works. Spaces should not present an issue with the code. Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 @GJones What would you recommend for those of us that don't have a SNAP drive or a Cache drive? I will gladly defer to someone more unraid-savvy, but I would suggest a single directory in the array that has the least contention with the storage for the sources. If you (can) store tv shows on disk1, disk2 and disk3, then make TEMPDISK="/mnt/disk4" because that will be the least problematic. Understand that HandBrakeCLI will not max out the disk speed unless your disks are horrid. However, you want to be reading from a different disk than you are using for writes. If you use all of them for the sources, make TEMPDISK point to the disk with the least amount of content. Link to comment
christuf Posted March 17, 2011 Share Posted March 17, 2011 You don't have to go down any further due to the way that find works. Thank you GJones - I was trying to be more specific to prevent the script from re-transcoding all my files without having to rename everything _H264. But I've take the advice given earlier and renamed all spaces to "_"... doesn't look as pretty, but it works! May I ask about the logic for putting the replaced files on the same disk as the original? Seems to me it might be easier to have them all kept centrally (on the cache drive for instance), rather than having to go through each disk to clear out? Maybe I'm missing something here though. Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 Normally in Linux I would recommend changing "This Series Name" to "This_Series_Name". I really don't want to convert all my spaces to _'s because I have a ton of files, but luckly all my episodes that have spaces in them are already done. I'm guessing for future MKV's and ISO's I should use _'s I've seen various ways of naming things like BattleStar_Galactica_-_S01E01.avi BattleStar.Galactica-_-S01E01.avi BattleStar-Galactica-S01E01.avi BattleStar-_-Galactica.S01E01.avi I honestly prefer to use _ for spaces. BattleStar_Galactica I haven't tried this script, but I think I might give it a spin today to see how it does. I have a buddy with a ton of DVD's and I'm struggling with the idea of figuring out how to use MKV's or ISO's, but if this works I wont have to. If you happen to have the option of starting from scratch, I can tell you an organization that works exceptionally well for XBMC (the one I use). /mnt /user /movies /Some Movie (1999) SOME_MOVIE.mkv movie.nfo /Some Other Movie (2007) Some_Other_Movie_00.mkv movie.nfo /movieshd /Some Epic (1999) SOME_EPIC_BLURAY.mkv movie.nfo /Blah Blah Movie (2007) Blah_Blah_Movie_BLURAY.mkv movie.nfo /tvshows Some Great TV Show GREAT_TV.s01e01.mkv GREAT_TV.s01e02.mkv ... GREAT_TV.s02e15.mkv Awesome Action Show Some_Series.s01e01.mkv Yes, you can make a videos share and put the movies and tv directories under that. I avoided that only because I wanted split levels handled a little differently across some other forms of videos. You can add season folders under each TV series, but I don't. XBMC (my media center of choice) doesn't much care and I lack the desire to add them at this point. I also separate HD movies from SD only because the storage requirements are different. I "manage" those differently across disks. The point is not "do this the way I do it." The point is that the script gives you a lot of flexibility in putting movies in a lot of different directories or one. Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 You don't have to go down any further due to the way that find works. Thank you GJones - I was trying to be more specific to prevent the script from re-transcoding all my files without having to rename everything _H264. But I've take the advice given earlier and renamed all spaces to "_"... doesn't look as pretty, but it works! May I ask about the logic for putting the replaced files on the same disk as the original? Seems to me it might be easier to have them all kept centrally (on the cache drive for instance), rather than having to go through each disk to clear out? Maybe I'm missing something here though. The goal was to put the final file "back" on the same disk as the original, while really putting it on the cache disk. To make a really complex topic short, unraid does not handle this the way I thought it would. I now specifically write them out to the cache drive and let unraids mover put them back later. This means an immediate move for the old file (final dir to replaced dir on the same disk) and the new file (from the transcode directory to the final directory on the cache drive). As to the overall goal, it is pretty straightforward. I pull movies in using MakeMKV (not running on the unraid box, but wouldn't that be nice). I store them immediately because I want to be able to watch the movie on my media center immediately. The last step in tcremkv and tcreblu is to update the library on the XBMC server: Store the "raw" mkv Index it in XBMC tcrepack or tcrepackblu finds the file and transcodes it Put the transcoded file back in the original directory Move the original file to the replaced directory (in case of a problem) Re-index to find the new file and remove the old (now missing) file Yes, now that I think of it, I could have put those in separate directories but I think I avoided that simply from maintenance. I don't really like looking in separate places (or configuring XBMC to do so) for movies. I separate kids' movies from other movies so I can serve them up to the kids in one chunk. HD movies are separated only because of different storage needs (balancing larger files). Link to comment
christuf Posted March 17, 2011 Share Posted March 17, 2011 GJones - I'm so sorry... I totally messed up what I meant, and just realized! What I really meant was is there a reason that the original files (non-transcoded) go into a "replaced" directory on the disk that they came from? I was thinking it would be easier to keep all the replaced originals in one central place. So sorry for the confusion! Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 NEW VERSION 095 Changed final directory processing to create the directory if needed on the cache drive and send the file there Fixed one typo that was a # instead of a " Fixed a missing tcLIBUPD in tcreblu Fixed a PATH problem in the cron Link to comment
GJones Posted March 17, 2011 Author Share Posted March 17, 2011 GJones - I'm so sorry... I totally messed up what I meant, and just realized! What I really meant was is there a reason that the original files (non-transcoded) go into a "replaced" directory on the disk that they came from? I was thinking it would be easier to keep all the replaced originals in one central place. So sorry for the confusion! Yes, there is a very complex technical reason for this. Just kidding. I like the file moves to take as little time as possible. The quicker moves happen, the less can go wrong. An on-disk move is instantaneous. Moving across disks takes far far longer. My goal is to get those files out of the "main" directory as fast as possible. I check my "replaced" directory about once a week to make sure that the transcodes had no issues and empty the directory out. If you are using user shares (and I am assuming you are), /mnt/user/replaced is one central location, even if it is across several disks. I am trying to accomplish both fast and convenient. Link to comment
aiden Posted March 18, 2011 Share Posted March 18, 2011 When you use ISO files, be careful. I specifically use MKV files because they have only one title in them at most times. ISO files do not. If you only change it to look for .iso files, it will always default to the initial title in the iso file. Ah... but when I back up my movies to ISO, I only grab the primary video track, and everything is remuxed into one .m2ts file. I know it's a little lame, but Dune players don't support subtitles over MKV, so I'm working with hardware constraints. Do subtitles and lossless audio represent any stumbling blocks? Link to comment
GJones Posted March 18, 2011 Author Share Posted March 18, 2011 When you use ISO files, be careful. I specifically use MKV files because they have only one title in them at most times. ISO files do not. If you only change it to look for .iso files, it will always default to the initial title in the iso file. Ah... but when I back up my movies to ISO, I only grab the primary video track, and everything is remuxed into one .m2ts file. I know it's a little lame, but Dune players don't support subtitles over MKV, so I'm working with hardware constraints. Do subtitles and lossless audio represent any stumbling blocks? Lossless audio and PGS subtitles are a problem only because HandBrake does not deal well with them. I use makemkv to pull the core audio and only forced subs. This lets HandBrakeCLI handle the audio sync correctly with the core DTS track and then I remux in the PGS subtitles if they are truly forced. I use mkvmerge (not part of this tool) to remux when necessary. It is rarely necessary for me. DVD subs work perfectly as-is. Link to comment
kizer Posted March 18, 2011 Share Posted March 18, 2011 Okkie I'm running 0.95 and I have a question. I set my TEMP location in the config file to /mnt/disk4/Downloads When I run either Run one job by: ts tcremkv /mnt/user/full/path/to/mkvfile.mkv if you have a cache drive OR Run one job by: ts tcremkv /mnt/diskX/full/path/to/mkvfile.mkv if you do not have a cache drive I get the following in my status when I check it. 0 running /tmp/ts-out.uoh8n0 tcremkv /mnt/user/Movies/thegreatest.mkv Shouldn't it be dumping to /mnt/disk4/Downloads not /tmp/ts-out.uoh8no The reason I ask is when I run it I watch the completion time climb and climb so I'm guessing its just eating up Ram and not working with the /mnt/disk4/Downloads folder Link to comment
GJones Posted March 18, 2011 Author Share Posted March 18, 2011 Okkie I'm running 0.95 and I have a question. I set my TEMP location in the config file to /mnt/disk4/Downloads When I run either Run one job by: ts tcremkv /mnt/user/full/path/to/mkvfile.mkv if you have a cache drive OR Run one job by: ts tcremkv /mnt/diskX/full/path/to/mkvfile.mkv if you do not have a cache drive I get the following in my status when I check it. 0 running /tmp/ts-out.uoh8n0 tcremkv /mnt/user/Movies/thegreatest.mkv Shouldn't it be dumping to /mnt/disk4/Downloads not /tmp/ts-out.uoh8no The reason I ask is when I run it I watch the completion time climb and climb so I'm guessing its just eating up Ram and not working with the /mnt/disk4/Downloads folder I assume you mean you are setting TEMPDISK to /mnt/disk4/Downloads and not TEMPDIR to /mnt/disk4/Downloads. The only thing going to /tmp/ts-out..... is the log. The log is using RAM but the output video file is being written to /mnt/disk4/Downloads/.transcode/thegreatest_VIDEO.mkv, using the directory you requested. You can read the log by typing ts -t (just like running tail -f /mnt/ts-out.logfilename, and will monitor the log until the command finishes or you hit CTRL-C). I will look into removing the log going to /tmp to save RAM, but it hadn't been an issue on my 4GB machine. Keep in mind that ts is part of the solution but not my program. Link to comment
kizer Posted March 18, 2011 Share Posted March 18, 2011 Ah figured it out. apparently I was using my windows machine to look in the /mnt/disk4/ folder and it was creating a hidden file and I didn't have show hidden files on. Now I feel kinda dumb LOL Encoding: task 1 of 1, 0.35 % (14.92 fps, avg 19.10 fps, ETA 06h16m58s) Some how I had multiple instances of HandBrake running all night because I was well stupid or something and noticed my CPU was running insanely hard all night. LOL Link to comment
christuf Posted March 18, 2011 Share Posted March 18, 2011 If you are using user shares (and I am assuming you are), /mnt/user/replaced is one central location, even if it is across several disks. I am using shares, but this folder doesn't appear in the list of network folders in OS X (not sure if you have any Mac experience). It does appear in the terminal when I browse to /mnt/user/replaced though... Should I delete the directory /mnt/user/replaced, and then create a share in the UnRaid interface? Link to comment
GJones Posted March 18, 2011 Author Share Posted March 18, 2011 If you are using user shares (and I am assuming you are), /mnt/user/replaced is one central location, even if it is across several disks. I am using shares, but this folder doesn't appear in the list of network folders in OS X (not sure if you have any Mac experience). It does appear in the terminal when I browse to /mnt/user/replaced though... Should I delete the directory /mnt/user/replaced, and then create a share in the UnRaid interface? Do not delete it. Just add the share if you want it to be a share. Then you can see it on the Mac as well. Link to comment
christuf Posted March 28, 2011 Share Posted March 28, 2011 Hi - me again! TC is working excellently, except for the daily cron aspect of it. A few days ago I put the dailyrepack it into the etc/cron.daily directory, and then clean forgot about it. I went to check today, and the ts has over 3000 entries (there are probably only 10 or so files that actually need encoding). It seems to be picking up the correct files for transcoding, but the are thousands of entries to /tmp directory... they're called ts-out.xxxxxx (where xxxxxx is a random six characters - like EtSwL4). The entries in ts look something like this: 3191 finished /tmp/ts-out.EtSwL4 0 4.06/0.25/0.04 /usr/local/bin/tcremkv /mnt/user/TV/Name/Season_x/Filename.mkv If I just call tscrepack from the command line everything works perfectly, so it seems to be something funky going on with the cron job. Grateful for any support - please let me know if you need any further details. Thank you! Chris Link to comment
GJones Posted March 28, 2011 Author Share Posted March 28, 2011 Cron does not see /usr/local/bin on the path. I will be posting a fix today. The failure is in it not seeing HandBrakeCLI in the path. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.