timmyx Posted December 18, 2020 Share Posted December 18, 2020 Hello,  I'm fairly new to unraid and definitely new to docker containers and linux scripts  But I use youtube-dl on my pc to regularly download new videos from channels I follow (just as safety measure that my inner /r/datahoarder tells me to do in case these channels ever go byebye 😆)  What I do now is call a .bat file on a schedule basis with all my arguments, naming structures, archives, etc. So it's fairly automated except that I would have to download a newer youtube-dl version manually  I've been setting up my first unraid server for the past couple of weeks now and this is something I'd like to move over to an unraid process. But I've tried a few different docker containers that I didn't have luck with:  youtube-dl-server: runs ok for but for single downloads, it can't handle customized output parameters nor can it listen to a playlist/channel for updates and automatically downloads youtube-dl material: promising, had an option to write archives automatically but I couldn't get the output parameters to work go auto yt: says from info that it can handle output options and archive writing, but I kept getting a 'There was an error getting channel metadata: From c.GetMetadata():' so I just ditched it  Alternatively, I've come around this and this but I wouldn't know how to setup or if it could manage to handle my script. So I was thinking... is there a way that I could have unraid read my batch files on a regular basis and run youtube-dl? Much like I do right now on windows  My batch file contains several commands like this:  youtube-dl --ignore-errors --no-continue --write-description --write-sub --download-archive "downloadsfolder/archive.txt" -f "(mp4)[height <=? 1080]" -o "downloadsfolder/%%(uploader)s - %%(upload_date)s - %%(title)s.%%(ext)s" https://www.youtube.com/channel/UCVI66VMGYg0dLPEkWPrCkBw for /R "downloadsfolder" %%f in (*.description) do ren "%%f" *.txt  tl;dr youtube-dl docker containers I tested can't handle some or all my output arguments for channels I regularly download to archive. is there a way to make unraid call a batch file (or similar) on a schedule basis and run youtube-dl to download newly added videos that are not listed in its output archive?  any help is really appreciated! 😊 Quote Link to comment
JonathanM Posted December 18, 2020 Share Posted December 18, 2020 Search Youtube dl material in the apps tab. https://forums.unraid.net/topic/87798-support-selfhostersnets-template-repository/ https://github.com/Tzahi12345/YoutubeDL-Material 1 Quote Link to comment
primeval_god Posted December 19, 2020 Share Posted December 19, 2020 On 12/18/2020 at 9:27 AM, timmyx said: tl;dr youtube-dl docker containers I tested can't handle some or all my output arguments for channels I regularly download to archive. is there a way to make unraid call a batch file (or similar) on a schedule basis and run youtube-dl to download newly added videos that are not listed in its output archive? You can use the User.Scripts plugin to run a bash script on a schedule. So far as running youtube-dl specifically the best way is within a docker container. You would use a docker run command in your script (probably with the --rm option to delete the container when done) to launch a youtubedl container from your script. Depending on the image you choose you may need to override the container entrypoint to directly launch youtubedl and then your would pass your arguments in the docker cmd. 1 Quote Link to comment
timmyx Posted December 21, 2020 Author Share Posted December 21, 2020 Thanks for the replies. I found a docker that's very promising to my use case  mikenye/youtube-dl - Docker Hub  All I need now is to figure out how to get it running Quote Link to comment
CS01-HS Posted December 21, 2020 Share Posted December 21, 2020 I may be going about this the wrong way but I have a Debian VM in unraid always running that even on my underpowered board uses barely any CPU (< 1% at idle.) With /mnt/user mapped in the VM it gives full access to the unraid shares and makes tasks like installing youtube-dl and other miscellaneous CLI apps much easier. 1 Quote Link to comment
mikenye Posted December 21, 2020 Share Posted December 21, 2020 (edited) Hi Timmy,  It’s mikenye here, author of the YouTube-DL docker container you want to try.  You DM’d me on discord but I can’t reply to you as we don’t share a server (I think you came and left?).  If you re-join the server I would be happy to help you.  -Mike Edited December 21, 2020 by mikenye 1 Quote Link to comment
CS01-HS Posted December 21, 2020 Share Posted December 21, 2020 (edited) Here for reference is my full setup within Debian: [debian@debian]:~ $ sudo crontab -l # Edit this file to introduce tasks to be run by cron. # # Each task to run has to be defined through a single line # indicating with different fields when the task will be run # and what command to run for the task # # To define the time you can provide concrete values for # minute (m), hour (h), day of month (dom), month (mon), # and day of week (dow) or use '*' in these fields (for 'any'). # # Notice that tasks will be started based on the cron's system # daemon's notion of time and timezones. # # Output of the crontab jobs (including errors) is sent through # email to the user the crontab file belongs to (unless redirected). # # For example, you can run a backup of all your user accounts # at 5 a.m every week with: # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/ # # For more information see the manual pages of crontab(5) and cron(8) # # m h dom mon dow  command # update youtube-dl once a week 05 00 * * 0 /usr/bin/wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl [debian@debian]:~ $ [debian@debian]:~ $ crontab -l # Edit this file to introduce tasks to be run by cron. # # Each task to run has to be defined through a single line # indicating with different fields when the task will be run # and what command to run for the task # # To define the time you can provide concrete values for # minute (m), hour (h), day of month (dom), month (mon), # and day of week (dow) or use '*' in these fields (for 'any'). # # Notice that tasks will be started based on the cron's system # daemon's notion of time and timezones. # # Output of the crontab jobs (including errors) is sent through # email to the user the crontab file belongs to (unless redirected). # # For example, you can run a backup of all your user accounts # at 5 a.m every week with: # 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/ # # For more information see the manual pages of crontab(5) and cron(8) # # m h dom mon dow  command # download nightly 00 01 *  *  *   /home/debian/scripts/youtube-playlist-download.sh > /var/log/youtube-playlist-download.log 2>&1 [debian@debian]:~ $ more /home/debian/scripts/youtube-playlist-download.sh #!/bin/bash # Download my youtube playlists # OPTIONS # YOUTUBE_CMD="/usr/local/bin/youtube-dl" SAVE_DIRECTORY="/mnt/unraid/Private/Youtube/" ARCHIVE_FILE="/home/debian/scripts/youtube-playlist-download.downloaded" YOUTUBE_USERNAME="PUT_YOUR_YOUTUBE_USERNAME_HERE" CMD_OPTIONS=(-i   --yes-playlist   --embed-thumbnail   --add-metadata   --embed-subs   --all-subs   -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4   --download-archive ${ARCHIVE_FILE}   ) ${YOUTUBE_CMD} ${CMD_OPTIONS[@]} -o "${SAVE_DIRECTORY}%(playlist)s/%(playlist_index)03d - %(title)s.%(ext)s" https://www.youtube.com/user/${YOUTUBE_USERNAME}/playlists exit 0 [debian@debian]:~ $  Edited December 21, 2020 by CS01-HS 1 Quote Link to comment
timmyx Posted December 24, 2020 Author Share Posted December 24, 2020 (edited) I'd like to thank everyone for the replies, I finally have it running (almost) seamlessly.  Here are the steps for anyone who wants to setup youtube-dl in unraid using mikenye's docker – shout out to mikenye for his support!  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––– How to add youtube-dl and run it your way Background None of the docker containers available in unRaid's repository from Community Applications worked for me the way I wanted (see first post) Repurposed first porst tl;dr - I wanted to be able to set a 'template' of commands that I already use with youtube-dl on Windows to check, download and write a list of downloaded videos from a list of channels, on a schedule and in unRaid instead I found plenty of other youtube-dl dockers in dockerHub but mikenye's was the one with the most information available and it's maintained quite regularly, so it was my choice While this how-to may be applicable to other youtube-dl docker containers, it is focused on mikenye's, so some steps and some arguments in the code may vary Resources mikenye/youtube-dl - Docker Hub youtube-dl - GitHub (reference) custom icon First things first This how-to assumes that you already have unRaid up and running with users and shares set You will need to install the official apps repository called Community Applications Check this video from Spaceinvader One until the 3 min mark and this post for the installation url Note that you will need unRaid 6.8.3 or newer to be able to use CA Once you have CA installed, you will need to also enable results from dockerHub: Then install mikenye/youtube-dl (dockerHub) and User Scripts (CA): Alternatively, you can install mikenye's docker from the Add Container option: Once you've completed the steps above, you will have most that is needed to get youtube-dl running Adding the script to run in unRaid Now we will set up the script to start youtube-dl automatically i.e. run the docker from unraid Follow these steps to add a new script: Now choose to edit the script: This is my script: Quote #!/bin/bash var=`date +"%FORMAT_STRING"` now=`date +"%m_%d_%Y"` now=`date +"%Y-%m-%d"` docker run \ --rm \ -i \ --name youtube-dl-cron \ -e PGID=100 \ -e PUID=99 \ -v /mnt/user/Archive:/workdir:rw \ -v /mnt/user/timmyx/Settings/youtube-dl:/settings:rw \ -v /mnt/user/timmyx/Settings/youtube-dl/ytdl.conf:/etc/youtube-dl.conf:ro \ mikenye/youtube-dl >> /mnt/user/timmyx/Settings/youtube-dl/logs/youtube-dl_${now}.log Arguments in details Red: used to add dates to the log files e.g. youtube-dl_2020-12-23.log Blue: are covered and explained by mikenye's documentation Orange: I added as reference to be used by ytdl.conf Note: change /mnt/user/timmyx/Settings/youtube-dl/ and /mnt/user/Archive to your desired folders Note: I tried to add /settings and /workdir through the docker options (see below) but they wouldn't work, so I forced them in the script If you want unRaid to run the script every so often, choose one of the preset schedules or use cron Configuring ytdl.conf The config file is used as your standard settings using youtube-dl's own arguments Note: all videos you download through the docker and through the user script from previous section will always use the options in this config file This is my config file: Quote --ignore-errors --geo-bypass --no-continue --no-progress --write-description --write-sub --sub-lang en --sub-format 'srt' --write-auto-sub --convert-subs srt --download-archive '/settings/downloaded.txt' --format 'bestvideo[ext=mp4,height<=?1080]+bestaudio[ext=m4a]/best[ext=mp4]/best' --output '/workdir/Youtube/%(uploader)s/%(uploader)s - %(upload_date)s - %(title)s [%(id)s].%(ext)s' --batch-file '/settings/channels.txt' Note: I will not go through details of each option but they are thoroughly explained in youtube-dl's documentation Note: For some reason, using --format "best[height<=?1080]" did not always download videos available in 1080p, so I had to force additional options Configuring channels.txt Used to easily add/remove channels to be downloaded from by the script/config file. This is what it should look like: Quote # 10 hour https://www.youtube.com/channel/UCVI66VMGYg0dLPEkWPrCkBw/videos # # channel name url # channel name url ... Or simply one channel per line Post-download folder permissions Note: before you do anything here know what this does here and here If you use Windows like I do, chances are you won't be able to modify any of the files downloaded by the docker due to folder permission issues (provided you are not logged in with root credentials in Windows of course) As a workaround, there is an option in unRaid that tweaks the settings and grants you write access again: You might need to go through this workaround again when new folders/channels are downloaded Wrapping things up This should be it. You now have mikenye's docker installed and running through custom settings and on a schedule (at least mine is :)) I used this image as icon to the docker container just so it looks pretty (I hope you won't mind mikenye!)  Additional options How to create nfo files for the downloaded content (this helps emby show correct Title and Description of downloaded videos) Follow this link to the walkthrough (tks hczv) Edited November 3, 2021 by timmyx 3 4 Quote Link to comment
tamito Posted January 30, 2021 Share Posted January 30, 2021 @Timmex Thank you for writing it up. I am able to follow you step and make it work. Do you have solution if some channel you want to download only couple of video instead of entire channels? Quote Link to comment
timmyx Posted February 1, 2021 Author Share Posted February 1, 2021 On 1/30/2021 at 2:18 PM, tamito said: @Timmex Thank you for writing it up. I am able to follow you step and make it work. Do you have solution if some channel you want to download only couple of video instead of entire channels? Hi, I'm sorry for the late response, you tagged someone else entirely 😆  To your question: youtube-dl has some native options but depending on your case I'd either do: a simulated run with archive options and remove all entries you actually want downloaded from the archive (this will make yt-dl write all strings from the channel but download none), or add all videos you want to a public playlist and have yt-dl fetch that list #1 is more work but will run great if you don't want to download specific videos in future releases (which would have you redo the procedure all the time)  #2 is the simplest, fastest and future-proof option so I'd stick with that  Hope this helps, feel free to reply if you need more assistance though Quote Link to comment
tamito Posted February 1, 2021 Share Posted February 1, 2021 @timmyx I tagged the wrong person.  Thank you for the solution. Do you encounter similar issue on downloading subtitle and cause too many connection error?  ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by ); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. WARNING: video doesn't have subtitles WARNING: No subtitle format found matching "srt" for language en, using vtt ERROR: unable to download video data: WARNING: video doesn't have subtitles  Quote Link to comment
timmyx Posted February 2, 2021 Author Share Posted February 2, 2021 The first error I've never had particularly, but I think there's some discussions on the official github support page, you might want to take a look there  The second error and the warnings about format/missing subs are fairly normal, some videos don't have subs (either preloaded or generated) Quote Link to comment
Kagami Hiiragi Posted June 4, 2021 Share Posted June 4, 2021 Hi @timmyx, great guide, managed to configure my own setup following it. On 12/24/2020 at 8:19 PM, timmyx said: Note: I tried to add /settings and /workdir through the docker options (see below) but they wouldn't work, so I forced them in the script That's because unraid will create pre-configured container with a docker create command and you will need to start it with a docker start command to apply settings. That way the run script basically boils down to docker start -a youtube-dl and you can edit everything else with unraid GUI. You can read about those docker commands here: https://docs.docker.com/engine/reference/commandline/create/ https://docs.docker.com/engine/reference/commandline/start/  I also had to modify the container and add aria2 downloader because youtube roughly throttles my downloading speed after ~10 videos. Example settings for this are: --external-downloader 'aria2c' --external-downloader-args '--console-log-level=warn -c -x 16 -s 16 -k 1M' It's not great either, but at least I can get 1MiB/s (without aria it's only about 50KiB/s). 1 2 Quote Link to comment
cheynolds Posted August 20, 2021 Share Posted August 20, 2021 Awesome work on putting this together, both to @timmyx and @mikenye - much appreciated!  I've implemented the combination of the posts in this thread but I get the following error in the syslog every 12-13 minutes.  crond[2845]: failed parsing crontab for user root: 6* * * /usr/local/emhttp/plugins/user.scripts/startCustom.php /boot/config/plugins/user.scripts/scripts/youtube-dl/script > /dev/null 2>&1  If I 'Run Script' it works fine, but cron doesn't seem to be able to initiate on its own. I have set the custom value to 0 6 * * *, with an intended run daily at 6am.  The contents of my script:  #!/bin/bash var=`date +"%FORMAT_STRING"` now=`date +"%m_%d_%Y"` now=`date +"%Y-%m-%d"` docker run \ --rm \ -i \ --name youtube-dl-cron \ -e PGID=100 \ -e PUID=99 \ -v /mnt/user/towerMEDIAvideos/YTDL:/workdir:rw \ -v /mnt/user/towerMEDIAvideos/YTDLsettings:/settings:rw \ -v /mnt/user/towerMEDIAvideos/YTDLsettings/ytdl.conf:/etc/youtube-dl.conf:ro \ mikenye/youtube-dl >> /mnt/user/towerMEDIAvideos/YTDLsettings/logs/youtube-dl_${now}.log  Any idea on what I'm missing? Thanks in advance. Hopefully it helps someone else down the line.  Quote Link to comment
eob157 Posted September 14, 2021 Share Posted September 14, 2021 Howdy @timmyx, When I install the mikenye container I am unable to get it to run in cli. This is the error I see: https://puu.sh/IbaG5/6c41622fca.png  Have you seen this before or am I not to worry about it? The container is running but whenever I open CLI I'm unable to enter commands.  Thanks for the help!  -EOB Quote Link to comment
hczv Posted October 6, 2021 Share Posted October 6, 2021 (edited) @timmyx Thanks for this guide, it really helped me. I tried a few different docker applications to do the same, but it just didnt work as intended. one thing i personally wanted to add to this guide is the creation of nfo files for the downloaded content, this helps emby show correct Title and Description of downloaded videos (dunno if it helps plex or other platforms).   First thing to do is download: YTDL-nfo into the defined settings folder:  also create the script post.sh in the defined settings folder: #!/bin/bash # Define argument as variable file=$1 # Have a lock so that it only installs requirements once lock=/post_install.lock # Remove file name from path, but keep path in variable file="${file%/*}/" # Check if the lock file exists, if it dosent; install ytdl-nfo if [ -e "$lock" ]; then ytdl-nfo "$file" else touch $lock apt update apt install python3-pip -y cd /settings/ytdl-nfo pip install -r requirements.txt pip install . ytdl-nfo "$file" fi This script installs and runs ytdl-nfo from the downloaded source. and since this is docker, it installs the ytdl-nfo once on each restart of the docker container. of course remember to set the script to executeable with "chmod +x post.sh"  now add two lines to the ytdl.conf file --write-info-json --exec 'bash /settings/post.sh {}' This writes an info.json file for each video (required by ytdl-nfo). It also runs post.sh each time a video is downloaded.  NOTE: there is a requirement to change the PUID in the container to 0 (root), this is unsafe but needed to allow the installation of pip and ytdl-nfo.  This is what it will look like in the log:  Future improvement would be to have ytdl-nfo preinstalled in the docker container Edited October 6, 2021 by hczv text correction 1 Quote Link to comment
timmyx Posted November 3, 2021 Author Share Posted November 3, 2021 On 9/13/2021 at 9:53 PM, eob157 said: Howdy @timmyx, When I install the mikenye container I am unable to get it to run in cli. This is the error I see: https://puu.sh/IbaG5/6c41622fca.png  Have you seen this before or am I not to worry about it? The container is running but whenever I open CLI I'm unable to enter commands.  Thanks for the help!  -EOB I never tried to run it through CLI but you can setup a script and run it through Scripts plugin  so intead of having a list of channels in ytdl.conf you paste individual links, I do that sometimes Quote Link to comment
timmyx Posted November 3, 2021 Author Share Posted November 3, 2021 On 10/6/2021 at 3:34 AM, hczv said: @timmyx Thanks for this guide, it really helped me. I tried a few different docker applications to do the same, but it just didnt work as intended. one thing i personally wanted to add to this guide is the creation of nfo files for the downloaded content, this helps emby show correct Title and Description of downloaded videos (dunno if it helps plex or other platforms).   First thing to do is download: YTDL-nfo into the defined settings folder:  also create the script post.sh in the defined settings folder: #!/bin/bash # Define argument as variable file=$1 # Have a lock so that it only installs requirements once lock=/post_install.lock # Remove file name from path, but keep path in variable file="${file%/*}/" # Check if the lock file exists, if it dosent; install ytdl-nfo if [ -e "$lock" ]; then ytdl-nfo "$file" else touch $lock apt update apt install python3-pip -y cd /settings/ytdl-nfo pip install -r requirements.txt pip install . ytdl-nfo "$file" fi This script installs and runs ytdl-nfo from the downloaded source. and since this is docker, it installs the ytdl-nfo once on each restart of the docker container. of course remember to set the script to executeable with "chmod +x post.sh"  now add two lines to the ytdl.conf file --write-info-json --exec 'bash /settings/post.sh {}' This writes an info.json file for each video (required by ytdl-nfo). It also runs post.sh each time a video is downloaded.  NOTE: there is a requirement to change the PUID in the container to 0 (root), this is unsafe but needed to allow the installation of pip and ytdl-nfo.  This is what it will look like in the log:  Future improvement would be to have ytdl-nfo preinstalled in the docker container  Awesome! I added this in my original post 1 Quote Link to comment
allenormond Posted June 10, 2022 Share Posted June 10, 2022 I struggled for awhile last night getting this to work in my system. I thought I'd pass along my notes just in case anybody else is encountering the same issues.  In the script I had to change: -e PUID=99 \ to -e PUID=100 \  I'll admit I have no idea if that's a good/bad idea.  I also had to change: -v /mnt/user/timmyx/Settings/youtube-dl/ytdl.conf:/etc/youtube-dl.conf:ro \ to -v /mnt/user/timmyx/Settings/youtube-dl/ytdl.conf:/etc/yt-dlp.conf:ro \ Which seems to be an issue.  In the end, I believe those were the only two changes that were needed to make it work for me. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.