youtube-dl strategy

[timmyx]

Hello,

 

I'm fairly new to unraid and definitely new to docker containers and linux scripts

 

But I use youtube-dl on my pc to regularly download new videos from channels I follow (just as safety measure that my inner /r/datahoarder tells me to do in case these channels ever go byebye 😆)

 

What I do now is call a .bat file on a schedule basis with all my arguments, naming structures, archives, etc. So it's fairly automated except that I would have to download a newer youtube-dl version manually

 

I've been setting up my first unraid server for the past couple of weeks now and this is something I'd like to move over to an unraid process. But I've tried a few different docker containers that I didn't have luck with:

 

• youtube-dl-server: runs ok for but for single downloads, it can't handle customized output parameters nor can it listen to a playlist/channel for updates and automatically downloads
• youtube-dl material: promising, had an option to write archives automatically but I couldn't get the output parameters to work
• go auto yt: says from info that it can handle output options and archive writing, but I kept getting a 'There was an error getting channel metadata: From c.GetMetadata():' so I just ditched it

 

Alternatively, I've come around this and this but I wouldn't know how to setup or if it could manage to handle my script. So I was thinking... is there a way that I could have unraid read my batch files on a regular basis and run youtube-dl? Much like I do right now on windows

 

My batch file contains several commands like this:

 

youtube-dl --ignore-errors --no-continue --write-description --write-sub --download-archive "downloadsfolder/archive.txt" -f "(mp4)[height <=? 1080]" -o "downloadsfolder/%%(uploader)s - %%(upload_date)s - %%(title)s.%%(ext)s" https://www.youtube.com/channel/UCVI66VMGYg0dLPEkWPrCkBw
  
for /R "downloadsfolder" %%f in (*.description) do ren "%%f" *.txt

 

tl;dr youtube-dl docker containers I tested can't handle some or all my output arguments for channels I regularly download to archive. is there a way to make unraid call a batch file (or similar) on a schedule basis and run youtube-dl to download newly added videos that are not listed in its output archive?

 

any help is really appreciated! 😊

[JonathanM]

Search Youtube dl material in the apps tab.

https://forums.unraid.net/topic/87798-support-selfhostersnets-template-repository/

https://github.com/Tzahi12345/YoutubeDL-Material

[primeval_god]

On 12/18/2020 at 9:27 AM, timmyx said:

tl;dr youtube-dl docker containers I tested can't handle some or all my output arguments for channels I regularly download to archive. is there a way to make unraid call a batch file (or similar) on a schedule basis and run youtube-dl to download newly added videos that are not listed in its output archive?

You can use the User.Scripts plugin to run a bash script on a schedule. So far as running youtube-dl specifically the best way is within a docker container. You would use a docker run command in your script (probably with the --rm option to delete the container when done) to launch a youtubedl container from your script. Depending on the image you choose you may need to override the container entrypoint to directly launch youtubedl and then your would pass your arguments in the docker cmd.

[timmyx]

Thanks for the replies. I found a docker that's very promising to my use case

 

mikenye/youtube-dl - Docker Hub

 

All I need now is to figure out how to get it running

[CS01-HS]

I may be going about this the wrong way but I have a Debian VM in unraid always running that even on my underpowered board uses barely any CPU (< 1% at idle.) With /mnt/user mapped in the VM it gives full access to the unraid shares and makes tasks like installing youtube-dl and other miscellaneous CLI apps much easier.

[mikenye]

Hi Timmy,

 

It’s mikenye here, author of the YouTube-DL docker container you want to try. 
 

You DM’d me on discord but I can’t reply to you as we don’t share a server (I think you came and left?). 
 

If you re-join the server I would be happy to help you. 
 

-Mike

Edited December 21, 2020 by mikenye

[CS01-HS]

Here for reference is my full setup within Debian:

[debian@debian]:~ $ sudo crontab -l
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

# update youtube-dl once a week
05  00  *  *  0  /usr/bin/wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl
[debian@debian]:~ $

[debian@debian]:~ $ crontab -l
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m  h  dom mon dow   command

# download nightly
00  01  *   *   *     /home/debian/scripts/youtube-playlist-download.sh > /var/log/youtube-playlist-download.log 2>&1


[debian@debian]:~ $ more /home/debian/scripts/youtube-playlist-download.sh
#!/bin/bash

# Download my youtube playlists

# OPTIONS
#
YOUTUBE_CMD="/usr/local/bin/youtube-dl"
SAVE_DIRECTORY="/mnt/unraid/Private/Youtube/"
ARCHIVE_FILE="/home/debian/scripts/youtube-playlist-download.downloaded"
YOUTUBE_USERNAME="PUT_YOUR_YOUTUBE_USERNAME_HERE"

CMD_OPTIONS=(-i
    --yes-playlist
    --embed-thumbnail
    --add-metadata
    --embed-subs
    --all-subs
    -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4
    --download-archive ${ARCHIVE_FILE}
    )

${YOUTUBE_CMD} ${CMD_OPTIONS[@]} -o "${SAVE_DIRECTORY}%(playlist)s/%(playlist_index)03d - %(title)s.%(ext)s" https://www.youtube.com/user/${YOUTUBE_USERNAME}/playlists

exit 0
[debian@debian]:~ $

 

Edited December 21, 2020 by CS01-HS

[timmyx]

I'd like to thank everyone for the replies, I finally have it running (almost) seamlessly.

 

Here are the steps for anyone who wants to setup youtube-dl in unraid using mikenye's docker – shout out to mikenye for his support!

 

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

How to add youtube-dl and run it your way

Background

• None of the docker containers available in unRaid's repository from Community Applications worked for me the way I wanted (see first post)
  • Repurposed first porst tl;dr - I wanted to be able to set a 'template' of commands that I already use with youtube-dl on Windows to check, download and write a list of downloaded videos from a list of channels, on a schedule and in unRaid instead
• I found plenty of other youtube-dl dockers in dockerHub but mikenye's was the one with the most information available and it's maintained quite regularly, so it was my choice
• While this how-to may be applicable to other youtube-dl docker containers, it is focused on mikenye's, so some steps and some arguments in the code may vary

Resources

• mikenye/youtube-dl - Docker Hub
• youtube-dl - GitHub (reference)
• custom icon

First things first

• This how-to assumes that you already have unRaid up and running with users and shares set
• You will need to install the official apps repository called Community Applications
  • Check this video from Spaceinvader One until the 3 min mark and this post for the installation url
  • Note that you will need unRaid 6.8.3 or newer to be able to use CA
• Once you have CA installed, you will need to also enable results from dockerHub:

• Then install mikenye/youtube-dl (dockerHub) and User Scripts (CA):

• Alternatively, you can install mikenye's docker from the Add Container option:

• Once you've completed the steps above, you will have most that is needed to get youtube-dl running

Adding the script to run in unRaid

• Now we will set up the script to start youtube-dl automatically i.e. run the docker from unraid 
• Follow these steps to add a new script:
  

• Now choose to edit the script:
  

• This is my script:

Quote

#!/bin/bash
var=`date +"%FORMAT_STRING"`
now=`date +"%m_%d_%Y"`
now=`date +"%Y-%m-%d"`
docker run \
--rm \
-i \
--name youtube-dl-cron \
-e PGID=100 \
-e PUID=99 \
-v /mnt/user/Archive:/workdir:rw \
-v /mnt/user/timmyx/Settings/youtube-dl:/settings:rw \
-v /mnt/user/timmyx/Settings/youtube-dl/ytdl.conf:/etc/youtube-dl.conf:ro \
mikenye/youtube-dl >> /mnt/user/timmyx/Settings/youtube-dl/logs/youtube-dl_${now}.log

Arguments in details

• Red: used to add dates to the log files e.g. youtube-dl_2020-12-23.log
• Blue: are covered and explained by mikenye's documentation
• Orange: I added as reference to be used by ytdl.conf
• Note: change /mnt/user/timmyx/Settings/youtube-dl/ and /mnt/user/Archive to your desired folders
• Note: I tried to add /settings and /workdir through the docker options (see below) but they wouldn't work, so I forced them in the script
  

• If you want unRaid to run the script every so often, choose one of the preset schedules or use cron

Configuring ytdl.conf

• The config file is used as your standard settings using youtube-dl's own arguments
• Note: all videos you download through the docker and through the user script from previous section will always use the options in this config file
• This is my config file:

Quote

--ignore-errors
--geo-bypass
--no-continue
--no-progress 
--write-description
--write-sub
--sub-lang en
--sub-format 'srt'
--write-auto-sub
--convert-subs srt
--download-archive '/settings/downloaded.txt'
--format 'bestvideo[ext=mp4,height<=?1080]+bestaudio[ext=m4a]/best[ext=mp4]/best'
--output '/workdir/Youtube/%(uploader)s/%(uploader)s - %(upload_date)s - %(title)s [%(id)s].%(ext)s'
--batch-file '/settings/channels.txt'

• Note: I will not go through details of each option but they are thoroughly explained in youtube-dl's documentation
• Note: For some reason, using --format "best[height<=?1080]" did not always download videos available in 1080p, so I had to force additional options

Configuring channels.txt

• Used to easily add/remove channels to be downloaded from by the script/config file.
• This is what it should look like:

Quote

# 10 hour
https://www.youtube.com/channel/UCVI66VMGYg0dLPEkWPrCkBw/videos
#
# channel name
url
# channel name
url

...

• Or simply one channel per line

Post-download folder permissions

• Note: before you do anything here know what this does here and here
• If you use Windows like I do, chances are you won't be able to modify any of the files downloaded by the docker due to folder permission issues (provided you are not logged in with root credentials in Windows of course)
• As a workaround, there is an option in unRaid that tweaks the settings and grants you write access again:

• You might need to go through this workaround again when new folders/channels are downloaded

Wrapping things up

• This should be it. You now have mikenye's docker installed and running through custom settings and on a schedule (at least mine is :))
• I used this image as icon to the docker container just so it looks pretty (I hope you won't mind mikenye!)

 

Additional options

• How to create nfo files for the downloaded content (this helps emby show correct Title and Description of downloaded videos)
  • Follow this link to the walkthrough (tks hczv)

Edited November 3, 2021 by timmyx

[tamito]

@Timmex Thank you for writing it up. I am able to follow you step and make it work. Do you have solution if some channel you want to download only couple of video instead of entire channels?

[timmyx]

On 1/30/2021 at 2:18 PM, tamito said:

@Timmex Thank you for writing it up. I am able to follow you step and make it work. Do you have solution if some channel you want to download only couple of video instead of entire channels?

Hi, I'm sorry for the late response, you tagged someone else entirely 😆

 

To your question: youtube-dl has some native options but depending on your case I'd either do:

1. a simulated run with archive options and remove all entries you actually want downloaded from the archive (this will make yt-dl write all strings from the channel but download none), or
2. add all videos you want to a public playlist and have yt-dl fetch that list

#1 is more work but will run great if you don't want to download specific videos in future releases (which would have you redo the procedure all the time)

 

#2 is the simplest, fastest and future-proof option so I'd stick with that

 

Hope this helps, feel free to reply if you need more assistance though

[tamito]

@timmyx I tagged the wrong person. 

 

Thank you for the solution. Do you encounter similar issue on downloading subtitle and cause too many connection error?

 

ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by ); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: video doesn't have subtitles
WARNING: No subtitle format found matching "srt" for language en, using vtt
ERROR: unable to download video data:
WARNING: video doesn't have subtitles

 

[timmyx]

The first error I've never had particularly, but I think there's some discussions on the official github support page, you might want to take a look there

 

The second error and the warnings about format/missing subs are fairly normal, some videos don't have subs (either preloaded or generated)

[Kagami Hiiragi]

Hi @timmyx, great guide, managed to configure my own setup following it.

On 12/24/2020 at 8:19 PM, timmyx said:

Note: I tried to add /settings and /workdir through the docker options (see below) but they wouldn't work, so I forced them in the script

That's because unraid will create pre-configured container with a docker create command and you will need to start it with a docker start command to apply settings.

That way the run script basically boils down to docker start -a youtube-dl and you can edit everything else with unraid GUI.

You can read about those docker commands here:

https://docs.docker.com/engine/reference/commandline/create/

https://docs.docker.com/engine/reference/commandline/start/

 

I also had to modify the container and add aria2 downloader because youtube roughly throttles my downloading speed after ~10 videos. Example settings for this are:

--external-downloader 'aria2c'
--external-downloader-args '--console-log-level=warn -c -x 16 -s 16 -k 1M'

It's not great either, but at least I can get 1MiB/s (without aria it's only about 50KiB/s).

[cheynolds]

Awesome work on putting this together, both to @timmyx and @mikenye - much appreciated!

 

I've implemented the combination of the posts in this thread but I get the following error in the syslog every 12-13 minutes.

 

crond[2845]: failed parsing crontab for user root: 6* * * /usr/local/emhttp/plugins/user.scripts/startCustom.php /boot/config/plugins/user.scripts/scripts/youtube-dl/script > /dev/null 2>&1

 

If I 'Run Script' it works fine, but cron doesn't seem to be able to initiate on its own. I have set the custom value to 0 6 * * *, with an intended run daily at 6am.

 

The contents of my script:

 

#!/bin/bash
var=`date +"%FORMAT_STRING"`
now=`date +"%m_%d_%Y"`
now=`date +"%Y-%m-%d"`
docker run \
--rm \
-i \
--name youtube-dl-cron \
-e PGID=100 \
-e PUID=99 \
-v /mnt/user/towerMEDIAvideos/YTDL:/workdir:rw \
-v /mnt/user/towerMEDIAvideos/YTDLsettings:/settings:rw \
-v /mnt/user/towerMEDIAvideos/YTDLsettings/ytdl.conf:/etc/youtube-dl.conf:ro \
mikenye/youtube-dl >> /mnt/user/towerMEDIAvideos/YTDLsettings/logs/youtube-dl_${now}.log

 Any idea on what I'm missing? Thanks in advance. Hopefully it helps someone else down the line.

 

[eob157]

Howdy @timmyx,

When I install the mikenye container I am unable to get it to run in cli. This is the error I see: https://puu.sh/IbaG5/6c41622fca.png

 

Have you seen this before or am I not to worry about it? The container is running but whenever I open CLI I'm unable to enter commands.

 

Thanks for the help!

 

-EOB

[hczv]

@timmyx

Thanks for this guide, it really helped me.

I tried a few different docker applications to do the same, but it just didnt work as intended.

one thing i personally wanted to add to this guide is the creation of nfo files for the downloaded content, this helps emby show correct Title and Description of downloaded videos (dunno if it helps plex or other platforms).

 

 

First thing to do is download: YTDL-nfo into the defined settings folder:

 

also create the script post.sh in the defined settings folder:

#!/bin/bash
# Define argument as variable
file=$1

# Have a lock so that it only installs requirements once
lock=/post_install.lock

# Remove file name from path, but keep path in variable
file="${file%/*}/"

# Check if the lock file exists, if it dosent; install ytdl-nfo
if [ -e "$lock" ]; then
    ytdl-nfo "$file"
else
    touch $lock
    apt update
    apt install python3-pip -y
    cd /settings/ytdl-nfo
    pip install -r requirements.txt
    pip install .
    ytdl-nfo "$file"
fi

This script installs and runs ytdl-nfo from the downloaded source. and since this is docker, it installs the ytdl-nfo once on each restart of the docker container.

of course remember to set the script to executeable with "chmod +x post.sh"

 

now add two lines to the ytdl.conf file

--write-info-json
--exec 'bash /settings/post.sh {}'

This writes an info.json file for each video (required by ytdl-nfo).

It also runs post.sh each time a video is downloaded.

 

NOTE: there is a requirement to change the PUID in the container to 0 (root), this is unsafe but needed to allow the installation of pip and ytdl-nfo.

 

This is what it will look like in the log:

 

Future improvement would be to have ytdl-nfo preinstalled in the docker container

Edited October 6, 2021 by hczv
text correction

[timmyx]

On 9/13/2021 at 9:53 PM, eob157 said:

Howdy @timmyx,

When I install the mikenye container I am unable to get it to run in cli. This is the error I see: https://puu.sh/IbaG5/6c41622fca.png

 

Have you seen this before or am I not to worry about it? The container is running but whenever I open CLI I'm unable to enter commands.

 

Thanks for the help!

 

-EOB

I never tried to run it through CLI but you can setup a script and run it through Scripts plugin

 

so intead of having a list of channels in ytdl.conf you paste individual links, I do that sometimes

[timmyx]

On 10/6/2021 at 3:34 AM, hczv said:

@timmyx

Thanks for this guide, it really helped me.

I tried a few different docker applications to do the same, but it just didnt work as intended.

one thing i personally wanted to add to this guide is the creation of nfo files for the downloaded content, this helps emby show correct Title and Description of downloaded videos (dunno if it helps plex or other platforms).

 

 

First thing to do is download: YTDL-nfo into the defined settings folder:

 

also create the script post.sh in the defined settings folder:

#!/bin/bash
# Define argument as variable
file=$1

# Have a lock so that it only installs requirements once
lock=/post_install.lock

# Remove file name from path, but keep path in variable
file="${file%/*}/"

# Check if the lock file exists, if it dosent; install ytdl-nfo
if [ -e "$lock" ]; then
    ytdl-nfo "$file"
else
    touch $lock
    apt update
    apt install python3-pip -y
    cd /settings/ytdl-nfo
    pip install -r requirements.txt
    pip install .
    ytdl-nfo "$file"
fi

This script installs and runs ytdl-nfo from the downloaded source. and since this is docker, it installs the ytdl-nfo once on each restart of the docker container.

of course remember to set the script to executeable with "chmod +x post.sh"

 

now add two lines to the ytdl.conf file

--write-info-json
--exec 'bash /settings/post.sh {}'

This writes an info.json file for each video (required by ytdl-nfo).

It also runs post.sh each time a video is downloaded.

 

NOTE: there is a requirement to change the PUID in the container to 0 (root), this is unsafe but needed to allow the installation of pip and ytdl-nfo.

 

This is what it will look like in the log:

 

Future improvement would be to have ytdl-nfo preinstalled in the docker container

 

Awesome! I added this in my original post

[allenormond]

I struggled for awhile last night getting this to work in my system. I thought I'd pass along my notes just in case anybody else is encountering the same issues.

 

In the script I had to change:

-e PUID=99 \

to 

-e PUID=100 \

 

I'll admit I have no idea if that's a good/bad idea.

 

I also had to change:

-v /mnt/user/timmyx/Settings/youtube-dl/ytdl.conf:/etc/youtube-dl.conf:ro \

to 

-v /mnt/user/timmyx/Settings/youtube-dl/ytdl.conf:/etc/yt-dlp.conf:ro \

Which seems to be an issue.

 

In the end, I believe those were the only two changes that were needed to make it work for me.