Jump to content
coppit

FileBot containers

270 posts in this topic Last Reply

Recommended Posts

FileBot is an automated file renamer and metadata fetcher.

 

Note: I used to have separate UI and non-UI containers. They have now been merged into one container that supports both methods of use.

 

THE UI IS DISABLED BY DEFAULT. To enable it, edit your filebot.conf, setting RUN_UI=true. Then restart the container.


You can use this container in automated mode, where you configure FileBot, and then drop files into a "watch dir" that will automatically be converted.

 

Or you can use it in UI mode, where you can either use a remote desktop client or a web browser to use the UI.

 

For more information on usage: https://hub.docker.com/r/coppit/filebot/

For source code: https://github.com/coppit/docker-filebot

This container is part of my template repository: https://github.com/coppit/docker-templates

 

Breaking change on June 26, 2016

 

In order to fix the automated FileBot so that it creates files with nobody:users ownership instead of root:root, I had to change a few things. Unfortunately existing users will need to do a few things to migrate their config files and containers to the new image:

 

  • Click on your container, then the "Advanced" button in the top right, and add new environment variables: "USER_ID=99", "GROUP_ID=100", and "UMASK=0000"
  • Run "chown nobody:users /path/to/config/*"
  • When you first run it, it will exit and complain in the logs about a filebot.sh version change. You'll need to edit your filebot.sh, bumping the version number to 2 and changing /root to /files
Edited by coppit

Share this post


Link to post

Yeah, I guess it should be filebot-cli or something. It's meant to be no-touch. You could compare the input and output directories, I suppose.

 

I've found UIs in docker containers to be really clunky (web UIs being the exception, of course). If I have some time I'll take a look at a UI, but I wouldn't count on it.

Share this post


Link to post

Is this going to check in the background the input directory for new files? What happens if torrent isn't completed in the input directory and filebot will want to copy it to new location?

Share this post


Link to post

I keep my incomplete files in a separate directory.

 

The change detection only triggers for new directories, files that were open for writing and have been closed, and for files moved into the directory. Opening or writing to a file doesn't trigger it, nor does reading a file. However, some other file in the directory could get written, causing FileBot to run on the whole thing. I don't know if filebot is smart enough to detect when files have opened file handles to them.

 

The safest thing would be to go from incomplete -> complete -> sorted, where complete is FileBot's input, and sorted is its output.

Share this post


Link to post

I've created a container with a UI. I'm looking for feedback.

 

I've updated the top post with info. It's also available via the "community applications" plugin UI.

Share this post


Link to post

Has anyone been successful with using more intricate rename lines other than the one originally put into the filebot.sh script here?

The original line for movie renaming is:  n.replaceAll(/[\`\u00b4\u2018\u2019\u02bb]/, "'"'"'").replaceAll(/[\u201c\u201d]/, '"'"'""'"'"')'

I'm having an enormously hard time (which may or may not be due to the fact that I'm not completely well versed on linux cli) trying to do anything other than:

 

{n}.{y}/{n.}.{y}.{vf}... etc etc, standard basic renaming.... however, what I'm looking to do is:

 

{n.replaceAll(/[:|]/, " - ").upperInitial().space('.')}.({y})/{n.replaceAll(/[:|]/, " - ").upperInitial().space('.')}.{y}.{vf}.{vc}.{ac}{'-CD'+pi}"

 

Essentially all the info after the name of the movie is easily done, but the naming of the movie to make sure no odd characters and anything that woudl throw it off or stop the process is removed.  So something like Naked.Gun.-.33.1/3.(Year)/Naked.Gun.-.33.1/3.(Year).720p.BRRip.x264.MP3.mkv  would be created.

 

If anyone has a moment to help me out with something like this, I'd greatly appreciate it. 

 

Share this post


Link to post

You should ask this in the filebot forum. There are also plenty of scripts available on that site.

Share this post


Link to post

You should ask this in the filebot forum. There are also plenty of scripts available on that site.

 

I did, and got my answer.  It was an issue with double quotes.

 

 

On the other hand though, I am having an issue that I imagine most other people using the docker are having too... All drives are spun down, file gets downloaded, then passed to filebot... which then spins up the drive it's writing to, essentially bypassing the cache drive.... why is this occurring?  Or why is it sendig a signal to spin up the drive?  Is there a method or command to throw in the script to disable this or is there something in one of the parameters thats doing this and I don't know it?

 

Here's the log on the filebot side:

 

[Jun 18 10:49:05] Detected new directory: Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek

[Jun 18 10:49:11] Input directory stabilized for 5 seconds. Running FileBot.

Locking /root/amc.log

Run script [fn:amc] at [Thu Jun 18 10:49:16 EDT 2015]

Parameter: ut_dir = /input

Parameter: ut_kind = multi

Parameter: music = n

Parameter: artwork = y

Parameter: deleteAfterExtract = y

Parameter: clean = y

Parameter: xbmc = 192.168.1.25

Parameter: excludeList = /config/amc-exclude-list.txt

Parameter: movieFormat = {n.replaceAll(/[:|?!]/,".-").upperInitial().space(".")}.{y}/\{n.replaceAll(/[:|?!]/,".-").upperInitial().space(".")}.{y}.{vf}.{vc}.{ac}{'-CD'+pi}

Using excludes: /config/amc-exclude-list.txt (14)

Input: /input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek/4cfd5a4f39cce4cef8acbef868856c413770b665.mp4

4cfd5a4f39cce4cef8acbef868856c413770b665.mp4 [series: Chef, movie: Chef (2014)]

Exclude Series: Chef

Group: [tvs:null, mov:chef 2014, anime:null] => [4cfd5a4f39cce4cef8acbef868856c413770b665.mp4]

Rename movies using [TheMovieDB]

Auto-detect movie from context: [/input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek/4cfd5a4f39cce4cef8acbef868856c413770b665.mp4]

Stripping invalid characters from new path: Chef.2014/\Chef.2014.1080p.x264.AAC

[M O V E] Rename [/input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek/4cfd5a4f39cce4cef8acbef868856c413770b665.mp4] to [/output/Chef.2014/Chef.2014.1080p.x264.AAC.mp4]

Processed 1 files

Fetching movie artwork for [Chef (2014)] to [/output/Chef.2014]

Fetching /output/Chef.2014/poster.jpg => {category: posters, width: 1400, height: 2100, language: en, url: http://image.tmdb.org/t/p/original/jc2BCqYtoyUM6rWdtW6KvJjL2Dq.jpg}

Fetching /output/Chef.2014/fanart.jpg => {category: backdrops, width: 1920, height: 1080, language: null, url: http://image.tmdb.org/t/p/original/sbH9aJV7fqhRaazCSWuOwNDIf1m.jpg}

Fanart not found: /output/Chef.2014/clearart.png / movieart

Fanart not found: /output/Chef.2014/logo.png / movielogo

Fetching /output/Chef.2014/disc.png => {type=moviedisc, id=82643, url=http://assets.fanart.tv/fanart/movies/212778/moviedisc/chef-5412e100d0e94.png, lang=en, likes=4, disc_type=bluray}

Notify XBMC: 192.168.1.25

Clean clutter files and empty folders

Delete /input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek

Delete /input

Done ?(?????)?

 

Command shown on unraid log that displays:

Jun 18 10:29:04 MediaNAS kernel: mdcmd (53): spindown 2

Jun 18 10:49:53 MediaNAS emhttp: /usr/bin/docker logs --tail=350 -f FileBot 2>&1

Jun 18 10:50:38 MediaNAS emhttp: /usr/bin/tail -n 42 -f /var/log/syslog 2>&1

Jun 18 10:50:55 MediaNAS emhttp: /usr/bin/docker logs --tail=350 -f FileBot 2>&1

Share this post


Link to post

On the other hand though, I am having an issue that I imagine most other people using the docker are having too... All drives are spun down, file gets downloaded, then passed to filebot... which then spins up the drive it's writing to, essentially bypassing the cache drive.... why is this occurring?  Or why is it sendig a signal to spin up the drive?  Is there a method or command to throw in the script to disable this or is there something in one of the parameters thats doing this and I don't know it?

Whether the cache drive gets used or when a drive is spun up has nothing to do with filebot or any docker for that matter.

 

If you are writing directly to one of the disks, then cache doesn't get used. Only user shares are cached. unRAID spins up drives when they are read or written.

Share this post


Link to post

On the other hand though, I am having an issue that I imagine most other people using the docker are having too... All drives are spun down, file gets downloaded, then passed to filebot... which then spins up the drive it's writing to, essentially bypassing the cache drive.... why is this occurring?  Or why is it sendig a signal to spin up the drive?  Is there a method or command to throw in the script to disable this or is there something in one of the parameters thats doing this and I don't know it?

Whether the cache drive gets used or when a drive is spun up has nothing to do with filebot or any docker for that matter.

 

If you are writing directly to one of the disks, then cache doesn't get used. Only user shares are cached. unRAID spins up drives when they are read or written.

 

That much I understand, however, if for example: I have nzbget download something, process it, then pass it to the array, using cache drive, with settings so all downloads are technically moved later, why does it not send the signal to spin up the drive, where as with filebot, this is happening? Is it because it's download the metadata separately and is causing this?

Share this post


Link to post

On the other hand though, I am having an issue that I imagine most other people using the docker are having too... All drives are spun down, file gets downloaded, then passed to filebot... which then spins up the drive it's writing to, essentially bypassing the cache drive.... why is this occurring?  Or why is it sendig a signal to spin up the drive?  Is there a method or command to throw in the script to disable this or is there something in one of the parameters thats doing this and I don't know it?

Whether the cache drive gets used or when a drive is spun up has nothing to do with filebot or any docker for that matter.

 

If you are writing directly to one of the disks, then cache doesn't get used. Only user shares are cached. unRAID spins up drives when they are read or written.

 

That much I understand, however, if for example: I have nzbget download something, process it, then pass it to the array, using cache drive, with settings so all downloads are technically moved later, why does it not send the signal to spin up the drive, where as with filebot, this is happening? Is it because it's download the metadata separately and is causing this?

What are your volume mappings?

Share this post


Link to post

On the other hand though, I am having an issue that I imagine most other people using the docker are having too... All drives are spun down, file gets downloaded, then passed to filebot... which then spins up the drive it's writing to, essentially bypassing the cache drive.... why is this occurring?  Or why is it sendig a signal to spin up the drive?  Is there a method or command to throw in the script to disable this or is there something in one of the parameters thats doing this and I don't know it?

Whether the cache drive gets used or when a drive is spun up has nothing to do with filebot or any docker for that matter.

 

If you are writing directly to one of the disks, then cache doesn't get used. Only user shares are cached. unRAID spins up drives when they are read or written.

 

That much I understand, however, if for example: I have nzbget download something, process it, then pass it to the array, using cache drive, with settings so all downloads are technically moved later, why does it not send the signal to spin up the drive, where as with filebot, this is happening? Is it because it's download the metadata separately and is causing this?

What are your volume mappings?

 

See attached, or:

 

Filebot:

/input /mnt/cache/appdata/Downloads/dst/Movies/

/output /mnt/user/Movies

/config /mnt/cache/appdata/filebot

 

NzbGet:

/data /mnt/user

/config /mnt/cache/appdata/nzbget/config

/downloads /mnt/cache/appdata/Downloads/

 

So with Nzbget and videosort script - downloads, pases to script, renames, finishes, then sends to /data... drive does not spin.

With Nzbget and filebot - downloads to /downloads, filebot picks up from there in /input, then /outputs with downloading metadata

VolMapping.jpg.bc4dfc339827c43dc101da5ca5200887.jpg

Share this post


Link to post
...

Filebot:

/input /mnt/cache/appdata/Downloads/dst/Movies/

/output /mnt/user/Movies

/config /mnt/cache/appdata/filebot

 

NzbGet:

/data /mnt/user

/config /mnt/cache/appdata/nzbget/config

/downloads /mnt/cache/appdata/Downloads/

 

So with Nzbget and videosort script - downloads, pases to script, renames, finishes, then sends to /data... drive does not spin.

With Nzbget and filebot - downloads to /downloads, filebot picks up from there in /input, then /outputs with downloading metadata

I use NZBGet but not any of these other things so excuse me if I miss this obvious.

 

You say videosort script sends to /data, which is mapped to /mnt/user. I assume you mean it is sending it to a subfolder of /data.

 

Filebot's /output is the Movies user share. Is that share configured to use the cache disk? Maybe something about that share's configuration is making it bypass cache. Post a screenshot of the Movies share page.

Share this post


Link to post

...

Filebot:

/input /mnt/cache/appdata/Downloads/dst/Movies/

/output /mnt/user/Movies

/config /mnt/cache/appdata/filebot

 

NzbGet:

/data /mnt/user

/config /mnt/cache/appdata/nzbget/config

/downloads /mnt/cache/appdata/Downloads/

 

So with Nzbget and videosort script - downloads, pases to script, renames, finishes, then sends to /data... drive does not spin.

With Nzbget and filebot - downloads to /downloads, filebot picks up from there in /input, then /outputs with downloading metadata

I use NZBGet but not any of these other things so excuse me if I miss this obvious.

 

You say videosort script sends to /data, which is mapped to /mnt/user. I assume you mean it is sending it to a subfolder of /data.

 

Filebot's /output is the Movies user share. Is that share configured to use the cache disk? Maybe something about that share's configuration is making it bypass cache. Post a screenshot of the Movies share page.

 

Correct, videosort is mapped in a way that it sends to a subfolder (Movies or Television)... so if I were to ever stop using sonarr, it would rename XXXXX, if it's a movie category or tv category, then sends to the appropriate directory on the array in /data (/mnt/user/________)

 

The movies user share as far as I know, is setting correctly, but you may have a better answer about that then I do.  It was the next place I was going to check.

 

See attached.  I do really appreciate the help!  It's driving me crazy what's causing this.

MoviesShare.jpg.1e17279dfa56d67743903b1d8f90e5a3.jpg

Share this post


Link to post

Share page look OK. How much free space is on the cache drive?

Share this post


Link to post

Share page look OK. How much free space is on the cache drive?

 

120gigs... lol.  It's not an issue of space.  I just don't know what could be causing this, and the only thing I can notice (at least based off the log) that would be causing it is FileBot, which I know, also makes no sense.  But it doesn't occur with just nzbget.

 

 

To summarize in case ayone else is looking at this page and gets confused what this post is about (or has any other input which I'd greatly appreciate any help to resolve):

 

-------------

 

I am having an issue that I imagine most other people using the docker are having too... All drives are spun down, file gets downloaded, then passed to filebot... which then spins up the drive it's writing to, essentially bypassing the cache drive.... why is this occurring?  Or why is it sendig a signal to spin up the drive?  Is there a method or command to throw in the script to disable this or is there something in one of the parameters thats doing this and I don't know it?

 

Here's the log on the filebot side:

 

[Jun 18 10:49:05] Detected new directory: Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek

[Jun 18 10:49:11] Input directory stabilized for 5 seconds. Running FileBot.

Locking /root/amc.log

Run script [fn:amc] at [Thu Jun 18 10:49:16 EDT 2015]

Parameter: ut_dir = /input

Parameter: ut_kind = multi

Parameter: music = n

Parameter: artwork = y

Parameter: deleteAfterExtract = y

Parameter: clean = y

Parameter: xbmc = 192.168.1.25

Parameter: excludeList = /config/amc-exclude-list.txt

Parameter: movieFormat = {n.replaceAll(/[:|?!]/,".-").upperInitial().space(".")}.{y}/\{n.replaceAll(/[:|?!]/,".-").upperInitial().space(".")}.{y}.{vf}.{vc}.{ac}{'-CD'+pi}

Using excludes: /config/amc-exclude-list.txt (14)

Input: /input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek/4cfd5a4f39cce4cef8acbef868856c413770b665.mp4

4cfd5a4f39cce4cef8acbef868856c413770b665.mp4 [series: Chef, movie: Chef (2014)]

Exclude Series: Chef

Group: [tvs:null, mov:chef 2014, anime:null] => [4cfd5a4f39cce4cef8acbef868856c413770b665.mp4]

Rename movies using [TheMovieDB]

Auto-detect movie from context: [/input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek/4cfd5a4f39cce4cef8acbef868856c413770b665.mp4]

Stripping invalid characters from new path: Chef.2014/\Chef.2014.1080p.x264.AAC

[M O V E] Rename [/input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek/4cfd5a4f39cce4cef8acbef868856c413770b665.mp4] to [/output/Chef.2014/Chef.2014.1080p.x264.AAC.mp4]

Processed 1 files

Fetching movie artwork for [Chef (2014)] to [/output/Chef.2014]

Fetching /output/Chef.2014/poster.jpg => {category: posters, width: 1400, height: 2100, language: en, url: http://image.tmdb.org/t/p/original/jc2BCqYtoyUM6rWdtW6KvJjL2Dq.jpg}

Fetching /output/Chef.2014/fanart.jpg => {category: backdrops, width: 1920, height: 1080, language: null, url: http://image.tmdb.org/t/p/original/sbH9aJV7fqhRaazCSWuOwNDIf1m.jpg}

Fanart not found: /output/Chef.2014/clearart.png / movieart

Fanart not found: /output/Chef.2014/logo.png / movielogo

Fetching /output/Chef.2014/disc.png => {type=moviedisc, id=82643, url=http://assets.fanart.tv/fanart/movies/212778/moviedisc/chef-5412e100d0e94.png, lang=en, likes=4, disc_type=bluray}

Notify XBMC: 192.168.1.25

Clean clutter files and empty folders

Delete /input/Chef.2014.1080p.BluRay.x264.YIFY-NZBgeek

Delete /input

Done ?(?????)?

 

Command shown on unraid log that displays:

Jun 18 10:29:04 MediaNAS kernel: mdcmd (53): spindown 2

Jun 18 10:49:53 MediaNAS emhttp: /usr/bin/docker logs --tail=350 -f FileBot 2>&1

Jun 18 10:50:38 MediaNAS emhttp: /usr/bin/tail -n 42 -f /var/log/syslog 2>&1

Jun 18 10:50:55 MediaNAS emhttp: /usr/bin/docker logs --tail=350 -f FileBot 2>&1

Share this post


Link to post

Something funny with your user shares? From console or telnet, what do you get with these?

ls -lah /mnt/user

ls -lah /mnt/user0

ls -lah /mnt/cache

Share this post


Link to post

Something funny with your user shares? From console or telnet, what do you get with these?

ls -lah /mnt/user

 

root@MediaNAS:~# ls -lah /mnt/user

total 21K

drwxrwxrwx 1 nobody users  26 Jun 18 10:49 ./

drwxr-xr-x 7 root  root  140 Jun 17 08:40 ../

drwxrwxrwx 1 nobody users  112 Feb 21 11:39 Backup\ Storage/

drwxrwxrwx 1 nobody users 1.2K May 10 03:40 Exercise/

drwxrwxrwx 1 nobody users  18 Jun 18 11:33 Movies/

drwxrwxrwx 1 nobody users 3.9K Jun 18 10:13 Television/

drwxrwxrwx 1 nobody users  76 Jun 17 14:56 appdata/

 

ls -lah /mnt/user0

 

root@MediaNAS:~# ls -lah /mnt/user0

total 26K

drwxrwxrwx 1 nobody users  192 Jun 18 10:12 ./

drwxr-xr-x 7 root  root  140 Jun 17 08:40 ../

drwxrwxrwx 1 nobody users  112 Feb 21 11:39 Backup\ Storage/

drwxrwxrwx 1 nobody users 1.2K May 10 03:40 Exercise/

drwxrwxrwx 1 nobody users  22K Jun 18 10:12 Movies/

drwxrwxrwx 1 nobody users 3.9K Jun 18 10:13 Television/

 

ls -lah /mnt/cache

 

root@MediaNAS:~# ls -lah /mnt/cache

total 16K

drwxrwxrwx 1 nobody users  26 Jun 18 10:49 ./

drwxr-xr-x 7 root  root  140 Jun 17 08:40 ../

drwxrwxrwx 1 nobody users  18 Jun 18 11:33 Movies/

drwxrwxrwx 1 nobody users  76 Jun 17 14:56 appdata/

 

Share this post


Link to post

Ok, so after some extensive testing... there is something within filebot absolutely sending a signal to spin up the drive it's about to write the files downloaded and processed by filebot to.

It's not xbmc notifications, it's not the moving of the files to the array.... it's the metadata being written when grabbing artwork and so on.

 

As you can see in this image take from command line, permissions are a problem, since the download is done by nzbget (nobody) and filebot writes as root, and also seems to be writing DIRECTLY to the array, rather than using the cache drive.

 

Untitled-1.jpg.0841871e6bd85065aef7d07c73157d79.jpg

Share this post


Link to post

See above for recent issues with this.

Coppit, or anyone else, do you experience the same issues?  Drive spins exactly as metadata starts to grab, unlike how it should be doing like with the movie, creating a link on the cache drive

Share this post


Link to post

It seems like the workaround is simple: Specify the output dir within your cache drive, and let mover handle moving it later.

 

Of course FileBot is going to write to the output dir, but from the docker container's perspective it can't control what unraid does with that write, or even that unraid exists. This seems like a general unraid<->docker issue that's not related to filebot. To confirm, you could modify your other containers to also use /mnt/user/... and I bet you'll see the same spin-up behavior.

 

I wonder for example if Docker's volume mapping somehow bypasses or interferes with unraid's cache drive implementation?

Share this post


Link to post

It seems like the workaround is simple: Specify the output dir within your cache drive, and let mover handle moving it later.

 

Of course FileBot is going to write to the output dir, but from the docker container's perspective it can't control what unraid does with that write, or even that unraid exists. This seems like a general unraid<->docker issue that's not related to filebot. To confirm, you could modify your other containers to also use /mnt/user/... and I bet you'll see the same spin-up behavior.

 

I wonder for example if Docker's volume mapping somehow bypasses or interferes with unraid's cache drive implementation?

 

Every other docker I have that points to the array (ie Nzbget or couchpotato or sonarr) all don't have the issue of spinning up the drive.

I did also try pointing filebot to /mnt/cache/Movies rather than to the array of the same folder, but the same thing actually occurred, drive spun up. 

I'm not blaming filebot or even this docker for that, a random spin up could even mean that the drive itself on the spindown delay needs to be reinforced again.... just saying, there's something odd about it when I don't have the issue with any other docker handling files that go to the array.

Share this post


Link to post

Does NzbGet write to its /data directory (which you mapped to /mnt/user)? And if so, are the files/directories that it writes to on some non-cache drive? What happens if you configure FileBot's output dir to be /mnt/user?

 

When I get a chance I'll do some experiments, but I don't know how filebot could be somehow telling unraid to spin up the drives, or what I could do to prevent that. The only thing I could imagine doing is filing a bug against unraid.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now