rwickra Posted March 26, 2014 Share Posted March 26, 2014 I'm wondering how the unRaiders out there are integrating their Seedboxes with unRaid home server. My unRaid server is my workhorse and it does a fine job automating downloads with Sickbeard, Sabnzbd and Couchpotato. But I am a member of a few private trackers and I use a seedbox for downloads. Unfortunately, getting this integrated with unraid hasn't been easy for me. Right now, here is my workflow: 1. Manually add .torrent files from the tracker to my seedbox via its web gui. 2. Wait for files to be downloaded to the seedbox. 3. Use an FTP client (CuteFTP, Filezilla) to transfer downloads to my home PC. 4. Rename files in my home PC and copy it to the correct network share on my unraid home server. Does anyone have a more streamlined way to accomplish this. I've thought about a few mechanisms but don't lack the know how to impement it seamlessly. OPTION 1: USE COUCHPOTATO/SICKBEARD TO PROCESS TORRENTS This works well in theory, but I don't think it has been implemented. My issue is that this process will make it difficult to seed downloaded torrents since the post-process system will make the files no longer available to the torrent client. This also seems a bit cumbersome although in theory, this is perhaps the most elegant way to make it work. I'm curious to find out if anyone has this mechanism working. OPTION 2: USE A SERVER-SIDE FTP CLIENT I'm not aware of an FTP client that can be used to FTP files directly from the seedbox onto an unraid share (like the cache drive), but if such an implementation existed, it might be amazing. I've thought about two ways to get around this but haven't had success with either. #1 -- Using RSYNC to sync remote folder (on seedbox) with temporary share (on cache drive). I don't know enough linux to get this to run on a regular basis. If anyone can suggest, I'd be very grateful. #2 -- Using BitTorrent Sync (BTSYNC) -- unraid has a BTSYNC plugin and I thought another option might be to use unraid to sync the seedbox to a temporary share on the cache drive, which I can then use to process files and move out of the cache drive. The problem with this is that there is a small bug with BTSYNC that keeps from completing the sync process. My synchronization stops halfway. The folks at BTSYNC think that they maybe due to a permissions issue. If anyone else has this setup working well (i.e. Torrenting via Seedbox to unraid) I would love to hear how they implemented their workflow. Quote Link to comment
drnackers Posted April 26, 2014 Share Posted April 26, 2014 Hey rwickra, I too had problems with the way to go about this - to add torrents I use 'Remote Torrent Adder' chrome extension and set it up for my Deluge Seedbox. I have my seedbox move once completed into the same directory structure as I have on my Unraid box. To get the files i ended up writing 2 scripts to using lftp and the mirror option which runs every 15 minutes from my Unraid box. I also had a problem with permissions once downloaded so in the script I have it change permissions of the Seedbox before downloading. Basically it runs, locks a file on the disk so it won't run again while it's still downloading and downloads 5 parts of each file simultaneously (which thoroughly maxes out my connection) I have attached the 2 files - both go into /boot/custom or flash/custom I have this line in my go file to add the cronjob at start up: # FTP Downloader cat /boot/custom/lftpcron >> /var/spool/cron/crontabs/root I hope they make sense, you will need to adjust the synctorrents.sh script to match your username, password, seedbox ftp address and the directory structures you use (have added comments to help you in the file) You will also need the lftp package in the extra folder ( on Unraid 5+ i used lftp-4.3.8-i486-1.txz and now on Unraid 6+ I am using lftp-4.4.9-x86_64-1.txz) just google those and you will find them. If I was smarter it could be made into a plugin - but sadly i haven't got those smarts! Attachments need to be renamed to lftpcron. and synctorrents.sh - remove the .txt's Good luck lftpcron.txt synctorrents.sh.txt Quote Link to comment
rwickra Posted April 26, 2014 Author Share Posted April 26, 2014 hey drnackers, thank you so much!! this is exactly what i was looking for. yes, a plugin maybe awesome -- and who knows someone might develop it, but it seems like most unraid-ers are focused on optimizing usenet access since torrents are so unreliable (trackers getting shut down left and right). i will give this a quick spin once my server is up and running again.. i'm having some major issues getting my system running, but once it is up to speed, i'll be checking your script out. thank you for sharing it. Quote Link to comment
pyrater Posted April 29, 2014 Share Posted April 29, 2014 Heres how i did mine and automated it. On your unraid box add the following scripts "putfileswatched.sh" (add this to a cron 1 per hr) This uploads all of the .torrent files on your unraid machine to the seedbox. #!/bin/bash login=pyrater pass=****** host=*********** remote_dir=/torrents/watch local_dir=/mnt/user/Data/Misc/Downloading/Blackhole trap "rm -f /tmp/syncwatched.lock" SIGINT SIGTERM if [ -e /tmp/syncwatched.lock ] then echo "syncwatched is running already." exit 1 else touch /tmp/syncwatched.lock lftp << EOF set ftp:ssl-protect-data yes set ftp:ssl-protect-list yes set mirror:use-pget-n 2 lftp -u $login,$pass $host mirror -c -P5 --Remove-source-files $remote_dir $local_dir quit EOF rm -f /tmp/syncwatched.lock rm -f /mnt/user/Data/Misc/Downloading/Blackhole/*.torrent exit 0 fi synctorrents.sh - this downloads all of the completed torrents from your seedbox. (using Deluge hardlinked for completed downloads) #!/bin/bash login=pyrater pass=******* host=******* remote_dir=/torrents/hardlinked local_dir=/mnt/user/Data/Misc/Downloading/Downloaded trap "rm -f /tmp/synctorrent.lock" SIGINT SIGTERM if [ -e /tmp/synctorrent.lock ] then echo "Synctorrent is running already." exit 1 else touch /tmp/synctorrent.lock lftp << EOF set ftp:ssl-protect-data yes set ftp:ssl-protect-list yes set mirror:use-pget-n 2 lftp -u $login,$pass $host mirror -c -P5 --Remove-source-files $remote_dir $local_dir quit EOF rm -f /tmp/synctorrent.lock chown -hR nobody /mnt/user/Data/Misc/Downloading/Downloaded/* chmod -R 777 /mnt/user/Data/Misc/Downloading/Downloaded/* /boot/extra/Scripts/unrarall -s --clean=all /mnt/user/Data/Misc/Downloading/Downloaded/ exit 0 fi End result = A copy of the media at both locations, one for you on your unraid machine for viewing, one on your seedbox for seeding. unrarall.zip Quote Link to comment
pyrater Posted April 29, 2014 Share Posted April 29, 2014 haha I see drnackers and i used a very similar setup, though his only needs one script so i would use his to keep it simple. Quote Link to comment
denzo Posted September 18, 2014 Share Posted September 18, 2014 I am trying to implement this but I'm having a hard time of it.....mostly because this is my first experience using the CLI. I have followed (drnackers) instructions and to "test it" I'm trying to run it using this command: root@UNRAID:/boot/custom# ./synctorrents.sh but I am getting this error: ./synctorrents.sh: line 21: lftp: command not found any ideas? Quote Link to comment
pyrater Posted September 18, 2014 Share Posted September 18, 2014 That error means you don't have lftp installed. Quote Link to comment
denzo Posted September 18, 2014 Share Posted September 18, 2014 I found the Slackware installer for lftp and installed it but now I am getting this message: lftp: error while loading shared libraries: libssl.so.1: cannot open shared object file: No such file or directory root@UNRAID:/boot/custom# any advise? Quote Link to comment
archedraft Posted September 18, 2014 Share Posted September 18, 2014 A google search indicates that error means you are missing a package called "openssl-solibs". Find that and install and then see if lftp will start. You may get another message similar to the one above which means lftp needs more packages to run. Quote Link to comment
denzo Posted September 18, 2014 Share Posted September 18, 2014 I looked long and hard but couldn't find it. When I did find it the instructions included the apt-get command to install it or it had to be compiled. I did find an old, old UnMenu page that mentions it (link) which looks to be about 5 years old....maybe I should pm Joe L. see if he has any advise. Quote Link to comment
archedraft Posted September 18, 2014 Share Posted September 18, 2014 Yeah it's hard to find the old packages for Slackware these days. Could build an Ubuntu VM though... Quote Link to comment
drnackers Posted September 18, 2014 Share Posted September 18, 2014 Hi Guys, Sorry I didn't post a link to the lftp package so here it is: http://mirrors.slackware.com/slackware/slackware64-current/slackware64/n/lftp-4.4.9-x86_64-1.txz.meta4 Hope that helps. Quote Link to comment
drnackers Posted September 18, 2014 Share Posted September 18, 2014 ... Probably should add that I saved the lftp package to /boot/custom or flash/custom I install the pkg from a line in the go file: installpkg /boot/custom/lftp-4.4.9-x86_64-1.txz Have fun! Quote Link to comment
denzo Posted September 20, 2014 Share Posted September 20, 2014 Try as I might I still can't get this to work . There has been progress though, I no longer get the lftp or the libssl.so.1 errors I was getting, but now I am getting this: mirror: Access failed: No such file (/mnt/user/Other/test_02) The odd thing is if I use this simple command it works fine: rsync -a -v -e ssh [email protected]:/home/denzoid/test_02/ /mnt/user/Other/test_02 But if I have the same directories in the script I get the "Access failed" error. Here's the script: #!/bin/sh login="denzoid" pass="1234567" host="sftp://ceedbox.com" trap "rm -f /tmp/synctorrent.lock" SIGINT SIGTERM if [ -e /tmp/synctorrent.lock ] then echo "Synctorrent is running already." exit 1 else touch /tmp/synctorrent.lock lftp <<EOF set ftp:ssl-auth TLS set ftp:ssl-force true set ftp:ssl-protect-list yes set ftp:ssl-protect-data yes set ssl:verify-certificate off lftp -u $login,$pass $host mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_media.log /home/denzoid/test_02/ mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_downloads.log /mnt/user/Other/test_02 quit 0 EOF rm -f /tmp/synctorrent.lock trap - SIGINT SIGTERM exit 0 fi I did strip out the permission lines from the original script because they caused (chmod) errors themselves. I also ran the "New Permissions" utility at some point. I am making the edits to the script using Notepad++. I've once again tried everything I could think of,but to no avail. What else should I try? TIA, denzoid Quote Link to comment
drnackers Posted September 23, 2014 Share Posted September 23, 2014 Try as I might I still can't get this to work . There has been progress though, I no longer get the lftp or the libssl.so.1 errors I was getting, but now I am getting this: mirror: Access failed: No such file (/mnt/user/Other/test_02) The odd thing is if I use this simple command it works fine: rsync -a -v -e ssh [email protected]:/home/denzoid/test_02/ /mnt/user/Other/test_02 But if I have the same directories in the script I get the "Access failed" error. Here's the script: #!/bin/sh login="denzoid" pass="1234567" host="sftp://ceedbox.com" trap "rm -f /tmp/synctorrent.lock" SIGINT SIGTERM if [ -e /tmp/synctorrent.lock ] then echo "Synctorrent is running already." exit 1 else touch /tmp/synctorrent.lock lftp <<EOF set ftp:ssl-auth TLS set ftp:ssl-force true set ftp:ssl-protect-list yes set ftp:ssl-protect-data yes set ssl:verify-certificate off lftp -u $login,$pass $host mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_media.log /home/denzoid/test_02/ mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_downloads.log /mnt/user/Other/test_02 quit 0 EOF rm -f /tmp/synctorrent.lock trap - SIGINT SIGTERM exit 0 fi I did strip out the permission lines from the original script because they caused (chmod) errors themselves. I also ran the "New Permissions" utility at some point. I am making the edits to the script using Notepad++. I've once again tried everything I could think of,but to no avail. What else should I try? TIA, denzoid hey denzoid, looks like your script only has the host folder not the target folder in these lines: mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_media.log /home/denzoid/test_02/ mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_downloads.log /mnt/user/Other/test_02 My one has both host and target folders: mirror -c --use-pget-n=5 --only-missing --log=/var/log/synctorrents_downloads.log /Downloads/ /mnt/user/Torrents Syntax: mirror (COMMANDS) --log=(YOUR LOG DESTINATION) /(SOURCE) /(Destination) Hope that helps Quote Link to comment
denzo Posted October 1, 2014 Share Posted October 1, 2014 drnackers, That was it! Thanks for all your help, it's much appreciated! Quote Link to comment
kenzo86 Posted October 1, 2014 Share Posted October 1, 2014 Legends! I have been searching for WEEKS for something like this! Great work @drnackers @pyrater! I am currently pre-clearing my drives, and once it is done I will give this script a shot...Just need to read up a little on this stuff as I am fairly new to linux. I know that I have to use notepad++ to compile/edit the script, any other tips? Few questions @drnackers @pyrater + anyone else who will kindly help 1) I would like to install the lftp to my ssd cache drive and not the flash (boot), Do I just need to create the "custom" folder on the ssd and put the files in there? 2) With Rar'd downloads - If I use the "unpack" feature on rutorrent (remotely), and set the unpack folder to "the torrents current dir" .....won't i get a hardlink containing BOTH the rar;d and unrar;d content in the sync folder which will be sent over to my NAS (double the data)? Is there a way around this? 3) Because the hardlink is still present on the seedbox, wont it re-download it on its next cycle as the movie on local nas would have moved from its processing folder to my Movies share? Is there a way of removing source files? tl;dr - is there a way of deleting source files after downloading using lftp to prevent it from redownloading? Many thanks! Quote Link to comment
kenzo86 Posted October 3, 2014 Share Posted October 3, 2014 EDIT anyone mind helping me get this cronjob running? Quote Link to comment
bender1 Posted May 29, 2015 Share Posted May 29, 2015 Just in case anyone else runs across the issue I had.... I used the lftp scripts, but they didn't work for me. chmod wasn't supported by my provider and every torrent that was uploaded didn't have the right permissions. Rtorrent would try to load it, but wasn't able to hash and it would fail. The answer, for me, turned out to be very simple. I added "-a" (without quotes) to the This. Also, using reverse mirror worked out better. mirror -c [b]-R -a[/b] --use-pget-n=5 --only-missing --log=/var/log/synctorrents_downloads.log /mnt/user/temp/torrents/blackhole /blackhole/ Before the -a, I was getting the same errors around directories not existing, access denied, etc... for the torrents directory, I also use --Remove-source-files to remove the torrents after they have been uploaded so they don't get uploaded again. For the mirror that pulls the media, I have headphones, couchpotato, or sonar processing which in turn moves the media from the temp directories to my media shares that plex can access. Hope this helps someone as I banged my head on this for 2 days. Still having an issue with directory structure and autolabel but that's easier to troubleshoot. http://lftp.yar.ru/lftp-man.html Quote Link to comment
Frostyfruit Posted January 19, 2018 Share Posted January 19, 2018 Hi all, I am using the above script but having an issue where Sonarr is picking up the file while it is downloading and copying to the folder incomplete. Is there anyway for lftp to lock the file while it is complete the transfer? Quote Link to comment
pyrater Posted January 19, 2018 Share Posted January 19, 2018 (edited) i no longer use this setup sorry. however it seems a simple changing of the folders would work. IE only look in the completed downloads folder. Edited January 19, 2018 by pyrater Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.