December 15, 201213 yr I've totally re-written the supplied screen script to be used via cron, or by the console, including options to trigger nzb imports and call the parsing helpers. If anyone is interested, I can post it after I test a little more. I am interested Cheers, Zenshi Me too, I think.... I gotta admit I'm an idiot with this stuff but, despite that handicap, I've gotten newznab setup and running and... I think an import running even. However, I ran it myself via console so, how do I automate this so it runs without me having to trigger anything manually? I've tried to read through the thread but I didn't see, or missed, anything that explains that part of it. Sorry but thank you for any help.
December 15, 201213 yr Alright, as there is interest...let me do more debugging and make sure I test everything I can. The script is over 100 lines long at this point and /should/ be a one-stop shop for running Newznab via whatever means. Right now it features: # * Lock file # Newznab does not protect the DB when it runs, having two scripts updating # the Database WILL cause corruption # * Logging # Normal update status is redirected to a configurable log file, can be used # with example logrotate.conf. # * Built for Cron # With only errors reported to the caller, the script is built to be run # from within cron # * Scheduled Optimization # * Advanced Newznab processing support # Enable/Disable nzb-importmodified.php and update_parsing.php # # Options: # # -q: Quiet Mode (Errors only to shell, all other messages to log) # -h: Help # -v: Version # -p: Enable enhanced post-processing # -i: Run nzb-importmodified.php # # Configuration # # Set to the directory where Newznab is installed NEWZNAB_BASE="/mnt/cache/AppData/Newznab" # # Set to the directory where NZBs to import are located IMPORT_DIR="/mnt/cache/AppData/Newznab/tempnzbs" # # Interval to run optimization OPT_INT=43200 Let me clean some things up and add a few little options, then I'll post it later tonight.
December 15, 201213 yr Alright, as there is interest...let me do more debugging and make sure I test everything I can. The script is over 100 lines long at this point and /should/ be a one-stop shop for running Newznab via whatever means. Right now it features: # * Lock file # Newznab does not protect the DB when it runs, having two scripts updating # the Database WILL cause corruption # * Logging # Normal update status is redirected to a configurable log file, can be used # with example logrotate.conf. # * Built for Cron # With only errors reported to the caller, the script is built to be run # from within cron # * Scheduled Optimization # * Advanced Newznab processing support # Enable/Disable nzb-importmodified.php and update_parsing.php # # Options: # # -q: Quiet Mode (Errors only to shell, all other messages to log) # -h: Help # -v: Version # -p: Enable enhanced post-processing # -i: Run nzb-importmodified.php # # Configuration # # Set to the directory where Newznab is installed NEWZNAB_BASE="/mnt/cache/AppData/Newznab" # # Set to the directory where NZBs to import are located IMPORT_DIR="/mnt/cache/AppData/Newznab/tempnzbs" # # Interval to run optimization OPT_INT=43200 Let me clean some things up and add a few little options, then I'll post it later tonight. Looking forward to it! Thanks!
December 16, 201213 yr are you calling the other php scrips in this all-for-one script you are making? cause in the last 3 days there have been script updates at least 2 times - you will be constantly updating this all-for-one if you are not calling the orig php scripts.... Myk
December 16, 201213 yr Author For those that have had to replace the lighttpd.cfg with the one I put in my dropbox, have you manually edited the rewrite rules previously or is it from the plug-in? It seems to work fine or my end but if it is getting botched on your ends I need to look into it.
December 16, 201213 yr For those that have had to replace the lighttpd.cfg with the one I put in my dropbox, have you manually edited the rewrite rules previously or is it from the plug-in? It seems to work fine or my end but if it is getting botched on your ends I need to look into it. Never touched it.
December 16, 201213 yr For those that have had to replace the lighttpd.cfg with the one I put in my dropbox, have you manually edited the rewrite rules previously or is it from the plug-in? It seems to work fine or my end but if it is getting botched on your ends I need to look into it. I also had not changed the lighttpd.cfg file but I had to replace as well. On another note I have hit a roadblock on the install. At step 2 of the newznab install I get a 500 - Internal Server Error. When I check my server log I also see: Dec 15 21:04:03 unraid kernel: php-cgi[26799]: segfault at 14 ip 081e0b69 sp bf9b1fc0 error 4 in php-cgi[8048000+391000] The script creates the database in mysql but it does not create the schema. Has anyone else run into this? Thanks in advance for your help.
December 16, 201213 yr Author I'll look at it tomorrow then. Probably didn't escape out a character somewhere. As for the MySQL issue, I haven't run across that and I've ran through the entire setup many times.
December 16, 201213 yr Influencer, great work on putting this plugin together. Can you post your directory structure that you use. I know your other plugins typically go in /mnt/cache/apps. This way every one could at least have a structure that works. Just to clarify, does your plugin install the required plugins if they are not present or do we need to install them first. Thanks.
December 16, 201213 yr Batt01, we will get into long support. I am an op in newznab and have gone as far as calling someone to help them and have done numerous join.me sessions. What is everyone having problems with? I know it inside n out. The torrent is slow, I have a larger set of nzbs (4 years worth) if someone needs. I also have an indexer I can add you folks to.
December 16, 201213 yr are you calling the other php scrips in this all-for-one script you are making? cause in the last 3 days there have been script updates at least 2 times - you will be constantly updating this all-for-one if you are not calling the orig php scripts.... Myk Just calling the newznab scripts, I'm not insane enough to try to keep up with all the changes
December 16, 201213 yr Right, who's going to allow access to the first awesome unRAID indexer? Not planing on opening one up. From what I read, hosting a website on uRaid is not a good idea. I would hate to get hacked and have someone blow up 10TB of data. Maybe with vpn. I have a unsed Qnap Nas, havent seen instructions on how to add newznab tho.
December 17, 201213 yr For those that have had to replace the lighttpd.cfg with the one I put in my dropbox, have you manually edited the rewrite rules previously or is it from the plug-in? It seems to work fine or my end but if it is getting botched on your ends I need to look into it. I also had not changed the lighttpd.cfg file but I had to replace as well. On another note I have hit a roadblock on the install. At step 2 of the newznab install I get a 500 - Internal Server Error. When I check my server log I also see: Dec 15 21:04:03 unraid kernel: php-cgi[26799]: segfault at 14 ip 081e0b69 sp bf9b1fc0 error 4 in php-cgi[8048000+391000] The script creates the database in mysql but it does not create the schema. Has anyone else run into this? Thanks in advance for your help. I managed to resolve the 500 error. I cleaned out all of my plugins and started again. I imagine I had some legacy code sitting around from a previous plugin install through the unMenu package manager. Anywho, I was able to get nn+ up and running. Looks like lots of releases are coming in. I am having a bit of trouble with the sickbeard integration though. I can see the releases I want on newznab but sickbeard cannot find them. I did some poking in the rss request to the newznab api from sickbeard and it looks like newznab is only returning 100 results. I got this config from the caps request to the api. (http://192.168.1.14:81/api?apikey=e84...&t=caps) <limits max="100" default="100"/> This seems like something you should be able to adjust in the options but I had no luck. Any suggestions on where else to look? Also, if it is configurable what is a good choice? Right now I have about a week of backfill for only 3 groups and I am getting over 1000 results.
December 17, 201213 yr Right, who's going to allow access to the first awesome unRAID indexer? There is an Unraiders' Index available. Check lounge forum for NZB Index, Private thread.
December 17, 201213 yr For those that have had to replace the lighttpd.cfg with the one I put in my dropbox, have you manually edited the rewrite rules previously or is it from the plug-in? It seems to work fine or my end but if it is getting botched on your ends I need to look into it. I also had not changed the lighttpd.cfg file but I had to replace as well. On another note I have hit a roadblock on the install. At step 2 of the newznab install I get a 500 - Internal Server Error. When I check my server log I also see: Dec 15 21:04:03 unraid kernel: php-cgi[26799]: segfault at 14 ip 081e0b69 sp bf9b1fc0 error 4 in php-cgi[8048000+391000] The script creates the database in mysql but it does not create the schema. Has anyone else run into this? Thanks in advance for your help. I managed to resolve the 500 error. I cleaned out all of my plugins and started again. I imagine I had some legacy code sitting around from a previous plugin install through the unMenu package manager. Anywho, I was able to get nn+ up and running. Looks like lots of releases are coming in. I am having a bit of trouble with the sickbeard integration though. I can see the releases I want on newznab but sickbeard cannot find them. I did some poking in the rss request to the newznab api from sickbeard and it looks like newznab is only returning 100 results. I got this config from the caps request to the api. (http://192.168.1.14:81/api?apikey=e84...&t=caps) <limits max="100" default="100"/> This seems like something you should be able to adjust in the options but I had no luck. Any suggestions on where else to look? Also, if it is configurable what is a good choice? Right now I have about a week of backfill for only 3 groups and I am getting over 1000 results. I had quite a few issues with SB searching NewzNab - what seemed to work in the in end was ticking the Sickbeard Index option in the search provider box in addition to your Newznab server and ensuring my episode naming convention wasn't set to custom in the post processing tab.
December 17, 201213 yr Author Every site I've checked has had a limit and max of 100, including sickbeards index. Influencer, great work on putting this plugin together. Can you post your directory structure that you use. I know your other plugins typically go in /mnt/cache/apps. This way every one could at least have a structure that works. Just to clarify, does your plugin install the required plugins if they are not present or do we need to install them first. Thanks. The structure I'm using(and this can really be any share) Newznab "root"(SERVER_ROOT setting in plugin settings): /mnt/cache/server/ SimpleFeatures webserver root: /mnt/cache/server/www The main thing to keep in mind is when unpacked newznabs "www" directory is one level down from where you set it, so the web server must be pointing there. The plug-in does not install by default, but if it detects the plug-ins are missing it will offer to install them. Even then you must configure them. Right now it looks for any mysql plug-in, which is the one in the unraid wiki as well as an updated version released by another user that can be found on the forums. The only web server plug-in it looks for(and I believe the only web server plug-in available) is SimpleFeatures web-server plug-in. If you manually install either a webserver or mysql it will fail, as well as if you use unmenu to install either. I'll probably change this in the future.
December 17, 201213 yr This can be a real hoss for Unraid. My dedicated feeding server uses 4Gb ram just for MySQL but I am also indexing like 51 groups
December 17, 201213 yr Author Well that depends, really if you think about it unraid is ideal for a newznab server. Most of us have installed way more ram than unraid will ever need. I currently have 8 GB installed (at one point 16 but that was just way too much overkill), and barely ever utilize more than a gig. Right now with sabnzbd running wide open, newznab indexing, streaming a HD movie, my wife listening to music from the server, couchpotato, sickbeard + other plug-ins running, I'm using a whopping 600MB of ram, leaving me with over 7 GB left doing nothing but being used as cache/buffer. If I assume cache_dirs is holding 1 GB of ram for itself(which I find unlikely but its possible), I still have 6 to play with. For a personal indexer ONLY indexing what I want, I'll probably never touch this.
December 17, 201213 yr Every site I've checked has had a limit and max of 100, including sickbeards index. From the sickbeard log it looked like it was building a cache of newznab by selecting all episodes through the api: http://192.168.1.14:81/api?age=1200&apikey=e84...&t=tvsearch&cat=5040%2C5030 then Sickbeard was using the cache to find episodes. As time goes by there will be a lot more than 100 items in the results. I didn't see anywhere where an api call was made that was more specific (e.g. filtering by show, season or episode). I guess I am missing something or just not understanding the processing steps correctly. Is there anywhere I can do some more reading on this?
December 17, 201213 yr Plugins working great, just a couple of small things and thought I would post on here incase anyone else had any issues with them. My mysql-bin files started to take over my cache drive as I dumped all the NZB's from the torrent into it. I'm not sure what the default setting in the conf is to keep them for but you may need to consider purging these if you are dumping a heap of NZB's into you database. Everything is going great after following everyones advise on here. my only tasks left to complete is 1. install Sphinx 2. install ffpmeg 3. setup SSL connection to the webserver. By any chance does anyone have any advise on these?? Cheers WIll
December 17, 201213 yr Ok, here is the cron script. Note: Insert standard "Don't blame me" disclaimer here. I put a lot of work into this, so if you edit and redistribute please leave in some mention of the origin . Additional items: I also setup the following file in /etc/logrotate.d: # This logname can be set in cron_newznab_process.sh # by setting the variable "LOG" # in the onfiguration section as follows: # # Log file # LOG="/var/log/cron_newznab_process.log" /var/log/cron_newznab_process.sh { create 644 root root notifempty daily rotate 3 missingok } And it is called by adding the following line to root's crontab: */10 * * * * /mnt/cache/AppData/Newznab/misc/update_scripts/nix_scripts/cront_newznab_process.sh -q -i That will fire the script off every 10 minutes, tell it to be "quiet" and to "Import" backlog data. If the script is already running, it will exit rather than start a duplicate process. Once the imports are finished, you can remove -i I'm still working out all the details, at some point I might pull all of this into a more simple solution..perhaps an optional bundle with the plugin. Using the screen solution is such a hack on a server like this. Changelog: v0.1: * Updated so each command appends to log, rather than replaces it. Initial Release cron_newznab_process-0.1.zip
December 17, 201213 yr Want to say thanks for all the work you've been doing Influencer for getting this plugin working. I am currently running the newznab_screen.sh script which seems to be the final part of this process and just waiting to see the results. I've encountered a couple of issues in which I've fixed from this thread and the other one. At the moment my server is using 50-60% processor whilst the script is running and I have 150MB of RAM free (400MB used and 3.45GB cached). Would this be sustainable? Also I'm only interested in TV shows and movies and have set up some of the groups. What are people recommending groups wise?
December 17, 201213 yr Also I'm only interested in TV shows and movies and have set up some of the groups. What are people recommending groups wise? That's actually an excellent question. Being a noob at this, I used NZB index sites a lot, but sadly, never paid much attention to the actual groups the media was coming from. Is there any sort of recommended list of groups we should be indexing for the maximum impact. Somehow I've ended up with quite a few foreign movies that I need to drop some groups to prevent and then I've had quite a bit of bad matches or garbage getting matched to actual films. All of them seem to be from the same groups. Instead of filtering what I have, a list of groups everybody else is using that are known to be good would be great for people trying to set this up on their own.
Archived
This topic is now archived and is closed to further replies.