Newznab

[BLKMGK]

Is anyone else besides myself doing an indexer install not on unRAID? I'm trying to build one in an Ubuntu VM instead of adding complexity to my unRAID box. Running into some issues and would be interested in networking with others!

 

Trying to use this guide: http://newznab.readthedocs.org/en/latest/guides/install_ubuntu-11.10/

 

Tried using their suggested alternate high perf database and couldn't get PHP to pass the preflight accessing SQL. Started fresh and had other issues, starting again from an updated Ubuntu here soon. These guys are using Sphinx and other things to try and speed things which sounds interesting to me. They don't explain some things very well though...

 

Anyway, interested in talking to others and figuring this out!

[graywolf]

Is anyone else besides myself doing an indexer install not on unRAID? I'm trying to build one in an Ubuntu VM instead of adding complexity to my unRAID box. Running into some issues and would be interested in networking with others!

 

Trying to use this guide: http://newznab.readthedocs.org/en/latest/guides/install_ubuntu-11.10/

 

Tried using their suggested alternate high perf database and couldn't get PHP to pass the preflight accessing SQL. Started fresh and had other issues, starting again from an updated Ubuntu here soon. These guys are using Sphinx and other things to try and speed things which sounds interesting to me. They don't explain some things very well though...

 

Anyway, interested in talking to others and figuring this out!

\

 

Try out http://www.howtogeek.com/120285/how-to-build-your-own-usenet-indexer/ they had pretty good instructions. Did it on Ubuntu 12.04.

 

You might want to go ahead and donate. The free version "worked" but produced few releases.

With the "donated" version, you get a ton of regexs that really make the process work well.

 

[prostuff1]

I split this off from the other thread.

 

Please do not add configuration questions to that one, it is for the discussion of nzb indexing sites that are not working, new ones that have been found, and that sort of stuff.

 

I am moving this to the customization forum.

[BLKMGK]

I split this off from the other thread.

 

Please do not add configuration questions to that one, it is for the discussion of nzb indexing sites that are not working, new ones that have been found, and that sort of stuff.

 

I am moving this to the customization forum.

 

Not sure this should've been moved from Lounge and to User Customizations since this isn't about modding an unRAID install and instead setting up a standalone server - a VM in my case.

[BLKMGK]

 

Try out http://www.howtogeek.com/120285/how-to-build-your-own-usenet-indexer/ they had pretty good instructions. Did it on Ubuntu 12.04.

 

You might want to go ahead and donate. The free version "worked" but produced few releases.

With the "donated" version, you get a ton of regexs that really make the process work well.

 

I'm using the paid version, it's certainly worth it if I can create anything that approaches what I've been getting from other sites!

 

So far that install guide looks much like the one I'm using with the exception of Sphinx and some other packages. I had really hoped to try Percona but wow that turned thorny when I tried it at first. Hopefully using the two guides I can get this rolling, I'll certainly try to post any gotcha's I run into. Sphinx is supposed to be an indexing service to speed things up so it sounded attractive for sure...

[lainie]

I am running Newznab+ in a Ubuntu 12.04 server in VirtualBox. Pretty much followed the steps at http://www.howtogeek.com/120285/how-to-build-your-own-usenet-indexer/ (that graywolf lists).

 

It connects to MySQL on my unRAID. Also, I mounted my /var/www/newznab directory to an UnRAID share. Aside from those 2 things, my install sticks with the howtogeek guide.

 

Once installed, I used http://www.tiag.me/how-to-backfill-newznab-safely-without-bloating-your-database/ running in screen to populate the db. Seems like many people are using "newznab.sh" instead, but this way is working for me. (Got the link from one of the newznab devs in IRC)

 

Have added unRAR, MediaInfo, FFmpeg & Lame from apt-get without any issues. Ran into some trouble with Sphinx. Grabbed version 2.0.6 deb file of Sphinx from their site. Got sphinx.conf generated following directions at http://newznab.readthedocs.org/en/latest/misc/sphinx/. Once I get to the step to start the daemon:

root@newz:/var/www/newznab/misc/sphinx# ./nnindexer.php daemon
Newznab full-text indexer utility

searchd: error while loading shared libraries: libmysqlclient.so.16: cannot open shared object file: No such file or directory

Perhaps this has something to do with my MySQL not being run locally? Found where it will generate errors until full and delta indexes are created, so I tried continuing on... but keep getting the "...libmysqlclient.so.16: cannot open shared..." error.

 

So unless I get that fixed, I will continue running without Sphinx. If anyone has suggestions, please let me know. If there are any questions for me (since I am not directly running Newznab on unRAID), let me know.... will help if I can.

[Talsit]

Does anyone know where the database resides? Running it on Ubuntu 12.04 in a VM with a 32 GB drive, currently at 50% full.

 

I want to move the whole install to an RDM and give it it's own 500GB HDD. (Noob instructions for moving the install on ESXI 5.1 would be much appreciated)

 

So far, works swimmingly, full integration into SB, SAB. Haven't tried CP, populating tv groups first.

[jangjong]

Does anyone know where the database resides? Running it on Ubuntu 12.04 in a VM with a 32 GB drive, currently at 50% full.

 

I want to move the whole install to an RDM and give it it's own 500GB HDD. (Noob instructions for moving the install on ESXI 5.1 would be much appreciated)

 

So far, works swimmingly, full integration into SB, SAB. Haven't tried CP, populating tv groups first.

 

I think for ubuntu, it should be /var/lib/mysql/

 

Also, if it's at 50% for 32GB, i would look in to this:

 

http://systembash.com/content/mysql-binary-log-file-size-huge/

 

For me, mysql created multiple binary log files that's over 1GB, which took about 6 - 7GB.

[Talsit]

Thanks, I'll give that a shot. I also need to look at how to backfill without bloat. Wanted to get it working first, then got so excited it was working that I forgot to go back and read the rest of the instructions on backfill!

[joelones]

There's also a comprehensive guide within the newznab install:

 

/var/www/newznab/misc/docgen/source/guides/install_ubuntu-11.10.rst

 

It's for 11.10 but I was able to easily adapt it to 12.04.

[mazma]

I have this working on an Umbuntu VM - the issue I keep getting is that Sickbeard ability to find files in the nzb index is pretty flakey. It mostly says it couldn't find the file even though I can clearly see it in my index.

 

Anyone else have this?

[mazma]

I am running Newznab+ in a Ubuntu 12.04 server in VirtualBox. Pretty much followed the steps at http://www.howtogeek.com/120285/how-to-build-your-own-usenet-indexer/ (that graywolf lists).

 

It connects to MySQL on my unRAID. Also, I mounted my /var/www/newznab directory to an UnRAID share. Aside from those 2 things, my install sticks with the howtogeek guide.

 

Once installed, I used http://www.tiag.me/how-to-backfill-newznab-safely-without-bloating-your-database/ running in screen to populate the db. Seems like many people are using "newznab.sh" instead, but this way is working for me. (Got the link from one of the newznab devs in IRC)

 

Have added unRAR, MediaInfo, FFmpeg & Lame from apt-get without any issues. Ran into some trouble with Sphinx. Grabbed version 2.0.6 deb file of Sphinx from their site. Got sphinx.conf generated following directions at http://newznab.readthedocs.org/en/latest/misc/sphinx/. Once I get to the step to start the daemon:

root@newz:/var/www/newznab/misc/sphinx# ./nnindexer.php daemon
Newznab full-text indexer utility

searchd: error while loading shared libraries: libmysqlclient.so.16: cannot open shared object file: No such file or directory

Perhaps this has something to do with my MySQL not being run locally? Found where it will generate errors until full and delta indexes are created, so I tried continuing on... but keep getting the "...libmysqlclient.so.16: cannot open shared..." error.

 

So unless I get that fixed, I will continue running without Sphinx. If anyone has suggestions, please let me know. If there are any questions for me (since I am not directly running Newznab on unRAID), let me know.... will help if I can.

 

I'm getting a syntax error on the newznab_screen.sh file - its being caused by the ; that appears in front of the NEWZNAB PATH variable - can you paste a copy of yours?

 

[lainie]

I'm getting a syntax error on the newznab_screen.sh file - its being caused by the ; that appears in front of the NEWZNAB PATH variable - can you paste a copy of yours?

 

This is the version of newznab_sceen.sh that I am running in screen. If you are using the one from http://www.tiag.me/how-to-backfill-newznab-safely-without-bloating-your-database/, make sure to use http://www.tiag.me/newznab_screen.txt instead of copy/pasting from the webpage... html made some changes.  Think the only difference in mine is the full path to php5 is specified.

 

#!/bin/bash
# call this script from within screen to get binaries, processes releases and 
# every half day get tv/theatre info and optimise the database

set -e

export NEWZNAB_PATH="/var/www/newznab/misc/update_scripts"
export NEWZNAB_SLEEP_TIME="5" # in seconds
LASTOPTIMIZE=`date +%s`

while :

do
CURRTIME=`date +%s`
cd ${NEWZNAB_PATH}
/usr/bin/php5 ${NEWZNAB_PATH}/update_binaries_threaded.php
/usr/bin/php5 ${NEWZNAB_PATH}/update_releases.php
/usr/bin/php5 /var/www/newznab/www/admin/nzb-importmodified.php /var/www/newznab/tempnzbs/ true
/usr/bin/php5 ${NEWZNAB_PATH}/update_releases.php

DIFF=$(($CURRTIME-$LASTOPTIMIZE))
if [ "$DIFF" -gt 43200 ] || [ "$DIFF" -lt 1 ]
then
LASTOPTIMIZE=`date +%s`
/usr/bin/php5 ${NEWZNAB_PATH}/optimise_db.php
/usr/bin/php5 ${NEWZNAB_PATH}/update_tvschedule.php
/usr/bin/php5 ${NEWZNAB_PATH}/update_theaters.php
fi

echo "waiting ${NEWZNAB_SLEEP_TIME} seconds..."
sleep ${NEWZNAB_SLEEP_TIME}

done

[BLKMGK]

Finally got my server rolling! The last part was figuring out that PHP was running out of memory and crashing in one area but once resolved it's up and running - yay! Now, I'd like to get hold of the motherload of NZB files that are apparently running around somewhere? Can someone point me to that, maybe via PM? Also, is there anything I need to do in the paid version to get the RegEx files to download and update? I've inserted my reg key into the config and overall it seems to be working. Whew! Lots of good info to be had it seems, really appreciate that everyone is sharing and working together.

 

I ended up using MySQL and not the high perf fork that one article suggested along with Apache. I do have Sphinx running as well as FFMPEG and the other toys added. Yet to see an NZB generated but it's early yet

 

P.S. Sickbeard currently throws errors trying to access my indexer too but I've put in the URL for th eweb page and the API key from my profile. Always something!

 

[Influencer]

With the plus version the regexes will update themselves when update_binaries is ran by the newznab script.

 

Once it finishes pulling down all the headers it will start making the nzbs.

[BetaQuasi]

I'm running one (plus) that has around 7 days of content now and it's working really well - followed the same guide as you BLKMGK.  Attempted the Percona fork, but it seems the php5-mysql components cause issues.  Back to regular MySQL and no probs at all.  On Ubuntu 11.10.  I did go with nginx though - I've got a few other sites running on it, and it rocks in terms of performance.

 

Sphinx, ffmpeg etc all running, though I doubt I'll ever make use of them between myself and my group of friends.  So far so good, none of us have missed a release in the last 24 hours   Not too concerned with backfilling, just with the daily stuff.

 

 

 

 

[BLKMGK]

The indexing capability of Sphinx looked like something worthwhile for sure. I've used the preview feature on other sites so having lame and whatnot running seemed prudent. Was interesting to build Deb packages from source!

 

Doubt I'll go over 7 days myself if I can wrangle a link to NZB files. I'm still running the stock update script for now, how exactly does this other prevent bloat and is the NZB import critical for that?

 

Sick erred when it tried to use my server. It should just need an API key and ip right? I'm not more that that to use other sites. Maybe it's because I had no NZB yet? Anyone else run into this and require tweaks?

 

P.S. A real quick update, we have NZB! W00T! Now to get them into Sick lol!

[BetaQuasi]

Add the site to sickbeard using:

 

http://myubernzbsite.com

 

and the api code as that appears on the profile page, not the full link if you cut and paste it.  Then restart sickbeard to have it start working - common with all newznab providers, have to restart mr sick to kick things into action.

[88keyz]

Joined the forum because there seems to be some good discussion happening here on newznab setup.  I'm running newznab+ on Ubuntu Server 12.04 and it has been running well for the last few days.  Getting lots of releases, perhaps too many, and there in lies my issue.  I am polling headers only for specific categories and I have been manually removing the releases from the other categories every day or so.  Is there a way to modify the update scripts so that releases that get categorized in to certain categories get automatically deleted?  So for example if I didn't want releases from the PC category (category 4000) to show up in my index at all, could it be automated to remove those releases on update?

 

Look forward to the discussion here.  There really doesn't appear to be much of a discussion community around newznab so far, at least that I can find, maybe this will become the place to go.

 

edit: Found the answer for anyone that might be interested. (thanks to sy on newznab chat on #IRC for all the help)

 

Just add the following line to /var/www/newznab/misc/testing/deletereleases.php

 

$sql = "select ID from releases where categoryID in (xxxx,xxxx,xxxx)";

 

Replace xxxx in the syntax above with the number of the category that you wish to empty and run the script with the following command:

 

/usr/bin/php5 /var/www/newznab/misc/testing/deletereleases.php

 

To have the code run automatically with your screen script simply add the above line to your /var/www/newznab/misc/update_scripts/nix_scripts/newznab_screen.sh file.

 

Now you have a clean index with only the categories that you want being shown.

[BLKMGK]

Add the site to sickbeard using:

 

http://myubernzbsite.com

 

and the api code as that appears on the profile page, not the full link if you cut and paste it.  Then restart sickbeard to have it start working - common with all newznab providers, have to restart mr sick to kick things into action.

 

Hrm, that link is 404 I have added the http://ipaddress for my site to Sick. I'm getting errors in Sick concerning access and I'm seeing the API requests go up so it's trying, we'll see. I've not restarted Sick I don't think.

 

88keyz - as for this being the goto place I hope not. This is a site primarily to support the unRAID NAS software (which rocks BTW) but many of us are also interested in obtaining content. Me personally I just hate having to get video off my TiVo! Sounds to me like the issue is you have too many groups activated, why not disable pulling from some of them? Mine only pulls form the groups I specify and so far is working well. Surely NewzNab has a forum yes? They certainly have an IRC channel that seems to be active, probably VERY active these days!

 

P.S. Disabled Sphinx. I was seeing errors in the screen script concerning it and from what little I've read it's not going to help me much on a single user system. I also found that running the screen as Sudo cleared many errors, should I need to do that?

[mazma]

I have 9500+ nzb's in my import folder - the script is running as noted on tiag.me. Thing is the number of tmp nzbs in my import folder aren't going down, I thought they got deleted as the process went along?

[Talsit]

I've got mine up and running.

 

New releases haven't been a problem and I'm just letting the software backfill some of my groups.

 

I have this running on an Ubuntu 12.04 Server VM with 2 processors, 2GB RAM and a 32 GB data drive.  I have peaked at 50% full on the drive, when I ran the optimise_db script it came back down to 19% full.  I've got three groups gradually backfilling to 1100 days and the rest of the groups are on 10 days.

 

I'm using the default Newznab screen script with no issues.  I have been getting utilization above 2.0 occassionally but that's probably due to the processing power needed to run the threaded update_binaries script.

 

So far, rock solid!

[CrashnBrn]

I'm running one (plus) that has around 7 days of content now and it's working really well - followed the same guide as you BLKMGK.  Attempted the Percona fork, but it seems the php5-mysql components cause issues.  Back to regular MySQL and no probs at all.  On Ubuntu 11.10.  I did go with nginx though - I've got a few other sites running on it, and it rocks in terms of performance.

 

Sphinx, ffmpeg etc all running, though I doubt I'll ever make use of them between myself and my group of friends.  So far so good, none of us have missed a release in the last 24 hours   Not too concerned with backfilling, just with the daily stuff.

 

For ffmpeg and the others did you compile the newer versions yourself or just apt-get the current ones?

[lainie]

Since my newznab is on Ubuntu, I just used apt-get for everything.

 

---

This is an off-topic question, but I am having trouble finding an answer (google's not being helpful). Typically with apt-get, you can use -y switch to automatically answer yes when installing. My ubuntu server is interpreting -y as a package and causing this error:

 

root@newz:~# apt-get install –y php5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package –y
root@newz:~#

 

It works as intended without the -y... but prompts me to answer Y/n. Not a huge deal, but would appreciate a pm with the answer if anyone knows why this is happening.

---

* Edit - to add replies:

 

CrashnBrn - "sudo apt-get install -y php5" did the exact same thing. I was already running as root so no change. Thank you for trying.

 

MyKroFt - "apt-get -y install php5" this worked. I've used "-y" after "install" on other systems with no issue. Was following the guide from HowToGeek & it has "-y" after "install"... weird... but Thanks, glad to have it not prompt me.

[Carpet3]

What are the risks with running your own indexing server?