unRAID with SABnzbd


Recommended Posts

The failed downloads were puzzling me so I did some testing.

 

I found I had 2 cases. The first case is a bad download that would not repair. The second case is a download with damage that can be repaired. Yet, when I check the directory I find it was repaired yet SAB quits processing the file and says the verification failed.

 

Here is part of the output from such a case. I first removed the repaired rar files and replaced them with the original files which get named with a .1 on the end. This put the files back to their original download state and they would not unzip.

 

Repair is required.
2 file(s) exist but are damaged.
16 file(s) are ok.
You have 235 out of 237 data blocks available.
You have 3 recovery blocks available.
Repair is possible.
You have an excess of 1 recovery blocks.
2 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 100000000 bytes to disk

Verifying repaired files:

Target: "tv_show.r02" - damaged. Found 14 of 15 data blocks.
Target: "tv_show.r12" - damaged. Found 14 of 15 data blocks.
Repair Failed.

 

However, in the directory I found the original files were renamed "tv_show.r02.1" and "tv_show.r12.1" and they were replaced with new files with the original name. Also, the new files will now allow a successfully unrar. So, the repair did not fail.

 

For fun, I ran the exact same par2 command again and get this output

 

Target: "tv_show.r02" - found.
Target: "tv_show.r12" - found.

Repair complete.

 

I found this very odd. It pointed to the par2 program having a bug. So, I did some searching and I found this -> par2cmdline-0.4-gcc4.patch

 

So, it appears there is a patch to the par2commandline we need to install. However, it's real late so I'll have to continue this tomorrow. I hope this helps so far. I'll try to muddle through the patch install tomorrow evening and see what happens if no-one else beats me to it.

 

Peter

 

Link to comment
  • Replies 734
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Well, that patch was for the source code and I couldn't find any info on what exactly it did.

 

I did some more searching and found this page;

 

http://www.chuchusoft.com/par2_tbb/download.html

 

When searching I came across the older versions where I found this in the notes;

 

"fixed a bug which affected the Linux and Mac versions whereby repairs would fail if the file being repaired was short or had one or two bad blocks (because the async write to the file's last byte was failing)."

 

Which led me to believe this could be a working version of par2. I downloaded the GNU/Linux 32-bit file and I replaced the par2 executable in /usr/bin with the par2 file from the download.

 

Here is a test result from a fairly messed up download. As you can see, SAB did download enough repair blocks and the new par2 executable returns a much better post repair result.

 

Before.

 

par2cmdline version 0.4, Copyright (C) 2003 Peter Brian Clements.
Modifications for concurrent processing, Unicode support, and hierarchial
directory support are Copyright (c) 2007-2008 Vincent Tan.
Concurrent processing utilises Intel Thread Building Blocks 2.0,
Copyright (c) 2007-2008 Intel Corp.
Executing using the 32-bit x86 (IA32) instruction set.

par2cmdline comes with ABSOLUTELY NO WARRANTY.

This is free software, and you are welcome to redistribute it and/or modify
it under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 2 of the License, or (at your
option) any later version. See COPYING for details.

Processing verifications and repairs concurrently.
Loading "show.vol00+1.par2".
Loaded 40 new packets
Loading "show.vol01+2.par2".
Loading "show.vol03+4.par2".
Loaded 1 new packets including 1 recovery blocks
Loading "show.vol07+8.par2".
Loaded 3 new packets including 3 recovery blocks
Loading "show.vol15+9.par2".
Loaded 7 new packets including 7 recovery blocks
Loaded 8 new packets including 8 recovery blocks

There are 19 recoverable files and 0 other files.
The block size used was 3456000 bytes.
There are a total of 238 data blocks.
The total size of the data files is 782774005 bytes.

Verifying source files:

Target: "show.nfo" - found.
Target: "show.r00" - damaged. Found 14 of 15 data blocks.
Target: "show.r01" - damaged. Found 14 of 15 data blocks.
Target: "show.r03" - damaged. Found 14 of 15 data blocks.
Target: "show.r02" - damaged. Found 14 of 15 data blocks.
Target: "show.r04" - damaged. Found 14 of 15 data blocks.
Target: "show.r05" - damaged. Found 14 of 15 data blocks.
Target: "show.r06" - damaged. Found 14 of 15 data blocks.
Target: "show.r07" - damaged. Found 14 of 15 data blocks.
Target: "show.r08" - damaged. Found 14 of 15 data blocks.
Target: "show.r10" - found.
Target: "show.r09" - damaged. Found 14 of 15 data blocks.
Target: "show.r11" - damaged. Found 14 of 15 data blocks.
Target: "show.r13" - found.
Target: "show.r12" - damaged. Found 14 of 15 data blocks.
Target: "show.r14" - damaged. Found 9 of 10 data blocks.
Target: "show.sfv" - found.
Target: "show.srr" - found.
Target: "show.rar" - damaged. Found 14 of 15 data blocks.

Scanning extra files:


Repair is required.
14 file(s) exist but are damaged.
5 file(s) are ok.
You have 224 out of 238 data blocks available.
You have 19 recovery blocks available.
Repair is possible.
You have an excess of 5 recovery blocks.
14 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 682763564 bytes to disk

Verifying repaired files:

Target: "show.r00" - damaged. Found 14 of 15 data blocks.
Target: "show.r01" - damaged. Found 14 of 15 data blocks.
Target: "show.r02" - damaged. Found 14 of 15 data blocks.
Target: "show.r03" - damaged. Found 14 of 15 data blocks.
Target: "show.r04" - damaged. Found 14 of 15 data blocks.
Target: "show.r05" - damaged. Found 14 of 15 data blocks.
Target: "show.r06" - found.
Target: "show.r07" - damaged. Found 14 of 15 data blocks.
Target: "show.r08" - damaged. Found 14 of 15 data blocks.
Target: "show.r09" - damaged. Found 14 of 15 data blocks.
Target: "show.r11" - damaged. Found 14 of 15 data blocks.
Target: "show.r12" - damaged. Found 14 of 15 data blocks.
Target: "show.r14" - damaged. Found 9 of 10 data blocks.
Target: "show.rar" - damaged. Found 14 of 15 data blocks.
Repair Failed.

 

After. The first part is the same so I did not repeat it.

 

Repair is required.
14 file(s) exist but are damaged.
5 file(s) are ok.
You have 224 out of 238 data blocks available.
You have 19 recovery blocks available.
Repair is possible.
You have an excess of 5 recovery blocks.
14 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 682763564 bytes to disk

Verifying repaired files:

Target: "show.r00" - found.
Target: "show.r01" - found.
Target: "show.r02" - found.
Target: "show.r03" - found.
Target: "show.r04" - found.
Target: "show.r05" - found.
Target: "show.r06" - found.
Target: "show.r07" - found.
Target: "show.r08" - found.
Target: "show.r09" - found.
Target: "show.r11" - found.
Target: "show.r12" - found.
Target: "show.r14" - found.
Target: "show.rar" - found.

Repair complete.

 

The next step was to actually try it with SAB again. Success! The previously failed download finished processing.

 

There is probably a proper package or something to get a working par2 but either I don't know where to look or it's simple difficult to get current info on the par2 versions out there. So, at this point I'll probably just put the par2 file on the flash drive and stick cp /boot/par2 /usr/bin/par2 into the go file. It'd be great if someone else comes up with a better solution.

 

In the last 44 downloads I've had 4 downloads that would fail due to this issue and this will fix them. I've also had a few that failed due to not having enough repair blocks and unfortunately this will not fix them. It seems to be older episodes that fail due to not having enough repair blocks.

 

Peter

 

 

Link to comment

lionelhutz -- what versions does par2 report before and after using the replacement version? I think the dependencies I use already has the fixed version.

 

Look at my last post in the first "code" section. The text that appears at the start of the par2 command is the same for both versions. I guarantee they are not the same though. The par2 included with the 2.0 dependencies file does not work on files that need to be repaired and the one I replaced it with works. I tested both with multiple par2 command lines and multiple nzb download attempts.

 

Peter

Link to comment

It's not hard to use.

Unzip the file into the root of the flash drive. Either in the flash share from the network or /boot from the server. I used 7-zip on W7 to do this.

Put the line "cp /boot/par2 /usr/bin/par2" into the go file right after the dependencies install.

Type "cp /boot/par2 /usr/bin/par2" at the command line to move the file without rebooting.

 

You could store the file in another place too if you wanted.

 

Note that you can still get failures due to not enough repair blocks but this takes care of the ones that say they failed to verify without any reason.

 

Peter

Link to comment

The text that appears at the start of the par2 command is the same for both versions. I guarantee they are not the same though. The par2 included with the 2.0 dependencies file does not work on files that need to be repaired and the one I replaced it with works.

Good to know. That probably means the 1.3 dependencies I'm using are also the 'broken' version. Thanks for your research! :-)

Link to comment

It's not hard to use.

Unzip the file into the root of the flash drive. Either in the flash share from the network or /boot from the server. I used 7-zip on W7 to do this.

Put the line "cp /boot/par2 /usr/bin/par2" into the go file right after the dependencies install.

Type "cp /boot/par2 /usr/bin/par2" at the command line to move the file without rebooting.

 

You could store the file in another place too if you wanted.

 

Note that you can still get failures due to not enough repair blocks but this takes care of the ones that say they failed to verify without any reason.

 

Peter

 

I just did what you posted and redownloaded 5 nzbs that had previously failed. All 5 of them completed successfully.

1 less manual headache to worry about.

 

Excellent research and greatly appreciate your letting us know.

Job well done.

Link to comment

Hello all.  I just migrated the duties of sab and sickbeard from my Mac Mini (HTPC) over to unRAID.  Seems to be working great so far.  Only issue I ran into was the known problem of the sickbeard cache folder using the flash drive.  I believe I have that all sorted out now.

 

What I am finding interesting is that I think sabnzbd (even idle) is preventing me from unmounting the array.  Every time I've tried to unmount, the cache drive seems to get stuck on unmounting.  If I browse to sab and shut it down, then the array becomes unmounted.  Anyone else experience this?  Am I just going to need to get in the habit of manually shutting down sab (and maybe even sickbeard) when I need to unmount/reboot?

 

Thanks!

Link to comment

Am I just going to need to get in the habit of manually shutting down sab (and maybe even sickbeard) when I need to unmount/reboot?

Yes.

 

Or, you can put a script into place that shuts it down for you when an attempt is made to stop the array.

In this post you can see how I have just such a script that starts and stops two processes on my server that would otherwise keep the disks from un-mounting.

http://lime-technology.com/forum/index.php?topic=5686.msg56731#msg56731

 

Once the 5.0 release is a bit further on you'll be able to tie into an event it supplies that occurs prior to the array stop.

 

Otherwise.... you'll need to manually stop any process keeping a disk busy before you can stop the array.

 

Joe L.

 

 

Link to comment

Guys,

 

I've been running SabNZBd (currently v0.5.5) from my cache disk for about 2 years now. Previously, my unRAID server would be on 24/7 for the lack of waking it up remotely (my Airport Extreme does not allow for forwarding magic packets to the broadcast address). Recently, I got hold of a cheap gigabit router with a custom firmware that allows me to wake up the server over the internet, so I've added the S3 sleep script from another thread to my go script.

 

I found however, that the server was unable to go to sleep because the cache disk wouldn't spin down with SabNZBd running. So I changed the path for the logs to the flash drive (instead of on the cache disk), but the cache disk still won't spin down.

 

The unRAID management GUI shows that some writes to the cache disk occur every few minutes or so, so that explains it's inability to spin down, but I'm not sure what causes this to happen. If I shutdown SabNZBd, the cache disk will spin down properly. I do not use a nzb blackhole, but the temp and cache folders are located on the cache disk.

Link to comment
It's not hard to use.

Unzip the file into the root of the flash drive. Either in the flash share from the network or /boot from the server. I used 7-zip on W7 to do this.

Put the line "cp /boot/par2 /usr/bin/par2" into the go file right after the dependencies install.

Type "cp /boot/par2 /usr/bin/par2" at the command line to move the file without rebooting.

 

This is what I did to create a new dependancies package

 

from telnet

tar -zxvf /boot/packages/SABnzbdDependencies-2.0-i486-unRAID.tgz

 

Now copy the new par2 file to /boot/usr/bin

 

tar -zcvf /boot/packages/SABnzbdDependencies-2.0test-i486-unRAID.tgz usr install

 

Modify the go file to point to the new dependencies package

Link to comment

Guys,

 

I've been running SabNZBd (currently v0.5.5) from my cache disk for about 2 years now. Previously, my unRAID server would be on 24/7 for the lack of waking it up remotely (my Airport Extreme does not allow for forwarding magic packets to the broadcast address). Recently, I got hold of a cheap gigabit router with a custom firmware that allows me to wake up the server over the internet, so I've added the S3 sleep script from another thread to my go script.

 

I found however, that the server was unable to go to sleep because the cache disk wouldn't spin down with SabNZBd running. So I changed the path for the logs to the flash drive (instead of on the cache disk), but the cache disk still won't spin down.

 

The unRAID management GUI shows that some writes to the cache disk occur every few minutes or so, so that explains it's inability to spin down, but I'm not sure what causes this to happen. If I shutdown SabNZBd, the cache disk will spin down properly. I do not use a nzb blackhole, but the temp and cache folders are located on the cache disk.

 

Programs like SabNZB will do this.  You have to quit the application if you want the server to sleep.  Transmission will do the same thing and keep the server awake.  You can not be downloading or seeding anything and Transmission will keep active the drive it is on.  If you want the sleep script to work you are probably going to have to modify it to kill the SabNZBd process before issuing the sleep command.  You will then have to restart SabNZB when the server wakes back up.

Link to comment

So it's logical to conclude that SABnzbd will also prevent my APC UPS from shutting down the server during a power outage?  That blows.

 

Not if you have the Power Down package installed from unMenu.  That powerdown package will be invoked by apc to shutdown the server, which kills all running process that may cause the server to not unmount any drives.

Link to comment

So it's logical to conclude that SABnzbd will also prevent my APC UPS from shutting down the server during a power outage?  That blows.

 

You should have your own /etc/rc.d/rc.local_shutdown script setup to shut down any and all processes needed. That's how my transmission, eggdrop, and sabnzbd are handled. There hasn't been any issues once it's properly setup.

 

It could look like this:

 

#!/bin/sh
#
# /etc/rc.d/rc.local_shutdown: local system shutdown script.

if [ -x /etc/rc.d/rc.transmission-daemon ]; then
  . /etc/rc.d/rc.transmission-daemon stop
fi

if [ -x /etc/rc.d/rc.sabnzbd ]; then
. /etc/rc.d/rc.sabnzbd stop
fi

if [ -x /etc/rc.d/rc.eggdrop ]; then
. /etc/rc.d/rc.eggdrop stop
fi

# kick off unraid powerdown
[ -x /etc/rc.d/rc.unRAID ] && /etc/rc.d/rc.unRAID stop

Link to comment

Yes.

 

Or, you can put a script into place that shuts it down for you when an attempt is made to stop the array.

In this post you can see how I have just such a script that starts and stops two processes on my server that would otherwise keep the disks from un-mounting.

http://lime-technology.com/forum/index.php?topic=5686.msg56731#msg56731

 

Once the 5.0 release is a bit further on you'll be able to tie into an event it supplies that occurs prior to the array stop.

 

Otherwise.... you'll need to manually stop any process keeping a disk busy before you can stop the array.

 

Joe L.

 

You should have your own /etc/rc.d/rc.local_shutdown script setup to shut down any and all processes needed. That's how my transmission, eggdrop, and sabnzbd are handled. There hasn't been any issues once it's properly setup.

 

It could look like this:

 

#!/bin/sh
#
# /etc/rc.d/rc.local_shutdown: local system shutdown script.

if [ -x /etc/rc.d/rc.transmission-daemon ]; then
 . /etc/rc.d/rc.transmission-daemon stop
fi

if [ -x /etc/rc.d/rc.sabnzbd ]; then
. /etc/rc.d/rc.sabnzbd stop
fi

if [ -x /etc/rc.d/rc.eggdrop ]; then
. /etc/rc.d/rc.eggdrop stop
fi

# kick off unraid powerdown
[ -x /etc/rc.d/rc.unRAID ] && /etc/rc.d/rc.unRAID stop

 

Thanks everyone!  My apologies as I'm still trying to wrap my head around this--and I love step-by-steps :)--but basically, what I would need to do is the following, correct?

 

- Create, for example, an rc.sabnzbd script that will launch sabnzbd called with a "start" command, and kill the process when called with a "stop" command.

- Edit rc.local_shutdown to make "stop" calls to the program-specific rc.* scripts

- Edit rc_files.auto_install (if I have one… I am at the airport right now so I can't look) to copy the appropriate rc.* scripts to /etc/rc.d

- unraid_addon_control.auto_install (which already exists in my go script) should take care of the rest...

 

Or am I completely off?  BRiT, any chance I could see your rc.sabnzbd file?  Thanks again!

 

(Edited to correct a typo).

Link to comment

Here's my full script. You will need to modify it.

 

I ported over a utility, start-stop-daemon, that takes care of the starting and stopping of programs as daemons while running under a different username.

 

You can likely replace a lot of this with simply spawning of SABNZBd directly, and just using the WGET line to stop it. That usually always works and never has to resort to using the line after it.

 

#! /bin/sh

# ----- CONFIGURATION -----
#
# The name of the user that should run SABnzbd
# It's RECOMENDED to run as it's own user.
USERNAME=download
GROUPNAME=download
#
# ----- *ADVANCED* CONFIGURATION -----
# Only change these options if you know what you are doing!
#
SAB_HOSTNAME=INSERT_IP_ADDRESS_HERE
SAB_PORT=8081
SAB_CONFIGNAME=/home/$USERNAME/.sabnzbd/sabnzbd.ini
SAB_ARGS="--server ${SAB_HOSTNAME}{SAB_PORT} --config-file ${SAB_CONFIGNAME}"
SAB_URL_SHUTDOWN="http://${SAB_HOSTNAME}{SAB_PORT}/sabnzbd/api?mode=shutdown&apikey=INSERT_API_KEY_HERE"
#
# ----- END OF CONFIGURATION -----
#
# PATH should only include /usr/* if it runs after the mountnfs.sh script.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SSD=/sbin/start-stop-daemon
WGET=/usr/bin/wget
SABNZBD=/usr/lib/python2.6/site-packages/SABnzbd-0.5.5/SABnzbd.py
DESC="usenet client"
NAME=SABnzbd
DAEMON=/usr/bin/python2.6
DAEMON_ARGS="$SABNZBD $SAB_ARGS"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/rc.d/rc.sabnzbd

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
[ -f /etc/default/rcS ] && . /etc/default/rcS

#
# Function that starts the daemon/service
#

do_start()
{
   # Return
   #   0 if daemon has been started
   #   1 if daemon was already running
   #   2 if daemon could not be started
   $SSD --chuid $USERNAME --group $GROUPNAME --start --pidfile $PIDFILE --make-pidfile \
           --name $NAME --exec $DAEMON --background --test -- $DAEMON_ARGS > /dev/null \
           || return 1
   $SSD --chuid $USERNAME --group $GROUPNAME --start --pidfile $PIDFILE --make-pidfile \
           --name $NAME --exec $DAEMON --background -- $DAEMON_ARGS \
           || return 2
}

#
# Function that stops the daemon/service
#
do_stop()
{
       # Return
       #   0 if daemon has been stopped
       #   1 if daemon was already stopped
       #   2 if daemon could not be stopped
       #   other if a failure occurred
       $WGET -q --delete-after $SAB_URL_SHUTDOWN
       $SSD --stop --quiet --retry=TERM/10/KILL/5 --pidfile $PIDFILE --exec $DAEMON

       RETVAL="$?"
       [ "$RETVAL" = 2 ] && return 2

       # Many daemons don't delete their pidfiles when they exit.
       rm -f $PIDFILE

       return "$RETVAL"
}

case "$1" in
 'start')
       echo "Starting $DESC" "$NAME..."
       do_start
       case "$?" in
               0|1) echo "   Starting $DESC $NAME succeeded" ;;
               *)   echo "   Starting $DESC $NAME failed" ;;
       esac
       ;;
 'stop')
       echo "Stopping $DESC $NAME..."
       do_stop
       case "$?" in
               0|1) echo "   Stopping $DESC $NAME succeeded" ;;
               *)   echo "   Stopping $DESC $NAME failed" ;;
       esac
       ;;

 'restart'|'force-reload')
       #
       # If the "reload" option is implemented then remove the
       # 'force-reload' alias
       #
       echo "Restarting $DESC $NAME..."
       do_stop
       case "$?" in
         0|1)
               do_start
               case "$?" in
                   0|1) echo "   Restarting $DESC $NAME succeeded" ;;
                   *)   echo "   Restarting $DESC $NAME failed: couldn't start $NAME" ;;
               esac
               ;;
         *)
               echo "   Restarting $DESC $NAME failed: couldn't stop $NAME" ;;
       esac
       ;;
 *)
       echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
       exit 3
       ;;
esac

Link to comment

Programs like SabNZB will do this.  You have to quit the application if you want the server to sleep.  Transmission will do the same thing and keep the server awake.  You can not be downloading or seeding anything and Transmission will keep active the drive it is on.  If you want the sleep script to work you are probably going to have to modify it to kill the SabNZBd process before issuing the sleep command.  You will then have to restart SabNZB when the server wakes back up.

 

I understand that if a download or upload is in progress, the cache disk is unable to spin down. But SabNZBd does not allow the cache disk to ever go to sleep, even though it's not downloading anything. The source files for Sab are located on the flash drive, so the application is not run from cache disk. The cache disk is only used for cache and temporary files while downloading...

 

And I'm pretty sure the cache disk spun down before...

 

While typing this message I googled what the cache folder in Sabnzbd was for. I assumed it was for temporary storage of downloads, but then I remembered that's what the incomplete folder is for. The cache folder only holds some info about the queue and other minor stuff and has only small files. And it actually does do small writes every now and then... I moved it to the flash drive (as well as the logs folder), so here's hoping that this will allow the cache disk to spin down.

Link to comment

@dimh

 

I have yet to add a cache drive to my system so I have installed sabnzbd in a folder on my array (using the dependancy package and instructions in this thread) This is also where the temporary files are stored.

 

My drive does spin down so your cache drive should too, there must be an issue with something.

Link to comment

@dimh

 

I have yet to add a cache drive to my system so I have installed sabnzbd in a folder on my array (using the dependancy package and instructions in this thread) This is also where the temporary files are stored.

 

My drive does spin down so your cache drive should too, there must be an issue with something.

 

Moving the cache folder to the flash drive now allows the cache drive to spin down... I'm a happy camper :)

Link to comment

@dimh

 

I have yet to add a cache drive to my system so I have installed sabnzbd in a folder on my array (using the dependancy package and instructions in this thread) This is also where the temporary files are stored.

 

My drive does spin down so your cache drive should too, there must be an issue with something.

 

Moving the cache folder to the flash drive now allows the cache drive to spin down... I'm a happy camper :)

 

If you use sabnzbd quite a lot this could give excessive read/writes on your flash drive?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.