PAR2_Shield - a python script to protect your files from bit-rot & some HD probs

[Wimpie]

This script is intended for protecting big files from bitrot and some HD problems (limited lost sectors, ...).

 

It works by creating a .par2 file with x recovery blocks.

 

Creating and checking is automatic, repairing is left completely to the user (with multipar or using any other .par2 utility). I feel to much can go wrong automatically repairing...

 

INSTALLING:

=========

I'm using unRAID 6.1.9, could be different for other versions.

 

Unzip the script, make it executeable (chmod +x PAR2_Shield.py) if needed.

Install python 3.x (using Nerd Tools).

Install a par2 program, eg: https://slackbuilds.org/repository/14.2/misc/par2cmdline/ . To automatically install this after a reboot: put this package in: '/boot/extra' directory.

 

USAGE:

======

usage: PAR2_Shield_v0.1.1.py [-h] [-f FILE_TYPE] [-s SLEEP] [-b BLOCKS] {create,check} folder

 

positional arguments:

  {create,check}        create : Create .par2 verify blocks if the file has none.

                                  check  : Check if the file and the .par2 file are OK.

  folder                      Directory to process, needs to end with '/'.

 

optional arguments:

  -h, --help            show this help message and exit

  -f FILE_TYPE, --file_type FILE_TYPE    Filetype to protect, eg: *.iso

  -s SLEEP, --sleep SLEEP                      Time when no par2 checking is allowed, stoptime-restarttime. Format: HH:MM-HH:MM with HH from 00 to 23 and MM from 00 to 59

  -b BLOCKS, --blocks BLOCKS              Number of recoveryblocks the create task will create (default: 50) (must be between 9 and 100)

 

Create examples:

To create .par2 files in a folder "/mnt/disk1/" for all "*.iso" and "*.mkv" and "*.wmv" files do:

PAR2_Shield create /mnt/disk1/ -f *.iso -f *.mkv -f *.wmv

 

Previous example used the default block number of 50, meaning 50 recovery blocks are created (and a maximum of 50 errors can be corrected). We now want to create 75 recovery blocks:

PAR2_Shield create /mnt/disk1/ -f *.iso -f *.mkv -f *.wmv - b 75

 

First example, but now we want do minimize impact on our server. Run it only between 00:00 hour and 06:00 hour.

PAR2_Shield create /mnt/disk1/ -f *.iso -f *.mkv -f *.wmv -s 06:00-23:59

 

Like previous example, but we want to run it between 01:00 and 06:00 hour.

PAR2_Shield create /mnt/disk1/ -f *.iso -f *.mkv -f *.wmv -s 00:00-01:00 -s 06:00-23:59

 

Check examples:

To check .par2 & protected ("*.iso") files in a folder "/mnt/disk1/" do:

PAR2_Shield create /mnt/disk1/ -f *.iso

 

Like previous example, but not only *.iso files but also *.mkv and *.wmv files.

PAR2_Shield create /mnt/disk1/ -f *.iso -f *.mkv -f *.wmv

 

Like first example, but only between 01:00 and 06:00

PAR2_Shield create /mnt/disk1/ -f *.iso -s 00:00-01:00 -s 06:00-23:59

 

OUTPUT:

=======

With the create task a .par2 file is created in the same directory as the protected file. A .par2 index file is also created by the 'par2cmdline' prog, but the script deletes this (not needed for this use).

 

Logging is written to "/mnt/tmp/PAR2_Shield_DATETIME.log. This is a full log (ALL_OK logging and ERROR logging).

 

ERROR logging is also done to the syslog.

 

NOTES / KNOWN ISSUES:

===================

The sleep function checks the time just before doing a create or check. This means that the script can run at a time when it should sleep on a slow system and big files. When the file is created or checked, the script will check the time again and go to sleep.

 

This script has been tested for following problems:

- Both .par2 and file are OK. Not a problem at all :-).

- File is corrupted, .par2 file OK

- File is OK, .par2 is corrupted

- File is OK, but has other filename than .par2 expects, .par2 file ok

- File is OK, .par2 file has other filename than expected, but is OK. (script will output a wrong filename for the protected file, but indicate that the .par2 is misnamed)

- Should be unicode safe, tested "vid_Ófærð.iso" as filename

- More than 1 to protect file in the same directory.

 

This script is intended for big files (20-50 Gb iso files, big .mkv files, etc...). It will probably also work for your photo collection, but you'll get lots of small files in those directories.

 

It may look that the script isn't doing anything, but on slow systems the creation of .par2 files can take a looong time. Checking on a full 3Tb drive takes 12 hours on my slow servers (see sig) and creation is many times slower.

 

Sleeping is used so the script works when the server is otherwise not used. The way this is programmed means that the server has to be on 24 hours a day. If you reboot, then the script will start from zero (when user runs the same command again).

 

The script doesn't use any throttling if you start to watch a movie, use the sleep function if your movies stutter or don't play well while running this script.

 

My advice is to run this on drive shares. Don't do this on user shares, otherwise you could have following situation: work on disk 1, finish, go to disk 5, disk 5 spins up while disk 1 spins down a bit later, finish disk 5 and go to disk 3, disk 3 spins up while disk 5 spins downa bit later, go to disk 1, disk 1 spins up....

 

par2cmdline is a single threaded program. You could run as much scripts in parallel as you have cpu cores. Just start them with a few seconds apart (to prevent them using the same log file).

 

The script views each to protect file in 32762 blocks, and builds 50 recovery blocks. In other words, 50 out of 32762 blocks can go bad in the protected file and still be repaired with these recovery blocks. The size of these recovery blocks will vary with the size of the file to protect.

 

Hope this is usefull for someone else.

Sugestions/bug reports/... are welcome...

PAR2_Shield_v0.1.1.zip

[Wimpie]

reserved

[mos65xx]

Thats cool and all but have you taken a look at SNAPRAID ?? UNRAID+SNAPRAID cant go wrong 

[JonathanM]

Thats cool and all but have you taken a look at SNAPRAID ?? UNRAID+SNAPRAID cant go wrong 

Could you elaborate on what you mean by that?