Several months ago I did read here in the forums that someone did suggest a PAR2 check and repair feature for unRAID. Did somebody install PAR2 on unRAID (what module?) and does somebody successfully use it?

 

Why I ask?

 

My two original unRAID boxes are on 5.0.5. They did never show any errors. All parity checks run without any errors. BUT. Within the last weeks some videos did show some glitches - even when copying it to the local HTPC and playing from there. I did watch that particular videos some weeks ago - and these errors didn't exist. In the meantime nothing was changed in the house. Still Gigabit LAN, same router, same switch, ...

 

So I did use the same BLURAY to re-load some ISOs and they play fine now. Seems that some bits did fall on my unRAID boxes without notification to unRAID and/or to me.

 

Now, I decided to use PAR2 to get some extra protection (5% is my idea) - but I don't know where to start.

 

Any help is highly appreciated.

 

TIA

 

To prevent Bit Rot, I brush my disks twice daily.

I don't actually know how to setup PAR2, but I am intrigued.

(and worried about bit decay.

 

There is a par2cmdline available for Linux. Search and I think you will find it. If not reply back and I will try to assist. You can also use the Windows tool "quickpar" to create the blocks but it is slower over the network. But as a GUI tool it is easier to use to verify the blocks and if you ever need to do an actual recovery.

 

Here is a post that I created after building a PAR set on a 1T drive.

 

http://lime-technology.com/forum/index.php?topic=3124.msg26018#msg26018

 

The hardest part will be getting the files you want to process in a flat directory structure (i.e., in one directory). I store files that way so wasn't as issue, but if you store them in a hierarchy it would take some effort to use the PAR tools. I did do as experiment (if memory serves) of creating soft links to files in a directory, creating a virtual flat structure and ran PAR2 on that and it worked.

 

There is one other issue. The tools need some room to do a recovery, so if your drive is almost full it can't do its magic. But there is an easy fix. If ever you need to do a recovery, you can move several of your large files to another disk. Quickpar will obviously not see them when it a sans the directory, but using the GUI tool you can manually add other files from other disks or directories. Once you do that it will be able to do its thing.

 

Let me know if you have any questions.

 

The hardest part will be getting the files you want to process in a flat directory structure (i.e., in one directory). I store files that way so wasn't as issue, but if you store them in a hierarchy it would take some effort to use the PAR tools. I did do as experiment (if memory serves) of creating soft links to files in a directory, creating a virtual flat structure and ran PAR2 on that and it worked.

 

I did the same thing. I created 1 directory that had symlinks (for 1 drive) using the md5 hash of the full path as the filename.

While it worked to flatten the directory structure, I quickly ran out of ram and crashed with the par2 cmd due to the magnitude of files involved.

 

I suppose this would work for movies, but for 250,000 mp3's it did not work.

I don't know if it was the magnitude of files and low memory pressure or a limit of ram and what was available for par2.

Perhaps when I go 64bit with unRAID I'll try again.

Here's a package. Put it into /boot/packages and it'll install when you boot. I believe this is still installed if you use Influencers SABnzbd packages, and possibly with other versions of that package as well.

 

http://code.google.com/p/unraid-greenleaf-repository/downloads/detail?name=par2cmdline_tbb-0.4-i486-1kh.tgz&can=2&q=

 

The hardest part will be getting the files you want to process in a flat directory structure (i.e., in one directory). I store files that way so wasn't as issue, but if you store them in a hierarchy it would take some effort to use the PAR tools. I did do as experiment (if memory serves) of creating soft links to files in a directory, creating a virtual flat structure and ran PAR2 on that and it worked.

 

I did the same thing. I created 1 directory that had symlinks (for 1 drive) using the md5 hash of the full path as the filename.

While it worked to flatten the directory structure, I quickly ran out of ram and crashed with the par2 cmd due to the magnitude of files involved.

 

I suppose this would work for movies, but for 250,000 mp3's it did not work.

I don't know if it was the magnitude of files and low memory pressure or a limit of ram and what was available for par2.

Perhaps when I go 64bit with unRAID I'll try again.

 

I don't think this will work Weebo. Your total number of blocks is limited to ~30k. Each file, no matter how small, is a block. With larger files, let's say 4G in size, you could only have 500 of them on a 2T disk. You'd have to pick a large block size of say 60M to keep the block count in range. And it should work. My experience was on 1T drives (maybe 2T too, don't remember) though. I haven't tried it (yet) on larger disks.

 

The obvious solution for you is to chunk up you music into sets of no more than ~30k files. Since the block size will be smaller your space requirements for also be less for each set. So if you theoretically could do your 250,000 files and wanted 50 recovery clocks it would take up the same amount of space as chunking it up into 10 sets of 25,000 files and creating 50 recovery blocks each.

The problem with any extended parity scheme -- whether a (hopefully) dual-parity for UnRAID (which would at least be fully automatic)  or a manually updated extra parity scheme like PAR/PAR2 is that it can only protect up to the number of bit errors you've stored error correction data for.

 

I suppose if you REALLY don't want to maintain a full set of backups it's better than nothing -- it at least lets you identify which files aren't good anymore -- but simply maintaining MD5's will let you do that; and a set of backups to restore corrupted/missing files doesn't have any limitations on just how much can be "corrected". 

 

The problem with any extended parity scheme -- whether a (hopefully) dual-parity for UnRAID (which would at least be fully automatic)  or a manually updated extra parity scheme like PAR/PAR2 is that it can only protect up to the number of bit errors you've stored error correction data for.

 

I suppose if you REALLY don't want to maintain a full set of backups it's better than nothing -- it at least lets you identify which files aren't good anymore -- but simply maintaining MD5's will let you do that; and a set of backups to restore corrupted/missing files doesn't have any limitations on just how much can be "corrected". 

 

I agree with everything you say. A backup is best. But it's a lot of time and expense. And it gets logistically complicated. If you have a fire or theft you could lose both. So an offsite backup is needed to protect you. Bandwidth becomes an issue. Maybe you take disks over to you neighbors or friends house from time to time, and spend a few hours drinking a brew waiting for it to finish. And then there is the cost. A 20T array would cost $800 in disks, maybe more. And that's disks only. The computer and controllers cost more.

 

Option 2 - clone your data disks. Put then in a safety deposit box. Seems easier to me. Periodically you'd need to take the disks home. My experience with disks that sit a long time is they stop working, so every six months or a year you'd need to take them home and check them out.

 

Or maybe there is a better way.

 

Md5 checksums would take say a few days of time to compute the first time, and with a clever plugin (which does t exist yet) maintain itself in your array's spare time. It would detect errors only. But that is very handy if you had a bad event and are staring at 3T worth of files and just want to know what's good!

 

PAR2 is more effort but takes a small amount of space, but often otherwise wasted space at the end of almost full drives. So I'll call it free. It takes more time to do and can't be done in real time. Maybe a ever plugin could detect new files and somehow do things overnight. But PAR2 blocks do more than md5. If you have even one of them it can tell you what is corrupted and renamed (recovery tools often loose file names). And with some more blocks you could recover from bit rot or the kinds of subtle corruption we've seem here. More blocks and you can recover a deleted file. The user can decide.

 

I am not an expert on 2 disk parity, but it will have operational impacts, maybe spinning up the entire array to do a write or slowing it down. Not sure the level of protection or visibility of file level corruption.

 

Running an unRaid array on a server with removable disks (don't have to open the case and risk knocking something loose) are very well protected. I'd say (these %s are Swags) - 94%.

 

Running an MD5 doesn't really add to the protecting but does make diagnosing things that do to wrong. Being able to check your files after a drive rebuild after a dirty shutdown and have the peace of mind is very valuable IMO. On my unscientific scale I'll give it 96%.

 

PAR2 sets covering all data, say enough to recover the largest file, provide a significant boost. You loose double disk failure, fire/theft, and big time corruption. I'd give it 98%.

 

Backups are the best. But are not real time. So you still are exposed for recent adds and changes. But you can recover from most anything with a complete backup. Call in 99.7%

 

Every user can decide what is right for them.

 

Update - my scale was based on ability to successfully recover in the event of a data loss event of some kind.

 

Here's a package. Put it into /boot/packages and it'll install when you boot. I believe this is still installed if you use Influencers SABnzbd packages, and possibly with other versions of that package as well.

 

http://code.google.com/p/unraid-greenleaf-repository/downloads/detail?name=par2cmdline_tbb-0.4-i486-1kh.tgz&can=2&q=

Wouldn't it need a script or something to install it from /boot/packages? I thought /boot/extra was for automatic installpkg as seen here.

I do not see that a Linux version is available, but for anyone interested in a windows gui version there is a currently supported update called MultiPar http://multipar.eu/ ... since quickpar has been abandonded.

 

The hardest part will be getting the files you want to process in a flat directory structure (i.e., in one directory). I store files that way so wasn't as issue, but if you store them in a hierarchy it would take some effort to use the PAR tools. I did do as experiment (if memory serves) of creating soft links to files in a directory, creating a virtual flat structure and ran PAR2 on that and it worked.

 

I did the same thing. I created 1 directory that had symlinks (for 1 drive) using the md5 hash of the full path as the filename.

While it worked to flatten the directory structure, I quickly ran out of ram and crashed with the par2 cmd due to the magnitude of files involved.

 

I suppose this would work for movies, but for 250,000 mp3's it did not work.

I don't know if it was the magnitude of files and low memory pressure or a limit of ram and what was available for par2.

Perhaps when I go 64bit with unRAID I'll try again.

 

I don't think this will work Weebo. Your total number of blocks is limited to ~30k. Each file, no matter how small, is a block. With larger files, let's say 4G in size, you could only have 500 of them on a 2T disk. You'd have to pick a large block size of say 60M to keep the block count in range. And it should work. My experience was on 1T drives (maybe 2T too, don't remember) though. I haven't tried it (yet) on larger disks.

 

The obvious solution for you is to chunk up you music into sets of no more than ~30k files. Since the block size will be smaller your space requirements for also be less for each set. So if you theoretically could do your 250,000 files and wanted 50 recovery clocks it would take up the same amount of space as chunking it up into 10 sets of 25,000 files and creating 50 recovery blocks each.

 

 

It's not really worth the effort.

It's far easier to spend $200 on another 4T drive, rsync it periodically and take it off site.

I use md5's to verify the files, rsync to back them up.  I learned par is pointless for me.

I thought I had real backups of the hard copy cd's, then learned they were even vulnerable to disaster.

What saved 'some' of my data was having a mirror copy in a different geographical location.

It's not really worth the effort.

It's far easier to spend $200 on another 4T drive, rsync it periodically and take it off site.

 

I agree.  Backups are clearly better.  And note that if you use enough PAR blocks to provide significant recovery capability, this starts to occupy a fair amount of space as well.  I think it's much better to use that extra space for actual backups 

 

 

I use md5's to verify the files, rsync to back them up.

 

I do pretty much the same, but don't use Linux for my backups.  I have MD5's for all files to verify integrity; and use SyncBack from a Windows box to do the backups.  I store the backups in a fireproof, waterproof safe -- not quite as good as offsite ... but I'm pretty comfortable with that level of protection.

 

... A 20T array would cost $800 in disks ...

 

True -- although that can largely be offset by using older disks that you're otherwise not using for the backups (e.g. when you upgrade your disks, just use the older disks for backups). 

 

Further ... when you consider how much time you've spent building up that 20TB of media (ripping disks; re-encoding; compressing; splitting series into episodes; cataloging; etc. -- depending on how you're storing everything) AND compare it to the overall cost of the hardware you're using both for your UnRAID server and the distribution/playback devices ... I think the cost of keeping everything well-backed up is pretty nominal.

 

... and if you start out maintaining backups, the incremental cost over time is pretty small.  That $800 estimate assumes you've done NO backups while you accumulated 20TB of data !!

 

I actually have my old array still functioning, with files as they were 4 years ago or something. It is a backup of maybe 1/2 my array. But I don't keep it offsite or anything. Weights a TON - probably 50 or 70 lbs. But that array is full of old 1T drives, and does not use 5in3s. Would be a super giant PITA to add more disks or upgrade the ones in their to larger ones. (You can see my old server in my sig link!)

 

In my earlier analysis, I remarked that backups only allow recovery in about 1.7% additional situations where a significant event (like a drive failing and crashing the server) occurs  (the % isn't scientific but just my guess). But I do have to admit, in those situations, the backup would protect you and unRAID/PAR2 would not. And it would protect you from a very large data loss (at least one disk).

Thanks for your ideas. I did install the par2cmd package and will start some testing ASAP.

 

On my two unRAID servers it's 99% static data so it will be a one-time job for most of the data. Backup is no solution for 74TB in total. But 7,4TB repair data is something that would fit.

 

Again, thanks a lot.

 

Backup is no solution for 74TB in total.

 

It may be a costly solution, but it's still the most certain way to protect your data.  And it may not be as costly as you think -- If you have the original BluRays and DVDs, and don't mind the potential hassle of re-ripping, re-rendering, cataloging, etc, then for all that data you're already "backed up".

 

 

But 7,4TB repair data is something that would fit.

 

You can indeed protect against a fairly significant amount of corruption with error correcting info totaling 10% of the data size ... but nothing close to the 98% bjp999 indicated.  I don't know what improvements PAR2 made in the error correction algorithms -- I believe it's simply a block-oriented approach that makes effectively smaller ("virtual") blocks from larger files, to improve the effective redundancy, but is still Reed-Solomon based.  A Reed-Solomon code with that much error correction data could indeed recover from some fairly significant data corruption (technically it can correct "ErrSize/2" errors, where that's the size of the error polynomial relative to the data block size) ... but will still be effectively useless if you lost an entire disk of data (which is, I think, the most likely event with UnRAID).

 

Keeping the PAR2 data will provide pretty good protection against "bit rot" or file corruption, but it's really not a good way to protect against disk failures.  What PAR/PAR2 was designed for was to provide recovery info for bad data transmissions -- where you have the data, but it's been corrupted.  They work very well for that purpose.

 

Bottom line:  If the goal is to actually protect your data, there's no substitute for backups.    If you simply want a way to confirm the integrity of your files, MD5s or SHA1's (or both) do a very good job of that (but as with anything less than a full backup, aren't perfect -- you CAN have a corrupted file that still passes an MD5 check).    If you want the ability to both check the integrity AND do SOME correction of any errors, then PAR2's will do that nicely.    Just don't be unrealistic in your expectations of the capability this will provide.

 

 

Keeping the PAR2 data will provide pretty good protection against "bit rot" or file corruption, ...

 

That's what I want, exactly. As I wrote in the original post I was hit by some minor but nasty file corruptions wthin files, something that unRAID can't detect. Parity check reports no errors. SMART reports no errors - but files that were perfect a month ago are corrupt now. As I said - it's 99% static data.

 

Creating all those data was so time consuming I don't want to do that again. Clicking on a title on my HTPC is much more fun than going down to the basement, search for that title, walk upstairs again and put it in the player, gasping for air ... and 5 minutes later my wife decides to watch a different movie ...

 

...going down to the basement, search for that title, walk upstairs again and put it in the player, gasping for air ... and 5 minutes later my wife decides to watch a different movie ...

Satisfying "True Love" is why we all need our unRAIDs.  Its too much work, otherwise.