Just a heads up.

 

In playing with making the md5 link files, then doing a par2, I realized.

It's going to take a day or so to do one disk!

 

Takes 12 hours or so to run an md5 sweep and link down a 1tb drive of movies using a 2.6Ghz Core 2 duo.

 

I may have to re-engineer this some how to take advantage of multiple cores. sheesh.

 

par2 does not look much faster.

Just a heads up.

 

In playing with making the md5 link files, then doing a par2, I realized.

It's going to take a day or so to do one disk!

 

Takes 12 hours or so to run an md5 sweep and link down a 1tb drive of movies using a 2.6Ghz Core 2 duo.

 

I may have to re-engineer this some how to take advantage of multiple cores. sheesh.

 

par2 does not look much faster.

If you were doing absolutely nothing but reading the 1TB drive at 100MB/s it would take several hours just to read the disk.  12 hours for an MD5 sounds about right... it is not a fast calculation.

Just a heads up.

 

In playing with making the md5 link files, then doing a par2, I realized.

It's going to take a day or so to do one disk!

 

Takes 12 hours or so to run an md5 sweep and link down a 1tb drive of movies using a 2.6Ghz Core 2 duo.

 

I may have to re-engineer this some how to take advantage of multiple cores. sheesh.

 

par2 does not look much faster.

 

I take it you've look at http://md5deep.sourceforge.net/ ?

 

Also, have a look at http://www.chuchusoft.com/par2_tbb/index.html

Just a heads up.

 

In playing with making the md5 link files, then doing a par2, I realized.

It's going to take a day or so to do one disk!

 

Takes 12 hours or so to run an md5 sweep and link down a 1tb drive of movies using a 2.6Ghz Core 2 duo.

 

I may have to re-engineer this some how to take advantage of multiple cores. sheesh.

 

par2 does not look much faster.

 

I take it you've look at http://md5deep.sourceforge.net/ ?

 

Also, have a look at http://www.chuchusoft.com/par2_tbb/index.html

 

I have looked at the md5deep program. It seems the same other then being able to recurse.

I do that with find in a loop which md5sums each file then creates the link files.

I may be able to rewrite the loop with named pipes so I can do two md5's at the same time.

 

 

What I have not seen is that version of par2cmdline

i.e. par2cmdline 0.4 with Intel Threading Building Blocks 2.2

With the threading built in, this could be a big help with par2.

It took over 13 hours to do 1tb of video.

What version md5sum are you using?  I've heard reports that md5sum in coreutils 8.4 is faster than 6.9.

 

What version md5sum are you using?  I've heard reports that md5sum in coreutils 8.4 is faster than 6.9.

 

 

md5sum (GNU coreutils) 6.9 ) This is what is in unRAID.

I am thinking back to 1960 and my first job programming a military machine that had a drum memory.  It was more a human simulation than a digital computer.  The memory forgot things, th paper tape reader was near sighted, the printer stuttered, . . . It was called NAVDAC (Navigation Data Assimilation Computer) but as the pogrammers referred to it "Not a Very Damm accurate Computer". Oh yeah the result of any division was an odd number!! Not a problem for large dividends with small divisors, but for small numbers....

 

Because of the memory's reliability, or lack thereof, we used a two dimensional parity check. This allowed for the correction of any single bit error anywhere on the drum.

 

The same thing could be done in unraid. On each disk in the array reserve a block lets say 4096 bits (any reasonable number will do) . Run a parity check on that disk by reading it in blocks of 4096 XORing all the blocks and setting the disks parity block to have even parity.  Now if there is a single bit error the exact bit that is wrong can be determined and fixed. Naturally each disks parity block is also XORed onto the parity drive. The details of the methodology are left to . . .

Before I get to 19 drives, I'm more likely to consider a second server based on some kind of risk evaluation...

 

Interesting, why this?

 

Sorry, I'm a little slow these days...

 

I do track all my hard drives (and one SSD  ), including the failure rates, which drives were in which machines, which drives came back from warranty replacements, how long they lasted, etc.  But as we know it's not so much the likelihood of a double hard-error occurring: it's the hard error + soft error problem.

 

When it comes to security, I look at three things when it comes to data: disclosure, destruction, and alteration.  A lot of this talk focuses on destruction (e.g. drive failure), but doesn't always address alteration (corruption). (I notice, though, that some of the recent posts talking about parcmdline/QuickPAR do explicitly or implicitly begin to address this.)

 

Anyway, the point is I'm also concerned that simple XOR parity by itself is too easily (by my subjective opinion) "fooled", meaning that, while unlikely, it's still possible for some bug in the system or unexpected use case to lead to bits getting flipped in such a way that parity checks OK but the original data is incorrect.  Having a second parity partially mitigates this concern in addition to the concern of a outright failure.

 

The bottom line is my data is sufficiently valuable and storage sufficiently cheap ($0.07/GB or less) such that when weighed against the cost of another server, I don't mind building another one and supporting Lime Technology by buying another license.  By the time I would get even close to 19 drives I'd probably have a spare mobo and memory laying around anyway just due to the natural upgrading of my other stuff (VMWare server, workstations...).  Heck, for that matter, I already have a spare case (the CoolerMaster 690 I replaced with the Antec Twelve Hundred).

 

Joe L made a good point about percentage of data lost.  It's not of much comfort when you lose data, though... and I guess everyone has to evaluate for themselves the value of that data.  Can it be reconstructed (e.g. from original CDs if you built an mp3 music collection), and what is that time worth?  And if it is just about percentage, would it make sense to actually have 19 x 2 TB drives even if you only have a few TBs of data? 

 

Of course, absolutely critical data (like personal photographs  ) I still back up to DVDs anyway, but the sheer amount of data is becoming, well, formidable, and I'm finding DVD media less reliable that unRAID these days.

 

Good stuff to think about. 

In playing with making the md5 link files, then doing a par2, I realized.

It's going to take a day or so to do one disk!

 

I may have to re-engineer this some how to take advantage of multiple cores. sheesh.

 

What I have not seen is that version of par2cmdline

i.e. par2cmdline 0.4 with Intel Threading Building Blocks 2.2

With the threading built in, this could be a big help with par2.

It took over 13 hours to do 1tb of video.

 

WeeboTech,

 

Your problem with par2 can be solved by using it in conjunction with 'find', which can recurse the directory tree and run par2 in each directory. Here is an example which prints an md5sum of each directory pathname:

 

find . -type d -exec echo {} `echo {} | md5sum` \;

 

Instead, if you want to print the md5sum's of all the files:

 

find . -type f -exec md5sum {} \;

 

However, this launches a new md5sum process (sequentially) for each file. When there are lots of small files, it can slow down. In that case, it is better to pass the list of filenames as an argument to md5sum and launch a single md5sum process. A naive way to do this is:

 

md5sum `find . -type f -print`

 

However, this breaks when the command line gets too long with lots of files. It also breaks when there are spaces in filenames. Instead, it is better to break it up using 'xargs' using a null character to separate filenames instead of a space:

 

find . -type f -print0 | xargs -q0 md5sum

 

Finally, if you want to run several md5sum's in parallel, use the GNU parallel command instead of xargs:

 

find . -type f -print0 | parallel -q0 md5sum

 

This will launch multiple md5sum processes, giving each a set of command line arguments. It also collects the output from each md5sum and merges it to give you the same output as if run sequentially with xargs. Note that both find and xargs are in unRAID, but parallel is not.

 

By the way, md5sum is disk-bound on my Atom CPU with 100MB/s disks, so running several in parallel won't help speed things up. However, running a separate md5sum for each disk in parallel should work just fine.

That's not the answer for my situation.

 

The issue is that I created 1 huge directory of md5sum names that symlink to the original file.

Then I run par2 on that directory. Sort of like flattening out the directory structure, then doing a par2 on it.

 

I kept running out of memory. I stopped for now, other things took priority.

I've already done the find through a loop to create the md5sums.

It's the par2 that fails.

 

If I wanted to speed up the md5sums even more I would split the process.

 

1 process to do the find and pipe to a named pipe

2 or more processes to read the named pipe for filenames and do the md5sums in parallel.

 

Because of the memory's reliability, or lack thereof, we used a two dimensional parity check. This allowed for the correction of any single bit error anywhere on the drum.

 

The same thing could be done in unraid.

 

Barrygordon, I was thinking exactly this same thing today.

 

WeeboTech and JoeL, your combined ideas of storing md5sum hashes of filenames and data contents is an excellent way protect integrity of the data on the disk. However, tripwire does pretty much the same thing (detect file changes), and that is probably implemented efficiently already.

 

I think it comes down to a fundamental problem with unRAID and RAID5: although you can detect an error, you do not know whether the error is in the parity drive or the data drives. Right now, unRAID always assumes the parity blocks are flawed and updates them if they mismatch the data drives. This is the smart thing to do, because unRAID is assuming the filesystem on each individual drive should be smart enough to protect its own data integrity. The parity drive contents are only used to reconstruct a replacement data drive. This does not offer any protection if a hard drive begins to suffer from write problems. In fact, it can potentially lead to corruption of the parity drive if what is written to the failing drive does not match what is read from the failing drive. This means all your other drives will not be protected.

 

A two-dimensional parity check is able to locate and correct this problem. Adding a second parity calculation like RAID6 is effectively doing the same thing, but it has higher overheads (you need two parity drives of the same size).

 

To protect data integrity, I would suggest adding a two-dimensional parity check. When the regular parity check fails, instead of assuming the parity data is corrupt like unRAID does today, instead it should access the second parity dimension to locate the failed region (be it on a parity disk or data disk) and repair it.

 

To see how this works, right think of a 2D array of data. Each row is a disk, with the last row being the parity disk. Right now, the parity disk holds column parity: you XOR all data in column 0 from the data disks and write this to column 0 of the parity disk. The 2nd dimension is across the rows. You could XOR all the blocks in a disk, and store a single block checksum for the entire disk. This way, when the parity check fails for a column, you simply scan each row and look for the disk that fails its own row-wise parity check. Of course, this would take forever if the entire drive must be read. Instead, break up the drive into groups of 32 columns, and store 1 block of parity data for every 32 data blocks on the data disk. The value 32 can be made bigger or smaller, it will slow performance (bigger) but be more storage-efficient.

 

The downside now is that each data disk needs its own parity data for the "rows" it contains. This is in addition to the parity disk. Where should we store this row-wise parity? We could change the 'md' driver, and reserve every 32nd block for the parity -- this would minimize head repositioning, but it means the disk is useless as soon as we take it out of the unRAID md driver environment (goes against unRAID philosophy, but most people will be OK with this because they know how to set up a new unRAID box to recover their data). Another option is to create a 2nd partition on the disk, and store all row-wise parity blocks in that 2nd partition. This would cause lots of head seeking and hit performance even more. A third option is to use a 2nd parity disk -- this is my preferred option, which I'll discuss in the next paragraph. A fourth option is to use the filesystem itself and store md5sum hashes of each file -- this is WeeboTech and JoeL's idea. The problem with the md5sum/filesystem approach is that it is a bit inefficient to keep the md5sum data up-to-date: you need to run an hourly job to find all changed files and recompute new md5sums; until that hourly job is run, the data is not protected. Also, it is difficult to locate the problem: you start by knowing the faulty column (from the parity disk), and must now map that sector number into a filename on the data disk, which is not an easy task.

 

My preferred solution is a 2nd parity drive, which can be much smaller than the first parity drive. This 2nd drive will not be used to reconstuct an already-failed data drive, it is only used in conjunction with a data drive that is starting to fail or has corrupted a few blocks (possibly due to filesystem bugs, hard drive problems, OS crashes, abrupt loss of power, etc). If your largest data drive is 1TB, and you have 16 data drives, you can use a 2nd parity drive of 1TB where each block is a checksum of 16 columns. If you want to use 500GB, simply checksum every 32 columns instead. Using a fast SSD of 50GB, you can checksum every 320 columns and avoid any slowdowns. The 2nd-dimension parity calculation can remain a simple XOR, you do not need anything fancy like reed-solomon codes.

 

This would be my most-preferred 'laundry list' item for unRAID  :-) as it enhances data protection, which is the reason I prefer it over regular RAID in the first place.

 

Guy

To protect data integrity, I would suggest adding a two-dimensional parity check. When the regular parity check fails, instead of assuming the parity data is corrupt like unRAID does today, instead it should access the second parity dimension to locate the failed region (be it on a parity disk or data disk) and repair it.

 

This 2nd drive will not be used to reconstuct an already-failed data drive, it is only used in conjunction with a data drive that is starting to fail or has corrupted a few blocks (possibly due to filesystem bugs, hard drive problems, OS crashes, abrupt loss of power, etc).

 

This would be my most-preferred 'laundry list' item for unRAID  :-) as it enhances data protection, which is the reason I prefer it over regular RAID in the first place.

 

To follow-up my own post: the 2nd parity drive here could be used to verify data integrity when reading one of the data drives without powering up any of the other drives (includign 1st parity drive). However, when it detects an error, it could power up the 1st parity drive and correct the error on the fly.

 

In this usage scenario, the 2nd drive is powered up often, but this gives you continuous data checking reliability that isn't currently present in unRAID. An SSD would simply make even more sense in this case (although, technically a big 32GB USB flash drive might work just as well, since this is not a bandwidth-hungry problem).

 

Guy

That's not the answer for my situation.

 

The issue is that I created 1 huge directory of md5sum names that symlink to the original file.

Then I run par2 on that directory. Sort of like flattening out the directory structure, then doing a par2 on it.

 

Why do you want to run par2 on a flat directory? You already saw that most shell utilities cannot handle a huge number of files.

 

I'm suggesting that you create a duplicate tree, or create a .md5sums subdirectory within in each directory, storing the md5sums in that subtree/directory. Then, use find to walk through the new subtree/directory to run par2.

 

Is there another reason for having a flat directory that I'm missing?

 

Guy

That's not the answer for my situation.

 

The issue is that I created 1 huge directory of md5sum names that symlink to the original file.

Then I run par2 on that directory. Sort of like flattening out the directory structure, then doing a par2 on it.

 

Why do you want to run par2 on a flat directory? You already saw that most shell utilities cannot handle a huge number of files.

 

I'm suggesting that you create a duplicate tree, or create a .md5sums subdirectory within in each directory, storing the md5sums in that subtree/directory. Then, use find to walk through the new subtree/directory to run par2.

 

Is there another reason for having a flat directory that I'm missing?

 

Guy

 

There have been a number of conversations about it. It was recommended to build the par2 set from the largest amount of data so that pieces could be recovered easier.

 

I had planed to do a par2 set per directory. It was suggested to have one flattened directory of symlinks and run the par2 on that as it would provide the largest ability to recover.

I know about tripwire.

 

My goal started out with a db search locate tool to index my whole unraid server.

Then add in the md5sum.  This would allow tracking or locating a files/name/position by md5sum in case you had to do a fsck and the files ended up in the lost+found directory.

The par2 suggestion came after that to verify or fill in possible holes during a fsck recovery situation.

 

From what I remember, the problem I had was not the number of files, it was that par2 was using huge amounts of ram to do the calculations and there were kernel settings preventing par2 from using the ram or swapping out.

I favor the same option as Guy. A SSD of a size determined by how many secondary parity columns are to be stored. Fast, Efficient, and provides single error correction with double error detection. (Error meaning a single bit error)

 

The issue I see is how to recognize that a bit has changed, that is when to do the parity check. When reading media data many single bit changes produces an imperceptible glitch. The glitch could be audio or video, in musc or in a picture, but without some sort of validity check being done by the file reader and incorporated in the streaming format it may never be noticied. Do modern disks provide such checking?

 

Now pragmatically if I can not see/hear the error do I really care?  Only in the sense that it might be an early warning of a failing drive.

 

I just ran my first unRaid parity chack since I stabilized my unraid system (4 2TB Data drives and 1 2 TB Parity drive) six months or so ago.  It took roughly 6 hours and reported no errors. In the past 6 months I have just used the system not doing anything special.  Been reliable as I need it to be as far as I can tell.

without some sort of validity check being done by the file reader and incorporated in the streaming format it may never be noticied. Do modern disks provide such checking?

Yes, in fact they do.  It is how they detect a un-readable sector.

 

When each sector is written a checksum is also written at the end of that sector.  When read back the checksum must match.  If not, a "raw" read error will result, and the sector re-read.  If, after several attempts at re-reading the sector it is still not matched, it is marked as pending re-allocation. (and a read failure returned to the OS) 

 

When unRAID gets a read error it re-constructs the required sector by reading the other disks in the array in combination with the parity disk and returns the re-constructed contents of the requested data to the program reading the disk.  It also then writes the re-constructed contents back to the disk to allow the SMART firmware the ability to re-allocate the sector if needed.  (It first tries to re-use the existing sector and re-write it)

 

All disks have raw read errors.  Some report them in the smart report, some do not.

 

When the SATA communications from the disk to the disk controller occur an additional checksum is added.  That too must match.  It verifies the SATA cable link.

 

There are occasional defective disks where the checksum matches, but the internal disk electronics will return an incorrect value (perhaps their internal buffer memory occasionally flips a bit due to noise on the power supply, or a marginal bit in the memory) When this occurs it is nearly impossible to detect, other than for the monthly parity checks occasionally finding random parity errors.  This is because all the checksums matched... for each of their individual steps, but a bit gets flipped between steps in the electronics.

Those disks should subsequently be tested as wheel-chocks, a purpose for which they are far better suited.. 

 

Joe L.

the wheel-chock test involves putting the disk under the wheel of your car, and driving over it again and again testing its ability to impede the motion of the car.

.

the wheel-chock test involves putting the disk under the wheel of your car, and driving over it again and again testing its ability to impede the motion of the car.

 

I think the ST-506 scored far better in wheel-chock tests than modern 3.5" HDs.

 

 

Thanks for the info Joe,  It has been quite a while since I dealt with hardware at that level. I suspected what you said was the case, as technology advances speed error rates usually goes up since tolerences are tighter and coding is used to get thing where they should be.

 

I have to schedule my parity checks for once per month, but other than that my unraid operation has been flawless. It did cost me quite a bit more than I needed to spend as I used an industrial case design with hot swap everything, and dual power supplies. The real expense was to individually Air condition my "Server Room" as I had never done that, it was just fed by the main HVAC.  Of course in the winter when it does get chilly in FL I would heat that room . . . The things one overlooks is amazing!!

+1 from me

 

Im building up my drives in a new norco 4224, but once i get past 12 drives ill start to get nervous about two simultaneous disk failures.

 

Regards

+1 from me

 

Im building up my drives in a new norco 4224, but once i get past 12 drives ill start to get nervous about two simultaneous disk failures.

 

Regards

 

If you spread the purchase of your drives, or buy different models and/or brands, and choose a "fill-up" plan for your shares (rather than "most-free"), the likelihood of two drives failing at the same time is really, really small.

 

Choosing "fill-up" will use only one drive for writing until it's full. For me, this takes about 2-3 months with a 1.5TB drive. And, if you're like me, you'll use the new drive more for reading too, since I tend to watch new movies more than older ones. So two drives reaching read/write cycles thresholds at the same time seems unlikely.

 

And in case two drives do fail simultaneously, at least you won't loose the data from the other drives, like with other RAID implementations.