Jump to content
sstouffer

unRaid and bitrot / filesystem checksums

14 posts in this topic Last Reply

Recommended Posts

According to this site: http://snapraid.sourceforge.net/compare.html unRAID doesn't have any kind of protection against bitrot or silent errors compared to some of the other options out there, like ZFS.  Is that true? Does the parity drive in any way detect and correct these errors, or is it not happening until a dual-parity solution happens?  I ask these questions because I'm currently running WHS v1 that has been giving me a few problems lately and I'm evaluating my other options out there.

 

Thanks!

Shawn

Share this post


Link to post

The parity drive can be used to detect and repair problems in sectors, but i don't think it's nearly as encompassing as ZFS.

 

Many users do monthly parity checks to insure the arrays and drives have good integrity.

Some users do periodic md5sums to detect issues at the file level.

 

I do know that upon a drive failure, all drives, plus parity are used to virtually present the failed drive and rebuild it upon new drive installation.

 

I'm, not sure of the exact rules of how parity correction works when single errors occur.

Perhaps someone else will post more detail on this particular subject.

Share this post


Link to post

If you want to do an md5 snapshot of each file on a particular drive:

 

find /mnt/disk1 -type f -exec md5sum {} \;>> /disk1_files.md5

 

Do it once a month.  It then isn't hard to use something like beyond compare to use that output file and compare it to an older output file from the same command and look for changed lines, ignoring new/deleted lines.

Share this post


Link to post

If you want to do an md5 snapshot of each file on a particular drive:

 

find /mnt/disk1 -type f -exec md5sum {} \;>> /disk1_files.md5

 

Do it once a month.  It then isn't hard to use something like beyond compare to use that output file and compare it to an older output file from the same command and look for changed lines, ignoring new/deleted lines.

 

How long does that take (and space) on a full 2TB drive (I know, depends on # of files, say 4,200 files)?

Share this post


Link to post

Speed is dependent on the number of bytes used on the disk and the disk read speed, because you are reading every byte of every file.

 

Size of the resulting file, a rough upper-estimate is 100 bytes per file assuming your FQFN's are less than 70 characters.... so not much.

Share this post


Link to post

If you want to do an md5 snapshot of each file on a particular drive:

 

find /mnt/disk1 -type f -exec md5sum {} \;>> /disk1_files.md5

 

Do it once a month.  It then isn't hard to use something like beyond compare to use that output file and compare it to an older output file from the same command and look for changed lines, ignoring new/deleted lines.

 

i sort of understand what you are talking about here, is there any chance of a better example with more verbose explanations please..

or maybe a wiki entry?

Share this post


Link to post

Check out md5deep also

 

http://md5deep.sourceforge.net/

 

Read the documentation about md5sum and md5deep. It's basically a checksum of your files.

The prior example uses find to traverse your filesystem and run md5sum on each fuile.

md5deep can do the traversal within the program

 

I have md5deep compiled for unRAID here:  http://code.google.com/p/unraid-weebotech/downloads/list

Share this post


Link to post

I've never experienced it. Although, video files are not a good candidate for detecting bit rot. The errors will not be noticeable when viewing the video.

Share this post


Link to post

I've never experienced it. Although, video files are not a good candidate for detecting bit rot. The errors will not be noticeable when viewing the video.

 

ahhh.. did not know that

Share this post


Link to post

I've never experienced it. Although, video files are not a good candidate for detecting bit rot. The errors will not be noticeable when viewing the video.

 

ahhh.. did not know that

 

This assumes bit-rot is a single bit error. Effects will be greater for for multiple bit errors.

 

If a b-frame (or slice) or p-frame has an error there's almost no chance of noticing. If an i-frame has an error it is unlikely you'll notice a transient effect but it will be fixed at the next i-frame, i.e, 1 to 15 frames later, depending on compression. Even in the worst case the error will effect the video for less than a second. Audio is even more robust to errors.

 

Compression algorithms are designed to be robust in case of transmission (or storage) error.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.