User ideas for handling potential unRAID 6 beta7/8 Data Corruption


Recommended Posts

If anyone has audio files that they would like to check, Foobar2000 has a File Integrity Validator.  Seems to work well.

 

http://www.foobar2000.org/components/view/foo_verifier

 

Thanks for the tip. I ran it on the archive of original audio recorded for my podcast (905 FLAC files, 97.7GB) and it only found one corrupt file. However, that file was already known to be corrupt as it was a failed upload from one of my co-hosts.

 

That's also a good thing, as the verifier correctly identified a known truncated file, suggesting none of those files have been truncated by the bug.

 

This suggest that none of my important source files have been corrupted. However, whether or not they have been jumbled up and have the wrong filename, as reported before is yet to be determined. At some point, I'll pull down all my backup files from OneDrive (some recent files have not been backed up, but most have) and do a byte compare.

 

The files were on a volume (drive) that had about 1 million writes to it since booting with Beta 6, so that's encouraging, I think.

That drive does have a lot of other files I care about, so it'll take a while to check the rest.

Link to comment

On the topic of error checking, given that a lot of the files we have on our servers are audio and video (audio was covered earlier with the foobar plugin), would this be an easy way to check video file integrity: http://superuser.com/questions/100288/how-can-i-check-the-integrity-of-a-video-file-avi-mpeg-mp4

 

You can use a feature in ffmpeg video converter: if you will specify it to recode video to nothing it will just read input file and report any errors that will appear. This is very fast process because video frames are just being read, checked and silently dropped.

 

Example command line: (for Linux)

 

ffmpeg -v error -i file.avi -f null - 2>error.log

-v error means a certain level of verbosity (to show some errors that are normally hidden because they don't affect playability a much).

 

You will get a full error log with some generic information about file ffmpeg will output, so this will probably require your attention, through filters can be written to perform batch check of similar files.

 

We'd just need a script to traverse through all directories, check every file, then parse any error messages out, right?

Link to comment

Perhaps a mod can split out the off-topic discussions in the other thread.

 

On with more data integrity checking -- I mentioned audio and video in other posts.

 

Some of my data is RAR'd. Here's a way to check all your RAR files for errors in one fell swoop: http://superuser.com/questions/359245/how-do-i-check-many-rar-files-for-corruption-at-once

 

ETA: That doesn't appear to work with RAR5 archives. Checking for another solution now...

Link to comment

OK, so all I've been testing so far is my own data, a small subset of the data on my server, but obviously it's the stuff I care about most.

 

I tested about 100GB of FLACs, plus some MP3s and WAVs. All are uncorrupted. There's no quick way of checking if filenames are corrupted/transposed, though.

 

I've also tested 82 RARs (28GB). No corruption:

 

9mPjwWG.png

 

If someone wants to put a little shell script that can run ffmepg recursively on /mnt/user/Movies, I'm up for trying that on the 24TB of Movies on my server! http://lime-technology.com/forum/index.php?topic=35161.msg327534#msg327534

 

Cheers,

 

Neil.

Link to comment

I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking?

 

No, because the checking is read only. I'm only checking shares that are made read-only (e.g. /mnt/user/Movies, not /mnt/disk6/Movies).

 

8EQZicr.png

Link to comment

I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking?

 

It depends on the root cause. In this case, if you mount the drive in read only mode, you cannot cause further corruption.

 

Since the drives are mounted with noatime, generally, there are no writes when reading/accessing the directories and files so we are pretty safe to do reads.

 

To be 100% sure, remount in readonly mode makes it safer.

Link to comment

I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking?

 

No, because the checking is read only. I'm only checking shares that are made read-only (e.g. /mnt/user/Movies, not /mnt/disk6/Movies).

 

8EQZicr.png

 

That would be the way to do it. 

 

Figured in case some of the less technical people (like me) wanted to check for corruption, the worst thing that could happen is introduce new corruption while checking for it.  Thanks for the reply.

Link to comment

i suggest bash + the file command + a bunch of file type specific tools could be used here.

 

7z alone could cover zip + rar + all the comic formats etc

 

Does linux 7z support RAR5? The release version for Windows doesn't. I didn't try the beta, though.

 

TBH i dont know. I never liked rars due to their license and only know them indirectly.

 

What I am suggesting isnt ideal but its a file level tool that could help if no FS level tool is available. And as i mentioned previously its not a bad tool regradless I for one would like to know what mp3s are corrupt etc regardless of how they got that way.

Link to comment
  • 3 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.