wgstarks Posted May 24, 2014 Share Posted May 24, 2014 I'm in the process of setting up monthly parity checks in Dynamix. Scheduling them is no problem but I don't know if I should do write corrections automatically? Link to comment
garycase Posted May 24, 2014 Share Posted May 24, 2014 A question that's often debated here Personally, I always do correcting checks. It's VERY likely that any detected errors are indeed on the parity disk -- otherwise the disks would show errors and UnRAID would have corrected the reads and/or disabled the disk if it failed on a write. Link to comment
SSD Posted May 24, 2014 Share Posted May 24, 2014 Truth is it doesn't much matter. If your array is healthy you should never get any. If your array is sick you will get some - but generally there is nothing to do but let unRAID correct them. So generally I do correcting checks for routine operations. It's when things get a little sideways that I opt for the non-correcting checks - and even then only for 1-2 minutes to make sure that parity is generally correct. But if 1-2 minutes in generating thousands and thousands of parity mismatches, that's when you don't want to run a correcting check. At those times, the parity may contain the key to recovering data. Link to comment
wgstarks Posted May 24, 2014 Author Share Posted May 24, 2014 Truth is it doesn't much matter. If your array is healthy you should never get any. If your array is sick you will get some - but generally there is nothing to do but let unRAID correct them. So generally I do correcting checks for routine operations. It's when things get a little sideways that I opt for the non-correcting checks - and even then only for 1-2 minutes to make sure that parity is generally correct. But if 1-2 minutes in generating thousands and thousands of parity mismatches, that's when you don't want to run a correcting check. At those times, the parity may contain the key to recovering data. If the parity check is running in the background how would I know how many mismatches there are? I guess I don't really understand what would cause this or how it should be expected and avoided? Link to comment
SSD Posted May 24, 2014 Share Posted May 24, 2014 For an automatic check, I'd just run correcting. Link to comment
superloopy Posted May 24, 2014 Share Posted May 24, 2014 I'd be running correcting checks monthly and a checksum check (you DO checksum your files, yes?) a couple of times a year to maintain integrity Any problems with data disks, restore from backups (you ARE backing up, yes?) Link to comment
wgstarks Posted May 24, 2014 Author Share Posted May 24, 2014 I'd be running correcting checks monthly and a checksum check (you DO checksum your files, yes?) a couple of times a year to maintain integrity Any problems with data disks, restore from backups (you ARE backing up, yes?) If you have a checksum error how do you repair it? No backup server yet. Link to comment
superloopy Posted May 24, 2014 Share Posted May 24, 2014 If you value your data (why wouldnt you?) you need to maintain backups. They are your only point of recovery if files go bad. Sent from my GT-N7100 using Tapatalk Link to comment
JonathanM Posted May 24, 2014 Share Posted May 24, 2014 If you have a checksum error how do you repair it?By restoring the file from your backup.No backup server yet.Unraid is not a backup. It cannot protect from users or applications overwriting, corrupting, or deleting files, it can only recover from a single physical drive failure. As to the original question, if you are regularly monitoring your servers health by checking smart status, looking at system logs, and generally keeping an eye on things, then a correcting check is probably best. If, on the other hand, you are going to set it up and never look at it until something starts acting funny, then I would do a non-correcting automatic check. Reason being, the parity check is likely the only time the server is really stressed fully, and it seems to bring failing hardware (drives, PSU, memory, etc) to the edge. If a drive is acting funny, I'd rather see the parity check errors and investigate the cause before blindly overwriting the only chance I may have at recovery. Link to comment
wgstarks Posted May 24, 2014 Author Share Posted May 24, 2014 I realize the difference between my unraid and a backup. Still there is the problem that backing up 12TB (currently) worth of videos would require an array of equivalent size. Off site would be the best location, so this would mean backing up over an internet connection (500kB/s if I'm lucky). I'm open to suggestions for cloud based backup that would provide cheap backup of what could conceivably wind up being over 100TB. I'm not aware of any but I know new options turn up often. Would be glad of any economical backup options but they really need to be free or very cheap. After all we're talking about a bunch of movies. HDTV rips mostly. I'd hate to lose them, but I think if my house burns down they'll be the least of my worries. Don't want to sound like I'm arguing against backups. I think they are a great idea. Just not sure they'll fit in my budget right now. Please feel free to provide backup suggestions though. I'm sure there are options I've never considered. Link to comment
SSD Posted May 24, 2014 Share Posted May 24, 2014 This issue of backups comes up from time to time. Although hard to debate that a full backup is great protection, most users here focus backup on things like digital pictures and important documents, not every ripped movie. But if such movies are irreplaceable or the effort involved is re-ripping is prohibitive in the event of a worst case disaster, backups are necessary. Subtle corruption that leads to data loss if possible but not likely, especially if you are monitoring and maintaining your array. But in some recovery scenarios users sometimes ask if there was any data corruption, and having the checksums is very valuable in those cases. A third option is building par2 blocks which provides some data correction capability. Link to comment
wgstarks Posted May 24, 2014 Author Share Posted May 24, 2014 Is there an add-on/script that will generate a list of checksums? I can see where it would be useful to at least know which files had become corrupted? Link to comment
BobPhoenix Posted May 24, 2014 Share Posted May 24, 2014 Is there an add-on/script that will generate a list of checksums? I can see where it would be useful to at least know which files had become corrupted? md5deep is available for 5.0.5 unRAID if you use the unMenu package manager. Link to comment
SSD Posted May 24, 2014 Share Posted May 24, 2014 Is there an add-on/script that will generate a list of checksums? I can see where it would be useful to at least know which files had become corrupted? md5deep is available for 5.0.5 unRAID if you use the unMenu package manager. Is there a reason to use md5deep over md5sum? I believe md5sum is preinstalled. I thought they were equivalent. Link to comment
BobPhoenix Posted May 24, 2014 Share Posted May 24, 2014 Is there an add-on/script that will generate a list of checksums? I can see where it would be useful to at least know which files had become corrupted? md5deep is available for 5.0.5 unRAID if you use the unMenu package manager. Is there a reason to use md5deep over md5sum? I believe md5sum is preinstalled. I thought they were equivalent. No reason didn't realize Dynamix had md5sum preinstalled. Plain unRAID has nothing which is why I suggested it. Link to comment
wgstarks Posted May 24, 2014 Author Share Posted May 24, 2014 Is there an add-on/script that will generate a list of checksums? I can see where it would be useful to at least know which files had become corrupted? md5deep is available for 5.0.5 unRAID if you use the unMenu package manager. Is there a reason to use md5deep over md5sum? I believe md5sum is preinstalled. I thought they were equivalent. No reason didn't realize Dynamix had md5sum preinstalled. Plain unRAID has nothing which is why I suggested it. What do I need to install With Dynamix? Only have the basic Dynamix installation so far. Had to wait til I could do a reboot to continue the process. Link to comment
SSD Posted May 24, 2014 Share Posted May 24, 2014 No reason didn't realize Dynamix had md5sum preinstalled. Plain unRAID has nothing which is why I suggested it. unRAID comes with md5sum installed by default. It is there with both 5.0.5 and 6.0b5a. Link to comment
BobPhoenix Posted May 24, 2014 Share Posted May 24, 2014 No reason didn't realize Dynamix had md5sum preinstalled. Plain unRAID has nothing which is why I suggested it. unRAID comes with md5sum installed by default. It is there with both 5.0.5 and 6.0b5a. News to me. Thanks. I always thought the script I copied used md5deep but I see I was wrong. Link to comment
BobPhoenix Posted May 25, 2014 Share Posted May 25, 2014 Just to update and round out on this. I'm blaming my memory loss on the massive doses of cold medication I've been taking. md5sum is built into unRAID so no need to install anything to get it at all md5deep is NOT needed. The script I use to automate using it was produced in this thread http://lime-technology.com/forum/index.php?topic=28168.0 Although I do see he was using md5deep in his script. I used md5sum because it was built in just modified the script for my use. He choose md5deep because it was suppose to have an auditing mode. So maybe there is a reason I'm sticking with md5sum like bjp999 was suggesting because it is built in. Hope that clarifies and helps. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.