garycase Posted April 4, 2014 Author Share Posted April 4, 2014 Understand. I agree there are (rare) cases where a non-correcting check is useful. In my case, if I'd had the same issue, and run a correcting check, I'd have run a checksum validation on the drive, identified the corrupted files, and simply replaced them from my backups. The real solution to this issue is dual parity ... then a parity error can be corrected with no "guesswork" as to where the error is Quote Link to comment
JonathanM Posted April 11, 2014 Share Posted April 11, 2014 People have lost data due to correcting parity checks.Current case study. http://lime-technology.com/forum/index.php?topic=32849.0 Granted this user didn't follow correct procedures for diagnosing issues, but unraid didn't help matters either. If the default action for an unclean shutdown was to boot to maintenance mode with no plugins and not automatically start the array or any automatic parity check, this particular user would be in a very different situation right now. The automatic start of a correcting parity check made a bad situation much worse. IMHO, correcting parity checks that run automatically are evil. Writing data to a disk without my explicit approval is BAD. Quote Link to comment
SSD Posted April 11, 2014 Share Posted April 11, 2014 I have made the following suggestion for a feature. Thoughts?? When a correcting parity check is requested, unRAID should start with a quick non-correcting check. Maybe do a several thousand blocks at the beginning, middle, and end of the disk. And if there are more than a few issues, report back to the user before starting the correcting check. I can see this protecting from a number of scenarios where you would not want to run a correcting check. This could actually be done as a stardard part of starting the array, and refuse to start it if the parity is off. This would force users to investigate only to find a warning on the unRAID page. Obviously there would still be an option to start it in maintenance mode, but at least a user knows there is a problem before any writes go to the array. Quote Link to comment
garycase Posted April 11, 2014 Author Share Posted April 11, 2014 Note that the guy in this recent case rebooted his array with the wrong USB flash drive, which had incorrect configuration information on it -- I'm not at all surprised this caused some issues. I agree that in this case it would make a lot more sense for UnRAID to give an "Invalid Configuration" message and simply stop ... but what's not clear is whether or not the bad configuration data was based on the same set of disks -- in which case the serial #'s would have been okay, and it may have simply started a parity check based on a drive that WAS the parity drive in an earlier config (e.g. before it was replaced with a 4TB drive). So it's not at all clear that what UnRAID did was "wrong" -- but it IS clear that the user messed up big time. And, of course, if he had backups ... I will NEVER understand why anyone who has data they don't want to lose doesn't back it up. Quote Link to comment
sureguy Posted April 11, 2014 Share Posted April 11, 2014 I will NEVER understand why anyone who has data they don't want to lose doesn't back it up. Because it hasn't happened to them yet. Sent from a mobile device, sorry for any typos. Quote Link to comment
garycase Posted April 12, 2014 Author Share Posted April 12, 2014 Tom hits the nail on the head ... No matter what storage system you use, unRaid or anything else, including cloud-based solutions, there are only two ways to ensure you don't lose something irreplaceable: 1. Keep your own redundant backups of critical files. 2. Rely on pure dumb luck... Quote Link to comment
garycase Posted April 12, 2014 Author Share Posted April 12, 2014 This was Tom's post where he made that quote: http://lime-technology.com/forum/index.php?topic=32859.0;topicseen Very interesting (and apparently mostly true) account of how Pixar nearly lost Toy Story II !! Quote Link to comment
page3 Posted May 24, 2014 Share Posted May 24, 2014 A shame that as backups are handled so badly by UnRAID one can only assume that LimeTech don't consider them important! We really do need a GUI interface for an easy backup solution, direct from an UnRaid box to an internal and/or external drive(s). Should be easy to set when and what is backed up and how. The fact that we don't have any sort of interface right now had lead me to consider other solutions many times, and is really the only area of UnRaid for which I am distinctly unhappy. Command line simply doesn't cut it for backups - we need a clear, easy to use interface that encourages them. I've tried many a time to get something up and running using existing command line magic, but never succeeded. My array remains without backup, not through lack of trying. I cannot be the only one? Quote Link to comment
SSD Posted May 24, 2014 Share Posted May 24, 2014 A shame that as backups are handled so badly by UnRAID one can only assume that LimeTech don't consider them important! We really do need a GUI interface for an easy backup solution, direct from an UnRaid box to an internal and/or external drive(s). Should be easy to set when and what is backed up and how. The fact that we don't have any sort of interface right now had lead me to consider other solutions many times, and is really the only area of UnRaid for which I am distinctly unhappy. Command line simply doesn't cut it for backups - we need a clear, easy to use interface that encourages them. I've tried many a time to get something up and running using existing command line magic, but never succeeded. My array remains without backup, not through lack of trying. I cannot be the only one? I don't understand what you are trying to do. It is true that unRAID does not come with a backup tool, but backing up the contents of the array to another server is as easy as copying the files. Most people use unRAID as a backup destination (i.e., backing up workstations). The ability to backup an entire array to an Internet site has feasibility issues due to the extremely long time it would take. I do not think this is a valid criticism of unRAID or its authors. Quote Link to comment
thestewman Posted August 6, 2014 Share Posted August 6, 2014 A shame that as backups are handled so badly by UnRAID one can only assume that LimeTech don't consider them important! We really do need a GUI interface for an easy backup solution, direct from an UnRaid box to an internal and/or external drive(s). Should be easy to set when and what is backed up and how. The fact that we don't have any sort of interface right now had lead me to consider other solutions many times, and is really the only area of UnRaid for which I am distinctly unhappy. Command line simply doesn't cut it for backups - we need a clear, easy to use interface that encourages them. I've tried many a time to get something up and running using existing command line magic, but never succeeded. My array remains without backup, not through lack of trying. I cannot be the only one? I would like to add another comment. I am a new Unraid user with no Linux experience. Unraid should be aware and probably is that the bulk of their OS sales and server hardware sales are to unexperienced users who cannot write code or scripts. That's why we purchased Unraid.And we are smart enough to realize that backups of everything are necessary because as was stated in the first post. Hard drives and systems fail. JonP and Tom provide us with a Unraid Backup program with a GUI interface. Also enable us with The ability to use plugin USB external drives that are not part of the array so we can take advantage of Unraid's speeds and convenience. Internal Hard drives not mounted on Unraid, not precleared and that are not part of the unraid array where we can backup Unraid files Quote Link to comment
garycase Posted August 6, 2014 Author Share Posted August 6, 2014 ... we are smart enough to realize that backups of everything are necessary Unfortunately not all users are as well-informed. Many seem to think "fault-tolerance" = no backup needed ... and don't realize that's not true until they experience a significant data loss. ... The ability to use plugin USB external drives that are not part of the array so we can take advantage of Unraid's speeds and convenience. This would indeed be nice as part of an overall backup utility which tracked newly added files wrote them to an external drive ... prompting for a drive swap whenever the external unit was filled. Internal Hard drives not mounted on Unraid, not precleared and that are not part of the unraid array where we can backup Unraid files This, however, is a different story. You don't want your backups on the same system that's being backed up -- one good electrical surge could wipe out both the primary data and the backups at the same time Nevertheless, it would still be better than no backups -- and as long as the system is UPS protected the risk is very small. [i know a lot of folks who backup their PC data to a 2nd drive in the same system ... usually UPS protected as well.] Quote Link to comment
thestewman Posted August 6, 2014 Share Posted August 6, 2014 ... we are smart enough to realize that backups of everything are necessary Unfortunately not all users are as well-informed. Many seem to think "fault-tolerance" = no backup needed ... and don't realize that's not true until they experience a significant data loss. ... The ability to use plugin USB external drives that are not part of the array so we can take advantage of Unraid's speeds and convenience. This would indeed be nice as part of an overall backup utility which tracked newly added files wrote them to an external drive ... prompting for a drive swap whenever the external unit was filled. Internal Hard drives not mounted on Unraid, not precleared and that are not part of the unraid array where we can backup Unraid files This, however, is a different story. You don't want your backups on the same system that's being backed up -- one good electrical surge could wipe out both the primary data and the backups at the same time Nevertheless, it would still be better than no backups -- and as long as the system is UPS protected the risk is very small. [i know a lot of folks who backup their PC data to a 2nd drive in the same system ... usually UPS protected as well.] Good points The last one about not using the same system was a point I overlooked. I was thinking of the convenience of backing up to a single drive when no external USB drive was available. Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night. Think Tom and company will write us a backup ? Quote Link to comment
garycase Posted August 6, 2014 Author Share Posted August 6, 2014 Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night. That's what I do these days. Think Tom and company will write us a backup ? I just use SyncBack on a Windows machine to do the backups. Wrote a little script to automatically turn on the backup server; then run Syncback to execute the backup "profile". I suspect it's not too difficult to write a simple Linux script that would execute at a specified time to run a sync between two servers ... but I'm not a "Linux guy" and haven't bothered to do this. It'd be nice if it was a built-in feature in UnRAID ... might even sell a few extra systems to those who'd like the convenience of an automated backup to a 2nd UnRAID server Quote Link to comment
nacat78 Posted August 7, 2014 Share Posted August 7, 2014 Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night. That's what I do these days. Think Tom and company will write us a backup ? I just use SyncBack on a Windows machine to do the backups. Wrote a little script to automatically turn on the backup server; then run Syncback to execute the backup "profile". I suspect it's not too difficult to write a simple Linux script that would execute at a specified time to run a sync between two servers ... but I'm not a "Linux guy" and haven't bothered to do this. It'd be nice if it was a built-in feature in UnRAID ... might even sell a few extra systems to those who'd like the convenience of an automated backup to a 2nd UnRAID server +1 That's what i am working on now, I've gotten a few people in my area to invest in unRaid setup and plan on making a larger backup server off-site to backup the unraid systems around my area.. would be nice to have a baked in feature that supports backup to another unraid setup. Quote Link to comment
JonathanM Posted August 7, 2014 Share Posted August 7, 2014 Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night. That's what I do these days. Think Tom and company will write us a backup ? I just use SyncBack on a Windows machine to do the backups. Wrote a little script to automatically turn on the backup server; then run Syncback to execute the backup "profile". I suspect it's not too difficult to write a simple Linux script that would execute at a specified time to run a sync between two servers ... but I'm not a "Linux guy" and haven't bothered to do this. It'd be nice if it was a built-in feature in UnRAID ... might even sell a few extra systems to those who'd like the convenience of an automated backup to a 2nd UnRAID server +1 That's what i am working on now, I've gotten a few people in my area to invest in unRaid setup and plan on making a larger backup server off-site to backup the unraid systems around my area.. would be nice to have a baked in feature that supports backup to another unraid setup. Making OpenVPN client and server available stock would be icing on top. You could set up the two units on the same gigabit switch for the initial data load, then cart one offsite and have it rsync the changes over vpn. Having that functionality tested and documented by limetech would put unraid into a whole new category of awesome. Quote Link to comment
garycase Posted August 7, 2014 Author Share Posted August 7, 2014 ]Making OpenVPN client and server available stock would be icing on top. You could set up the two units on the same gigabit switch for the initial data load, then cart one offsite and have it rsync the changes over vpn. Having that functionality tested and documented by limetech would put unraid into a whole new category of awesome. Definitely a great idea => Being able to have the backup server offsite would be really neat. Clearly that can be done already with add-ons, but to have it built-in to UnRAID would add a bit layer of trust and reliability to it (No worry about an add-on not working due to some change in future releases). Quote Link to comment
ZeroT Posted August 15, 2014 Share Posted August 15, 2014 I am right in the middle of planning a new home NAS build, primarily for the same as most, movies, pictures etc. My current plan is a Plus licence unRAID box with parity, swap and 'smaller' data drives, lets say 3 x 2tb. Then I was planning a FreeNAS box just using UFS with fewer larger drives, eg 2 x 3tb. This way my unRAID box as a daily NAS has that extra fault tolerance from the extra drives and can sync to the FreeNAS box which is used purely as the backup box with the same (or more) capacity, this allows the backup box not to need any extra licencing with potential for more larger drives for backing up other machines/unRAID's in the future if needed. Any thoughts/warnings on this approach? Quote Link to comment
garycase Posted August 15, 2014 Author Share Posted August 15, 2014 My only real question is why use such small drives? 3 x 2TB provides either 4 or 6 TB of storage (depending on whether you're talking data drives or including parity in that list). Either of these could be matched by a single 6TB drive (with one more for parity). Ditto the 2nd box ... the higher capacity drives provide much more storage with fewer drives and fewer SATA ports. Quote Link to comment
ZeroT Posted August 16, 2014 Share Posted August 16, 2014 My only real question is why use such small drives? 3 x 2TB provides either 4 or 6 TB of storage (depending on whether you're talking data drives or including parity in that list). Either of these could be matched by a single 6TB drive (with one more for parity). Ditto the 2nd box ... the higher capacity drives provide much more storage with fewer drives and fewer SATA ports. I simply don't need bigger drives that's all, I am currently aiming for an 8tb (4x2tb) data unRAID plus 2tb parity & 120gb SSD cache, and a 12tb (4x3tb) freeNAS for backups. My current server data is all held on a single 2tb drive! Quote Link to comment
thestewman Posted August 29, 2014 Share Posted August 29, 2014 Yes, I've said often that the one missing component in my backup strategy is an offsite backup. I DO have offsite backups of all my personal data [financial records; scans of all important records (actually far more than you need to keep); pictures; etc.]. But the backups of my UnRAID data are not kept offsite. They ARE, however, rather robust. I have 2 primary UnRAID servers -- one for media and one for "everything else". Together they're a bit over 40TB. These are both automatically backed up to a 3rd UnRAID server which is at the far end of the house. In addition, there's a complete set of backups on an additional set of disks that are stored in WeibeTech DriveBoxes which are kept in a UL-certified Class 125 3-hr waterproof and fireproof safe (keeps contents below 125 degrees for up to 3 hrs when exposed to 1925 degree heat). While not as good as an offsite backup, I feel pretty comfortable that I am FAR better backed up than the vast majority of UnRAID users :) [And FWIW, we live less than 2 minutes away from the nearest fire station ... so I suspect my 3 hr safe is plenty to protect against fire damage.] ... and EVERY disk on all of the UnRAID servers and on the extra backup disks has an MD5 checksum stored with it, so I can easily run a verification to confirm whether or not there has been any corruption of the file. In addition, EVERY computer I have is protected with a good UPS. So the likelihood of, for example, losing 2 drives at a time on more than one of the UnRAID boxes, is VERY low. I stand by my comment that I have a VERY low tolerance for risk ... I think my backup strategy supports that comment Gary Being a noobie and lacking any real expertise I need to know how i can get this done. These are both automatically backed up to a 3rd UnRAID server Tom hasn't expressed any interest to do this. I have a 24 drive system and want to buy another 24 drive system and have a differential or incremental backup done everynight Stew Quote Link to comment
garycase Posted August 29, 2014 Author Share Posted August 29, 2014 Gary Being a noobie and lacking any real expertise I need to know how i can get this done. These are both automatically backed up to a 3rd UnRAID server ... I have a 24 drive system and want to buy another 24 drive system and have a differential or incremental backup done everynight I use a schedule task on a Windows PC and SyncBack SE to do the backups. SE and Pro are paid versions, but the free version will work just fine [ http://www.2brightsparks.com/freeware/freeware-hub.html ] The scheduled task runs a simple batch file that does the following: (a) Sends a WOL packet to turn on the backup server (b) Waits 2 minutes (plenty of time for the server to boot ... time yours to see if that's enough for it) © Starts SyncBack to run the backup profiles [i have several profiles that all run in a "Group" profile (If you get Syncback, you'll know what that means) (d) Turns off the backup server This, of course, requires that you have another PC that's always on (although it can be in sleep mode, with the task scheduled to wake up the PC when it runs). You could do this same basic thing with a Cron job under UnRAID using WOL and RSync, but I'm not a "Linux guy" so I can't help with setting that up. I suspect others would if asked nicely I REALLY like SyncBack, which has a myriad of options that make it very simple to make the backup job work exactly like you want it to. Quote Link to comment
talmania Posted August 30, 2014 Share Posted August 30, 2014 Very good write-up, wiki-worthy! The necessity of backups can't be stressed enough. I know a guy who lost all record and photographs of the first five years of his son's life due to a HDD failure - he didn't have backups, and they were gone. I can't even imagine how that made him feel. And this is exactly why there are clean rooms and data recovery services. Yes it will cost an arm and a leg but I've had to send drives out for corporate executives 4-5 times in my early career and EVERY time the data recovered was near 100%. I hope your friend realized these services were available. $1k would be a very small price to pay for that stuff! :'( Quote Link to comment
Brucey7 Posted August 30, 2014 Share Posted August 30, 2014 I use another unRAID user. We mirror our movie/TV collections and periodically sync our machines. All my personal stuff, photos etc, I keep on multiple systems and in the cloud. Quote Link to comment
thestewman Posted August 30, 2014 Share Posted August 30, 2014 I use another unRAID user. We mirror our movie/TV collections and periodically sync our machines. All my personal stuff, photos etc, I keep on multiple systems and in the cloud. And how are you doing the backups and Sync ? Quote Link to comment
Brucey7 Posted August 30, 2014 Share Posted August 30, 2014 Partly it's a real pain. Annually, we bring a server in the car (that's the painful bit) and then sync it using Beyond Compare (which is a stunning piece of software). Monthly, we use Beyond Compare to copy all new/modified files/folders since the last sync date to a new disk and post that. I dare say we could do it with a VPN but our internet lines are maxed out 24/7, mine with movies, and his with TV series. Other methods considered, have been replacing a disk/rebuilding parity on the donor machine and posting the pulled disk. Popping the pulled disk into the recipient machine and doing a new config and rebuilding parity. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.