Recommended Posts

Understand.  I agree there are (rare) cases where a non-correcting check is useful.

 

In my case, if I'd had the same issue, and run a correcting check, I'd have run a checksum validation on the drive, identified the corrupted files, and simply replaced them from my backups.

 

The real solution to this issue is dual parity ... then a parity error can be corrected with no "guesswork" as to where the error is  :)

 

Link to comment
  • Replies 117
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

People have lost data due to correcting parity checks.
Current case study. http://lime-technology.com/forum/index.php?topic=32849.0 Granted this user didn't follow correct procedures for diagnosing issues, but unraid didn't help matters either.

 

If the default action for an unclean shutdown was to boot to maintenance mode with no plugins and not automatically start the array or any automatic parity check, this particular user would be in a very different situation right now. The automatic start of a correcting parity check made a bad situation much worse.

 

IMHO, correcting parity checks that run automatically are evil. Writing data to a disk without my explicit approval is BAD.

Link to comment

I have made the following suggestion for a feature. Thoughts??

 

When a correcting parity check is requested, unRAID should start with a quick non-correcting check. Maybe do a several thousand blocks at the beginning, middle, and end of the disk. And if there are more than a few issues, report back to the user before starting the correcting check. I can see this protecting from a number of scenarios where you would not want to run a correcting check.

 

This could actually be done as a stardard part of starting the array, and refuse to start it if the parity is off. This would force users to investigate only to find a warning on the unRAID page. Obviously there would still be an option to start it in maintenance mode, but at least a user knows there is a problem before any writes go to the array.

 

Link to comment

Note that the guy in this recent case rebooted his array with the wrong USB flash drive, which had incorrect configuration information on it -- I'm not at all surprised this caused some issues.  I agree that in this case it would make a lot more sense for UnRAID to give an "Invalid Configuration" message and simply stop ... but what's not clear is whether or not the bad configuration data was based on the same set of disks -- in which case the serial #'s would have been okay, and it may have simply started a parity check based on a drive that WAS the parity drive in an earlier config (e.g. before it was replaced with a 4TB drive).    So it's not at all clear that what UnRAID did was "wrong" -- but it IS clear that the user messed up big time.    And, of course, if he had backups ...  8)

 

I will NEVER understand why anyone who has data they don't want to lose doesn't back it up.

 

Link to comment

Tom hits the nail on the head ...

 

No matter what storage system you use, unRaid or anything else, including cloud-based solutions, there are only two ways to ensure you don't lose something irreplaceable:

1. Keep your own redundant backups of critical files.

2. Rely on pure dumb luck...

Link to comment
  • 1 month later...

A shame that as backups are handled so badly by UnRAID one can only assume that LimeTech don't consider them important!

 

We really do need a GUI interface for an easy backup solution, direct from an UnRaid box to an internal and/or external drive(s). Should be easy to set when and what is backed up and how.

 

The fact that we don't have any sort of interface right now had lead me to consider other solutions many times, and is really the only area of UnRaid for which I am distinctly unhappy.

 

Command line simply doesn't cut it for backups - we need a clear, easy to use interface that encourages them.

 

I've tried many a time to get something up and running using existing command line magic, but never succeeded. My array remains without backup, not through lack of trying. I cannot be the only one?

Link to comment

A shame that as backups are handled so badly by UnRAID one can only assume that LimeTech don't consider them important!

 

We really do need a GUI interface for an easy backup solution, direct from an UnRaid box to an internal and/or external drive(s). Should be easy to set when and what is backed up and how.

 

The fact that we don't have any sort of interface right now had lead me to consider other solutions many times, and is really the only area of UnRaid for which I am distinctly unhappy.

 

Command line simply doesn't cut it for backups - we need a clear, easy to use interface that encourages them.

 

I've tried many a time to get something up and running using existing command line magic, but never succeeded. My array remains without backup, not through lack of trying. I cannot be the only one?

 

I don't understand what you are trying to do. It is true that unRAID does not come with a backup tool, but backing up the contents of the array to another server is as easy as copying the files.

 

Most people use unRAID as a backup destination (i.e., backing up workstations).

 

The ability to backup an entire array to an Internet site has feasibility issues due to the extremely long time it would take.

 

I do not think this is a valid criticism of unRAID or its authors.

Link to comment
  • 2 months later...

A shame that as backups are handled so badly by UnRAID one can only assume that LimeTech don't consider them important!

 

We really do need a GUI interface for an easy backup solution, direct from an UnRaid box to an internal and/or external drive(s). Should be easy to set when and what is backed up and how.

 

The fact that we don't have any sort of interface right now had lead me to consider other solutions many times, and is really the only area of UnRaid for which I am distinctly unhappy.

 

Command line simply doesn't cut it for backups - we need a clear, easy to use interface that encourages them.

 

I've tried many a time to get something up and running using existing command line magic, but never succeeded. My array remains without backup, not through lack of trying. I cannot be the only one?

 

I would like to add another comment.

I am a new Unraid user with no Linux experience. Unraid should be aware and probably is that the bulk of their OS sales and server hardware sales are to unexperienced users who cannot write code or scripts.

That's why we purchased Unraid.And we are smart enough to realize that backups of everything are necessary because as was stated in the first post. Hard drives and systems fail.

JonP and Tom provide us with a Unraid Backup program with a GUI interface.

Also enable us with

The ability to use plugin USB external drives that are not part of the array so we can take advantage of Unraid's speeds and convenience.

Internal Hard drives not mounted on Unraid, not precleared and that are not part of the unraid array

where we can backup Unraid files

 

Link to comment

... we are smart enough to realize that backups of everything are necessary

 

Unfortunately not all users are as well-informed.    Many seem to think "fault-tolerance" = no backup needed ... and don't realize that's not true until they experience a significant data loss.

 

 

... The ability to use plugin USB external drives that are not part of the array so we can take advantage of Unraid's speeds and convenience.

 

This would indeed be nice as part of an overall backup utility which tracked newly added files wrote them to an external drive ... prompting for a drive swap whenever the external unit was filled.

 

 

Internal Hard drives not mounted on Unraid, not precleared and that are not part of the unraid array

where we can backup Unraid files

 

This, however, is a different story.  You don't want your backups on the same system that's being backed up -- one good electrical surge could wipe out both the primary data and the backups at the same time  :)    Nevertheless, it would still be better than no backups -- and as long as the system is UPS protected the risk is very small.      [i know a lot of folks who backup their PC data to a 2nd drive in the same system ... usually UPS protected as well.]

 

Link to comment

... we are smart enough to realize that backups of everything are necessary

 

Unfortunately not all users are as well-informed.    Many seem to think "fault-tolerance" = no backup needed ... and don't realize that's not true until they experience a significant data loss.

 

 

... The ability to use plugin USB external drives that are not part of the array so we can take advantage of Unraid's speeds and convenience.

 

This would indeed be nice as part of an overall backup utility which tracked newly added files wrote them to an external drive ... prompting for a drive swap whenever the external unit was filled.

 

 

Internal Hard drives not mounted on Unraid, not precleared and that are not part of the unraid array

where we can backup Unraid files

 

This, however, is a different story.  You don't want your backups on the same system that's being backed up -- one good electrical surge could wipe out both the primary data and the backups at the same time  :)    Nevertheless, it would still be better than no backups -- and as long as the system is UPS protected the risk is very small.      [i know a lot of folks who backup their PC data to a 2nd drive in the same system ... usually UPS protected as well.]

 

Good points

 

The last one about not using the same system was a point I overlooked.

I was thinking of the convenience of backing up to a single drive when no external USB drive  was available.

Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night.

Think Tom and company will write us a backup ?

Link to comment

Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night.

 

That's what I do these days.

 

Think Tom and company will write us a backup ?

 

I just use SyncBack on a Windows machine to do the backups.  Wrote a little script to automatically turn on the backup server; then run Syncback to execute the backup "profile".    I suspect it's not too difficult to write a simple Linux script that would execute at a specified time to run a sync between two servers ... but I'm not a "Linux guy" and haven't bothered to do this.    It'd be nice if it was a built-in feature in UnRAID ... might even sell a few extra systems to those who'd like the convenience of an automated backup to a 2nd UnRAID server  :)

Link to comment

Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night.

 

That's what I do these days.

 

Think Tom and company will write us a backup ?

 

I just use SyncBack on a Windows machine to do the backups.  Wrote a little script to automatically turn on the backup server; then run Syncback to execute the backup "profile".    I suspect it's not too difficult to write a simple Linux script that would execute at a specified time to run a sync between two servers ... but I'm not a "Linux guy" and haven't bothered to do this.    It'd be nice if it was a built-in feature in UnRAID ... might even sell a few extra systems to those who'd like the convenience of an automated backup to a 2nd UnRAID server  :)

 

+1

That's what i am working on now, I've gotten a few people in my area to invest in unRaid setup and plan on making a larger backup server off-site to backup the unraid systems around my area.. would be nice to have a baked in feature that supports backup to another unraid setup.

Link to comment

Personally I was thinking of getting an identical hardware copy of my Unraid system that would backup incrementally every night.

 

That's what I do these days.

 

Think Tom and company will write us a backup ?

 

I just use SyncBack on a Windows machine to do the backups.  Wrote a little script to automatically turn on the backup server; then run Syncback to execute the backup "profile".    I suspect it's not too difficult to write a simple Linux script that would execute at a specified time to run a sync between two servers ... but I'm not a "Linux guy" and haven't bothered to do this.    It'd be nice if it was a built-in feature in UnRAID ... might even sell a few extra systems to those who'd like the convenience of an automated backup to a 2nd UnRAID server  :)

 

+1

That's what i am working on now, I've gotten a few people in my area to invest in unRaid setup and plan on making a larger backup server off-site to backup the unraid systems around my area.. would be nice to have a baked in feature that supports backup to another unraid setup.

Making OpenVPN client and server available stock would be icing on top. You could set up the two units on the same gigabit switch for the initial data load, then cart one offsite and have it rsync the changes over vpn. Having that functionality tested and documented by limetech would put unraid into a whole new category of awesome.
Link to comment

]Making OpenVPN client and server available stock would be icing on top. You could set up the two units on the same gigabit switch for the initial data load, then cart one offsite and have it rsync the changes over vpn. Having that functionality tested and documented by limetech would put unraid into a whole new category of awesome.

 

Definitely a great idea => Being able to have the backup server offsite would be really neat.  Clearly that can be done already with add-ons, but to have it built-in to UnRAID would add a bit layer of trust and reliability to it (No worry about an add-on not working due to some change in future releases).

 

Link to comment
  • 2 weeks later...

I am right in the middle of planning a new home NAS build, primarily for the same as most, movies, pictures etc.

 

My current plan is a Plus licence unRAID box with parity, swap and 'smaller' data drives, lets say 3 x 2tb.

 

Then I was planning a FreeNAS box just using UFS with fewer larger drives, eg 2 x 3tb.

 

This way my unRAID box as a daily NAS has that extra fault tolerance from the extra drives and can sync to the FreeNAS box which is used purely as the backup box with the same (or more) capacity, this allows the backup box not to need any extra licencing with potential for more larger drives for backing up other machines/unRAID's in the future if needed.

 

Any thoughts/warnings on this approach?

Link to comment

My only real question is why use such small drives?    3 x 2TB provides either 4 or 6 TB of storage (depending on whether you're talking data drives or including parity in that list).    Either of these could be matched by a single 6TB drive (with one more for parity).    Ditto the 2nd box ... the higher capacity drives provide much more storage with fewer drives and fewer SATA ports.

 

Link to comment

My only real question is why use such small drives?    3 x 2TB provides either 4 or 6 TB of storage (depending on whether you're talking data drives or including parity in that list).    Either of these could be matched by a single 6TB drive (with one more for parity).    Ditto the 2nd box ... the higher capacity drives provide much more storage with fewer drives and fewer SATA ports.

 

I simply don't need bigger drives that's all, I am currently aiming for an 8tb (4x2tb) data unRAID plus 2tb parity & 120gb SSD cache, and a 12tb (4x3tb) freeNAS for backups.

 

My current server data is all held on a single 2tb drive!

Link to comment
  • 2 weeks later...

Yes, I've said often that the one missing component in my backup strategy is an offsite backup.    I DO have offsite backups of all my personal data [financial records; scans of all important records (actually far more than you need to keep);  pictures; etc.].    But the backups of my UnRAID data are not kept offsite.  They ARE, however, rather robust.  I have 2 primary UnRAID servers -- one for media and one for "everything else".    Together they're a bit over 40TB.    These are both automatically backed up to a 3rd UnRAID server which is at the far end of the house.  In addition, there's a complete set of backups on an additional set of disks that are stored in WeibeTech DriveBoxes which are kept in a UL-certified Class 125 3-hr waterproof and fireproof safe (keeps contents below 125 degrees for up to 3 hrs when exposed to 1925 degree heat).    While not as good as an offsite backup, I feel pretty comfortable that I am FAR better backed up than the vast majority of UnRAID users  :) :)  [And FWIW, we live less than 2 minutes away from the nearest fire station ... so I suspect my 3 hr safe is plenty to protect against fire damage.]

 

... and EVERY disk on all of the UnRAID servers and on the extra backup disks has an MD5 checksum stored with it, so I can easily run a verification to confirm whether or not there has been any corruption of the file.

 

In addition, EVERY computer I have is protected with a good UPS.    So the likelihood of, for example, losing 2 drives at a time on more than one of the UnRAID boxes, is VERY low.

 

I stand by my comment that I have a VERY low tolerance for risk ... I think my backup strategy supports that comment  :)

 

Gary

Being a noobie and lacking any real expertise I need to know how i can get this done.

 

These are both automatically backed up to a 3rd UnRAID server

 

Tom hasn't expressed any interest to do this.

I have a 24 drive system and want to buy another 24 drive system and have a differential or incremental backup done everynight

 

Stew

Link to comment

Gary

Being a noobie and lacking any real expertise I need to know how i can get this done.

 

These are both automatically backed up to a 3rd UnRAID server

 

...

I have a 24 drive system and want to buy another 24 drive system and have a differential or incremental backup done everynight

 

 

I use a schedule task on a Windows PC and SyncBack SE to do the backups.  SE and Pro are paid versions, but the free version will work just fine  [ http://www.2brightsparks.com/freeware/freeware-hub.html ]

 

The scheduled task runs a simple batch file that does the following:

 

(a)  Sends a WOL packet to turn on the backup server

(b)  Waits 2 minutes (plenty of time for the server to boot ... time yours to see if that's enough for it)

©  Starts SyncBack to run the backup profiles [i have several profiles that all run in a "Group" profile (If you get Syncback, you'll know what that means)

(d)  Turns off the backup server

 

This, of course, requires that you have another PC that's always on (although it can be in sleep mode, with the task scheduled to wake up the PC when it runs).

 

You could do this same basic thing with a Cron job under UnRAID using WOL and RSync, but I'm not a "Linux guy" so I can't help with setting that up.  I suspect others would if asked nicely  :)  I REALLY like SyncBack, which has a myriad of options that make it very simple to make the backup job work exactly like you want it to.

 

Link to comment

Very good write-up, wiki-worthy!

 

The necessity of backups can't be stressed enough. I know a guy who lost all record and photographs of the first five years of his son's life due to a HDD failure - he didn't have backups, and they were gone. I can't even imagine how that made him feel.

 

And this is exactly why there are clean rooms and data recovery services.  Yes it will cost an arm and a leg but I've had to send drives out for corporate executives 4-5 times in my early career and EVERY time the data recovered was near 100%.  I hope your friend realized these services were available.  $1k would be a very small price to pay for that stuff!  :'(

Link to comment

Partly it's a real pain.

 

Annually, we bring a server in the car (that's the painful bit) and then sync it using Beyond Compare (which is a stunning piece of software).

Monthly, we use Beyond Compare to copy all new/modified files/folders since the last sync date to a new disk and post that.

 

I dare say we could do it with a VPN but our internet lines are maxed out 24/7,  mine with movies, and his with TV series.

 

Other methods considered, have been replacing a disk/rebuilding parity on the donor machine and posting the pulled disk. Popping the pulled disk into the recipient machine and doing a new config and rebuilding parity.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.