Recommended Posts

  • Replies 117
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Clearly a lot of people do that; but some of us only rip our personal DVDs (I even have a couple hundred I ripped from old VHS tapes) or legal recordings of movies from TV.    While even that is of questionable legality (different courts have ruled differently); there's NO doubt that it's not legal to provide copies to others.

 

For those who get their content via illegal downloads, clearly they don't care about the legality of sharing this content with friends.  I'm well aware that a lot of UnRAID collections are built in this way.

 

 

Link to comment

You would be astounded at what goes on in Thailand.

 

There is a company listed on the Thai stock exchange with stores in every major city.  All they sell are pirated VCD's, DVD's and Blu-Ray's.  The printing on the boxes is low resolution, although usually the discs themselves are perfect. 

 

You raise an interesting point.  If you previously bought a VHS tape, you have a non-exclusive licence to the movie.  I wonder what the legal situation would be if you downloaded a high definition version of it.

Link to comment
  • 2 weeks later...

After reading this thread and thinking about the data I have that's important to me (music files) I would like to start to backup my drives.  Is there a way to plug an external HDD directly to the chassis where the unRAID system resides and copy the drives locally?  I think I would rather do this than copy the files over the network.

Link to comment
  • 1 month later...

Great stuff Gary.

 

I'm in the process of doing exactly the same for my data.  I have to admit I've been a slack in keeping my data safe, but 2 drives went down without warning, making recovery difficult (one even passes a SMART test! and the other only a minor issue) in my PC last week, and luckily I had backups of the important stuff.  Still it was a tense few days, and I've purchased a new backup server so that I now have 2 servers (unraid and Synology) to backup to each other, and then a further backup and offsite backup.

 

It might seem OCD to some, but it's insurance against losing photos, and important documents. :)

 

Thanks for the checksum info, I'll get on to that next because I was wondering how to go about checking the data integrity! :)

Link to comment

It might seem OCD to some, but it's insurance against losing photos, and important documents. :)

 

Thanks for the checksum info, I'll get on to that next because I was wondering how to go about checking the data integrity! :)

 

 

If you can organize the files into folders that change and do not change, you can use par2 to protect the non changing files also.

it works like a hash but a little better. The hash will tell you there is change, the par2 file will allow you to fix bit rot or recover missing files.

This is up to a selected specific percentage of recovery.

 

 

hash tells you something is wrong.  Par2 tells you it's wrong and recovers.

Link to comment

Par2 does seem to provide some good recovery capabilities, but aside from the fact that it gives you a high probability of recovery without the need for total replication, is there any reason you suggest using it instead of full backups?    At the very modest cost of drives these days, it seems like a full backup is a decidedly better alternative ... especially if you maintain MD5s on all of your files (and all of the backups).

 

Just seems like Murphy would almost guarantee that the first time you had significant data loss it would happen to be something that your par2 couldn't fix  :)

 

Link to comment

Par2 does seem to provide some good recovery capabilities, but aside from the fact that it gives you a high probability of recovery without the need for total replication, is there any reason you suggest using it instead of full backups?    At the very modest cost of drives these days, it seems like a full backup is a decidedly better alternative ... especially if you maintain MD5s on all of your files (and all of the backups).

 

Just seems like Murphy would almost guarantee that the first time you had significant data loss it would happen to be something that your par2 couldn't fix  :)

 

I did not say, or mean to imply, that full backups should not be done.

 

you can use par2 to protect the non changing files also.

 

Point is par2 can do the hash/checksum and recover from a small percentage of bitrot, mistaken changes, data loss or corruption.

This would even protect your backups. if let's say a few sectors cannot be read.

 

While you can maintain md5's of the files, you cannot recover from them, with par2 you can detect (as in the md5) and recover from a small percentage of corruption at the same time.  I'm considering to contact the author of corz and see if he'll add a par2 option.

Link to comment
  • 1 month later...

I don't like the idea of losing any of the data I've spent so many years collecting ... whether it's family pictures, financial records, or just all the music and movies I've collected over the years.    When you factor in the cost and time involved in that, a couple thousand bucks to have it VERY well backed up is really no big deal.

 

I am just blown away by this sentence. This is just how I feel. LOL - and until now I wasn't sure I did! It has just cemented my plan for a backup server! I don't have the money to keep it off site like I would like but baring some act of god or some idiot with a grudge against my country I'll feel safe!

Link to comment
  • 5 weeks later...

Thanks for the information on Backups.

 

I'm planning on building a new unRAID box as a backup, and a second Pro license. I'm wondering if it's a good idea to use the product below to duplicate all my disks first, around 10 3TB HDD's:

 

http://www.newegg.com/Product/Product.aspx?Item=0Y5-0001-00014

 

I'd love to keep a backup of my UNraid server and the OP's original suggestions made sense but by the time I'm done finishing doing a complete back up of all my 30-6=24TB on  DVD's I would be a very old man. If I really wanted to do it right, I would just make a duplicate UNraid server and sync them together with some kind of automation process. I have been lucky to only lose single hard drives at a time.

 

Link to comment

...  by the time I'm done finishing doing a complete back up of all my 30-6=24TB on  DVD's I would be a very old man ...

 

Not sure why you think that.  24TB isn't really all that much these days ... on 4 6TB drives.    A small 2nd UnRAID server with that much (or more capacity); and it takes ~ 5 minutes of "your time" to backup your entire collection [Clearly the copy would take a few days cross the network, but that doesn't require any action on your part except to initiate the copy.]

 

Link to comment

...  by the time I'm done finishing doing a complete back up of all my 30-6=24TB on  DVD's I would be a very old man ...

 

Not sure why you think that.  24TB isn't really all that much these days ... on 4 6TB drives.    A small 2nd UnRAID server with that much (or more capacity); and it takes ~ 5 minutes of "your time" to backup your entire collection [Clearly the copy would take a few days cross the network, but that doesn't require any action on your part except to initiate the copy.]

 

I tend to agree here.  you can trigger an rsync over the "local" network and let it fly, check on it every day and just let it fly.

You can calculate how long it takes with a best of case of around 90MB/s.  With the right tuning of kernel and rsyncd.conf files along with the proper command line, this will be faster and easier then realized.

 

The trick is to tune the kernel for maximum buffering on the writes and open the TCP buffer/windows with the sockopts settings.

 

I've moved 4TB overnight with ease.

With rsync you can stop and restart the rsync and it will only rsync the files that have changed.

Link to comment

...  by the time I'm done finishing doing a complete back up of all my 30-6=24TB on  DVD's I would be a very old man ...

 

Not sure why you think that.  24TB isn't really all that much these days ... on 4 6TB drives.    A small 2nd UnRAID server with that much (or more capacity); and it takes ~ 5 minutes of "your time" to backup your entire collection [Clearly the copy would take a few days cross the network, but that doesn't require any action on your part except to initiate the copy.]

He's talking about backing up onto actual DVD's.  Would take forever.  Beyond that, its actually cheaper to back it up onto other hard drives.

Link to comment

...  by the time I'm done finishing doing a complete back up of all my 30-6=24TB on  DVD's I would be a very old man ...

 

Not sure why you think that.  24TB isn't really all that much these days ... on 4 6TB drives.    A small 2nd UnRAID server with that much (or more capacity); and it takes ~ 5 minutes of "your time" to backup your entire collection [Clearly the copy would take a few days cross the network, but that doesn't require any action on your part except to initiate the copy.]

He's talking about backing up onto actual DVD's.  Would take forever.  Beyond that, its actually cheaper to back it up onto other hard drives.

 

 

Duah, thanks for the clarification!

Link to comment

I've seen some people suggest using rsync to set an automated backup of your unraid box to another unraid box (or other linux box).

 

If you use an automated rsync process isn't there potential for corruption to be mirrored into the backup?

 

If there are goals in having a backup are to protect against the following right?

 

[*]Software Errors – File system corruption or bit rot or some other sort of internal error which silently (or not so silently) corrupts your data. 

[*]Hardware Failure – Protect against hard disk or other mechanical error

[*]Human Error – Protect against accidental deletion or other human initiated mistakes which destroy your data.

 

My concern with automated rsync processes is that that rsync will update files if they have changed from an earlier rsync run. If the reason the file has changed is silent corruption won’t that silent corruption be copied into the backup diminishing your ability to recover?

Maybe rsync doesn't work that way... or maybe this can be avoided by properly configuring rsync?

 

Any advice on how to properly do this would be much appreciated, because I've def seen people use the --delete command (which deletes files on the destination which don't exist on the source) Which totally removes the ability to recover from human error should you not catch your mistake before your automated process runs. Any other pitfalls to avoid?

 

Link to comment

With rsync, files are usually compares by size & modification time.

So silent corruption may not cause the files to be rsynced to the backup.

 

There's another strategy with the -c option which does a checksum compare rather then size/time.

That would surely propagate the silent corruption.

 

Accidental deletion protection. 

In this case, for crucial files that need to be kept over a time period or 'deltas' there is the --link-dest option.

 

This can be done locally disk to disk, or it can be done remotely if the remote server 'pulls' the files.

With this option you can link the current destination directory to new name. All files recursively are linked from an old directory to a new directory before the rsync is executed.

The rsync is then executed on the source to the new name. If there are any chances they are copied over. If there are no changes at all, the new directory looks like the old directory.

 

This has the benefit of using 1x the space for the whole tree, then the space required for each file's change.

I've used this to mirror source trees and carry deltas. It can be done daily, hourly, monthly whatever is choosen.

You do this by managing the source directory name and destination directory name on the backup volume.

 

This article has a good description of the process.

http://goodcode.io/blog/easy-backups-using-rsync/

Link to comment

With rsync, files are usually compares by size & modification time.

So silent corruption may not cause the files to be rsynced to the backup.

 

There's another strategy with the -c option which does a checksum compare rather then size/time.

That would surely propagate the silent corruption.

 

Accidental deletion protection. 

In this case, for crucial files that need to be kept over a time period or 'deltas' there is the --link-dest option.

 

This can be done locally disk to disk, or it can be done remotely if the remote server 'pulls' the files.

With this option you can link the current destination directory to new name. All files recursively are linked from an old directory to a new directory before the rsync is executed.

The rsync is then executed on the source to the new name. If there are any chances they are copied over. If there are no changes at all, the new directory looks like the old directory.

 

This has the benefit of using 1x the space for the whole tree, then the space required for each file's change.

I've used this to mirror source trees and carry deltas. It can be done daily, hourly, monthly whatever is choosen.

You do this by managing the source directory name and destination directory name on the backup volume.

 

This article has a good description of the process.

http://goodcode.io/blog/easy-backups-using-rsync/

 

I want to say thank you so much for this! Really good stuff! The link is also awesome and set me on the right path.

My only problem is that My wife and I live in a small apartment, (just starting out), so I'm not sure that I can convince her that a second backup server is what we need. She's onboard with the primary one.  I could possibly put a remote server in at my parent’s house... but we'll have to see about that as well. Also I assume there are a few additional steps when performing back-ups across the internet?

 

Link to comment

You don't have to have a second backup server. It can be a another disk that is mounted on demand in the script.

 

rsync from one location to the other with the link-dest and a dated directory.

It can be a usb mounted disk as well, but you'll want it to be a usb 3 or eSATA disk for speed.

 

As far as internet access.

Yes, you might have to set up ssh with a tunnel in order to encrypt the data and provide remote access to the other server.

using ssh and port tunneling you can even proxy the webGui over the ssh tunnel, but that's a more involved process for another thread.  Might be better off setting up a VPN.

 

 

Link to comment

 

Also I assume there are a few additional steps when performing back-ups across the internet?

 

You would need to setup a VPN at your in laws house so that you could connect to their network and backup your files. A VPN will give you a secure connection. Also your backup transfer speed will be dependent on your homes upload speed. So it would be best to make the first backup at your home then move the server to your in laws.

Link to comment

Thanks again for the advice, I feel like I'm moving in the right direction now.

I guess I need to figure out if I want to have a second unraid box or if I don't have enough data to warrent that yet and back up via an USB3 HDD attached outside of the array.

 

Oddball question. Should you backup \flash ? In the event of a usbdrive failure I realize that a new key will be required (I have a backup of that anyway... because why not...) but is there any other settings or files which having a backup would help the process of getting up and running again? If not the whole flash are there important files we should be looking at backing up?

Link to comment

Oddball question. Should you backup \flash ? In the event of a usbdrive failure I realize that a new key will be required (I have a backup of that anyway... because why not...) but is there any other settings or files which having a backup would help the process of getting up and running again? If not the whole flash are there important files we should be looking at backing up?

 

 

I rsync my flash periodically as there are some configuration files, plugins, extras, etc, etc.

In addition it has the superblock which is the layout of the array itself.

Keep in mind it might be out of date from when you've last made a change to the array, so you'll want to do the rsync/backup anytime you change the array.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.