Backing up your data

[Juscuz17]

What is the standard back up procedure when you build an Unraid machine (or any NAS build)? I have parity, I am probably going to add double parity. The real question is when you have a 36TB NAS, do you build a second 36TB NAS to back it up? I keep reading on here that full back up is best, and I get that, but aren't we building these setups to have one big dependable storage area?

[Vr2Io]

2 hours ago, Juscuz17 said:

do you build a second 36TB NAS to back it up?

Yes, because parity mainly used to against disk failure only. With backup, I even think I don't need parity, I will turn parity to data disk if I run out of space. ( But I won't build in separate NAS, just have full backup ) 

Edited August 11, 2023 by Vr2Io

[madmin]

To my understanding:

 

Backups protect from data loss. If you delete finally or overwrite a file, you can restore it from a backup. But if your file was corrupted by a disk error or the like, you would not know, and with the next backup, you would update an intact file with the corrupted one - so the intact file would be gone (That's why there are strategies to keep multiple backups, like 3 of the past days, 1 from the past week, and one from the past month, in case you detect the problem). Depending on backup timing, you could also lose data from corruption that has not been backed up yet (e.g. files from that day before a nightly backup).

 

Parity or RAID would detect such data corruption and restore the file immediately from is twin (before tge backup is updated with corrupted data). So it's especially critical to secure real time monitoring data against disk failures. But if you yourself, or a ransomware, deletes, overwrites, or encrypt the file, this would be mirrored immediately to the twin - so the files would be lost (without a backup to restore it from).

 

That's why critical data should be protected by both. If I can afford only one for securing private data, I would pick backups, as a rule of thumb.

 

But with the next backup, deleted/ overwritten files would be lost in the backup, too, though. Ransomware or local catastrophes might even destroy a local backup directly. So if your serious about it, it makes sense to also keep a backup at a remote place,in longer cycles...

[dboonthego]

A good back up (preferably stored in another location) will save you from theft, natural disaster, or doing something stupid like writing a script to move all your files and forget the trailing slash/ in the destination.

[Lolight]

On 8/11/2023 at 12:41 AM, Juscuz17 said:

The real question is when you have a 36TB NAS, do you build a second 36TB NAS to back it up?

Multiple backups (3-2-1) is a preferred and recommended strategy only for your irreplaceable or hard to replace data which usually consists just a small portion of a common consumer NAS system.

[Michael_P]

22 hours ago, madmin said:

Parity or RAID would detect such data corruption and restore the file immediately

 

Doesn't work like that, Parity has no idea what a 'file' is and wouldn't 'know' there's a difference in bits until a scan is done and if the parity calculation differs, it'll tell you something doesn't add up, but it won't know which is correct so it'll just fail the disk. Same with RAID, it'll just mirror the corrupted data.

 

Parity/RAID is for rebuilding a faild disk. If you need data integrity, you need checksums and backups

[madmin]

28 minutes ago, Michael_P said:

Parity/RAID is for rebuilding a faild disk.

Ok, sorry for the misunderstanding. In this case, I see even less sense in it,  in comparison to a backup (you can rebuild from backup, too, so it seems top be just a matter of backup frequency: Parity/RAID for instant mirroring of immediate data, including all mistakes I make, vs. backup for delayed mirroring of slowly changing data that can revert mistakes I make).

 

This actually makes me reconsider if I really need a parity disk for personal data use...

[binhex]

Ok, sorry for the misunderstanding. In this case, I see even less sense in it,  in comparison to a backup (you can rebuild from backup, too, so it seems top be just a matter of backup frequency: Parity/RAID for instant mirroring of immediate data, including all mistakes I make, vs. backup for delayed mirroring of slowly changing data that can revert mistakes I make).
 
This actually makes me reconsider if I really need a parity disk for personal data use...

Parity is still worthwhile as it allows you to recover from physical hard drive failure with no data loss, but parity is not backup as you have found out.

Sent from my 22021211RG using Tapatalk

[Frank1940]

What I do when I make a backup of the personal data files is to make a dated backup in a separate folder/directory each time (~ 1st of each month) to my Unraid server.  Then periodically, I will go through and delete selected folders as they age.  Eventually, I have about one folder per calendar year where that year is about ten years in the past. 

 

I define 'personal data files' as those files whose content was generated by user and is generally unique to that user.   That data might also be regularly updated with new information.  It is  generally dynamic.  

 

Items like personal pictures are static data.  While the collection might be added to, each individual item is generally unchanged as time passes.   I move these items to a more static backup Unraid location.  For those items, I do make make periodic backups of that Unraid location for an 'off-site' storage.  The monthly personal data files are included in this off-site transfer.   (I use USB HDs as the media in this case.  I always have two drives in that offsite location and one drive on site.)  

[madmin]

7 minutes ago, binhex said:

Parity is still worthwhile as it allows you to recover from physical hard drive failure with no data loss

Yes, I think it comes down to whether securing the data loss (since the last backup, typically daily, maybe even less) is worth the extra drive.

 

Personally, I worry more about losing my complete history before that than about losing a day's spare time work. Then again, everybody's use case and  financial priorities are different...

[Michael_P]

Parity/RAID is strictly for high availability, so you can continue using the system while the failed disk(s) rebuild. There is absolutely zero data security, as it can be deleted/corrupted/changed at any time. 

 

Figure out what's really important and 3-2-1 back up that. 

[Frank1940]

1 hour ago, Michael_P said:

There is absolutely zero data security, as it can be deleted/corrupted/changed at any time. 

 

This is so true as there is the additional possibility that the server could be destroyed fire, flooding, lightning, or theft to mention a few scenarios beyond the mere failure of a hard drive...

[madmin]

2 hours ago, Michael_P said:

Parity/RAID is strictly for high availability, so you can continue using the system while the failed disk(s) rebuild.

That was gold - I much better understand now what its purpose is (and what it is not). Thanks!

[Juscuz17]

Thanks for all the responses! I really only need to back up stuff I can't replace. I was stuck on I don't want to lose everything, but all my media can be found again if it was lost.  My pictures and any documents don't take up much space, that's easy to manage a 3-2-1 back up.

 

I also plan to go double parity. Another drive is not that expensive, and I will be covered if a second drive fails.

[ConnerVT]

There is a lot of wisdom in this thread.

[kizer]

Personally and I do mean personally. I've been using unraid since 2009 and I have had 3 builds using the same USB. The only thing I have 2 of is cases. Everything else has been swapped in and out of 1 case.  Motherboard, ram, yadda yadda. So what I did I picked through my pile of goodies and slapped together a second system and purchased 3 x 12TB drives and I'm still using my 6 x 4TB drives in my primary because I can swap them in and out as I need and expand their sizes when ever I need. That doesn't benefit my Backup server stuck at 24TB, but none the less I'm not full yet so I'll have to deal later. 

 

Every day my some files on my Cache are backed up to another SSD in the same machine with a script I wrote which is nothing fancy, but it does what I need to it to. Also every day at 8AM when I know Plex is done doing all of its normal stuff my backup server connects via unassigned Devices to my Primary, backups all my media and other various files then disconnects. Its been running flawlessly for months. I've been meaning to change up the code some so when I turn on the Backup server it performs the backup and shuts its self off, but I keep forgetting. I intend to relocate my backup else where out of the house possibly into the garage hence the turning it on so it doesn't sweat to death or freeze or I might take it out to the shed we are building and insulating. I'm guessing it will end up in the shed honestly since it will be in a better environment temperature wise. 

 

None the less save those old builds. You don't need lots of horsepower to do a backup especially if your primary is your media server or whatever its doing for you.