New to NAS - how does the data security of unRAID compare to solutions with checksum before writes to parity?

[RiverElf]

Hello there!

 

I'm new to NAS and I'm currently looking into the different server software solutions that exist. In my research thus far unRAID looks very appealing to me.

After having read about it, one thing I'm a bit concerned about are silent errors - the so called bitrot. As I understand it, solutions that checksums data before it is written to parity have more protection against this type of data corruption? In a worst case scenario I guess one could also unknowingly copy bitrot to ones backup drives? However, I'm not sure how big the risk of bitrot is? I don't think I've ever encountered it on my desktop PC, and I'm not sure if the problem is overstated? I mean, if a high percentage unRAID users experienced bitrot, then these forums wouldn't have so many happy and active users, would they? If you could help me understand how unRAID's data security compares to solutions which use checksums before data is written to parity and use the so called scrubs - I would very much appreciate it.

 

I found this comparison: http://www.snapraid.it/compare.html

 

Thanks for your help!

 

 

PS: If this thread is in the wrong forum please feel free to move it.

 

Edited March 3, 2017 by RiverElf
Edited for clarity

[garycase]

There's a plugin for UnRAID [Dynamix File Integrity] that will automatically generate checksums for your data;  or you can use any of several other checksum generation utilities to generate them yourself.

 

Note, however, that a checksum simply lets you identify that you have a corrupted file -- it does NOT correct that file.

 

No matter which NAS system you choose to use, remember that a RAID system is NOT a substitute for backups.    THAT is the real solution to bitrot (and any other problem that results in data corruption/loss).    Checksums are of course still a good idea, as they'll let you know when a file is corrupted -- and you can then restore it from a backup.

 

[Frank1940]

In my opinion, 'bitrot' is the biggest FUD (Fear, Uncertainty, Doubt) none-issue out there!  The read error rate on hard drives, while a very, very  small percentage of all reads, probably happens on most hard drives several times a day.  This fact has been known for years.  Drives manufacturers has implemented several layers of data error detection and correction to find and correct most of them.  If you look at the SMART attributes ( say, at Wikipedia), you will see that they actually flag some sectors as unreliable and schedule them for action at some point in the future.

 

Having said all of that, Is there a possibility that so many errors occur in a data block that the detection scheme erroneously assumes that the data is correct data ? Absolutely!!!!  That is why some knowledgeable folks have a secondary (and, in some case, tertiary) backup(s) of the really important stuff that can't be restored by other means. Graycase mentioned the file checksum utility.  Depending on your level of paranoia, that will provide you with a level of confidence that you haven't had any bitrot occur. But your backup is your safety security blanket. 

 

But the five plus years that I here been watching this board, I have have not heard of a single case of even suspected bitrot....

Edited March 3, 2017 by Frank1940

[garycase]

5 hours ago, Frank1940 said:

But the five plus years that I here been watching this board, I have have not heard of a single case of even suspected bitrot....

 

Agree -- I've been using UnRAID for ~ 8 years and have been pretty active on this board for about half that time, and have never seen any confirmed instance of data loss due to bitrot.  

 

[RiverElf]

@garycase: Thanks for reminding me of the checksum plugins for unRAID. As you mention they appear to only detect corrupt files, not correct them. The data integrity systems of ZFS and Btrfs and SnapRAID also use checksums for detecting errors, but they appear to also be able to fix these silent errors before they are carried on to parity, and before they could be copied to a backup. I can see that I didn't mention this error correction part of the data integrity system used by some of the other NAS solutions in my original post, although I intended to. The silent error correction feature is what arose my initial concern about bitrot in unRAID vs these other systems. The link in my first post shows a comparison table made by the author of SnapRAID were he points out some of these differences. And yes, I am aware that RAID or parity only protects against drive failure, not theft, fire, user error and so forth - only an off site backup would help in those scenarios. I'm very, very pleased to hear that you haven't heard of any cases of bitrot in your time on this board garycase.  It does appear to be somewhat overstated then.

 

@Frank1940: Thanks for pointing out that error correction features are also built into hard drives. Being new to NAS it's not always easy to know how critically important each feature of the different NAS solutions are. Although I must admit I like the idea of data integrity checks through checksum and silent error correction as used by ZFS, Btrfs and SnapRAID, I am more drawn to unRAID as it has many of the other features I desire, like: to read a single file in unRAID only the storage drive where it is saved must spin up. In ZFS and Btrfs all drives must spin up as all files are striped across the drives. And if redundancy should somehow fail only the data on the faulty drive(s) is lost in unRAID where all data is lost on all drives with ZFS and Btrfs. And then there is the ability to use drives of different sizes in unRAID which is very nice feature for a home user like myself who already has a stack of drives and likes to add drives incrementally. Only SnapRAID appears to have some features similar to unRAID while also having data integrity through checksum and silent error correction before parity. Come to think of it, SnapRAID is also open source, perhaps unRAID could adopt a few of its alluring data integrity features in a future upgrade? Thanks for adding to the choir of happy unRAID users Frank

----------------------------------

Thanks to both of you for taking time to comment! An active forum with users happy to help and welcome newcomers into the flock is so valuable!

PS: I'm working this weekend which is why I'm a bit slow to reply.

[garycase]

While it's possible to include enough extra bits to allow some degree of error correction; anything short of a full backup can NOT correct all errors.    i.e. don't be fooled by the ZFS and Btrfs error correction bits -- you absolutely still need a backup if your data is important to you.

 

[RiverElf]

@garycase: Thanks for pointing that out.  I may have overestimated what these error correction features are able to do. Yes, I do know that backup is important, but I must admit, that I'm not very good at it.  Perhaps it's because I've only done it fully manually. I really should get back at it again, and perhaps limit my backups to the most important data, so I won't frown over the cost of having to buy twice the drive space for all my files. Do you have a some kind of semi-automated backup system that works for you? I'll need to learn more about backup solutions and see which one might be suitable for me. I remember using SyncBackFree for quite a while.

Edited March 5, 2017 by RiverElf

[garycase]

6 minutes ago, RiverElf said:

Do you have a some kind of semi-automated backup system that works for you?

My backups are fully automated.

I have a scheduled task that runs on a Windows machine that effectively does the following (it's more complex than just this, but I'll keep the outline simple):

 

(a)  Turns on the server via WOL

(b)  Turns on the backup server via WOL

(c)  Waits 2 minutes (to allow the servers to boot

(d)  Runs a SyncBack group profile that does all of the backups

(e)  Turns off the servers via ssh

 

I backup my entire server; but you could easily use this process to only back up selective shares if you have data you really don't care if you lose.   As I'm sure you know, SyncBack is very flexible in what you can do with the backup profiles.   I figure anything I consider important enough to keep on a fault-tolerant server is important enough I don't want to lose it.

 

[RiverElf]

Ah, such a fully automated backup system sounds very nice. Yes, I remember SyncBack to be quite flexible, although I only learned to use the features I thought I needed at the time. I have a laptop with two drives and had SyncBack copy certain folders from C -> D just to prevent data loss if I had a drive fail. I remember I could set it up so if I deleted a file in the source folder, it would also delete it in the destination, which is what I wanted. I also had it run as a scheduled task every two hours or so. As far as I remember I copied files manually to an external drive. I'm not sure why I didn't semi-automate that as I could have made a shortcut on the desktop to a profile that would have done that for me when clicked. Perhaps the drive letters changed if I happened to have a usb stick attached before the external drive, well I can't remember right now.

 

Yes, ideally I would want to backup all my data as well. I do a bit photography and video, and those files take up a lot of space. I'd also like to upgrade to a camera capable of 4K sometime in the future, so that will only increase file sizes. If I chose to not backup all my files it would be because of the cost of building a second server, but I'll look into that. I guess one of them could be built as a dedicated server, and the other could be less so. Thanks for your input @garycase

 

By the way, do both of your servers run unRAID?

Edited March 5, 2017 by RiverElf

[garycase]

50 minutes ago, RiverElf said:

I remember I could set it up so if I deleted a file in the source folder, it would also delete it in the destination, which is what I wanted.

I think that's a dangerous choice.   One key advantage of a backup is if you inadvertently delete or modify a file you can still recover it from your backup.   With this choice, if you inadvertently deleted a file (or files) ... and didn't realize it right away ... then when your next backup runs it will delete your backup file as well.    I think a better choice is to manually delete files from your backup that you know you no longer need.

 

53 minutes ago, RiverElf said:

By the way, do both of your servers run unRAID?

Yes.

 

[RiverElf]

Yes, as far as I recall SyncBack also gave a warning when you chose that setting - just to make sure you knew what you were asking it to do. However, at that time what I wanted was a mirror of my folders. I might do it differently today. When Windows XP no longer was supported I installed Ubuntu on the laptop, and I didn't bother to look for a backup/sync tool which could do the same. I wasn't using the computer that much anymore, I thought. However, last year something happened whereafter Ubuntu could no longer boot. And on that laptop I just happened to have the only copy of a spreadsheet with information on who was on the photos I took at a conference and what they were saying. Fortunately over the weekend I learned how to create a rescue drive on a usb stick, which was then able to fix the startup issues. Boy was I relieved. I've also lost some images due to a failing card reader corrupting the photos at both ends while they were being copied to the computer. And on one occasion last year my computer wouldn't read an SD card and wanted to reformat it. After trying out different recovery software I finally found one that was able to recover the images and videos. So these days I'm all for dual card slots in cameras and servers with redundancy. So I ought to implement a proper backup scheme as well.

[CHBMB]

On 03/03/2017 at 6:18 PM, garycase said:

have been pretty active on this board

 

Understatement of the year award goes to @garycase with a post count of 13158..... 

[garycase]

6 minutes ago, CHBMB said:

 

Understatement of the year award goes to @garycase with a post count of 13158..... 

 

[Obi_One]

Hello,

 

I have a question direct at Garycase,:

 

it's seem that you power down your primary server instead of keep it online.

is there a reason that you do that other than to save power usage or was it some other reason ?.

 

I keep my online 24/7 and available at all time. I try to build the server with lowest idle power as I can so I can keep it online all the time.

 

[garycase]

As I noted above ...

 

17 hours ago, garycase said:

(it's more complex than just this, but I'll keep the outline simple):

 

 

I actually have 3 servers involved in the backups -- two main servers and a backup server.    One of the primary servers is on 24/7 (and is a very low power unit that idles at ~ 20 w);  the other one is only on in the evenings; and the backup server is only on when there's an active backup.    So the "real" process is that 3 servers are turned on via WOL  (one of which WILL already be on, and another one MAY already be on, but I send both WOL commands just to be sure);  and two servers are shut down via SSL after the backups are completed.    The backup task runs once/week or "on demand" (by double-clicking an icon on my desktop).

 

I also have a test server that I occasionally backup via a separate task --- but generally it doesn't have any data that really matters, so it's a manual task that I run as needed.