JamesAlexander Posted December 24, 2019 Share Posted December 24, 2019 (edited) Hi, I'm in the process of building a NAS at home for my storage needs. I take a lot of photos and want redundancy to prevent file loss. It would also be great to have a media server and backup my smartphone remotely. Now, some questions. How does Unraid save data exactly? Does it use the principle of drive pooling? Can a single drive be taken out and read like a normal one? How is data loss prevented? What advantages would it have over Stablebit Drivepool? Thank you in advance for your insights. Edited December 24, 2019 by JamesAlexander Quote Link to comment
trurl Posted December 24, 2019 Share Posted December 24, 2019 I'll answer in a slightly different order since it makes more sense that way. 29 minutes ago, JamesAlexander said: Can a single drive be taken out and read like a normal one? Yes each disk is an independent filesystem containing complete files that can be accessed on any linux. 29 minutes ago, JamesAlexander said: How does Unraid save data exactly? Does it use the principle of drive pooling? Each disk is an independent filesystem, so each disk contains complete files. But folders can span disks, and these spanned folders are presented as network shares. 31 minutes ago, JamesAlexander said: How is data loss prevented? 1 (or 2) parity disk allows Unraid to calculate the data for 1 (or 2) missing disk to be calculated from parity plus all remaining disks. Here is the wiki on single parity: https://wiki.unraid.net/UnRAID_6/Overview#Parity-Protected_Array Dual parity is similar but the parity2 calculation is independent and somewhat more complicated. But, parity is NOT a substitute for backups. Plenty of ways to lose files besides disk failure, including simple user error. You must always have another copy of anything important and irreplaceable. 38 minutes ago, JamesAlexander said: What advantages would it have over Stablebit Drivepool? Not familiar with that solution so I will just mention some advantages Unraid has over RAID. In addition to each disk being an independent filesystem which can be accessed by itself on any linux, Unraid also allows different sized disks to be mixed in the parity array, and you can easily replace or add disks without rebuilding the entire array. The tradeoff vs RAID is exactly that each disk is an independent filesytem, there is no striping so each file can only be accessed at single disk speed instead of striped RAID where multiple disks are accessed in parallel. Writes are actually somewhat slower than single disk speed since Unraid maintains parity in realtime. There is also cache or cache pool, optionally SSDs, which can allow faster I/O, including writes which are later moved to the slower parity array. 1 Quote Link to comment
lotetreemedia Posted December 24, 2019 Share Posted December 24, 2019 6 hours ago, JamesAlexander said: Hi, I'm in the process of building a NAS at home for my storage needs. I take a lot of photos and want redundancy to prevent file loss. It would also be great to have a media server and backup my smartphone remotely. Now, some questions. How does Unraid save data exactly? Does it use the principle of drive pooling? Can a single drive be taken out and read like a normal one? How is data loss prevented? What advantages would it have over Stablebit Drivepool? Thank you in advance for your insights. Did a little write up w while back hope it helps: https://www.makeuseof.com/tag/unraid-ultimate-home-nas/ 1 Quote Link to comment
JamesAlexander Posted December 24, 2019 Author Share Posted December 24, 2019 (edited) @techsperion, thank you for all this very useful information. Would I be correct in saying that Unraid stores a given file as copies on multiple drives? Also, is it possible to skip copies or redundancy for some unimportant files that I can afford to lose? Edited December 24, 2019 by JamesAlexander Quote Link to comment
JamesAlexander Posted December 24, 2019 Author Share Posted December 24, 2019 5 hours ago, techsperion said: Did a little write up w while back hope it helps: https://www.makeuseof.com/tag/unraid-ultimate-home-nas/ Currently reading. Quote Link to comment
JonathanM Posted December 24, 2019 Share Posted December 24, 2019 50 minutes ago, JamesAlexander said: Would I be correct in saying that Unraid stores a given file as copies on multiple drives? No. There is only one drive that holds any particular file. Quote Link to comment
itimpi Posted December 24, 2019 Share Posted December 24, 2019 50 minutes ago, JamesAlexander said: Would I be correct in saying that Unraid stores a given file as copies on multiple drives? Also, is it possible to skip copies or redundancy for some unimportant files that I can afford to lose? No - a file is only ever stored in one place. Redundancy against hardware failure (in particular disk failures) is a separate topic and functions independently of individual files. Quote Link to comment
lotetreemedia Posted December 24, 2019 Share Posted December 24, 2019 @techsperion, thank you for all this very useful information. Would I be correct in saying that Unraid stores a given file as copies on multiple drives? Also, is it possible to skip copies or redundancy for some unimportant files that I can afford to lose? Unraid stores a single file. However there’s enough data on the parity drive alone to recover that file if it got destroyed as part of a drive failure. I’m not familiar with ignoring files but there’s some things to note: 1) you could insert a drive and not add it to the array. Then store all your not so useful files here. I believe you can do this as part of the unassigned devices plugin. 2) the cache is also a drive that you can choose to have unprotected files on. When you create a share you choose weather you want unRAID to write to the cache first then move it to the protected array later. Or skip the cache altogether, or only use the cache. There’s way more experienced people that patrol this forum who will have better insight than me Quote Link to comment
S80_UK Posted December 24, 2019 Share Posted December 24, 2019 (edited) 1 hour ago, techsperion said: Unraid stores a single file. However there’s enough data on the parity drive alone to recover that file if it got destroyed as part of a drive failure. Sorry - but the second sentence is not correct and is actually misleading. @trurl already put it very succinctly... 13 hours ago, trurl said: 1 (or 2) parity disk allows Unraid to calculate the data for 1 (or 2) missing disk to be calculated from parity plus all remaining disks. Here is the wiki on single parity: https://wiki.unraid.net/UnRAID_6/Overview#Parity-Protected_Array Dual parity is similar but the parity2 calculation is independent and somewhat more complicated. To rewrite your original statement... However there’s enough data on the parity drive in conjunction with the data on all of the the other data drives which do not contain that file to recover that file if it got destroyed as part of a drive failure. Edited December 24, 2019 by S80_UK Typo corrected Quote Link to comment
BRiT Posted December 24, 2019 Share Posted December 24, 2019 To be pedantic, it can not recover at the file level, it is at the device level that just happens to contain files. Unraid and Parity is not able to recover previous versions of files. One still needs to plan for backups accordingly. 1 Quote Link to comment
lotetreemedia Posted December 24, 2019 Share Posted December 24, 2019 To rewrite your original statement... However there’s enough data on the parity drive in conjunction with the data on all of the the other data drives which do not contain that file to recover that file if it got destroyed as part of a drive failure. Ah so it’s like traditional RAID in that sense. You learn something new every day ta! Quote Link to comment
trurl Posted December 24, 2019 Share Posted December 24, 2019 4 hours ago, techsperion said: However there’s enough data on the parity drive alone to recover that file And if you think about it for a moment, it would not be possible for parity by itself to recover a file from any possible disk that might fail. For that to happen, parity would effectively be a backup of the complete array. There is no magic that would allow it to have the capacity for that. 1 Quote Link to comment
JamesAlexander Posted December 25, 2019 Author Share Posted December 25, 2019 (edited) 12 hours ago, jonathanm said: No. There is only one drive that holds any particular file. Ok, so this is like software RAID5? This is where parity data fits into the picture? 11 hours ago, techsperion said: Unraid stores a single file. However there’s enough data on the parity drive alone to recover that file if it got destroyed as part of a drive failure. I’m not familiar with ignoring files but there’s some things to note: 1) you could insert a drive and not add it to the array. Then store all your not so useful files here. I believe you can do this as part of the unassigned devices plugin. 2) the cache is also a drive that you can choose to have unprotected files on. When you create a share you choose weather you want unRAID to write to the cache first then move it to the protected array later. Or skip the cache altogether, or only use the cache. There’s way more experienced people that patrol this forum who will have better insight than me Right, so there is a way to store but not protect against drive failure. I will find these more experienced people. 10 hours ago, S80_UK said: To rewrite your original statement... However there’s enough data on the parity drive in conjunction with the data on all of the the other data drives which do not contain that file to recover that file if it got destroyed as part of a drive failure. How does the data on each drive on top of parity function? How many drive failures can the system handle? One or more? I suppose this depends on what resources are available for parity? 9 hours ago, BRiT said: To be pedantic, it can not recover at the file level, it is at the device level that just happens to contain files. Unraid and Parity is not able to recover previous versions of files. One still needs to plan for backups accordingly. My question comes from what I understand of drive pooling which is just file level copies on separate drives. So no versioning like with Freenas/ZFS. I have dropped FreeNAS because it is apparently necessary to setup the final system and it is also difficult to grow storage. Unraid seems more lightweight and flexible. 7 hours ago, trurl said: And if you think about it for a moment, it would not be possible for parity by itself to recover a file from any possible disk that might fail. For that to happen, parity would effectively be a backup of the complete array. There is no magic that would allow it to have the capacity for that. I will read more about parity. This is a kind of error detection? Edited December 25, 2019 by JamesAlexander Quote Link to comment
lotetreemedia Posted December 25, 2019 Share Posted December 25, 2019 I will read more about parity. This is a kind of error detection? RAID5 splits the data over the array. A single file can belong on many drives. That’s not the case with Unraid. Unraid stores a single file On a single drive. Your array can suffer the loss of however many parity drives you have. Scenario: you have a total of 5 drives with the parity drive equal to or larger in size than the largest drive.Eg:Parity >= 3TBDrive1 = 3TBDrive2 = 3TBDrive3 = 3TBDrive4 = 3TBIf one drive dies and you slot in another, you’ll recover your data.If you have 2 parity drives:Parity1 >= 3TBParity2 >= 3TBDrive1 = 3TBDrive2 = 3TBDrive3 = 3TBYou can lose two drives.Parity in a nutshell is a mathematical computation of the data stored across the array. It uses just binary but to make it easier to understand:If you have an equation:1 + 1 + 1 + x = 3What will x be equal to? I’m over simplifying it but you get the picture. Just Imagine those as your drives.FreeNAS can grow the array size but it’s recommended that you do all the drives which is expensive. So if you had the scenario above you’d unplug the first drive, then insert a new larger one and let the array resilver / rebuild. Then unplug the next drive, plug in a larger one and let the array resilver. Rinse and repeat.In contrast Unraid doesn’t really care that you have mixed drive sizes, as long as your parity is your largest drive you’ll be golden. A lot of this is covered in the article I published above Quote Link to comment
lotetreemedia Posted December 25, 2019 Share Posted December 25, 2019 Also you may have come across this statement before:RAID be it on FreeNAS / Unraid / the millennium falcon is not a substitute for a good backup strategy.RAID is more of a high availability / performance thing. If you have a treasured repository of cat memes that you absolutely cannot lose, make sure you have thought about on-site / off-site backups.Parity must not be relied on to keep your data safe. Sorry to come across as a little alarmist but this must be said Quote Link to comment
trurl Posted December 25, 2019 Share Posted December 25, 2019 4 hours ago, techsperion said: as long as your parity is your largest drive Doesn't have to be the largest, each parity drive has to be at least as large as the largest single data drive. Quote Link to comment
trurl Posted December 25, 2019 Share Posted December 25, 2019 9 hours ago, JamesAlexander said: Right, so there is a way to store but not protect against drive failure. I will find these more experienced people. Don't really understand that first sentence. And it was in response to a quote that had some things wrong as far as how parity works. As for those more experienced people, some of them have already replied on this thread. 9 hours ago, JamesAlexander said: I will read more about parity. This is a kind of error detection? Parity can detect and correct a single bit error. Dual parity can detect and correct 2 bit errors. Parity is very common in computing and communications, and it is always conceptually the same though the specific implementation may differ in some respects. If you have a group of bits, determine if their sum is even or odd then include an additional bit that reflects that even or odd. Then if a bit is missing you can figure out what it was based on whether it should be even or odd. I already posted a link to the wiki on parity above. Quote Link to comment
trurl Posted December 25, 2019 Share Posted December 25, 2019 And I missed this one 22 hours ago, jonathanm said: No. There is only one drive that holds any particular file. 9 hours ago, JamesAlexander said: Ok, so this is like software RAID5? This is where parity data fits into the picture? No it is not like RAID5, what he was describing is one of the things that makes Unraid very different from any RAID. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.