How does Unraid work, exactly?

[EqualEgg]

Hi,

 

I’m trying to understand how Unraid works exactly under the hood. I’ve been looking into different solutions, such as Snapraid, mdraid/mdadm, ZFS, Btrfs, Synology. For most of these systems, I think, I have a reasonable understanding of how they work. But I still have questions about Unraid.

 

This is what I know so far:

 

• Unraid works on the block level, and uses 1 or 2 parity disk(s). It writes the parity data to the parity disk(s). Thus, if a disk fails in the array, it can recover the data. Clear and understandable.
• It works on the block level when calculating parity, but you still have to format the individual disks, and put a filesystem on them. – And this is where I get lost.
• Unraid supports btrfs. You can format disks with btrfs. But it’s also known that Unraid has no integrity checks. – Another thing I can’t wrap my head around, since btrfs does have integrity checking features.

 

Questions:

 

• Does Unraid have an intermediate layer between the filesystem (and system calls to that filesystem) and the underlying block devices? Does it intercept the block level (i.e. not file system level) writes, and is this the point where it does its magic?
  • The documentation talks about disks and block level parity. Then boom, all of a sudden, filesystems and shares. I’m curious about what’s in between!
• Why is it said that Unraid has no integrity checks while it supports btrfs? (Talking about the base system, not plugins.) I understand that on the block level, where it calculates parity, there are no integrity checks or checksums. This is fine. But when a btrfs filesystem is read, it does perform integrity checks, regardless of the underlying block device, or underlying Unraid layer. So, my data would still have integrity. Am I getting it wrong? Am I missing something?

 

I understand that Unraid is proprietary. There is no problem with that. But it’s a little bit frustrating that there is literally zero information on how it works internally. This is also a problem with Synology by the way. If I don’t know how something works, I find it difficult to fully trust it. I understand that it works well, and it’s stable and reliable, but still.

 

Thanks in advance for any replies or explanation!

Edited December 22, 2023 by EqualEgg
grammar

[itimpi]

Unraid has a custom md driver that works at the block level for array drives below any file system so is file system agnostic.

 

Unraid has no automatic file integrity checks that covers all file systems supported, but if you use a file system such as btrfs or ZFS which does have such checks then you get them for the drives using those file system types.

 

[EqualEgg]

2 hours ago, itimpi said:

Unraid has a custom md driver that works at the block level for array drives below any file system so is file system agnostic.

 

Unraid has no automatic file integrity checks that covers all file systems supported, but if you use a file system such as btrfs or ZFS which does have such checks then you get them for the drives using those file system types.

 

 

Thanks for the response, @itimpi!

 

Ok, that explains it. I mean, this was sort of my suspicion. But still, it’s nice to hear it from someone who knows about it more than me.

 

Now, if it works like this, my only remaining question is, how is reliability and robustness guaranteed? If there is any error during the recovery process, that could potentially damage the filesystem on the disk. Btrfs is fragile, but that’s just a feeling based on the many Reddit comments about broken btrfs volumes. (XFS may be more resilient at the expense of no integrity.) We could of course say that any redundancy, even without strong guarantees, is better than none. Is that what Unraid is?

 

By the way, even if it’s like that, don’t misunderstand, I’m not talking down what Unraid does. I’m just trying to better understand it. And the above concerns are equally true for Synology’s solution, and for pure mdraid too. Considering that, Unraid is better.

[MAM59]

1 hour ago, EqualEgg said:

Now, if it works like this, my only remaining question is, how is reliability and robustness guaranteed?

Straightforward as before: just Block Level.

UNRAID does not care what is in the Block, it has no idea of Filesystems, just adds up the blocks and saves the result on the parity(-ies).

If you use filesystems that are "more clever" within the array, you add their features to reliability and robustness.

In case of a drive failure, UNRAID can restore the old content. This should work unless the former disk has been overwritten with garbage before.

So, if your btfrs ran wild and killed its tables, UNRAID cannot help you.

Thats why its usually better to keep things simple and use something "stupid" like xfs.

ZFS should be better than BTFRS they say, but the current implementation is really bad running on an UNRAID array (ZFS uses idle time to rearrange and optimize the disks, since it does not know about UNRAID parity running above, it produces excessive write operations on the parity drive, slowing the system down to its knees. Unless they find a solution for this, stay away from zfs (can be used within pools instead, without UNRAID parity).

 

The main advantage for the UNRAID idea is the enourmous flexibility. You can use disks of any size in your array, Using any filesystem you want. The only restriction is that the parity drive needs to be the largest disk of all (because it needs to provide room for every block of every data disk).

So, even if your parity crashes, or your UNRAID fails totally. Every data disk can be mounted on every other computer because the data disks are independent of each others.

 

 

[trurl]

12 minutes ago, MAM59 said:

the parity drive needs to be the largest disk of all

Each parity disk needs to be at least as large as the largest single array data disk.

[EqualEgg]

50 minutes ago, MAM59 said:

In case of a drive failure, UNRAID can restore the old content. This should work unless the former disk has been overwritten with garbage before.

So, if your btfrs ran wild and killed its tables, UNRAID cannot help you.

Thats why its usually better to keep things simple and use something "stupid" like xfs.

 

I get it, but it wasn’t my main question though. Maybe I wasn’t clear enough, I’m sorry.

 

My question was regarding the reliability and robustness specifically of the recovery process. For example, a disk in the array dies suddenly. The filesystem on it was btrfs. Unraid restores the disk, block by block, but there is one, tiny, single bit error that slips in, and gets written to the ‘recovered’ disk. This is not unimaginable. Can be caused by a controller error, a cable issue etc. That one single bit error makes btrfs go crazy once the filesystem is mounted again, i.e. the recovery was unsuccessful for reasons outside of btrfs’ control.

 

But your comment on using something simple and stupid still applies regardless, I guess. XFS is less likely to go crazy, and is probably more resilient against total failure.

 

50 minutes ago, MAM59 said:

ZFS should be better than BTFRS they say, but the current implementation is really bad running on an UNRAID array (ZFS uses idle time to rearrange and optimize the disks, since it does not know about UNRAID parity running above, it produces excessive write operations on the parity drive, slowing the system down to its knees. Unless they find a solution for this, stay away from zfs (can be used within pools instead, without UNRAID parity).

 

 

Understandable. And makes sense.

 

However, I’m not sure about the ZFS rearranging stuff. Maybe on raid-z1 and raid-z2? But in that case, why would you use Unraid under it? Or if for single-drive volumes, I’m not sure how different ZFS is from btrfs in terms of workload. Both are copy-on-write. But if you say ZFS is worse, I believe you.

 

Drive sizes and flexibility etc. etc. That’s fine, I get that part.

 

Edit: Thanks for your comment btw!

Edited December 22, 2023 by EqualEgg

[JorgeB]

7 minutes ago, EqualEgg said:

That one single bit error makes btrfs go crazy once the filesystem is mounted again, i.e. the recovery was unsuccessful for reasons outside of btrfs’ control.

Very unlikely that a single error would destroy a btrfs filesystem, the metadata it's redundant so no problem if it was from there, if it's in the data, the filesystem would still mount and you can then scrub it to find the corruption.

[MAM59]

1 hour ago, EqualEgg said:

But in that case, why would you use Unraid under it?

Wrong direction 🙂

UNRAID array is ABOVE everything else. Thats why ZFS (if single or raidz) has no idea about it and cannot take it into respect.

 

 

[EqualEgg]

1 hour ago, JorgeB said:

Very unlikely that a single error would destroy a btrfs filesystem, the metadata it's redundant so no problem if it was from there, if it's in the data, the filesystem would still mount and you can then scrub it to find the corruption.

 

Well, I’m getting offtopic here, it has nothing to do with Unraid, but I really don’t know what to think about btrfs! The number of comments I have read about btrfs volumes crashing for no reason is really alarming. I just cannot trust it. I mean I haven’t used it much, apart from my Synology NAS, where it worked fine, but I have a very uneasy feeling about it. Part of the reason why I’m moving away from Synology too. For me it’s either ZFS-raid, or something simpler like Unraid, Snapraid, simple mdraid, or just no raid with XFS, ext4 or ZFS, depending on the use-case. I have made a decision to avoid btrfs for now!

 

But your comment makes a lot of sense, and if everything works as it should, you are probably right!

[EqualEgg]

57 minutes ago, MAM59 said:

Wrong direction 🙂

UNRAID array is ABOVE everything else. Thats why ZFS (if single or raidz) has no idea about it and cannot take it into respect.

 

I don't know what you mean. If you imply that it makes sense to combine ZFS Raid-z1 or Raid-z2 with Unraid, then I disagree. I don't really feel like convincing you, or explaining why, but if that's what you meant, then with all respect, I think you are wrong on this one.

[MAM59]

7 hours ago, EqualEgg said:

It works on the block level when calculating parity, but you still have to format the individual disks, and put a filesystem on them. – And this is where I get lost.

yeah you got lost. so rethink now.

UNRAID secures the block level, the individual disks contain (normal) filesystems which may (or may not) provide additional benefits on file level.

 

The UNRAID part is that much seperated that you are able to pull out a disk from this array and use it standalone anywhere (aka: UNRAID writes NOTHING to the data disks, it just takes care of the parity) (* and before someone comlains, yeah it also allows shares to be split among drives but automatically joined together for presentation to the network, but that is another different layer).

15 minutes ago, EqualEgg said:

If you imply that it makes sense to combine ZFS Raid-z1 or Raid-z2 with Unraid, then I disagree

no! absolutely not!

I assure you, here is a ZFS-free zone! I've tested it, it was bad, I reverted to XFS :-)))

 

Some people wanted it because the main disadvantage of UNRAID (in my opinion) is the missing read cache. You can write fast (if you have a proper cache drive), but once the data has been moved to the array, reading it back is drive-slow. ZFS contains a RAM cache called ARC. People hoped it would cure the problem, but the results are not convincing.

 

[EqualEgg]

4 minutes ago, MAM59 said:

yeah you got lost. so rethink now.

UNRAID secures the block level, the individual disks contain (normal) filesystems which may (or may not) provide additional benefits on file level.

 

The UNRAID part is that much seperated that you are able to pull out a disk from this array and use it standalone anywhere (aka: UNRAID writes NOTHING to the data disks, it just takes care of the parity) (* and before someone comlains, yeah it also allows shares to be split among drives but automatically joined together for presentation to the network, but that is another different layer).

no! absolutely not!

I assure you, here is a ZFS-free zone! I've tested it, it was bad, I reverted to XFS :-)))

 

Some people wanted it because the main disadvantage of UNRAID (in my opinion) is the missing read cache. You can write fast (if you have a proper cache drive), but once the data has been moved to the array, reading it back is drive-slow. ZFS contains a RAM cache called ARC. People hoped it would cure the problem, but the results are not convincing.

 

 

With the first part, I got lost not with what it does, but how it does it. But thanks for explaining anyway.

 

For the second thing: Ok then, we are on the same page.

 

 

[trurl]

5 minutes ago, MAM59 said:

UNRAID secures the block level

For disks in the parity array. It is possible to have multi-disk btrfs or ZFS pools outside the parity array.