Triple Cach Disk Setup - looking for experience and insight

[Glassed Silver]

Hey guys!

 

I'm considering going from no cache disks to triple cache disks.

I'm playing with the thought of getting 3x 860 EVOs with 500GB each. That'd yield me 750GB of usable space, I'm not worried at all about losing some usable space at all, since it does give me parity on the cache drives through the btrfs pooling.

 

Now the question is, for more, but not that much more I could be getting 2x 1TB drives which would be mirrored and yield me 250GB more space, but I'd be in the conundrum of "which drive to trust if one of them throws errors" (basically the old RAID-1 weakness), would I not? Or is btrfs implementing this method of mirroring in a way that makes it safer?

Basically, despite planning to backup appdata to array with the CA Backup / Restore Appdata plugin by Andrew Zawadzki I still want the most robust cache drive setup, since I already despise the idea of having storage that isn't parity protected.... >_>

 

If I had the chance it'd be read-only area. I guess... VMs kinda need it as well and some databases might profit from it as well. Still don't even know how much use of it I'd make. I don't see myself using it much for shares. HDD speed is fair enough for the documents and media and if there ever is a need for faster storage, then I guess I'd use it for that, but that's so much of a secondary thought...

 

Cheers and thank you for any insight, because the topics I could find about this are super old and I'd hope there's a bit more field experience by now.

[trurl]

23 minutes ago, Glassed Silver said:

don't even know how much use of it I'd make. I don't see myself using it much for shares.

Typically you will want appdata, domains, system shares with all files on SSD cache and configured to stay on cache. This allows dockers / VMs to perform better since writes aren't affected by array parity, and it allows array disks to spin down since dockers / VMs will generally keep some files on these shares open all the time.

[Glassed Silver]

26 minutes ago, trurl said:

Typically you will want appdata, domains, system shares with all files on SSD cache and configured to stay on cache. This allows dockers / VMs to perform better since writes aren't affected by array parity, and it allows array disks to spin down since dockers / VMs will generally keep some files on these shares open all the time.

I know about these things, but I honestly don't really care about the performance of most of my dockers.

 

Back to my question though, because no matter the capacity I will invest in, that aside, is triple ripe enough for production? Which of them is better? Dual or triple?

I'm talking strictly about reliability here. I don't care if my youtube downloader feeds into 100MB/s HDD space or SATA-saturating SSDs or 2/3 saturated. My internet connection is 100MBit/s. Now I know there are still many other apps that could greatly improve and I would rather have wiggle room anyhow, so the capacity is really much more of a secondary thought here. I'm not entirely budget constrained.

[JonathanM]

28 minutes ago, Glassed Silver said:

Back to my question though, because no matter the capacity I will invest in, that aside, is triple ripe enough for production? Which of them is better? Dual or triple?

3 drives in BTRFS RAID1 which is the default Unraid setting is still only double protection. The RAID level is what determines the redundancy. Triple requires you to manually specify RAID1c3, you will need to research the command line needed to do that at the balance prompt. If you specify c3 for both data and metadata, you would end up with 500GB, not 750GB of available space on 3 500GB drives.

[Glassed Silver]

57 minutes ago, jonathanm said:

3 drives in BTRFS RAID1 which is the default Unraid setting is still only double protection. The RAID level is what determines the redundancy. Triple requires you to manually specify RAID1c3, you will need to research the command line needed to do that at the balance prompt. If you specify c3 for both data and metadata, you would end up with 500GB, not 750GB of available space on 3 500GB drives.

Ah, command line sounds like it'd be more of an edge case and hence less well supported... Not really feeling like going down that route.

So how about a scenario where unRAID notices a difference between both drives? How are errors handled, because if full physical drive failure is the only thing I'd be protected against then that'd be a bummer...

I'm already a bit puzzled by what to do when unRAID sees sync errors on my array when all drives are in good health... I run another parity check and fix errors and keep going, hoping nothing blows up.

(not that I'm indifferent to my data, but I haven't really caught the memo I think what to do in a case like that I guess... maybe I'm the only one, could be I missed something big.)

 

Cheers for all guidance, really. Seems like I'm gonna save myself the third drive then if it doesn't add anything really given I don't want an edge case setup that is harder to get support for.

[tjb_altf4]

2 hours ago, Glassed Silver said:

Ah, command line sounds like it'd be more of an edge case and hence less well supported... Not really feeling like going down that route.

RAID1c3 is available in the GUI from 6.9 beta releases.

Edited October 3, 2020 by tjb_altf4

[JonathanM]

16 minutes ago, tjb_altf4 said:

RAID1c3 is available in the GUI from 6.9 beta releases.

Cool! I knew there was a push to get the metadata moved to c3, so they also added the option for data c3?

I haven't played with BTRFS options for several years, after a near data loss incident I've stayed with XFS.

[tjb_altf4]

32 minutes ago, jonathanm said:

Cool! I knew there was a push to get the metadata moved to c3, so they also added the option for data c3?

I haven't played with BTRFS options for several years, after a near data loss incident I've stayed with XFS.

 I though I saw raid1c3/raid1c4 in beta25 balance options when I spun it up on a test machine, but definitely shown in beta29 notes

Quote

ANOTHER known issue: we have added additional btrfs balance options:

raid1c3

raid1c4

and modified the raid6 balance operation to set meta-data to raid1c3 (previously was raid1).

 

[JorgeB]

8 hours ago, Glassed Silver said:

So how about a scenario where unRAID notices a difference between both drives? How are errors handled, because if full physical drive failure is the only thing I'd be protected against then that'd be a bummer...

If any corruption is detected in a redundant btrfs member it's automatically corrected on read (also during a scrub).

[trurl]

16 hours ago, Glassed Silver said:

I'm already a bit puzzled by what to do when unRAID sees sync errors on my array when all drives are in good health... I run another parity check and fix errors and keep going, hoping nothing blows up.

(not that I'm indifferent to my data, but I haven't really caught the memo I think what to do in a case like that I guess... maybe I'm the only one, could be I missed something big.)

Unless you have some specific reason to suspect a specific data disk, such as immediately after a data disk rebuild, you have no choice but to correct parity.