Cache Pools, ZFS & NVME

[jvanoort]

I'm new to Unraid and I've been watching Spaceinvader One tutorials on YouTube to get my initial Unraid setup completed.  My array currently consists of a 20 TB parity with (2) 18 TB array drives both formatted in XFS.  I plan to add another HDD in the near future formatted in ZFS for sending snapshot backups from the Cache pool(s).  I have (3) 2TB Samsung 990 NVME drives.  I currently have each drive formatted in ZFS but in separate Cache pools so the I/O would be faster with separate lanes for each.  It essentially mirrors this video from Spaceinvader with the caveat that I formatted in ZFS.

After doing some additional research on this Forum, I'm thinking I should setup the (3) NVME drives in a Raidz configuration to allow for some redundancy if a drive fails and for the self-healing abilities of ZFS, etc.  The main functions my server will perform is a Plex server on docker and running BlueIris on a Windows VM.  I'll likely expand the functionality of the server in the future, but these are mission critical.  Right now, I'm jus trying to get my Array and Cache Pool(s) setup as best as possible.  Server specs are as follows:

Mobo: Asus ProArt Z790

Processor: Intel Core i-9 14900k

Ram: 128 GB DDR 5

Hard Drives as listed above.

 

I'm looking for opinions/advice on whether to leave (3) Cache pools separate or if I should run the (3) NVME's in a RaidZ as one Cache pool.  Thanks!

[mavrrick]

I think it really comes down to do you want the benefits of the ZFS file system like compression, dedup, redundancy, bitrot protection ect. ZFS bring some cool features, but also some interesting challenges. 

 

Performance wize you probably wouldn't see any improvements with the NVME in a RaidZ array, but this is the case with most raid solutions with NVME. You would likely see it with spinning disk, but it is unlikely with NVME's as their performance is just so much faster then what the standard install of ZFS is optimized for. That doesn't mean it will be horrible, but just not be as fast as each drive can run alone. I have 3 Crucial P3Plus 4TB drives in a ZFS RAIDZ1 Pool and they seem to fluctuate between 800MB and 2400MB per second. That box though is a much weaker setup then what you listed above. The box is a CWWK Mini PC which can only use NVME drives so I am kind of stuck in that setup. 

 

One of the best things about this setup is the ZFS Snapshots and on disk compression. The fact I can backup all of the docker containers in seconds and then replicate it if I want is nice. It is also nice vs the older CA Appdata backup plugin. That plugin creates full zip files of the Appdata for each day the backup runs. That adds up quickly. The Snapshots are only keeping Delta data so much smaller footprint. That allows me to to keep daily, weekly and monthly Snapshots. 

 

Disk compression for non Plex data can also add up and make a good difference depending on how much of that you have.

 

ZFS by default will use only 1/8th of your system memory, but if you use some of the more advanced features of the file system it can start to get memory hungry so keep that in mind as well. 

 

There is also a complication of ZFS expansion. Right now you can't just add a drive if you want to change stuff. You have to add it with the same basic dimensions of the previous disk setup in the ZFS pool ie if you setup 3 drives in a Raidz1 array, to expand the pool you would need to add another RaidZ1 array to the pool to expand it.

 

So to summarize it is a trade off. Use ZFS and get a bit less IO but fault tolerance, or no fault tolerance and get the IO. My case it is a no brainier, but your situation is a bit different with the fact you also have the 3 spinning disk.