unconventional unraid setup

[escapement]

I'm a long time unraid user, but no guru for sure.  I currently have a traditional unraid server setup (hpz820 dual-xeon, 512gb ram, 4x14tb hdd in array,  4tb nvme cache, 10gb networking, dual graphics cards for passthrough).  I use this mostly for media storage, docker (starr stack, nextcloud, etc..) and as a vm host for a mix of linux and windows vms that I use for my daily driver (linux) and some retro gaming and not too demanding steam games. Xeons, ddr3 registered ram and pcie-3 aren't great for gaming but they server for this purpose.  I 

 

I'm currently setting up a new server which I intend on using for running development vms for windows and linux, database serving and video transcoding.  It will have an amd graphics card to pass through to vms and an nvidia tesla p40 for number crunching.  I'm trying to leverage new technology that has arisen since I set up my last server...  Namely, cheap nvme drives and zfs.  I originally was going to use dual nvme drives for the array and 4xhdd for a raidz zfs pool, but after doing  research,  found nvme drives aren't good in pools due to the trim/wear and tear issues.

 

The host is a dell t7910 workstation...  dual 20 core xeon, 128gb ram, expandable to 2tb ... probably going to bring it up to 512gb which is affordable.  This machine supports pcie bifurcation and has a lot of slots, so I can jam a bunch of nvme drives into it. 

 

After thinking about it, this is the new layout I came up with:

 

* a single 512gb nvme for the array...  no parity.  The array won't be used much so not too worried about trim.  Since there is no parity, I think I can take it off line and trim it if I need to... Well at least I think I can but may be wrong about this... but not much of an issue.

 

* a cache pool made up of 3x4tb nvme drives set up in raidz, giving me 8tb storage with redundancy.

 

* 4x10tb hdd zpool set up in raidz for main storage.

 

The system, domains and appdata shares will be stored on the cache drive with secondary storage on the zpool.  I doubt I will fill up the cache, so it will never be written to the zpool, but I can use plugins or zfs replication to back them up to the zpool.  I can create shares either on the zpool or cache/zpool. 

 

It seems like this has advantages over my traditional setup.  I have redundancy in the cache pool and the main storage is interleaved so it should be faster than an unraid array.  I can also move the cache shares to the zpool to empty the cache drive, destroy the cache pool and create a new one with more nvme drives if I need to in the future, so it's expandable.  The move the shares back to the new cache pool.

 

A disadvantage I can see is that zpools are always spun up, so it will use more power and I don't have fine grained control over which share is stored on which drive.

 

Is this a reasonable setup?  Are there any pitfalls I'm missing?  As I said, I'm not an unraid expert, but a long time user and would appreciate some feedback from those of you out there who are.

[JorgeB]

That's a perfectly valid setup.