Optimizing Unraid 7 Beta 1 with All-ZFS Setup: 4x 12TB HDDs + 1TB NVMe Cache Configuration Issues

[donglobal]

Hi everyone,

 

I'm seeking advice on optimizing my Unraid ZFS pool setup. I'm currently running Unraid 7 beta 1 with the following hardware:

• Four 12TB HDDs
• One 1TB NVMe

Goals:

• Utilize the HDDs as the main storage pool.
• Use the NVMe as a cache drive where data is first written before being moved to the HDD pool for faster access.
• Leverage a 10GBe connection to my local switch for high-speed access.

 

Current Setup:

 

1. First Pool (HDDs):

• Configured as a ZFS RAIDZ array.
• Issue: Default shares (Appdata, Domains, ISOS, System) are not automatically created.

2. Second Pool (Cache):

• Configured with the single NVMe drive.
• Default shares are created when setting up this pool first with the name 'Cache.'

3. Share Settings:

• Allocated the NVMe (CACHE) as the primary drive.
• HDD pool as the secondary drive for each share.

4. Special Metadata Storage Attempt:

• Tried using the NVMe for Special Metadata storage in the HDD pool.
• Issue: Default shares were still not created, so I abandoned this setup.

 

Problems Faced:

• Default shares not being created when setting up the ZFS RAIDZ pool first.
• Uncertainty about the current setup's efficiency with two separate pools (one for HDDs, one for NVMe).
• Need guidance on the optimal file structure before populating drives with data and installing apps.

 

Questions:

•  Is there a way to force the creation of default shares when setting up a ZFS RAIDZ pool?
• Would it be more efficient to combine the NVMe and HDDs into a single pool using a different configuration?
• How should I configure the NVMe to best serve as a cache drive in a ZFS setup?
• Has anyone successfully set up a similar configuration, and what best practices or tips can you share?

 

I want to ensure my setup is optimized for performance and reliability before I start adding data and installing applications. Any advice or experiences you can share would be greatly appreciated.

 

Thank you!

 

[JorgeB]

16 minutes ago, donglobal said:

Issue: Default shares (Appdata, Domains, ISOS, System) are not automatically created.

This is by design, if there's no array those shares will be created in the first available pool, in alphabetical order, you can move them to the other one, but note that for best performance, most of those shares should be in a flash based pool, especially the domains share for VMs.

 

19 minutes ago, donglobal said:

• Tried using the NVMe for Special Metadata storage in the HDD pool.
• Issue: Default shares were still not created, so I abandoned this setup.

They should have been be created in the remaining pool, if you want you can try again, or just move the shares to the other pool using the mover, services need to be disabled first.

 

 

 

 

 

[donglobal]

21 minutes ago, JorgeB said:

This is by design, if there's no array those shares will be created in the first available pool, in alphabetical order, you can move them to the other one, but note that for best performance, most of those shares should be in a flash based pool, especially the domains share for VMs.

 

They should have been be created in the remaining pool, if you want you can try again, or just move the shares to the other pool using the mover, services need to be disabled first.

 

 

 

 

 

Questions for Clarification:

 

Creating Shares Manually:

• If I recreate a fresh pool with the HDDs, should I manually create the default shares (Appdata, Domains, ISOS, System) in this pool?
• Should these shares be created before or after setting up the NVMe as special metadata storage?

Using NVMe as Special Metadata Storage:

• After creating the shares manually, can I then add the NVMe as special metadata storage under the HDD pool?
• Will this ensure that the shares are created in the HDD pool and benefit from the NVMe’s performance for metadata storage?

Optimal Configuration:

• Is it more optimal to create two separate pools (one with the HDDs and one with the NVMe) or combine them into a single pool with NVMe as special metadata storage?
• For performance, especially for VMs, should the `domains` share specifically be placed in the NVMe pool, or can it remain in the combined pool with special metadata storage?
• I want to ensure I’m setting this up in the most efficient way for performance and reliability before populating the drives with data and installing apps.

The two pool setup is mapped like the attached as present

 

Thank you for your assistance JorgeB!

 

[Kilrah]

A ZFS metadata vdev will not hold any data, as its name implies it just stores zfs metadata. It makes no sense in a home server application, plus it becomes critical as in if it fails the entire pool is lost regardless of the reduncancy on the HDDs. 

 

You want 2 pools and the default shares apart maybe from isos to be set to only reside on the SSD pool.

[donglobal]

1 minute ago, Kilrah said:

A ZFS metadata vdev will not hold any data, as its name implies it just stores zfs metadata. It makes no sense in a home server application, plus it becomes critical as in if it fails the entire pool is lost regardless of the reduncancy on the HDDs. 

 

You want 2 pools and the default shares apart maybe from isos to be set to only reside on the SSD pool.

Thanks for the advice Kilrah

[JorgeB]

1 hour ago, donglobal said:

If I recreate a fresh pool with the HDDs, should I manually create the default shares (Appdata, Domains, ISOS, System) in this pool?

They should be automatically created when you enable the docker and VMs services, those services come disabled by default, for a clean 7.0.0 install.

 

1 hour ago, donglobal said:

Using NVMe as Special Metadata Storage:

• After creating the shares manually, can I then add the NVMe as special metadata storage under the HDD pool?

You can, and this will help performance, like folder listing or any other metadata intensive works, but a 120 GB SSD would be enough, unless you use many small files, in that case you can make the small files, up to a size, go to the special vdev, but like mentioned by @Kilrah it should have the same redundancy as the main pool, in this case a mirror, if you lose the special vdev, you lose the whole pool.

 

 

 

[donglobal]

3 minutes ago, JorgeB said:

They should be automatically created when you enable the docker and VMs services, those services come disabled by default, for a clean 7.0.0 install.

 

You can, and this will help performance, like folder listing or any other metadata intensive works, but a 120 GB SSD would be enough, unless you use many small files, in that case you can make the small files, up to a size, go to the special vdev, but like mentioned by @Kilrah it should have the same redundancy as the main pool, in this case a mirror, if you lose the special vdev, you lose the whole pool.

 

 

 

Great advice JorgeB, you have made me rethink the structure of this disk layout again. In this implementation I will not be running any VDevs as the processor is not that strong so will mainly be used for locally run docker containers, and a file server for working documents, photo archive, videos and a timeMachine Backup of a local machine.

 

I plan to make a more beefier machine  for virtual machines and heavier workloads once I have proven the reliability of running this one for a few months. 

 

Thanks @jorgeB & @Kilrah for all your assistance.