Got some new parts and recycled my old machine and now I need advice how to best setup arrays and pools.

Usage is dockers, VMs (work+lite gaming), home automation with HomeAssistant, home security with coral TPU (offloading footage to another NAS) etc... 

I have secondary backup for important work and family data arranged on a separate machine in house so don't need too much parity wise.

Machine would be on 24/7. Electricity cost is not an issue.

 

CPU: i9-13900k

RAM: at the moment 64Gb, going to 128Gb soon

 

NVMe

1x Samsung SSD 970 EVO Plus - 2TB  (VM pass through)
1x SanDisk Extreme Pro  - 1 TB  (VM pass through)
1x SanDisk Extreme Pro - 500 GB  (was cache, xfs)

NVMe were set this way in old machine.

No more slots for extra NVMe are available on motherboard.

 

HDD

3x Seagate 16TB Segate Exos, 7200rpm  (new drives)

1x WD Blue 4TB, 5400rpm (formerly parity)

3x WD Blue 3TB, 5400rpm (were in array, xfs)

1x Toshiba 2TB, 7400 rpm (was in array, xfs)

Motherboard can handle 6 more drives.

 

I was keeping emulated VM HDD drives on cache (smallest NVMe) and would just pass through biger NVMe directly to VMs as extra drives. Worked like a charm. Is this best way to do it?

How should I format HDDs, ZFS/xfs/btrfs and what combo? Is it sane to have parity protected Array with smaller HDDs and a ZFS pool with 3x 16TB Seagates? What FS should I use for array drives? How do I arrange VMs and shares in that case?

What would you do? Any advice is wellcome.

Edited April 21, 20242 yr by Lavoslav
solved

if you wish to maintain NVME for VM passthorugh you can.

In my setup I ahve 2 disk to compelte array needs, a NVME as a single disk cached and my main data on a zfs pool


I would recommend the cache dis be a single NVME drive for unraid system imaged of the docker.xfs image file and virtlib.img files locations.
It is recommended that each "array /cache / zfs have a min of 2 disk.

This is for redundancy backup and configuration of protected files

I find it easier to use the unraid array for NVME/ssd for use as virtual disk for VM when needed.
The Cached disk for docker system appdata when needed.
You can tell setting and other plugins/apps to use the zfs pool instead. It is more how you want to use and access the data.

I use a rsync command to copy and backup the app data and some setting of the Vm on my zfs pool.

Everything else server or data wise in on my zfs storage. If you can 3-4 disk a raidz1 hopefully a raid z2 depending on size of each disk. (

With the number of disk you can potentially use. I would still recommend a similar setup for the 321 rule.

 

What is the 3-2-1 Rule?

The 3-2-1 Rule is a data protection strategy that recommends having three copies of your data, stored on two different types of media, with one copy kept off-site.

As a widely embraced data backup strategy, the 3-2-1 Rule prescribes:

Maintain three copies of your data: This includes the original data and at least two copies.

Use two different types of media for storage: Store your data on two distinct forms of media to enhance redundancy.

Keep at least one copy off-site: To ensure data safety, have one backup copy stored in an off-site location, separate from your primary data and on-site backups.

This rule is a robust guideline for data protection, ensuring redundancy, resilience, and the ability to recover data even in the face of unexpected events or disasters.

Unless told otherwise at disk assignment and creation.
By default, the cache disk is btrfs the array uses xfs and a zfs type is required to be made under pools for the time being.

There are pros and cons of each linux file system type.

See
https://www.salvagedata.com/btrfs-zfs-xfs-ext4-how-are-they-different/

https://serverfault.com/questions/1000767/ext4-vs-xfs-vs-btrfs-vs-zfs-for-nas

in the end, it what you want to do and how you want to interact with the hardware.

Edited April 21, 20242 yr by bmartino1

HDD

Zfs Raid z1
3x Seagate 16TB Segate Exos, 7200rpm  (new drives)

Zfs raid z1
3x WD Blue 3TB, 5400rpm (were in array, xfs)

Unraid disk 1 array
1x Toshiba 2TB, 7400 rpm (was in array, xfs)
Unraid disk1 party 1
1x WD Blue 4TB, 5400rpm (formerly parity)

NVME
cahcc disk
1x SanDisk Extreme Pro - 500 GB  (was cache, xfs)

keep your VM passthorugh.

 

On 4/20/2024 at 6:05 AM, Lavoslav said:

Machine would be on 24/7. Electricity cost is not an issue.

Thats great, but do you have UPS ? For non-UPS protect system, I will config limit / minimize continue writing ( i.e. security camera recording ) to one NVMe / disks instead most of storage. This will major prevent FS corruption in suddenly power off.

 

On 4/20/2024 at 6:05 AM, Lavoslav said:

I was keeping emulated VM HDD drives on cache (smallest NVMe)

This not a problem if you never mind the warning.

 

On 4/20/2024 at 6:05 AM, Lavoslav said:

How should I format HDDs, ZFS/xfs/btrfs and what combo?

I like btrfs more then ZFS, mainly bcz better performance.

 

On 4/20/2024 at 6:05 AM, Lavoslav said:

Is it sane to have parity protected Array with smaller HDDs and a ZFS pool with 3x 16TB Seagates?

That depends on what capacity you need and how you define pool / array is main / backup. For me, I use array as backup tier, but it also share for some user because array has better user access control.

As I want to transfer file quickly, so pool are 1st place to store file then periodic sync to array. ( I don't like mover design, so array and pool always independent )

Edited April 21, 20242 yr by Vr2Io

Guys, thank you very much for replies. You've broken me out of my "thought loop" so to speak and given me great pointers how to proceed. 

 

@bmartino1

Tyvm for the screenshot of your main tab and the writeup. Helped greatly to visualise things better.

As for 3-2-1- backup, I am set for my needs, though I don't do the last part of 3-2-1 offsite but rather have a solution where I monthly copy stuff I need on disks I keep safely stored and not used in any other way.

 

@Vr2Io

As far as power protection goes setup is as follows:

UnRaid - 1500VA ups, set to poweroff server if it hits 5 minutes runtime capacity.

Backup machine - 650VA UPS. Backup machine is low powered QNAP (not using their terrible OS though). Almost always turned off. Wakes on Lan when backup needs to run and powers down when done. Set to poweroff at 2 minutes remaining runtime.

Bare metal and TV - 1000VA, run untill dry, which is never since they are turned on only when in use.

POE capable switch, POE powered AP and POE cameras (powered through switch) and Modem, have it's own 1500VA, they run untill dry

 

I both love you and hate you for saying "...how you define pool / array is main / backup. For me, I use array as backup tier,..", I've never thought abut this in a way you framed it. For me it was all on the array except NVMe directly passed to VMs. That sentence, combined with bmartino1's suggestion, solved it for me.

 

As far as disk setup goes I'm gonna go very close to what @bmartino1 wrote in second post. For now 2 of the 3TB drives are gona go into array as xfs and third will go in array as zfs for replication, but I'll get 3 more larger drives at some later date for a second ZFS pool since @Vr2Io gave me some food for thought. 

 

Unraid array

- party
1x WD Blue 4TB

- array

2x WD Blue 3TB, as xfs

1x WD Blue 3TB, as ZFS

1x Toshiba 2TB, as xfs
 

Cache disk

1x SanDisk Extreme Pro - 500 GB, as ZFS

 

ZFS Pool raidz1 HDD

3x Seagate 16TB Segate Exos

 

NVMe disks direct pass through to VM's

 

Future expansion another ZFS raidz1 3 disk pool.

 

Marking this as solved, but if anybody has anything to add or point out, feel free. Parhaps it can help somebody else in similar situation.