Hi All,

I have a high I/O workload, and I'm wanting to streamline the system with a fast NVMe tier that machines can write finished files to quickly, which Unraid then automatically moves over to a bigger ZFS pool in the background.

I have 3 enterprise U.2 7.68TB NVMe drives to use for this fast target and my idea is:

1. Migrate our ZFS pool from TrueNAS to Unraid

2. Create a RAIDZ1 pool with the 3 NVMe drives with:
   - An exclusive share on it as a write-only drop zone (so bypasses FUSE for full speed).
   - A normal share with the NVMe pool as primary and the main ZFS pool as secondary target.

3. A little script that watches the exclusive share and when a file finishes writing, moves it into the user share. Since both are on the same physical drives, the move should be instant and the file stays on NVMe until the mover shifts it later.

E.g.

Writes to/mnt/nvmeRaidZ1/ExclusiveShare ---> Script Immediately moves completed file to /mnt/user/NormalDataShare (where this share is on the same nvmeRaidZ1 drives) ---> Mover handles from there.

My questions and concerns are

1. Is moving a file from an exclusive share into a user share on the same pool actually instant, and will the mover then handle it normally?

2. Anyone doing something similar, if so, how well does it hold up?

3. Is there a simpler built-in way to get "fast write tier that drains to bulk" that I'm missing?

4. is FUSE going to actually cause a noticeable slowdown for a RAIDZ1 pool of U.2. NVME's?

---

Extra Info/Context:

I currently have the ZFS pool on TrueNAS as a VM on Unraid. I've not migrated because I was waiting for a significant enough reason to do so - which I think could be this. If Unraid isn't the right solution for this, I'm open to other suggestions, even keeping TN as a VM and setting up something on that.

Hardware Deets:

• Single-socket EPYC server, plenty of PCIe/RAM

• 3× enterprise U.2 NVMe (planned as a RAIDZ1 pool — the fast tier)

• ~300TB ZFS pool (the bulk/capacity tier)

• Unraid 7.3.0

• Dual 25GB SFP uplinks.

Edited July 16Jul 16 by Crad
added more context.

9 minutes ago, Crad said:

Is moving a file from an exclusive share into a user share on the same pool actually instant, and will the mover then handle it normally?

Not unless you "hack" things a bit, on zfs different shares are separate datasets, so from ZFS' perspective they're entirely separate and moving data between them means rewriting everything.

10 minutes ago, Crad said:

is FUSE going to actually cause a noticeable slowdown for a RAIDZ1 pool of U.2. NVME's?

Depends what you're writing, for large files probably not, for small ones definitely. The overhead is mostly on file open.

If FUSE is an issue then make only the exclusive share, don't use the stock mover but write your own script to offload to array instead of moving to another share and then use unraid's mover.

Edited July 16Jul 16 by Kilrah

2 minutes ago, Kilrah said:

Not unless you "hack" things a bit, on zfs different shares are separate datasets, so from ZFS' perspective they're entirely separate and moving data between them means rewriting everything.

Depends what you're writing, for large files probably not, for small ones definitely. The overhead is mostly on file open.

That's good to know. 95%+ of my I/O is large files.

file open overhead is tricky, because all the actively used data will be on ZFS pools, not the main Array, so hopefully no Fuse overhead there?

6 minutes ago, Crad said:

so hopefully no Fuse overhead there?

If they're on exclusive shares no overhead, otherwise there's some regardless of location. If the share is not exclusive fuse always has to look on all drives (both from array and all pools) for where the file actually is/has to be created following the allocation rules.

Edited July 16Jul 16 by Kilrah

5 minutes ago, Kilrah said:

If they're on exclusive shares no overhead, otherwise there's some regardless of location. If the share is not exclusive fuse always has to look on all drives (both from array and all pools) for where the file actually is/has to be created following the allocation rules.

Ahh of course. So does that mean fuse overhead will be lower if the share is only stored on ZFS pools? (e.g. share is on ZFS nvme pool + ZFS hdd pool). The theory being that the zfs drives are always spinning and should be faster to assess the file structure?

Not that I know of, but I've never tested for how things scale with the number of drives/storage locations. I'd expect "fuse/not fuse" has more impact that "fuse with few drives/fuse with more drives".

Thanks, I'll have to do some testing.