I have a zpool of 3x2TB NVME drives. I would like to add a fourth. They are all the same drive from the same manufacturer. I'm trying to use zfs attach and getting an error of

cannot attach nvme-eui.6479a78dd00000d8 to raidz1-0: can only attach to mirrors and top-level disks

I've tried doing it from the GUI as well and just get 'invalid expansion'.

What is the supported way to do this?

 

Thank you.

did you add nvme to disk 1 array?

ZFS using the smallest disk size when determining disk space if you have 3x2TB and one dies and you add 1TB disk it will drop the 2TB disk to 1TB space.

in unraid for zfs it should be a pool device of 3 disk in raid z1 based on your setup... Please post a diag file not enoth info..

 

The issue lies in the fundamental design of ZFS and how RAIDZ pools work, particularly in Unraid. Here's a clear explanation of what's happening and why the error occurs:

Why You Can't Add a Single Disk to a RAIDZ Pool

RAIDZ Expansion Rules:

ZFS RAIDZ pools (RAIDZ1, RAIDZ2, RAIDZ3) are designed to work with a fixed number of devices at creation.

Once a RAIDZ pool is created, you cannot simply add a single disk to the RAIDZ pool to expand its capacity. This is because RAIDZ is based on striping parity across all drives, and changing the pool's geometry (adding a single drive) would disrupt the parity structure.

Error Explanation:

The error cannot attach ... can only attach to mirrors and top-level disks indicates that zfs attach is only valid for mirrored vdevs, not RAIDZ pools.

Unraid ZFS Implementation:

Unraid supports ZFS but does not modify ZFS’s core behavior, meaning RAIDZ pools follow ZFS's standard rules.

How to Expand Your Pool

Currently, ZFS does not support dynamic expansion of RAIDZ pools by adding a single drive. However, there are a few options to expand or modify your setup:

Option 1: Create a New RAIDZ Pool

Add the new NVMe drive as part of a new RAIDZ pool and then combine the pools using ZFS's dataset-level flexibility.

This approach keeps your existing pool intact and lets you use the additional storage as a separate pool.

Option 2: Replace Drives with Larger Ones

You can replace each 2TB NVMe drive in your RAIDZ pool with a larger drive, one at a time.

Once all drives are replaced, the pool will expand to the size of the smallest new drive.

Option 3: Back Up, Recreate, and Restore

Back up the data from your existing pool.

Destroy the current RAIDZ pool.

Create a new RAIDZ pool with all four drives.

Restore your data to the new pool.

Option 4: Wait for RAIDZ Expansion Support

Recent ZFS updates (starting in version 2.0) have introduced the groundwork for RAIDZ expansion, but it is not yet a stable feature in Unraid or most ZFS setups. Future updates may allow for adding single drives to RAIDZ pools.

Important Considerations

Resilvering Limitations: If you could attach a drive (like in mirrored vdevs), ZFS would automatically begin resilvering. But for RAIDZ, adding a drive isn't allowed because it cannot maintain parity integrity during such an operation.

Best Practices: Always ensure you have backups before making changes to your pool structure. Operations like recreating the pool or replacing drives come with risks.

Simplified Explanation for the User

"ZFS RAIDZ1 pools, like the one you're using, don't allow adding single drives because the way parity and striping are calculated is fixed when the pool is created. This means you can't expand your pool by attaching a single drive.

You have a few options:

Create a new pool with the extra drive and manage them together.

Replace each drive with a larger one, one at a time, to expand capacity.

Back up your data, recreate the pool with all drives, and restore your data.

Unfortunately, RAIDZ dynamic expansion isn't supported yet in ZFS, but it's something that's being worked on for future releases."

5 hours ago, bfenty said:

What is the supported way to do this?

RAIDZ expansion is currently not supported, it will be once OpenZFS 2.3 is out, and Unraid is updated to include it.

Thanks. Not the answer I wanted to hear but I appreciate the clarity. 
 

is there an unsupported way to do what I’m trying to do? or an I just SOL for now?

6 hours ago, bmartino1 said:

did you add nvme to disk 1 array?

ZFS using the smallest disk size when determining disk space if you have 3x2TB and one dies and you add 1TB disk it will drop the 2TB disk to 1TB space.

in unraid for zfs it should be a pool device of 3 disk in raid z1 based on your setup... Please post a diag file not enoth info..

 

The issue lies in the fundamental design of ZFS and how RAIDZ pools work, particularly in Unraid. Here's a clear explanation of what's happening and why the error occurs:

Why You Can't Add a Single Disk to a RAIDZ Pool

RAIDZ Expansion Rules:

ZFS RAIDZ pools (RAIDZ1, RAIDZ2, RAIDZ3) are designed to work with a fixed number of devices at creation.

Once a RAIDZ pool is created, you cannot simply add a single disk to the RAIDZ pool to expand its capacity. This is because RAIDZ is based on striping parity across all drives, and changing the pool's geometry (adding a single drive) would disrupt the parity structure.

Error Explanation:

The error cannot attach ... can only attach to mirrors and top-level disks indicates that zfs attach is only valid for mirrored vdevs, not RAIDZ pools.

Unraid ZFS Implementation:

Unraid supports ZFS but does not modify ZFS’s core behavior, meaning RAIDZ pools follow ZFS's standard rules.

How to Expand Your Pool

Currently, ZFS does not support dynamic expansion of RAIDZ pools by adding a single drive. However, there are a few options to expand or modify your setup:

Option 1: Create a New RAIDZ Pool

Add the new NVMe drive as part of a new RAIDZ pool and then combine the pools using ZFS's dataset-level flexibility.

This approach keeps your existing pool intact and lets you use the additional storage as a separate pool.

Option 2: Replace Drives with Larger Ones

You can replace each 2TB NVMe drive in your RAIDZ pool with a larger drive, one at a time.

Once all drives are replaced, the pool will expand to the size of the smallest new drive.

Option 3: Back Up, Recreate, and Restore

Back up the data from your existing pool.

Destroy the current RAIDZ pool.

Create a new RAIDZ pool with all four drives.

Restore your data to the new pool.

Option 4: Wait for RAIDZ Expansion Support

Recent ZFS updates (starting in version 2.0) have introduced the groundwork for RAIDZ expansion, but it is not yet a stable feature in Unraid or most ZFS setups. Future updates may allow for adding single drives to RAIDZ pools.

Important Considerations

Resilvering Limitations: If you could attach a drive (like in mirrored vdevs), ZFS would automatically begin resilvering. But for RAIDZ, adding a drive isn't allowed because it cannot maintain parity integrity during such an operation.

Best Practices: Always ensure you have backups before making changes to your pool structure. Operations like recreating the pool or replacing drives come with risks.

Simplified Explanation for the User

"ZFS RAIDZ1 pools, like the one you're using, don't allow adding single drives because the way parity and striping are calculated is fixed when the pool is created. This means you can't expand your pool by attaching a single drive.

You have a few options:

Create a new pool with the extra drive and manage them together.

Replace each drive with a larger one, one at a time, to expand capacity.

Back up your data, recreate the pool with all drives, and restore your data.

Unfortunately, RAIDZ dynamic expansion isn't supported yet in ZFS, but it's something that's being worked on for future releases."

Thanks for the lengthy reply. My understanding is that ZFS now supports officially the ability to expand a pool. It sounds like that’s not yet implemented in unraid. Annoying, because I had data on that disk that I couldn’t back up elsewhere before creating the pool. My plan was to create the pool, copy over the data, then add the disk to the pool. 

6 minutes ago, bfenty said:

My understanding is that ZFS now supports officially the ability to expand a pool. 

Not yet. 2.3 is afaik still a release candidate

37 minutes ago, Mainfrezzer said:

2.3 is afaik still a release candidate

Correct, TrueNAS already supports it, but they are running a pre-release zfs version, I don't expect Unraid to support it before openZFS 2.3 stable is out.

The same question, created raidz1 pool from 3 x 2TB nvme diff manufactures and realized that motherboard has one more free slot, wanted to add #4 2TB drive, but don`t want to copy all data away from pool to array, then re-create and copy back...

10 hours ago, d3m3zs said:

The same question

Same answer, see above.