Hello! I stumbled upon the following issue -- I am using customer-grade NVME drives for my cache pool in Unraid (HP EX950); not so long ago my pool one of the drives died and I was struggling to replace my raidz2 array drive with ssd from a different vendor (WD SN750).

The problem is that new WD SN750 has slightly smaller size which causes normal zfs replace procedure to fail. (Old drives - 2048408248320 bytes, new drive - 2000398934016 bytes)

I wanted to solve this with over-provisioning - do not allocate partition on a whole drive, to keep leeway for different drive vendors.

I tried to manually partition drives prior to zfs pool creation in unraid UI, eg:

parted -s /dev/nvme3n1 unit s mklabel msdos mkpart primary 2048 3900000000 # ~1.99TB

But the problem is, unraid unconditionally overrides partition table:

sfdisk --quiet --label dos /dev/nvme1n1 <<< 'start=2048' # from logs; uses WHOLE drive for zfs disk

Is there any way to force unraid to keep disk in pre-partitioned state prior to zfs pool initialization / replacement disk resilvering?

P.S. nvme namespaces is not a solution, as my customer drives simply do not support namespace management.

P.P.S. I also use luks encryption on drives

Edit: Unraid version: 7.2.4; redundancy mode is raidz1

Edited June 21Jun 21 by pacmancoder

IIRC , Unraid will always repartition a pool device if the existing partition is not the default layout, you could create the parrion and format it ZFS outside the GUI, then just import the pool using the GUI, that would work.

@JorgeB Well... it is definitely an option, but more like last resort - I came to Unraid because of usability/user friendliness, so but it is not ideal to manually manage zpool, re-partition, luks encrypt/open, re-import pool in UI etc. each time disaster struck... It just wastes a lot of my time and makes me want to try other options like TrueNAS saddly ☹️.

I'll try to update to latest 7.3.1, maybe things have changed, changelogs mentioned custom partitioning on array drives, maybe pool drives were affected too?

UPD: No, update to 7.3.1 didn't help

Edited June 21Jun 21 by pacmancoder

This is typically not needed, so it's not really a feature that someone has requested or is on LT's radar, you can create a feature reuqest here if you like: https://product.unraid.net/b/unraid-os-feature-requests

54 minutes ago, pacmancoder said:

try other options like TrueNAS

Alwsys good to try other options, but IIRC TrueNAS is the same regarding this.

So, I basically spent my whole weekend but got it working the way I like, maybe someone would find these crude scripts useful:

prepare_drive.sh

#!/bin/bash

# Erases and initializes NVME device for use in zfs pool.
# Creates fixed parition size (set by G_PARTITION_SIZE). Current value is
# set for 2TB drives, set this value accordingly for your specific device size
# (average size minus a few leeway GB))
# Assumes your LUKS encryption key resides in G_KEYFILE

set -e

G_PARTITION_SIZE=1996800000000 # About ~4Gb overprovisioning to be safe
G_BLOCK_SIZE=$(( G_PARTITION_SIZE / 512 ))
G_KEYFILE=/path/to/keyfile

if [[ ! $1 =~ ^nvme[0-9]+n1$ ]]; then
    echo "ERROR: Invalid nvme device, expected nvmeXn1 format ($(basename $0) nvme0n1)"
    exit 1
fi

DEVICE_NAME=$1
DEVICE_PATH="/dev/${DEVICE_NAME}"

if [[ ! -e $DEVICE_PATH ]]; then
    echo "ERROR: Device ${DEVICE_PATH} do not exist!"
    exit 1
fi


DEVICE_MODEL=$(cat /sys/block/${DEVICE_NAME}/device/model)
DEVICE_SERIAL=$(cat /sys/block/${DEVICE_NAME}/device/serial)
DEVICE_SIZE_BLOCKS=$(cat /sys/block/${DEVICE_NAME}/size)
DEVICE_SIZE=$(( DEVICE_SIZE_BLOCKS * 512 ))

echo "===> DEVICE INFO"
echo -e "\tPath: ${DEVICE_PATH}"
echo -e "\tModel: ${DEVICE_MODEL}"
echo -e "\tSerial: ${DEVICE_SERIAL}"
echo -e "\tSize: " $(numfmt --to=si $DEVICE_SIZE) "(${DEVICE_SIZE} bytes)"

if [[ $DEVICE_SIZE -lt $G_PARTITION_SIZE ]]; then
    echo "ERROR: Drive is too small, please select another drive or update G_PARTITION_SIZE in script"
    exit 1
fi

echo " ===> WARNING"
echo -e "\tThis actions is IRREVERSIBLE!"
echo -e "\tAre you SURE this device is safe to ERASE?"
echo -e "\tEnter device name again ($DEVICE_NAME)"

read -p "Device name: " CONFIRMATION

if [[ $CONFIRMATION != $DEVICE_NAME ]]; then
    echo "INCORRECT DEVICE NAME, ABORTING..."
    exit 1
fi

echo "===> REMOVING OLD FILESYSTEM(S)"
if [[ -e "${DEVICE_PATH}p1" ]]; then
    echo -ne "\tFound device first partition, removing..."
    wipefs --all "${DEVICE_PATH}p1"
    echo "OK"
fi
echo -ne "\tRemoving partition table..."
wipefs --all --no-act "${DEVICE_PATH}"
echo "OK"

echo "===> PERFORMING TRIM"
echo -ne "\tTrimming..."
blkdiscard -f ${DEVICE_PATH}
echo "OK"

echo "===> REFORMATTING DRIVE"
echo -e "\tOverprovisioned disk partition size:" $G_PARTITION_SIZE
echo -e "\tOverprovisioned disk block count:" $G_BLOCK_SIZE
echo -en "\tFormatting..."
parted -s ${DEVICE_PATH} unit s mklabel gpt mkpart primary 2048 ${G_BLOCK_SIZE}
echo "OK"

echo "===> ENABLING LUKS ENCRYPTION"
echo -e "\tUsing keyfile: ${G_KEYFILE}"
echo -ne "\tEnabling..."
cryptsetup -q luksFormat --key-file ${G_KEYFILE} "${DEVICE_PATH}p1"
echo "OK"

echo " ===> SUCCESS!"
echo -e "\t DISK IS READY TO BE USED IN ZFS ARRAY"

make_pool.sh

#!/bin/bash

# Script requires:
# 0. Prepare each drive with prepare_drive.sh
# 1. drive.list file in script folder (new-line separated list of nvme drives (e.g. nvme0n1))
# 2. open_luks.sh script in script folder (see below)
# 3. close_luks.sh script in script folder
#
# Set G_RAID/G_POOL_NAME to desired values

G_RAID="raidz2"
G_POOL_NAME="cache_nvme"

BASEDIR=$(dirname "$0")
mapfile -t DRIVE_LIST < $BASEDIR/drive.list

echo "WARNING! THIS WILL DESTROY ALL DATA ON THE FOLLOWING DRIVES:"
for x in ${DRIVE_LIST[@]}; do
    echo $x
done
echo "NOTE: DRIVES SHOULD BE PREPARED VIA SEPARATE SCRIPT PRIOR TO THIS OPERATION"
read -p "ARE YOU SURE? (WRITE yes IN CAPITAL LETERS): " ANSWER
if [[ $ANSWER != "YES" ]]; then
    echo "ABORTING..."
    exit 1
fi

echo "===> OPENING LUKS"

MAPPED=()
for x in "${DRIVE_LIST[@]}"; do
  MAPPED+=("/dev/mapper/${x}p1")
done

for x in ${DRIVE_LIST[@]}; do
    echo -ne "\tMAPPING ${x}..."
    bash $BASEDIR/open_luks.sh $x && echo -e "\tOK" #REM
done

echo "===> MAKING ZPOOL"
echo -e "\tzfs commands in progress.."
zpool create \
    -o ashift=12 \
    -O dnodesize=auto \
    -O acltype=posixacl \
    -O xattr=sa \
    -O utf8only=on \
    -O compression=lz4 \
    -O atime=off \
    ${G_POOL_NAME} \
    ${G_RAID} \
    ${MAPPED[@]}
zpool status ${G_POOL_NAME}
zpool list ${G_POOL_NAME}
echo -e "\tOK"

echo "===> DEACTIVATING ZPOOL"
zpool export ${G_POOL_NAME}
echo -e "\tOK"

echo "===> CLOSING LUKS"
for x in ${DRIVE_LIST[@]}; do
    echo -ne "\tUNMAPPING ${x}..."
    bash $BASEDIR/close_luks.sh $x && echo -e "\tOK" #REM
done

echo "POOL CREATION SUCCESSFULL!"
echo "NOW YOU CAN IMPORT IT IN UNRAID"

open_luks.sh

#!/bin/bash

# replace G_KEYFILE with your path
# usage `open_luks.sh nvme0n1`

set -e

G_KEYFILE=/path/to/keyfile

if [[ ! $1 =~ ^nvme[0-9]+n1$ ]]; then
    echo "ERROR: Invalid nvme device, expected nvmeXn1 format ($(basename $0) nvme0n1)"
    exit 1
fi

DEVICE_NAME=$1
DEVICE_PATH="/dev/${DEVICE_NAME}"

if [[ ! -e $DEVICE_PATH ]]; then
    echo "ERROR: Device ${DEVICE_PATH} do not exist!"
    exit 1
fi

cryptsetup luksOpen --key-file "${G_KEYFILE}" "/dev/${DEVICE_NAME}p1" "${DEVICE_NAME}p1" 

close_luks.sh

#!/bin/bash

# usage `close_luks.sh nvme0n1`

set -e

if [[ ! $1 =~ ^nvme[0-9]+n1$ ]]; then
    echo "ERROR: Invalid nvme device, expected nvmeXn1 format ($(basename $0) nvme0n1)"
    exit 1
fi

DEVICE_NAME=$1
DEVICE_PATH="/dev/${DEVICE_NAME}"

if [[ ! -e $DEVICE_PATH ]]; then
    echo "ERROR: Device ${DEVICE_PATH} do not exist!"
    exit 1
fi

cryptsetup luksClose "${DEVICE_NAME}p1"

start_pool_maintenance.sh

#!/bin/bash

# reqires correct drive.list(currently alive) and G_POOL_NAME & open_luks.sh script

G_POOL_NAME="cache_nvme"

BASEDIR=$(dirname "$0")
mapfile -t DRIVE_LIST < $BASEDIR/drive.list

echo "===> OPENING LUKS"

MAPPED=()
for x in "${DRIVE_LIST[@]}"; do
  MAPPED+=("/dev/mapper/${x}p1")
done

for x in ${DRIVE_LIST[@]}; do
    echo -ne "\tMAPPING ${x}..."
    bash $BASEDIR/open_luks.sh $x && echo -e "\tOK" #REM
done

echo "===> IMPORTING ZPOOL"
zpool import ${G_POOL_NAME}
echo -e "\tOK"

stop_pool_maintenance.sh

#!/bin/bash

# reqires correct drive.list(currently active in zpool) and G_POOL_NAME & close_luks.sh script

G_POOL_NAME="cache_nvme"

BASEDIR=$(dirname "$0")
mapfile -t DRIVE_LIST < $BASEDIR/drive.list

echo "===> DEACTIVATING ZPOOL"
zpool export ${G_POOL_NAME}
echo -e "\tOK"

echo "===> CLOSING LUKS"
for x in ${DRIVE_LIST[@]}; do
    echo -ne "\tUNMAPPING ${x}..."
    bash $BASEDIR/close_luks.sh $x && echo -e "\tOK" #REM
done

README.md

```markdown
HOW TO REPLACE DRIVE
===================
1. Prepare new drive with prepare_drive.sh script:
    `bash prepare_drive.sh nvme10n1`
2. Open ZPool for maintenance via start_pool_maintenance.sh script:
    - ensure drive.list file have only alive drives
    `bash start_pool_maintenance.sh`
    - Import in forced mode if zpool export was previously failed:
        `zpool import -f cache_nvme`
3. Identify failed drive, notice old device GUID:
    - `zpool status cache_nvme`
4. Offline failed drive if not already:
    - `zpool offline cache_nvme /dev/mapper/nvme2n1p1`
    - `bash close_luks.sh <old_drive>` -- close luks encryption
5. Replace failed drive in zpool; use the old device''s GUID from `zpool status` output
    - bash open_luks.sh <new_drive>
    - `zpool replace cache_nvme <old-device-guid> /dev/mapper/nvme2n1p1
6. Monitor resilver progress:
    - `watch -n 5 zpool status cache_nvme`
    - DO NOT TURN OFF SERVER WHILE IT IS RESILVERING!
7. Close ZPool maintenance:
    - add new drive to drive.list to keep maintenance scripts running
    - `bash stop_pool_maintenance.sh`
8. Remove old pool from Unraid UI (Dont worry, it will not destroy any data)
9. Create new pool with same name and new disk set, FS should be set to AUTO!
10. Done. start array.
```

Edited June 21Jun 21 by pacmancoder

16 hours ago, pacmancoder said:

I wanted to solve this with over-provisioning

This seems a good idea, most people will simple recreate the pool and copy back data, appreciate for those script.