Hi all,

I have a ZFS pool I created a few weeks ago - 5x20TB raidz1 pool. I believe my Rosewill hotswap enclosure had an iffy connection to SATA so one of the drives fell offline. I also upgraded to 7.1.2 and then 7.1.3 during this time but I'm not sure this is related. When I rebooted my other zpool was fine but the other had the ZFS Pool "Unmountable: Unsupported or No File System" error.

The system cannot find the bad zpool, named "hdd-one" at all. I've read a lot about possibilities and tried many of them, though I'm avoiding things like zpool import -f because I've read about possible data loss.

At the same time this happened I had a possible power spike on my other server and lost a few drives there. I believe they are totally (electrically) there.

What logs/data/info do I need to post here to try to fix this long enough to pull the data? Again, I'd have a backup but that got fried.

Thanks for any and all constructive help.

Please post the diagnostics.

Thanks for having a look. Note that I had an all-purpose ssd pool (single drive) formatted as ZFS that I used as cache and appdata, though I had no apps yet. I detached that to reduce complexity.

unraid-2-diagnostics-20250608-0744.zip

Post the output from zdb -l /dev/sdl1, and thge same for /dev/sdm1 /dev/sdn1 /dev/sdk1

root@unRAID-2:~# zdb -l /dev/sdl1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'hdd-one'

state: 0

txg: 309179

pool_guid: 13744187982852489828

errata: 0

hostname: 'unRAID-2'

top_guid: 432963734000058533

guid: 3428341674918404786

vdev_children: 1

vdev_tree:

type: 'raidz'

id: 0

guid: 432963734000058533

nparity: 1

metaslab_array: 65

metaslab_shift: 34

ashift: 12

asize: 100002919874560

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 3428341674918404786

path: '/dev/sdk1'

whole_disk: 0

DTL: 6343

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 2669037195086011101

path: '/dev/sdh1'

whole_disk: 0

DTL: 6342

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 7092376628010463207

path: '/dev/sdg1'

whole_disk: 0

DTL: 404

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 14126259427063294583

path: '/dev/sdn1'

whole_disk: 0

DTL: 6341

create_txg: 4

children[4]:

type: 'disk'

id: 4

guid: 3883892125914957370

path: '/dev/sdm1'

whole_disk: 0

DTL: 6340

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2 3

root@unRAID-2:~# /dev/sdm1 /dev/sdn1 /dev/sdk1

bash: /dev/sdm1: Permission denied

root@unRAID-2:~# /dev/sdm1

bash: /dev/sdm1: Permission denied

root@unRAID-2:~# /dev/sdn1

bash: /dev/sdn1: Permission denied

root@unRAID-2:~# /dev/sdk1

bash: /dev/sdk1: Permission denied

Thanks for the help.

Edited June 9, 20251 yr by BKTEK

6 hours ago, JorgeB said:

and thge same for /dev/sdm1 /dev/sdn1 /dev/sdk1

Sorry if I wasn't clear, by the same for these I meant:

zdb -l /dev/sdm1

zdb -l /dev/sdn1

zdb -l /dev/sdk1

root@unRAID-2:~# zdb -l /dev/sdm1

zdb -l /dev/sdn1failed to unpack label 0

failed to unpack label 1

failed to unpack label 2

failed to unpack label 3

root@unRAID-2:~# zdb -l /dev/sdn1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'hdd-one'

state: 0

txg: 309179

pool_guid: 13744187982852489828

errata: 0

hostname: 'unRAID-2'

top_guid: 432963734000058533

guid: 14126259427063294583

vdev_children: 1

vdev_tree:

type: 'raidz'

id: 0

guid: 432963734000058533

nparity: 1

metaslab_array: 65

metaslab_shift: 34

ashift: 12

asize: 100002919874560

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 3428341674918404786

path: '/dev/sdk1'

whole_disk: 0

DTL: 6343

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 2669037195086011101

path: '/dev/sdh1'

whole_disk: 0

DTL: 6342

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 7092376628010463207

path: '/dev/sdg1'

whole_disk: 0

DTL: 404

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 14126259427063294583

path: '/dev/sdn1'

whole_disk: 0

DTL: 6341

create_txg: 4

children[4]:

type: 'disk'

id: 4

guid: 3883892125914957370

path: '/dev/sdm1'

whole_disk: 0

DTL: 6340

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2 3

root@unRAID-2:~# zdb -l /dev/sdk1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'hdd-one'

state: 0

txg: 309179

pool_guid: 13744187982852489828

errata: 0

hostname: 'unRAID-2'

top_guid: 432963734000058533

guid: 2669037195086011101

vdev_children: 1

vdev_tree:

type: 'raidz'

id: 0

guid: 432963734000058533

nparity: 1

metaslab_array: 65

metaslab_shift: 34

ashift: 12

asize: 100002919874560

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 3428341674918404786

path: '/dev/sdk1'

whole_disk: 0

DTL: 6343

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 2669037195086011101

path: '/dev/sdh1'

whole_disk: 0

DTL: 6342

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 7092376628010463207

path: '/dev/sdg1'

whole_disk: 0

DTL: 404

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 14126259427063294583

path: '/dev/sdn1'

whole_disk: 0

DTL: 6341

create_txg: 4

children[4]:

type: 'disk'

id: 4

guid: 3883892125914957370

path: '/dev/sdm1'

whole_disk: 0

DTL: 6340

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2 3

sdm is missing the label, or the device ID changed, post the output from zpool import

root@unRAID-2:~# zpool import

pool: hdd-one

id: 13744187982852489828

state: FAULTED

status: One or more devices contains corrupted data.

action: The pool cannot be imported due to damaged devices or data.

The pool may be active on another system, but can be imported using

the '-f' flag.

see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E

config:

hdd-one FAULTED corrupted data

raidz1-0 DEGRADED

sdl1 ONLINE

sdk1 ONLINE

sdg1 ONLINE

sdn1 ONLINE

sdm1 UNAVAIL invalid label

pool: ssd-one

id: 14352649158317043617

state: ONLINE

action: The pool can be imported using its name or numeric identifier.

config:

ssd-one ONLINE

sdb1 ONLINE

root@unRAID-2:~#

zdb -l /dev/sdg1 please

root@unRAID-2:~# zdb -l /dev/sdg1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'hdd-one'

state: 0

txg: 309086

pool_guid: 13744187982852489828

errata: 0

hostname: 'unRAID-2'

top_guid: 432963734000058533

guid: 7092376628010463207

vdev_children: 1

vdev_tree:

type: 'raidz'

id: 0

guid: 432963734000058533

nparity: 1

metaslab_array: 65

metaslab_shift: 34

ashift: 12

asize: 100002919874560

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 3428341674918404786

path: '/dev/sdk1'

whole_disk: 0

DTL: 6343

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 2669037195086011101

path: '/dev/sdh1'

whole_disk: 0

DTL: 6342

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 7092376628010463207

path: '/dev/sdg1'

whole_disk: 0

DTL: 404

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 14126259427063294583

path: '/dev/sdn1'

whole_disk: 0

DTL: 6341

create_txg: 4

children[4]:

type: 'disk'

id: 4

guid: 3883892125914957370

path: '/dev/sdm1'

whole_disk: 0

DTL: 6340

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2

------------------------------------

LABEL 3

------------------------------------

version: 5000

name: 'hdd-one'

state: 0

txg: 309179

pool_guid: 13744187982852489828

errata: 0

hostname: 'unRAID-2'

top_guid: 432963734000058533

guid: 7092376628010463207

vdev_children: 1

vdev_tree:

type: 'raidz'

id: 0

guid: 432963734000058533

nparity: 1

metaslab_array: 65

metaslab_shift: 34

ashift: 12

asize: 100002919874560

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 3428341674918404786

path: '/dev/sdk1'

whole_disk: 0

DTL: 6343

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 2669037195086011101

path: '/dev/sdh1'

whole_disk: 0

DTL: 6342

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 7092376628010463207

path: '/dev/sdg1'

whole_disk: 0

DTL: 404

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 14126259427063294583

path: '/dev/sdn1'

whole_disk: 0

DTL: 6341

create_txg: 4

children[4]:

type: 'disk'

id: 4

guid: 3883892125914957370

path: '/dev/sdm1'

whole_disk: 0

DTL: 6340

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 3

Thanks for your continued help.

7 minutes ago, BKTEK said:

txg: 309086

This is the problem, this last deviec is not on the same transaction group has the other ones:

2 hours ago, BKTEK said:

txg: 309179

This, together with sdm having an invalid label, means that you have a raidz1 pool with two bad devices, so it cannot be imported.

You can try to rewind the pool so see if it can be imported with an earlier txg, but seconds/minutes of data would be lost:

First, offline the device with the bad label so that it doesn't interfere:

echo 1 > /sys/block/sdm/device/delete

Then type:

zpool import -F hdd-one

If that doesn't work, you can try as a last ditch effort to add -X, it will try more extreme measures, but can rewind to an inconsistent txg:

zpool import -FX hdd-one

Option 1 didn't work so it's the nuclear option now. It's thinking about it currently. I'll update when I know more. Thanks again.

root@unRAID-2:~# zpool import -F hdd-one

cannot import 'hdd-one': I/O error

Destroy and re-create the pool from

a backup source.

root@unRAID-2:~# zpool import -FX hdd-one

cannot import 'hdd-one': one or more devices is currently unavailable

1 hour ago, BKTEK said:

Destroy and re-create the pool from

I'm afraid this will be the only option.

Shame. Still not sure how that could have gone so bad with so many things happening at the same time (other server getting popped). Thank you for the help. Guess I'll need to move on.