Replaced a disk in a zfs pool, eesilvering was done and things looked good, or so I thought.
However, the old disk remained in the pool as "FAULTED"

Tried different things, replace, remove, detach and finally leabelclear, which seems to have done something, but now I can not start the array anymore because of "Wrong Pool State zraid - invalid config"

After I did a labelclear, since it was angry about the label, array won't start anymore and zpool import does not look fun...

Before labelclear:
config:

NAME STATE READ WRITE CKSUM

zraid DEGRADED 0 0 0

raidz1-0 DEGRADED 0 0 0

16668474481916585699 FAULTED 0 0 0 was /dev/sda1

sda1 ONLINE 0 0 0

sdc1 ONLINE 0 0 0

sdd1 ONLINE 0 0 0

sdf1 ONLINE 0 0 0

sde1 ONLINE 0 0 0

After:

config:

zraid UNAVAIL insufficient replicas

raidz1-0 UNAVAIL insufficient replicas

16668474481916585699 FAULTED corrupted data

sda1 FAULTED corrupted data

sdc1 ONLINE

sdd1 ONLINE

sdf1 ONLINE

sde1 ONLINE

Please advise.

Thanks.

For which device did you run labeclear? I wish you had asked for help earlier. There are now two invalid disks for a raidz1 pool, so the pool cannot be imported.

I ran it for sda ... Yeah I also wish I'd done that

Should not listen to random posts on the internet.

It seems however that all labels (/dev assignments?) has been changed around...

So the pool is now unrecovereable?

Edited May 23May 23 by ambreswe

Possibly, try

zpool import -F pool_name

if that fails, post the output from

zdb -l /dev/sda1

7 minutes ago, JorgeB said:

Possibly, try

zpool import -F pool_name

if that fails, post the output from

zdb -l /dev/sda1

root@unraid:~# zpool import -F zraid

cannot import 'zraid': one or more devices is currently unavailable

root@unraid:~# zdb -l /dev/sda1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'zraid'

state: 0

txg: 0

pool_guid: 4182961424354843335

errata: 0

min_alloc: 4096

max_alloc: 4096

hostname: 'unraid'

top_guid: 14611536735580611996

guid: 11461170224768266555

vdev_children: 1

vdev_tree:

type: 'raidz'

id: 0

guid: 14611536735580611996

nparity: 1

raidz_expand_txgs[0]: 3936041

raidz_expand_txgs[1]: 3970252

metaslab_array: 256

metaslab_shift: 34

ashift: 12

asize: 48009350479872

min_alloc: 4096

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 16668474481916585699

path: '/dev/sda1'

whole_disk: 0

not_present: 1

DTL: 30187

create_txg: 4

children[1]:

type: 'replacing'

id: 1

guid: 666396365514631839

whole_disk: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 11461170224768266555

path: '/dev/sdb1'

whole_disk: 0

DTL: 30186

children[1]:

type: 'disk'

id: 1

guid: 6689874950903492856

path: '/dev/sda1'

whole_disk: 0

DTL: 147

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 17863017234158965939

path: '/dev/sdc1'

whole_disk: 0

DTL: 30185

create_txg: 4

children[3]:

type: 'disk'

id: 3

guid: 18345851527532164897

path: '/dev/sdd1'

whole_disk: 0

DTL: 30184

create_txg: 4

children[4]:

type: 'disk'

id: 4

guid: 6909868677911286036

path: '/dev/sdf1'

whole_disk: 0

DTL: 392

create_txg: 4

children[5]:

type: 'disk'

id: 5

guid: 12718430070841080607

path: '/dev/sde1'

whole_disk: 0

DTL: 24

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

org.openzfs:raidz_expansion

create_txg: 0

labels = 0 1 2 3

Thanks for helping

Long shot but... if I remove the replacement disk, and inset the old one, can that in any way help the situation?

It may be worth a try, but before that, post the output from

zpool import -d /dev/disk/by-id/

5 minutes ago, JorgeB said:

It may be worth a try, but before that, post the output from

zpool import -d /dev/disk/by-id/

pool: zraid

id: 4182961424354843335

state: UNAVAIL

status: One or more devices contains corrupted data.

action: The pool cannot be imported due to damaged devices or data.

see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E

config:

zraid UNAVAIL insufficient replicas

raidz1-0 UNAVAIL insufficient replicas

16668474481916585699 FAULTED corrupted data

sda1 FAULTED corrupted data

wwn-0x5000cca261c3595e-part1 ONLINE

wwn-0x5000cca261c33ef0-part1 ONLINE

wwn-0x5000cca261c34ce1-part1 ONLINE

wwn-0x5000cca261c34a7f-part1 ONLINE

I also tried putting the old drive back to see if there's any difference in the zpool import command and yeah, there is.

Is this something we can work with?

pool: zraid

id: 4182961424354843335

state: FAULTED

status: One or more devices contains corrupted data.

action: The pool cannot be imported due to damaged devices or data.

see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E

config:

zraid FAULTED corrupted data

raidz1-0 DEGRADED

sdb1 ONLINE

sda1 FAULTED corrupted data

sdc1 ONLINE

sdd1 ONLINE

sdf1 ONLINE

sde1 ONLINE

Again, thanks for helping, much appreciated.

I don't see a way to import the pool with the new drive.

The old one shows online but may be out of date. You try to go back a few TXGs to see if can be imported (with the old drive connected).

zpool import -FX zraid

If that still fails, I'm afraid you may need to destroy the pool and recover from a backup.

8 minutes ago, JorgeB said:

I don't see a way to import the pool with the new drive.

The old one shows online but may be out of date. You try to go back a few TXGs to see if can be imported (with the old drive connected).

zpool import -FX zraid

If that still fails, I'm afraid you may need to destroy the pool and recover from a backup.

Put the command in and didn't get the prompt back, yet... I hope this works!

Sadly, I think I'm screwed.

root@unraid:~# zpool import -FX zraid

cannot import 'zraid': one or more devices is currently unavailable

I'll start the tedious process of trying to restore everything then I guess...

Thanks for the help!