Background:

• upgraded cache pool SSD's to a bigger ones.   Did the necessary to move all files on cache to array (only "appdata" and "system" were on cache)
• new cache pool has 2 SSDs in a pool formatted to zfs.
• changed the mover direction for "appdata" and "system" now Array->Cache
• initiated mover to move the two back from Array to cache

 

What happened:

• log shows a log of zio errors on one SSD in the pool.
• now the zpool "cache" is suspended.
• followed suggestion from "zpool status -x" and did a "zpool clear cache".
• seems not much use ...

  pool: cache
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
config:

        NAME        STATE     READ WRITE CKSUM
        cache       UNAVAIL      0     0     0  insufficient replicas
          mirror-0  UNAVAIL      0     0     0  insufficient replicas
            sdi1    FAULTED      6     0     0  too many errors
            sdj1    FAULTED      3     0     0  too many errors

errors: No known data errors

Not sure what could be done now.   All files in "system" are still in Array.   Some files in "appdata" are already moved to Cache.

But mover seem hanged, cannot stop array (grey out) and "move" is also grey out.  See screenshot 

vault-diagnostics-20241208-2004.zip

Edited December 8, 20241 yr by dandus

Update:

- now my unraid is back on, cache pool is normal, parity checking in progress

 

What has been done:

- at command prompt did a ”mover stop"

- this revived the button to stop the array

- tried stopping the array but can't umount /mnt/user.  A "ps X" and "lsof" shows that the mover process is holding on to /mnt/user.  But the mover process is in the uninterruptible D status and cannot be killed

- forced a power recycle

Edited December 8, 20241 yr by dandus

I did a smart test on the SSD in the cache pool, healthy!  Why were there so much zio error

Dec  8 19:35:06 vault kernel: ata1: COMRESET failed (errno=-16)
Dec  8 19:35:06 vault kernel: ata1: reset failed, giving up
Dec  8 19:35:06 vault kernel: ata1.00: disable device
Dec  8 19:35:06 vault kernel: ata1: EH complete
...
Dec  8 19:35:06 vault kernel: ata2: COMRESET failed (errno=-16)
Dec  8 19:35:06 vault kernel: ata2: reset failed, giving up
Dec  8 19:35:06 vault kernel: ata2.00: disable device
Dec  8 19:35:06 vault kernel: ata2: EH complete

 

Both cache devices dropped offline, could be a power/connection issue.

2 hours ago, JorgeB said:

Dec  8 19:35:06 vault kernel: ata1: COMRESET failed (errno=-16)
Dec  8 19:35:06 vault kernel: ata1: reset failed, giving up
Dec  8 19:35:06 vault kernel: ata1.00: disable device
Dec  8 19:35:06 vault kernel: ata1: EH complete
...
Dec  8 19:35:06 vault kernel: ata2: COMRESET failed (errno=-16)
Dec  8 19:35:06 vault kernel: ata2: reset failed, giving up
Dec  8 19:35:06 vault kernel: ata2.00: disable device
Dec  8 19:35:06 vault kernel: ata2: EH complete

 

Both cache devices dropped offline, could be a power/connection issue.

I have noticed this.   The SATA and power connectors to the new SSDs are the same as for the old ones.   I will try to open up, unplug and plug again.