I had a lot of snapshots and decided to get rid of them with
zfs list -H -o name -t snapshot | grep autosnap | xargs -n1 zfs destroy
This spat out a few "snapshot not found" errors and the system got really slow until I noticed some apps could no longer write files.
System no longer responded and I had to power cycle it.
Now system hangs at boot but I'm still able to ssh into the machine. Everything seems to hang waiting for
zpool import -a
Commands like zpool status don't seem to work. /proc/spl/kstat/zfs/dbgmsg contains the following
timestamp message
1681372320 spa.c:6242:spa_tryimport(): spa_tryimport: importing tank
1681372320 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADING
1681372320 vdev.c:160:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/ata-ST12000NM001G-2MV103_ZL274PFX-part1': best uberblock found for spa $import. txg 18136025
1681372320 spa_misc.c:418:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=18136025
1681372320 spa.c:8360:spa_async_request(): spa=$import async request task=4096
1681372321 spa.c:8360:spa_async_request(): spa=$import async request task=2048
1681372321 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): LOADED
1681372321 spa_misc.c:418:spa_load_note(): spa_load($import, config trusted): UNLOADING
1681372321 spa.c:6098:spa_import(): spa_import: importing tank
1681372321 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1681372321 vdev.c:160:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/ata-ST12000NM001G-2MV103_ZL274PFX-part1': best uberblock found for spa tank. txg 18136025
1681372321 spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=18136025
1681372321 spa.c:8360:spa_async_request(): spa=tank async request task=4096
1681372341 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): read 182 log space maps (4100 total blocks - blksz = 131072 bytes) in 17406 ms
1681372371 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): spa_load_verify found 0 metadata errors and 53 data errors
1681372371 mmp.c:240:mmp_thread_start(): MMP thread started pool 'tank' gethrtime 354061113656
1681372371 spa.c:8360:spa_async_request(): spa=tank async request task=1
1681372371 spa.c:2606:spa_livelist_delete_cb(): deleting sublist (id 95993) from livelist 95991, 0 remaining
1681372371 spa.c:8360:spa_async_request(): spa=tank async request task=2048
1681372371 spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADED
1681372371 spa.c:8360:spa_async_request(): spa=tank async request task=4096
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 937, smp_length 135136, unflushed_allocs 442368, unflushed_frees 12580790272, freed 0, defer 0 + 0, unloaded time 354384 ms, loading_time 51 ms, ms_max_size 14897127424, max size error 14082514944, old_weight 840000000000001, new_weight 840000000000001
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 1105, smp_length 387240, unflushed_allocs 188547072, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 354436 ms, loading_time 64 ms, ms_max_size 16458719232, max size error 16458719232, old_weight 840000000000001, new_weight 840000000000001
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 1220, smp_length 518048, unflushed_allocs 303104, unflushed_frees 14799437824, freed 0, defer 0 + 0, unloaded time 354500 ms, loading_time 74 ms, ms_max_size 15683502080, max size error 12946800640, old_weight 840000000000001, new_weight 840000000000001
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 2374, smp_length 326880, unflushed_allocs 2367995904, unflushed_frees 258826240, freed 0, defer 0 + 0, unloaded time 354575 ms, loading_time 104 ms, ms_max_size 12846678016, max size error 12846579712, old_weight 840000000000001, new_weight 840000000000001
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 49, smp_length 384, unflushed_allocs 1228546048, unflushed_frees 90112, freed 0, defer 0 + 0, unloaded time 354679 ms, loading_time 56 ms, ms_max_size 6055149568, max size error 6055059456, old_weight 800000000000001, new_weight 800000000000001
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 2478, smp_length 207680, unflushed_allocs 2539520, unflushed_frees 8653791232, freed 0, defer 0 + 0, unloaded time 354735 ms, loading_time 51 ms, ms_max_size 5210308608, max size error 1588199424, old_weight 800000000000001, new_weight 800000000000001
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 2480, smp_length 142144, unflushed_allocs 712704, unflushed_frees 8535842816, freed 0, defer 0 + 0, unloaded time 354787 ms, loading_time 76 ms, ms_max_size 3770638336, max size error 1008377856, old_weight 7c0000000000002, new_weight 7c0000000000002
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 2497, smp_length 1480, unflushed_allocs 5153120256, unflushed_frees 2602729472, freed 0, defer 0 + 0, unloaded time 354863 ms, loading_time 122 ms, ms_max_size 3899867136, max size error 2694348800, old_weight 7c0000000000002, new_weight 7c0000000000002
1681372372 metaslab.c:2437:metaslab_load_impl(): metaslab_load: txg 18136027, spa tank, vdev_id 0, ms_id 2093, smp_length 66136, unflushed_allocs 315039744, unflushed_frees 15409152, freed 0, defer 0 + 0, unloaded time 354985 ms, loading_time 40 ms, ms_max_size 3293323264, max size error 3293306880, old_weight 7c0000000000001, new_weight 7c0000000000001
1681372372 spa_history.c:307:spa_history_log_sync(): txg 18136027 open pool version 5000; software version zfs-2.1.9-0-g92e0d9d18-dist; uts Tower 5.19.17-Unraid #2 SMP PREEMPT_DYNAMIC Wed Nov 2 11:54:15 PDT 2022 x86_64
1681372398 dsl_scan.c:3433:dsl_process_async_destroys(): freed 27648 blocks in 25843ms from free_bpobj/bptree txg 18136027; err=85