Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

System fails to mount array, kernel panic on boot after update

sens_dep
in General Support

Featured Replies

updated earlier tonight and now my system fails to mount. Tried rolling back, disabling docker, removing a stick of ram, etc... not sure how to proceed. diag attached. showing weird drive data in the dashboard too:

 

image.thumb.png.78e95c562b648dce892b5c420cafa951.png

 

Intel 14th gen failure?

 

here is the panic when starting the array: 

Sep 15 19:48:02 GhostNAS emhttpd: shcmd (432): udevadm settle
Sep 15 19:48:02 GhostNAS emhttpd: Opening encrypted volumes...
Sep 15 19:48:02 GhostNAS emhttpd: shcmd (433): touch /boot/config/forcesync
Sep 15 19:48:02 GhostNAS emhttpd: Mounting disks...
Sep 15 19:48:02 GhostNAS emhttpd: mounting /mnt/disk1
Sep 15 19:48:02 GhostNAS emhttpd: shcmd (434): mkdir -p /mnt/disk1
Sep 15 19:48:02 GhostNAS emhttpd: shcmd (435): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1
Sep 15 19:48:02 GhostNAS kernel: SGI XFS with ACLs, security attributes, no debug enabled
Sep 15 19:48:02 GhostNAS kernel: XFS (md1p1): Mounting V5 Filesystem
Sep 15 19:48:03 GhostNAS kernel: XFS (md1p1): Starting recovery (logdev: internal)
Sep 15 19:48:03 GhostNAS emhttpd: shcmd (436): xfs_growfs /mnt/disk1
Sep 15 19:48:03 GhostNAS kernel: XFS (md1p1): Ending recovery (logdev: internal)
Sep 15 19:48:03 GhostNAS root: meta-data=/dev/md1p1             isize=512    agcount=13, agsize=268435455 blks
Sep 15 19:48:03 GhostNAS root:          =                       sectsz=512   attr=2, projid32bit=1
Sep 15 19:48:03 GhostNAS root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Sep 15 19:48:03 GhostNAS root:          =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
Sep 15 19:48:03 GhostNAS root: data     =                       bsize=4096   blocks=3418095603, imaxpct=5
Sep 15 19:48:03 GhostNAS root:          =                       sunit=0      swidth=0 blks
Sep 15 19:48:03 GhostNAS root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Sep 15 19:48:03 GhostNAS root: log      =internal log           bsize=4096   blocks=521728, version=2
Sep 15 19:48:03 GhostNAS root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Sep 15 19:48:03 GhostNAS root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Sep 15 19:48:03 GhostNAS emhttpd: mounting /mnt/disk3
Sep 15 19:48:03 GhostNAS emhttpd: shcmd (437): mkdir -p /mnt/disk3
Sep 15 19:48:03 GhostNAS emhttpd: shcmd (438): mount -t xfs -o noatime,nouuid /dev/md3p1 /mnt/disk3
Sep 15 19:48:03 GhostNAS kernel: XFS (md3p1): Mounting V5 Filesystem
Sep 15 19:48:03 GhostNAS kernel: XFS (md3p1): Starting recovery (logdev: internal)
Sep 15 19:48:03 GhostNAS emhttpd: shcmd (439): xfs_growfs /mnt/disk3
Sep 15 19:48:03 GhostNAS kernel: XFS (md3p1): Ending recovery (logdev: internal)
Sep 15 19:48:03 GhostNAS root: meta-data=/dev/md3p1             isize=512    agcount=8, agsize=268435455 blks
Sep 15 19:48:03 GhostNAS root:          =                       sectsz=512   attr=2, projid32bit=1
Sep 15 19:48:03 GhostNAS root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Sep 15 19:48:03 GhostNAS root:          =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
Sep 15 19:48:03 GhostNAS root: data     =                       bsize=4096   blocks=1953506633, imaxpct=5
Sep 15 19:48:03 GhostNAS root:          =                       sunit=0      swidth=0 blks
Sep 15 19:48:03 GhostNAS root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Sep 15 19:48:03 GhostNAS root: log      =internal log           bsize=4096   blocks=521728, version=2
Sep 15 19:48:03 GhostNAS root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Sep 15 19:48:03 GhostNAS root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Sep 15 19:48:03 GhostNAS emhttpd: mounting /mnt/disk4
Sep 15 19:48:03 GhostNAS emhttpd: shcmd (440): mkdir -p /mnt/disk4
Sep 15 19:48:03 GhostNAS emhttpd: shcmd (441): mount -t xfs -o noatime,nouuid /dev/md4p1 /mnt/disk4
Sep 15 19:48:03 GhostNAS kernel: XFS (md4p1): Mounting V5 Filesystem
Sep 15 19:48:04 GhostNAS kernel: XFS (md4p1): Starting recovery (logdev: internal)
Sep 15 19:48:04 GhostNAS emhttpd: shcmd (442): xfs_growfs /mnt/disk4
Sep 15 19:48:04 GhostNAS kernel: XFS (md4p1): Ending recovery (logdev: internal)
Sep 15 19:48:04 GhostNAS root: meta-data=/dev/md4p1             isize=512    agcount=8, agsize=268435455 blks
Sep 15 19:48:04 GhostNAS root:          =                       sectsz=512   attr=2, projid32bit=1
Sep 15 19:48:04 GhostNAS root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Sep 15 19:48:04 GhostNAS root:          =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
Sep 15 19:48:04 GhostNAS root: data     =                       bsize=4096   blocks=1953506633, imaxpct=5
Sep 15 19:48:04 GhostNAS root:          =                       sunit=0      swidth=0 blks
Sep 15 19:48:04 GhostNAS root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Sep 15 19:48:04 GhostNAS root: log      =internal log           bsize=4096   blocks=521728, version=2
Sep 15 19:48:04 GhostNAS root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Sep 15 19:48:04 GhostNAS root: realtime =none                   extsz=4096   blocks=0, rtextents=0
Sep 15 19:48:04 GhostNAS emhttpd: mounting /mnt/cache
Sep 15 19:48:04 GhostNAS emhttpd: shcmd (443): mkdir -p /mnt/cache
Sep 15 19:48:04 GhostNAS emhttpd: shcmd (444): /usr/sbin/zpool import -f -N -o autoexpand=on  -d /dev/nvme0n1p1 -d /dev/nvme1n1p1 811097023252368398 cache
Sep 15 19:48:10 GhostNAS kernel: VERIFY3(size <= rt->rt_space) failed (281442900340736 <= 16139841536)
Sep 15 19:48:10 GhostNAS kernel: PANIC at range_tree.c:436:range_tree_remove_impl()
Sep 15 19:48:10 GhostNAS kernel: Showing stack for process 19845
Sep 15 19:48:10 GhostNAS kernel: CPU: 8 PID: 19845 Comm: z_wr_iss Tainted: P           O       6.1.79-Unraid #1
Sep 15 19:48:10 GhostNAS kernel: Hardware name: Micro-Star International Co., Ltd. MS-7D29/MEG Z690I UNIFY (MS-7D29), BIOS 1.I0 07/26/2024
Sep 15 19:48:10 GhostNAS kernel: Call Trace:
Sep 15 19:48:10 GhostNAS kernel: <TASK>
Sep 15 19:48:10 GhostNAS kernel: dump_stack_lvl+0x44/0x5c
Sep 15 19:48:10 GhostNAS kernel: spl_panic+0xd0/0xe8 [spl]
Sep 15 19:48:10 GhostNAS kernel: ? bt_grow_leaf+0xc3/0xd6 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? zfs_btree_find_in_buf+0x4c/0x94 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? zfs_btree_find+0x16d/0x1b0 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? rs_get_start+0xc/0x1d [zfs]
Sep 15 19:48:10 GhostNAS kernel: range_tree_remove_impl+0x77/0x406 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? zio_wait+0x1ee/0x1fd [zfs]
Sep 15 19:48:10 GhostNAS kernel: space_map_load_callback+0x70/0x79 [zfs]
Sep 15 19:48:10 GhostNAS kernel: space_map_iterate+0x2d3/0x324 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? spa_stats_destroy+0x16c/0x16c [zfs]
Sep 15 19:48:10 GhostNAS kernel: space_map_load_length+0x93/0xcb [zfs]
Sep 15 19:48:10 GhostNAS kernel: metaslab_load+0x33b/0x6e3 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? slab_post_alloc_hook+0x4d/0x15e
Sep 15 19:48:10 GhostNAS kernel: ? rs_get_start+0xc/0x1d [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl]
Sep 15 19:48:10 GhostNAS kernel: ? __kmem_cache_alloc_node+0x118/0x147
Sep 15 19:48:10 GhostNAS kernel: metaslab_activate+0x36/0x1f1 [zfs]
Sep 15 19:48:10 GhostNAS kernel: metaslab_alloc_dva+0x8bc/0xfce [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? preempt_latency_start+0x2b/0x46
Sep 15 19:48:10 GhostNAS kernel: metaslab_alloc+0x107/0x1fd [zfs]
Sep 15 19:48:10 GhostNAS kernel: zio_dva_allocate+0xee/0x73f [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? kmem_cache_free+0xc9/0x154
Sep 15 19:48:10 GhostNAS kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl]
Sep 15 19:48:10 GhostNAS kernel: ? preempt_latency_start+0x2b/0x46
Sep 15 19:48:10 GhostNAS kernel: ? _raw_spin_lock+0x13/0x1c
Sep 15 19:48:10 GhostNAS kernel: ? _raw_spin_unlock+0x14/0x29
Sep 15 19:48:10 GhostNAS kernel: ? tsd_hash_search+0x70/0x7d [spl]
Sep 15 19:48:10 GhostNAS kernel: zio_execute+0xb1/0xdf [zfs]
Sep 15 19:48:10 GhostNAS kernel: taskq_thread+0x266/0x38a [spl]
Sep 15 19:48:10 GhostNAS kernel: ? wake_up_q+0x44/0x44
Sep 15 19:48:10 GhostNAS kernel: ? zio_subblock+0x22/0x22 [zfs]
Sep 15 19:48:10 GhostNAS kernel: ? taskq_dispatch_delay+0x106/0x106 [spl]
Sep 15 19:48:10 GhostNAS kernel: kthread+0xe4/0xef
Sep 15 19:48:10 GhostNAS kernel: ? kthread_complete_and_exit+0x1b/0x1b
Sep 15 19:48:10 GhostNAS kernel: ret_from_fork+0x1f/0x30
Sep 15 19:48:10 GhostNAS kernel: </TASK>


 

ghostnas-diagnostics-20240915-2140.zip

Edited by sens_dep

  • Quote
    • Author

    Update. Was able to start the array after mounting the cache in read only
     

    1. disable auto start of array
    2. reboot
    3. mount cache read only: 
      zpool import -o readonly=on cache
    4. start the array
    5. seems fine?

    would the path here be to backup all the cache and reformat?

     

    thanks

  • Quote
    • Community Expert
    6 hours ago, sens_dep said:

    would the path here be to backup all the cache and reformat?

    Yep

  • Quote
    • Author
    2 hours ago, JorgeB said:

    Yep

    What causes this? It seemed nominal before I triggered the update. Any way to monitor this / be more prepared in the future?

  • Quote
    • Community Expert

    Difficult to say for certain, could be a hardware issue, like bad RAM, device/controller firmware, or a ZFS issue/bug, but if it keeps happening I would bet on the former.

  • Quote

    • Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Reply to this topic...
    Go to topic listing

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.