Jump to content
We're Hiring! Full Stack Developer ×
  • [6.12.2] unable to mount zfs: removing nonexistent segment from range tree


    chuckbucket
    • Closed

    I am unable to mount my ZFS pool, starting the array in Maintenance mode work. Running in safemode sometimes work. My system has been less stable since upgrading to 6.12.x, but has improved since I changed docker to use ipvlan. Things were running well until yesterday, the system took a hard crash and have been unable to mount the ZFS shares since.

     

    The diagnostic tool did not grab my entire log file, so I manually added it to the zip.

     

    Jul  7 02:07:24 MCP emhttpd: mounting /mnt/cache
    Jul  7 02:07:24 MCP emhttpd: shcmd (176): mkdir -p /mnt/cache
    Jul  7 02:07:24 MCP emhttpd: shcmd (177): /usr/sbin/zpool import -N -o autoexpand=on  -d /dev/nvme0n1p1 -d /dev/nvme1n1p1 -d /dev/sdf1 12239425747297594656 cache
    Jul  7 02:07:32 MCP kernel: PANIC: zfs: removing nonexistent segment from range tree (offset=9901bf8881000 size=5008000)
    Jul  7 02:07:32 MCP kernel: Showing stack for process 3781
    Jul  7 02:07:32 MCP kernel: CPU: 5 PID: 3781 Comm: metaslab_group_ Tainted: P           O       6.1.36-Unraid #1
    Jul  7 02:07:32 MCP kernel: Hardware name: Gigabyte Technology Co., Ltd. H470M DS3H/H470M DS3H, BIOS F22 11/23/2021
    Jul  7 02:07:32 MCP kernel: Call Trace:
    Jul  7 02:07:32 MCP kernel: <TASK>
    Jul  7 02:07:32 MCP kernel: dump_stack_lvl+0x44/0x5c
    Jul  7 02:07:32 MCP kernel: vcmn_err+0x86/0xc3 [spl]
    Jul  7 02:07:32 MCP kernel: ? put_cpu_partial+0x6e/0x9f
    Jul  7 02:07:32 MCP kernel: ? bt_grow_leaf+0xc3/0xd6 [zfs]
    Jul  7 02:07:32 MCP kernel: ? bt_grow_leaf+0xc3/0xd6 [zfs]
    Jul  7 02:07:32 MCP kernel: ? zfs_btree_find_in_buf+0x4c/0x94 [zfs]
    Jul  7 02:07:32 MCP kernel: ? zfs_btree_find_in_buf+0x4c/0x94 [zfs]
    Jul  7 02:07:32 MCP kernel: zfs_panic_recover+0x6b/0x86 [zfs]
    Jul  7 02:07:32 MCP kernel: range_tree_remove_impl+0xc3/0x406 [zfs]
    Jul  7 02:07:32 MCP kernel: ? zio_wait+0x1ee/0x1fd [zfs]
    Jul  7 02:07:32 MCP kernel: space_map_load_callback+0x70/0x79 [zfs]
    Jul  7 02:07:32 MCP kernel: space_map_iterate+0x2d3/0x324 [zfs]
    Jul  7 02:07:32 MCP kernel: ? spa_stats_destroy+0x16c/0x16c [zfs]
    Jul  7 02:07:32 MCP kernel: space_map_load_length+0x93/0xcb [zfs]
    Jul  7 02:07:32 MCP kernel: metaslab_load+0x33b/0x6e3 [zfs]
    Jul  7 02:07:32 MCP kernel: ? _raw_spin_unlock_irqrestore+0x24/0x3a
    Jul  7 02:07:32 MCP kernel: ? __wake_up_common_lock+0x88/0xbb
    Jul  7 02:07:32 MCP kernel: metaslab_preload+0x4c/0x97 [zfs]
    Jul  7 02:07:32 MCP kernel: taskq_thread+0x266/0x38a [spl]
    Jul  7 02:07:32 MCP kernel: ? wake_up_q+0x44/0x44
    Jul  7 02:07:32 MCP kernel: ? taskq_dispatch_delay+0x106/0x106 [spl]
    Jul  7 02:07:32 MCP kernel: kthread+0xe4/0xef
    Jul  7 02:07:32 MCP kernel: ? kthread_complete_and_exit+0x1b/0x1b
    Jul  7 02:07:32 MCP kernel: ret_from_fork+0x1f/0x30
    Jul  7 02:07:32 MCP kernel: </TASK>

    mcp-diagnostics-20230707-1414.zip




    User Feedback

    Recommended Comments

    Looks like the pool is corrupted, see if it can be imported in read-only mode, if yes backup and re-create:

     

    zpool import -o readonly=on cache

     

    If it mounts read-only you can then start the array to access your other storage, but cache will still appear unmountable in the GUI.

    Link to comment

    I could not directly fix the issue, but I was able to access recover all of my data.

     

    I used the terminal:


    @MCP:~# zpool import

       pool: cache
         id: 12239425747297594656
      state: ONLINE
     action: The pool can be imported using its name or numeric identifier.
     config:
    
            cache          ONLINE
              raidz1-0     ONLINE
                nvme0n1p1  ONLINE
                nvme1n1p1  ONLINE
                sde1       ONLINE

    'zpool settings' for whatever reason thought nothing was wrong with my zfs pool, but mounting it with 'zpool import cache' caused a kernel panic which caused the entire system to hang,


    @MCP:~# zpool import -o readonly=on cache

    This allowed me to mount the cache pool as read only. I then used the file manager to copy all of my data to a USB drive. Then I deleted the zfs cache pool and re created it.

     

    I don't know what caused my pool to corrupt, but the issue has been solved. I will set the status to closed.

    Link to comment

    I have the same problem. Not sure what happened but all my pools got corrupted and had files corrupted in them. I suspect some kind of memory error but from my tests all hw is ok. I'm using most enterprise hw and have battery backup.BRTFS had fixed the errors on the fs and all files are ok but ZFS all those corrupted files are lost(can't even copy them,disks are ok). These are backup files so no big deal but I'm getting rid of ZFS because of this behavior . 

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.

×
×
  • Create New...