January 2, 20251 yr Hello, I was troubleshooting sporadic lockup issues in my setup, which I believe I have finally tracked down to corruption of a vdisk file that resides in my cache pool of 3x1tb NVMe drives in raidz1. After a scrub/clear I always get the following, two checksum issues on every disk: root@BaradDur:~# zpool status pool: cache state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: scrub repaired 192K in 00:01:20 with 0 errors on Thu Jan 2 13:53:21 2025 config: NAME STATE READ WRITE CKSUM cache ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 nvme1n1p1 ONLINE 0 0 2 nvme2n1p1 ONLINE 0 0 2 nvme0n1p1 ONLINE 0 0 2 errors: No known data errors I found the messed up file by process of elimination - I have moved everything else off of that pool successfully. Any attempt to start that vm, copy the img file locally via mover/mc/etc. or remotely via smb causes the server to lock up / all shares become inaccessible / etc. Dec 31 15:55:00 BaradDur kernel: VERIFY3(size <= rt->rt_space) failed (196608 <= 193536) Dec 31 15:55:00 BaradDur kernel: PANIC at range_tree.c:436:range_tree_remove_impl() Dec 31 15:55:00 BaradDur kernel: Showing stack for process 21163 Dec 31 15:55:00 BaradDur kernel: CPU: 2 PID: 21163 Comm: dsl_scan_iss Tainted: P O 6.1.74-Unraid #1 Dec 31 15:55:00 BaradDur kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X399 Taichi, BIOS P4.03 01/18/2024 Dec 31 15:55:00 BaradDur kernel: Call Trace: Dec 31 15:55:00 BaradDur kernel: <TASK> Dec 31 15:55:00 BaradDur kernel: dump_stack_lvl+0x44/0x5c Dec 31 15:55:00 BaradDur kernel: spl_panic+0xd0/0xe8 [spl] Dec 31 15:55:00 BaradDur kernel: ? preempt_latency_start+0x2b/0x46 Dec 31 15:55:00 BaradDur kernel: ? _raw_spin_lock+0x13/0x1c Dec 31 15:55:00 BaradDur kernel: ? _raw_spin_unlock+0x14/0x29 Dec 31 15:55:00 BaradDur kernel: ? slab_free_freelist_hook.constprop.0+0x3b/0xaf Dec 31 15:55:00 BaradDur kernel: ? _raw_spin_unlock_irqrestore+0x24/0x3a Dec 31 15:55:00 BaradDur kernel: ? kmem_cache_free+0x10f/0x154 Dec 31 15:55:00 BaradDur kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl] Dec 31 15:55:00 BaradDur kernel: ? slab_post_alloc_hook+0x4d/0x15e Dec 31 15:55:00 BaradDur kernel: range_tree_remove_impl+0x77/0x406 [zfs] Dec 31 15:55:00 BaradDur kernel: ? kmem_cache_free+0xc9/0x154 Dec 31 15:55:00 BaradDur kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl] Dec 31 15:55:00 BaradDur kernel: scan_io_queues_run_one+0x3c5/0x586 [zfs] Dec 31 15:55:00 BaradDur kernel: taskq_thread+0x269/0x38a [spl] Dec 31 15:55:00 BaradDur kernel: ? wake_up_q+0x44/0x44 Dec 31 15:55:00 BaradDur kernel: ? taskq_dispatch_delay+0x106/0x106 [spl] Dec 31 15:55:00 BaradDur kernel: kthread+0xe7/0xef Dec 31 15:55:00 BaradDur kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 31 15:55:00 BaradDur kernel: ret_from_fork+0x22/0x30 Dec 31 15:55:00 BaradDur kernel: </TASK> The behavior is very repeatable - past a certain point copying that file the WebUI reports CPU2 at 100%, then services start to become unresponsive and more cores will go to 100%. If I'm already ssh'd into the server I can interact with it minimally, for example I can issue a reboot command and it will tell me it's going down NOW - but it doesn't actually do anything past that. The only solution I have found is hard reboot the server. In the mean time, I have added a regular SSD for cache in a new pool and am moving all of my AppData over there to get my hosted services running again. I have run memtest86+ overnight and all tests were successful. All NVMe disks recently reported no errors on extended SMART test. I was NOT running scrubs on an automated schedule prior to this, but that is now scheduled. Does anyone have any recommendations on how to proceed from here? Is there any way to look at recovering that VM Disk? I fully embrace that it's my fault for not backing it up before this, and It's not the end of the world if it is totally lost, but I would like to explore any options How do I prevent this from occurring in the future? Anything that could indicate what caused corruption across the entire pool? If it is totally hooped - how do I proceed safely destroying that pool and re-adding it? Thank you! baraddur-diagnostics-20250102-1339.zip
January 3, 20251 yr Community Expert Probably won't help in this case but try mounting the pool read-only with: zpool import -o readonly=on cache The try copying that file
January 3, 20251 yr Author 7 hours ago, JorgeB said: Probably won't help in this case but try mounting the pool read-only with: zpool import -o readonly=on cache The try copying that file Thanks for the response! I stopped the array, mounted the pool as readonly, and attempted to copy it off to a remote smb share using MC. Sadly it still stalls at 37% and dumps the following into syslog: Jan 3 12:10:14 BaradDur kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Jan 3 12:10:14 BaradDur kernel: (detected by 2, t=60002 jiffies, g=24323593, q=180027 ncpus=16) Jan 3 12:10:14 BaradDur kernel: rcu: All QSes seen, last rcu_preempt kthread activity 60002 (4378039330-4377979328), jiffies_till_next_fqs=3, root ->qsmask 0x0 Jan 3 12:10:14 BaradDur kernel: rcu: rcu_preempt kthread starved for 60002 jiffies! g24323593 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 Jan 3 12:10:14 BaradDur kernel: rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior. Jan 3 12:10:14 BaradDur kernel: rcu: RCU grace-period kthread stack dump: Jan 3 12:10:14 BaradDur kernel: task:rcu_preempt state:R running task stack:0 pid:15 ppid:2 flags:0x00004000 Jan 3 12:10:14 BaradDur kernel: Call Trace: Jan 3 12:10:14 BaradDur kernel: <TASK> Jan 3 12:10:14 BaradDur kernel: __schedule+0x5b2/0x612 Jan 3 12:10:14 BaradDur kernel: ? _raw_spin_unlock_irqrestore+0x24/0x3a Jan 3 12:10:14 BaradDur kernel: ? __mod_timer+0x207/0x232 Jan 3 12:10:14 BaradDur kernel: ? rcu_gp_init+0x494/0x494 Jan 3 12:10:14 BaradDur kernel: schedule+0x8e/0xcc Jan 3 12:10:14 BaradDur kernel: schedule_timeout+0x9d/0xd7 Jan 3 12:10:14 BaradDur kernel: ? __next_timer_interrupt+0xf6/0xf6 Jan 3 12:10:14 BaradDur kernel: rcu_gp_fqs_loop+0x12d/0x475 Jan 3 12:10:14 BaradDur kernel: rcu_gp_kthread+0x151/0x16d Jan 3 12:10:14 BaradDur kernel: kthread+0xe7/0xef Jan 3 12:10:14 BaradDur kernel: ? kthread_complete_and_exit+0x1b/0x1b Jan 3 12:10:14 BaradDur kernel: ret_from_fork+0x22/0x30 Jan 3 12:10:14 BaradDur kernel: </TASK> Jan 3 12:10:14 BaradDur kernel: rcu: Stack dump where RCU GP kthread last ran: Jan 3 12:10:14 BaradDur kernel: CPU: 2 PID: 125681 Comm: z_rd_int_1 Tainted: P O 6.1.74-Unraid #1 Jan 3 12:10:14 BaradDur kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X399 Taichi, BIOS P4.03 01/18/2024 Jan 3 12:10:14 BaradDur kernel: RIP: 0010:abd_fletcher_4_iter+0x0/0x95 [zcommon] Jan 3 12:10:14 BaradDur kernel: Code: 48 81 fe ff 01 00 00 48 89 d1 48 89 f2 48 89 fe 77 0a 48 89 cf e8 1a f9 ff ff eb 07 31 ff e8 93 fe ff ff 31 c0 e9 86 a9 7e e1 <0f> 1f 44 00 00 41 56 41 55 49 89 f5 49 83 e5 c0 41 54 49 89 d4 55 Jan 3 12:10:14 BaradDur kernel: RSP: 0018:ffffc900177af898 EFLAGS: 00000206 Jan 3 12:10:14 BaradDur kernel: RAX: 000000000001f000 RBX: 0000000000001000 RCX: 0000000000000000 Jan 3 12:10:14 BaradDur kernel: RDX: ffffc900177af960 RSI: 0000000000000000 RDI: 0000000000000000 Jan 3 12:10:14 BaradDur kernel: RBP: 0000000000000000 R08: ffffc900177af960 R09: 000002a5c22f0400 Jan 3 12:10:14 BaradDur kernel: R10: ffff88816d144e00 R11: 0000000000010000 R12: 0000000000000000 Jan 3 12:10:14 BaradDur kernel: R13: ffff8881c59ac068 R14: ffffffffa041773a R15: ffffc900177af960 Jan 3 12:10:14 BaradDur kernel: FS: 0000000000000000(0000) GS:ffff88903d080000(0000) knlGS:0000000000000000 Jan 3 12:10:14 BaradDur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 3 12:10:14 BaradDur kernel: CR2: 0000561c68d45f70 CR3: 000000023e770000 CR4: 00000000003506e0 Jan 3 12:10:14 BaradDur kernel: Call Trace: Jan 3 12:10:14 BaradDur kernel: <IRQ> Jan 3 12:10:14 BaradDur kernel: ? rcu_check_gp_kthread_starvation+0xec/0xfd Jan 3 12:10:14 BaradDur kernel: ? rcu_sched_clock_irq+0x87e/0xa45 Jan 3 12:10:14 BaradDur kernel: ? do_set_msr+0x12/0x12 [kvm] Jan 3 12:10:14 BaradDur kernel: ? notifier_call_chain+0x38/0x5a Jan 3 12:10:14 BaradDur kernel: ? timekeeping_update+0xe8/0x117 Jan 3 12:10:14 BaradDur kernel: ? tick_init_jiffy_update+0x7c/0x7c Jan 3 12:10:14 BaradDur kernel: ? update_process_times+0x62/0x81 Jan 3 12:10:14 BaradDur kernel: ? tick_sched_timer+0x43/0x71 Jan 3 12:10:14 BaradDur kernel: ? __hrtimer_run_queues+0xeb/0x190 Jan 3 12:10:14 BaradDur kernel: ? hrtimer_interrupt+0x9c/0x16e Jan 3 12:10:14 BaradDur kernel: ? __sysvec_apic_timer_interrupt+0xc5/0x12f Jan 3 12:10:14 BaradDur kernel: ? sysvec_apic_timer_interrupt+0x80/0xa6 Jan 3 12:10:14 BaradDur kernel: </IRQ> Jan 3 12:10:14 BaradDur kernel: <TASK> Jan 3 12:10:14 BaradDur kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20 Jan 3 12:10:14 BaradDur kernel: ? fletcher_4_incremental_byteswap+0x2f/0x2f [zcommon] Jan 3 12:10:14 BaradDur kernel: ? fletcher_4_incremental_byteswap+0x2f/0x2f [zcommon] Jan 3 12:10:14 BaradDur kernel: ? fletcher_4_incremental_byteswap+0x2f/0x2f [zcommon] Jan 3 12:10:14 BaradDur kernel: abd_iterate_func+0x8c/0xef [zfs] Jan 3 12:10:14 BaradDur kernel: abd_fletcher_4_impl+0x33/0x46 [zfs] Jan 3 12:10:14 BaradDur kernel: abd_fletcher_4_native+0x55/0x73 [zfs] Jan 3 12:10:14 BaradDur kernel: ? ___slab_alloc+0x256/0x6fe Jan 3 12:10:14 BaradDur kernel: ? mempool_alloc+0x5e/0x149 Jan 3 12:10:14 BaradDur kernel: ? mempool_alloc+0x5e/0x149 Jan 3 12:10:14 BaradDur kernel: ? __kmem_cache_alloc_node+0x118/0x147 Jan 3 12:10:14 BaradDur kernel: ? __blk_rq_map_sg+0x2a4/0x339 Jan 3 12:10:14 BaradDur kernel: ? dma_direct_map_sg+0x1a4/0x1d6 Jan 3 12:10:14 BaradDur kernel: ? preempt_latency_start+0x2b/0x46 Jan 3 12:10:14 BaradDur kernel: ? _raw_spin_lock_irqsave+0x2c/0x37 Jan 3 12:10:14 BaradDur kernel: ? wbt_issue+0x16/0x3a Jan 3 12:10:14 BaradDur kernel: zio_checksum_error_impl+0x30b/0x41e [zfs] Jan 3 12:10:14 BaradDur kernel: ? preempt_latency_start+0x2b/0x46 Jan 3 12:10:14 BaradDur kernel: ? _raw_spin_lock+0x13/0x1c Jan 3 12:10:14 BaradDur kernel: ? _raw_spin_unlock+0x14/0x29 Jan 3 12:10:14 BaradDur kernel: ? nvme_queue_rqs+0x13c/0x176 [nvme] Jan 3 12:10:14 BaradDur kernel: ? blk_mq_flush_plug_list+0xbd/0x217 Jan 3 12:10:14 BaradDur kernel: ? __blk_flush_plug+0xc9/0x11a Jan 3 12:10:14 BaradDur kernel: zio_checksum_error+0x82/0xb9 [zfs] Jan 3 12:10:14 BaradDur kernel: raidz_checksum_verify+0x39/0x69 [zfs] Jan 3 12:10:14 BaradDur kernel: vdev_raidz_io_done+0x1f3/0x7eb [zfs] Jan 3 12:10:14 BaradDur kernel: ? kmem_cache_free+0x10f/0x154 Jan 3 12:10:14 BaradDur kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl] Jan 3 12:10:14 BaradDur kernel: ? percpu_counter_add_batch+0x85/0xa2 Jan 3 12:10:14 BaradDur kernel: ? preempt_latency_start+0x2b/0x46 Jan 3 12:10:14 BaradDur kernel: zio_vdev_io_done+0x13b/0x199 [zfs] Jan 3 12:10:14 BaradDur kernel: zio_execute+0xb4/0xdf [zfs] Jan 3 12:10:14 BaradDur kernel: taskq_thread+0x269/0x38a [spl] Jan 3 12:10:14 BaradDur kernel: ? wake_up_q+0x44/0x44 Jan 3 12:10:14 BaradDur kernel: ? zio_subblock+0x22/0x22 [zfs] Jan 3 12:10:14 BaradDur kernel: ? taskq_dispatch_delay+0x106/0x106 [spl] Jan 3 12:10:14 BaradDur kernel: kthread+0xe7/0xef Jan 3 12:10:14 BaradDur kernel: ? kthread_complete_and_exit+0x1b/0x1b Jan 3 12:10:14 BaradDur kernel: ret_from_fork+0x22/0x30 Jan 3 12:10:14 BaradDur kernel: </TASK>
January 3, 20251 yr Community Expert You can try upgrading to v7, it uses OpenZFS 2.2, maybe it will have the issue fixed, if not I'm afraid that only using a file recovery app
January 3, 20251 yr Author 3 hours ago, JorgeB said: You can try upgrading to v7, it uses OpenZFS 2.2, maybe it will have the issue fixed, if not I'm afraid that only using a file recovery app Thanks for responding again. Do I understand correctly from the release notes that v7 is one-way for ZFS pools and v7 itself is currently only in RC? If so, and i'm not missing something, I think I'll hold off on upgrading and try something like DMDE instead. Is there another file recovery app that you have heard good things about or had good experiences with in similar instances of file corruption?
January 4, 20251 yr Community Expert 11 hours ago, d3fc0n0wltraps said: Do I understand correctly from the release notes that v7 is one-way for ZFS pools and v7 itself is currently only in RC? It's only one way if you upgrade the pools, but you don't need to do that.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.