Codeman20400

I was able to get all of the dockers to start, including Plex. seems the issue was a V1 space cache on the unassigned drive https://forums.unraid.net/topic/131232-unraid-os-version-6115-available/?do=findComment&comment=1194660 ran the command to clear it on the UD, and the Plex docker started normally, finally. (With the array stopped: btrfs check --clear-space-cache v2 /dev/sdX1 If this is an old pool that might have v1 space cache also run: btrfs check --clear-space-cache v1 /dev/sdX1 It won't hurt if there's no v1 cache)

No dice, it seems changing to ipvlan didn't help nor hurt - as soon as the plex docker started, i got another error Dec 5 10:48:41 Tachikoma kernel: nvidia-uvm: Loaded the UVM driver, major device number 242. Dec 5 10:49:45 Tachikoma kernel: rcu: INFO: rcu_preempt self-detected stall on CPU Dec 5 10:49:45 Tachikoma kernel: rcu: 11-....: (59998 ticks this GP) idle=a33/1/0x4000000000000000 softirq=181650/181652 fqs=14809 Dec 5 10:49:45 Tachikoma kernel: (t=60001 jiffies g=624985 q=572966 ncpus=32) Dec 5 10:49:45 Tachikoma kernel: NMI backtrace for cpu 11 Dec 5 10:49:45 Tachikoma kernel: CPU: 11 PID: 9 Comm: kworker/u64:0 Tainted: P O 5.19.17-Unraid #2 Dec 5 10:49:45 Tachikoma kernel: Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 Dec 5 10:49:45 Tachikoma kernel: Workqueue: btrfs-cache btrfs_work_helper Dec 5 10:49:45 Tachikoma kernel: Call Trace: Dec 5 10:49:45 Tachikoma kernel: <IRQ> Dec 5 10:49:45 Tachikoma kernel: dump_stack_lvl+0x44/0x5c Dec 5 10:49:45 Tachikoma kernel: nmi_cpu_backtrace+0xda/0x104 Dec 5 10:49:45 Tachikoma kernel: ? lapic_can_unplug_cpu+0x95/0x95 Dec 5 10:49:45 Tachikoma kernel: nmi_trigger_cpumask_backtrace+0x72/0x107 Dec 5 10:49:45 Tachikoma kernel: trigger_single_cpu_backtrace+0x2a/0x31 Dec 5 10:49:45 Tachikoma kernel: rcu_dump_cpu_stacks+0xa0/0xd6 Dec 5 10:49:45 Tachikoma kernel: rcu_sched_clock_irq+0x44b/0xaf7 Dec 5 10:49:45 Tachikoma kernel: ? perf_event_task_tick+0x5e/0x1e3 Dec 5 10:49:45 Tachikoma kernel: ? tick_nohz_restart_sched_tick+0x83/0x83 Dec 5 10:49:45 Tachikoma kernel: update_process_times+0x62/0x81 Dec 5 10:49:45 Tachikoma kernel: tick_sched_timer+0x43/0x71 Dec 5 10:49:45 Tachikoma kernel: __hrtimer_run_queues+0x110/0x1b7 Dec 5 10:49:45 Tachikoma kernel: hrtimer_interrupt+0x9c/0x16e Dec 5 10:49:45 Tachikoma kernel: __sysvec_apic_timer_interrupt+0xc5/0x12f Dec 5 10:49:45 Tachikoma kernel: sysvec_apic_timer_interrupt+0x80/0xa6 Dec 5 10:49:45 Tachikoma kernel: </IRQ> Dec 5 10:49:45 Tachikoma kernel: <TASK> Dec 5 10:49:45 Tachikoma kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Dec 5 10:49:45 Tachikoma kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7f/0x1d0 Dec 5 10:49:45 Tachikoma kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 0f Dec 5 10:49:45 Tachikoma kernel: RSP: 0018:ffffc9000009bc60 EFLAGS: 00000202 Dec 5 10:49:45 Tachikoma kernel: RAX: 0000000000000101 RBX: ffffc9000009bcc8 RCX: 000000000ba9d30b Dec 5 10:49:45 Tachikoma kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffc9000009bcc8 Dec 5 10:49:45 Tachikoma kernel: RBP: ffff888a8614d620 R08: 0000000000000000 R09: 0000000000000200 Dec 5 10:49:45 Tachikoma kernel: R10: ffff888a8614d620 R11: 0000000000030eec R12: 0000000000000001 Dec 5 10:49:45 Tachikoma kernel: R13: ffff888b697d7a78 R14: ffff88810a78d000 R15: 0000000036681000 Dec 5 10:49:45 Tachikoma kernel: do_raw_spin_lock+0x14/0x1a Dec 5 10:49:45 Tachikoma kernel: __btrfs_remove_free_space_cache+0xe/0x2d Dec 5 10:49:45 Tachikoma kernel: load_free_space_cache+0x230/0x2dd Dec 5 10:49:45 Tachikoma kernel: ? kmem_cache_free+0xdb/0x176 Dec 5 10:49:45 Tachikoma kernel: caching_thread+0x7e/0x43d Dec 5 10:49:45 Tachikoma kernel: ? move_linked_works+0x2f/0x6a Dec 5 10:49:45 Tachikoma kernel: ? pwq_adjust_max_active+0x88/0xa7 Dec 5 10:49:45 Tachikoma kernel: btrfs_work_helper+0x114/0x2a5 Dec 5 10:49:45 Tachikoma kernel: process_one_work+0x1ab/0x295 Dec 5 10:49:45 Tachikoma kernel: worker_thread+0x18b/0x244 Dec 5 10:49:45 Tachikoma kernel: ? rescuer_thread+0x281/0x281 Dec 5 10:49:45 Tachikoma kernel: kthread+0xe7/0xef Dec 5 10:49:45 Tachikoma kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 5 10:49:45 Tachikoma kernel: ret_from_fork+0x22/0x30 Dec 5 10:49:45 Tachikoma kernel: </TASK> I do currently have that docker set to host, would adjusting that make a difference? EDIT: looking it over, would this be something to do with having the plex appdata on a mounted unassigned device?

this is post reset and post array start, with Dockers currently disabled. going to try starting Docker, and turning off autostart for the Plex container, see if that is the culprit tachikoma-diagnostics-20221205-1035.zip EDIT: was able to start up all of the other dockers I normally run aside from Plex, seems stable for the moment

I've started to see a crash when starting dockers on my freshly updated server - I was able to get the array started normally after disabling the docker service, and everything seemed normal. Once I reenabled Docker, it started to go through the list, but partway through I get a hard crash, and the server is completely unresponsive until a hard reboot. I was able to grab a segment of the syslog from the crash, and am working to get the full diag file, but i saw this after my Plex container started - . Dec 5 09:42:05 Tachikoma kernel: nvidia-uvm: Loaded the UVM driver, major device number 242. Dec 5 09:42:09 Tachikoma rc.docker: PlexMediaServer: started succesfully! Dec 5 09:42:09 Tachikoma kernel: br-64a9fd434999: port 14(vethf5fee39) entered blocking state Dec 5 09:42:09 Tachikoma kernel: br-64a9fd434999: port 14(vethf5fee39) entered disabled state Dec 5 09:42:09 Tachikoma kernel: device vethf5fee39 entered promiscuous mode Dec 5 09:42:11 Tachikoma kernel: eth0: renamed from veth6d72832 Dec 5 09:42:11 Tachikoma kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf5fee39: link becomes ready Dec 5 09:42:11 Tachikoma kernel: br-64a9fd434999: port 14(vethf5fee39) entered blocking state Dec 5 09:42:11 Tachikoma kernel: br-64a9fd434999: port 14(vethf5fee39) entered forwarding state Dec 5 09:42:11 Tachikoma rc.docker: swag: started succesfully! Dec 5 09:42:11 Tachikoma kernel: docker0: port 6(vethd18c7fe) entered blocking state Dec 5 09:42:11 Tachikoma kernel: docker0: port 6(vethd18c7fe) entered disabled state Dec 5 09:42:11 Tachikoma kernel: device vethd18c7fe entered promiscuous mode Dec 5 09:42:11 Tachikoma kernel: docker0: port 6(vethd18c7fe) entered blocking state Dec 5 09:42:11 Tachikoma kernel: docker0: port 6(vethd18c7fe) entered forwarding state Dec 5 09:42:12 Tachikoma kernel: docker0: port 6(vethd18c7fe) entered disabled state Dec 5 09:42:12 Tachikoma avahi-daemon[3298]: Joining mDNS multicast group on interface vethf5fee39.IPv6 with address fe80::60de:2dff:fe8d:4448. Dec 5 09:42:12 Tachikoma avahi-daemon[3298]: New relevant interface vethf5fee39.IPv6 for mDNS. Dec 5 09:42:12 Tachikoma avahi-daemon[3298]: Registering new address record for fe80::60de:2dff:fe8d:4448 on vethf5fee39.*. Dec 5 09:42:22 Tachikoma kernel: ------------[ cut here ]------------ Dec 5 09:42:22 Tachikoma kernel: NETDEV WATCHDOG: eth1 (igb): transmit queue 4 timed out Dec 5 09:42:22 Tachikoma kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:529 dev_watchdog+0x145/0x1b3 Dec 5 09:42:22 Tachikoma kernel: Modules linked in: nvidia_uvm(PO) xt_mark veth xt_nat nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle iptable_mangle iptable_nat vhost_net tun vhost vhost_iotlb tap xfs xt_MASQUERADE ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 md_mod ipmi_devintf ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc bonding tls ixgbe xfrm_algo mdio igb nvidia_drm(PO) nvidia_modeset(PO) x86_pkg_temp_thermal intel_powerclamp coretemp nvidia(PO) kvm_intel mgag200 drm_shmem_helper kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel ipmi_ssif crypto_simd cryptd rapl drm_kms_helper intel_cstate drm mpt3sas isci intel_uncore libsas i2c_i801 backlight i2c_algo_bit i2c_smbus syscopyarea ahci raid_class sysfillrect input_leds sysimgblt libahci joydev led_class i2c_core scsi_transport_sas fb_sys_fops acpi_ipmi wmi Dec 5 09:42:22 Tachikoma kernel: ipmi_si button unix [last unloaded: xfrm_algo] Dec 5 09:42:22 Tachikoma kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P O 5.19.17-Unraid #2 Dec 5 09:42:22 Tachikoma kernel: Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 Dec 5 09:42:22 Tachikoma kernel: RIP: 0010:dev_watchdog+0x145/0x1b3 Dec 5 09:42:22 Tachikoma kernel: Code: f7 c5 00 00 75 26 4c 89 ef c6 05 f1 f7 c5 00 01 e8 d4 31 fb ff 44 89 f1 4c 89 ee 48 c7 c7 c3 6e 15 82 48 89 c2 e8 f5 af 0f 00 <0f> 0b 4c 89 ef e8 1c fe ff ff 48 8b 83 88 fc ff ff 4c 89 ef 44 89 Dec 5 09:42:22 Tachikoma kernel: RSP: 0018:ffffc90006534eb0 EFLAGS: 00010282 Dec 5 09:42:22 Tachikoma kernel: RAX: 0000000000000000 RBX: ffff888169424448 RCX: 0000000000000027 Dec 5 09:42:22 Tachikoma kernel: RDX: 0000000000000103 RSI: ffffffff820d7be1 RDI: 00000000ffffffff Dec 5 09:42:22 Tachikoma kernel: RBP: 0000000000000004 R08: 0000000000000000 R09: ffffffff828653f0 Dec 5 09:42:22 Tachikoma kernel: R10: 00003fffffffffff R11: 75712074696d736e R12: ffff88816942439c Dec 5 09:42:22 Tachikoma kernel: R13: ffff888169424000 R14: 0000000000000004 R15: ffffffff8172073a Dec 5 09:42:22 Tachikoma kernel: FS: 0000000000000000(0000) GS:ffff888a57ac0000(0000) knlGS:0000000000000000 Dec 5 09:42:22 Tachikoma kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 5 09:42:22 Tachikoma kernel: CR2: 000014ca731ec000 CR3: 0000000a9dcb0001 CR4: 00000000000606e0 Dec 5 09:42:22 Tachikoma kernel: Call Trace: Dec 5 09:42:22 Tachikoma kernel: <IRQ> Dec 5 09:42:22 Tachikoma kernel: ? netif_tx_lock+0x1e/0x1e Dec 5 09:42:22 Tachikoma kernel: call_timer_fn+0x6f/0x10d Dec 5 09:42:22 Tachikoma kernel: __run_timers+0x144/0x184 Dec 5 09:42:22 Tachikoma kernel: ? timerqueue_add+0x6a/0x76 Dec 5 09:42:22 Tachikoma kernel: ? enqueue_hrtimer+0x77/0x82 Dec 5 09:42:22 Tachikoma kernel: run_timer_softirq+0x2b/0x43 Dec 5 09:42:22 Tachikoma kernel: __do_softirq+0x129/0x288 Dec 5 09:42:22 Tachikoma kernel: __irq_exit_rcu+0x79/0xb8 Dec 5 09:42:22 Tachikoma kernel: sysvec_apic_timer_interrupt+0x85/0xa6 Dec 5 09:42:22 Tachikoma kernel: </IRQ> Dec 5 09:42:22 Tachikoma kernel: <TASK> Dec 5 09:42:22 Tachikoma kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Dec 5 09:42:22 Tachikoma kernel: RIP: 0010:cpuidle_enter_state+0x11b/0x1e4 Dec 5 09:42:22 Tachikoma kernel: Code: 5b fa a1 ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 9d a9 a6 ff fb 0f 1f 44 00 00 <45> 85 ed 0f 88 9e 00 00 00 48 8b 04 24 49 63 cd 48 6b d1 68 49 29 Dec 5 09:42:22 Tachikoma kernel: RSP: 0018:ffffc900000f3e98 EFLAGS: 00000246 Dec 5 09:42:22 Tachikoma kernel: RAX: ffff888a57ac0000 RBX: 0000000000000002 RCX: 0000000000000000 Dec 5 09:42:22 Tachikoma kernel: RDX: 0000000000000003 RSI: ffffffff820d7be1 RDI: ffffffff820d80c1 Dec 5 09:42:22 Tachikoma kernel: RBP: ffff888a57af5b88 R08: 0000000000000002 R09: 0000000000000002 Dec 5 09:42:22 Tachikoma kernel: R10: 0000000000000020 R11: 000000000000007f R12: ffffffff82315740 Dec 5 09:42:22 Tachikoma kernel: R13: 0000000000000002 R14: 000015066edf74de R15: 0000000000000000 Dec 5 09:42:22 Tachikoma kernel: ? cpuidle_enter_state+0xf5/0x1e4 Dec 5 09:42:22 Tachikoma kernel: cpuidle_enter+0x2a/0x38 Dec 5 09:42:22 Tachikoma kernel: do_idle+0x187/0x1f5 Dec 5 09:42:22 Tachikoma kernel: cpu_startup_entry+0x1d/0x1f Dec 5 09:42:22 Tachikoma kernel: start_secondary+0xeb/0xeb Dec 5 09:42:22 Tachikoma kernel: secondary_startup_64_no_verify+0xce/0xdb Dec 5 09:42:22 Tachikoma kernel: </TASK> Dec 5 09:42:22 Tachikoma kernel: ---[ end trace 0000000000000000 ]--- Dec 5 09:42:33 Tachikoma kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 16-... 24-... } 21427 jiffies s: 1713 root: 0x2/. Dec 5 09:42:33 Tachikoma kernel: rcu: blocking rcu_node structures (internal RCU debug): l=1:16-31:0x101/. Dec 5 09:42:33 Tachikoma kernel: Task dump for CPU 16: Dec 5 09:42:33 Tachikoma kernel: task:kworker/u64:4 state:R running task stack: 0 pid: 3647 ppid: 2 flags:0x00004008 Dec 5 09:42:33 Tachikoma kernel: Workqueue: btrfs-cache btrfs_work_helper Dec 5 09:42:33 Tachikoma kernel: Call Trace: Dec 5 09:42:33 Tachikoma kernel: <TASK> Dec 5 09:42:33 Tachikoma kernel: ? __schedule+0x59e/0x5f6 Dec 5 09:42:33 Tachikoma kernel: ? chksum_update+0x13/0x1e Dec 5 09:42:33 Tachikoma kernel: ? crc32c+0x2f/0x62 Dec 5 09:42:33 Tachikoma kernel: ? kmem_cache_debug_flags+0xc/0x1d Dec 5 09:42:33 Tachikoma kernel: ? setup_object+0x17/0x51 Dec 5 09:42:33 Tachikoma kernel: ? new_slab+0x1cf/0x1df Dec 5 09:42:33 Tachikoma kernel: ? ___slab_alloc+0x288/0x590 Dec 5 09:42:33 Tachikoma kernel: ? __load_free_space_cache+0x215/0x452 Dec 5 09:42:33 Tachikoma kernel: ? __slab_free+0x83/0x29a Dec 5 09:42:33 Tachikoma kernel: ? __load_free_space_cache+0x345/0x452 Dec 5 09:42:33 Tachikoma kernel: ? sysvec_apic_timer_interrupt+0x92/0xa6 Dec 5 09:42:33 Tachikoma kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20 Dec 5 09:42:33 Tachikoma kernel: ? native_queued_spin_lock_slowpath+0x87/0x1d0 Dec 5 09:42:33 Tachikoma kernel: ? do_raw_spin_lock+0x14/0x1a Dec 5 09:42:33 Tachikoma kernel: ? __btrfs_remove_free_space_cache+0xe/0x2d Dec 5 09:42:33 Tachikoma kernel: ? load_free_space_cache+0x230/0x2dd Dec 5 09:42:33 Tachikoma kernel: ? kmem_cache_free+0xdb/0x176 Dec 5 09:42:33 Tachikoma kernel: ? caching_thread+0x7e/0x43d Dec 5 09:42:33 Tachikoma kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a Dec 5 09:42:33 Tachikoma kernel: ? btrfs_work_helper+0x114/0x2a5 Dec 5 09:42:33 Tachikoma kernel: ? process_one_work+0x1ab/0x295 Dec 5 09:42:33 Tachikoma kernel: ? worker_thread+0x18b/0x244 Dec 5 09:42:33 Tachikoma kernel: ? rescuer_thread+0x281/0x281 Dec 5 09:42:33 Tachikoma kernel: ? kthread+0xe7/0xef Dec 5 09:42:33 Tachikoma kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 5 09:42:33 Tachikoma kernel: ? ret_from_fork+0x22/0x30 Dec 5 09:42:33 Tachikoma kernel: </TASK> Dec 5 09:42:33 Tachikoma kernel: Task dump for CPU 24: Dec 5 09:42:33 Tachikoma kernel: task:kworker/u64:3 state:R running task stack: 0 pid: 3646 ppid: 2 flags:0x00004008 Dec 5 09:42:33 Tachikoma kernel: Workqueue: btrfs-cache btrfs_work_helper Dec 5 09:42:33 Tachikoma kernel: Call Trace: Dec 5 09:42:33 Tachikoma kernel: <TASK> Dec 5 09:42:33 Tachikoma kernel: ? chksum_update+0x13/0x1e Dec 5 09:42:33 Tachikoma kernel: ? crc32c+0x2f/0x62 Dec 5 09:42:33 Tachikoma kernel: ? kmem_cache_debug_flags+0xc/0x1d Dec 5 09:42:33 Tachikoma kernel: ? setup_object+0x17/0x51 Dec 5 09:42:33 Tachikoma kernel: ? new_slab+0x1cf/0x1df Dec 5 09:42:33 Tachikoma kernel: ? ___slab_alloc+0x288/0x590 Dec 5 09:42:33 Tachikoma kernel: ? __load_free_space_cache+0x215/0x452 Dec 5 09:42:33 Tachikoma kernel: ? __slab_free+0x83/0x29a Dec 5 09:42:33 Tachikoma kernel: ? __load_free_space_cache+0x345/0x452 Dec 5 09:42:33 Tachikoma kernel: ? __load_free_space_cache+0x3fb/0x452 Dec 5 09:42:33 Tachikoma kernel: ? sysvec_apic_timer_interrupt+0x92/0xa6 Dec 5 09:42:33 Tachikoma kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20 Dec 5 09:42:33 Tachikoma kernel: ? native_queued_spin_lock_slowpath+0x85/0x1d0 Dec 5 09:42:33 Tachikoma kernel: ? do_raw_spin_lock+0x14/0x1a Dec 5 09:42:33 Tachikoma kernel: ? __btrfs_remove_free_space_cache+0xe/0x2d Dec 5 09:42:33 Tachikoma kernel: ? load_free_space_cache+0x230/0x2dd Dec 5 09:42:33 Tachikoma kernel: ? kmem_cache_free+0xdb/0x176 Dec 5 09:42:33 Tachikoma kernel: ? caching_thread+0x7e/0x43d Dec 5 09:42:33 Tachikoma kernel: ? move_linked_works+0x2f/0x6a Dec 5 09:42:33 Tachikoma kernel: ? pwq_adjust_max_active+0x88/0xa7 Dec 5 09:42:33 Tachikoma kernel: ? btrfs_work_helper+0x114/0x2a5 Dec 5 09:42:33 Tachikoma kernel: ? process_one_work+0x1ab/0x295 Dec 5 09:42:33 Tachikoma kernel: ? worker_thread+0x18b/0x244 Dec 5 09:42:33 Tachikoma kernel: ? rescuer_thread+0x281/0x281 Dec 5 09:42:33 Tachikoma kernel: ? kthread+0xe7/0xef Dec 5 09:42:33 Tachikoma kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 5 09:42:33 Tachikoma kernel: ? ret_from_fork+0x22/0x30 Dec 5 09:42:33 Tachikoma kernel: </TASK> Dec 5 09:42:36 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:42:42 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:42:47 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:42:52 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:42:58 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:43:04 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:43:09 Tachikoma kernel: ixgbe 0000:02:00.0 eth4: initiating reset due to tx timeout Dec 5 09:43:12 Tachikoma kernel: rcu: INFO: rcu_preempt self-detected stall on CPU Dec 5 09:43:12 Tachikoma kernel: rcu: 16-....: (59999 ticks this GP) idle=19f/1/0x4000000000000000 softirq=333249/333258 fqs=14611 Dec 5 09:43:12 Tachikoma kernel: (t=60000 jiffies g=2141645 q=493736 ncpus=32) Dec 5 09:43:12 Tachikoma kernel: NMI backtrace for cpu 16 Dec 5 09:43:12 Tachikoma kernel: CPU: 16 PID: 3647 Comm: kworker/u64:4 Tainted: P W O 5.19.17-Unraid #2 Dec 5 09:43:12 Tachikoma kernel: Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 Dec 5 09:43:12 Tachikoma kernel: Workqueue: btrfs-cache btrfs_work_helper Dec 5 09:43:12 Tachikoma kernel: Call Trace: Dec 5 09:43:12 Tachikoma kernel: <IRQ> Dec 5 09:43:12 Tachikoma kernel: dump_stack_lvl+0x44/0x5c Dec 5 09:43:12 Tachikoma kernel: nmi_cpu_backtrace+0xda/0x104 Dec 5 09:43:12 Tachikoma kernel: ? lapic_can_unplug_cpu+0x95/0x95 Dec 5 09:43:12 Tachikoma kernel: nmi_trigger_cpumask_backtrace+0x72/0x107 Dec 5 09:43:12 Tachikoma kernel: trigger_single_cpu_backtrace+0x2a/0x31 Dec 5 09:43:12 Tachikoma kernel: rcu_dump_cpu_stacks+0xa0/0xd6 Dec 5 09:43:12 Tachikoma kernel: rcu_sched_clock_irq+0x44b/0xaf7 Dec 5 09:43:12 Tachikoma kernel: ? tick_nohz_restart_sched_tick+0x83/0x83 Dec 5 09:43:12 Tachikoma kernel: update_process_times+0x62/0x81 Dec 5 09:43:12 Tachikoma kernel: tick_sched_timer+0x43/0x71 Dec 5 09:43:12 Tachikoma kernel: __hrtimer_run_queues+0x110/0x1b7 Dec 5 09:43:12 Tachikoma kernel: hrtimer_interrupt+0x9c/0x16e Dec 5 09:43:12 Tachikoma kernel: __sysvec_apic_timer_interrupt+0xc5/0x12f Dec 5 09:43:12 Tachikoma kernel: sysvec_apic_timer_interrupt+0x80/0xa6 Dec 5 09:43:12 Tachikoma kernel: </IRQ> Dec 5 09:43:12 Tachikoma kernel: <TASK> Dec 5 09:43:12 Tachikoma kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Dec 5 09:43:12 Tachikoma kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7f/0x1d0 Dec 5 09:43:12 Tachikoma kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 0f Dec 5 09:43:12 Tachikoma kernel: RSP: 0018:ffffc9000f6e7c60 EFLAGS: 00000202 Dec 5 09:43:12 Tachikoma kernel: RAX: 0000000000000101 RBX: ffffc9000f6e7cc8 RCX: 0000000080240011 Dec 5 09:43:12 Tachikoma kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffc9000f6e7cc8 Dec 5 09:43:12 Tachikoma kernel: RBP: ffff8881b83a6690 R08: ffff8881b83a6690 R09: 0000000080240011 Dec 5 09:43:12 Tachikoma kernel: R10: ffff8881b83a6690 R11: 0000000000030ec0 R12: 0000000000000001 Dec 5 09:43:12 Tachikoma kernel: R13: ffff88831f920238 R14: ffff888105d45000 R15: 0000000036681000 Dec 5 09:43:12 Tachikoma kernel: do_raw_spin_lock+0x14/0x1a Dec 5 09:43:12 Tachikoma kernel: __btrfs_remove_free_space_cache+0xe/0x2d Dec 5 09:43:12 Tachikoma kernel: load_free_space_cache+0x230/0x2dd Dec 5 09:43:12 Tachikoma kernel: ? kmem_cache_free+0xdb/0x176 Dec 5 09:43:12 Tachikoma kernel: caching_thread+0x7e/0x43d Dec 5 09:43:12 Tachikoma kernel: ? _raw_spin_unlock_irqrestore+0x1a/0x3a Dec 5 09:43:12 Tachikoma kernel: btrfs_work_helper+0x114/0x2a5 Dec 5 09:43:12 Tachikoma kernel: process_one_work+0x1ab/0x295 Dec 5 09:43:12 Tachikoma kernel: worker_thread+0x18b/0x244 Dec 5 09:43:12 Tachikoma kernel: ? rescuer_thread+0x281/0x281 Dec 5 09:43:12 Tachikoma kernel: kthread+0xe7/0xef Dec 5 09:43:12 Tachikoma kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 5 09:43:12 Tachikoma kernel: ret_from_fork+0x22/0x30 Dec 5 09:43:12 Tachikoma kernel: </TASK> Dec 5 09:43:12 Tachikoma kernel: Sending NMI from CPU 16 to CPUs 24: Dec 5 09:43:12 Tachikoma kernel: NMI backtrace for cpu 24 Dec 5 09:43:12 Tachikoma kernel: CPU: 24 PID: 3646 Comm: kworker/u64:3 Tainted: P W O 5.19.17-Unraid #2 Dec 5 09:43:12 Tachikoma kernel: Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 03/04/2015 Dec 5 09:43:12 Tachikoma kernel: Workqueue: btrfs-cache btrfs_work_helper Dec 5 09:43:12 Tachikoma kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7f/0x1d0 Dec 5 09:43:12 Tachikoma kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 0f Dec 5 09:43:12 Tachikoma kernel: RSP: 0018:ffffc9000f6dfc60 EFLAGS: 00000202 Dec 5 09:43:12 Tachikoma kernel: RAX: 0000000000000101 RBX: ffffc9000f6dfcc8 RCX: 0000000080240020 Dec 5 09:43:12 Tachikoma kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffc9000f6dfcc8 Dec 5 09:43:12 Tachikoma kernel: RBP: ffff88834fd33e00 R08: ffff88834fd33e00 R09: 0000000080240020 Dec 5 09:43:12 Tachikoma kernel: R10: ffff88834fd33e00 R11: 0000000000030eec R12: 0000000000000001 Dec 5 09:43:12 Tachikoma kernel: R13: ffff8882f63267f8 R14: ffff888105d45000 R15: 000000002e370000 Dec 5 09:43:12 Tachikoma kernel: FS: 0000000000000000(0000) GS:ffff889457c00000(0000) knlGS:0000000000000000 Dec 5 09:43:12 Tachikoma kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 5 09:43:12 Tachikoma kernel: CR2: 00007fff0cb5ab88 CR3: 000000000220a001 CR4: 00000000000606e0 Dec 5 09:43:12 Tachikoma kernel: Call Trace: Dec 5 09:43:12 Tachikoma kernel: <TASK> Dec 5 09:43:12 Tachikoma kernel: do_raw_spin_lock+0x14/0x1a Dec 5 09:43:12 Tachikoma kernel: __btrfs_remove_free_space_cache+0xe/0x2d Dec 5 09:43:12 Tachikoma kernel: load_free_space_cache+0x230/0x2dd Dec 5 09:43:12 Tachikoma kernel: ? kmem_cache_free+0xdb/0x176 Dec 5 09:43:12 Tachikoma kernel: caching_thread+0x7e/0x43d Dec 5 09:43:12 Tachikoma kernel: ? move_linked_works+0x2f/0x6a Dec 5 09:43:12 Tachikoma kernel: ? pwq_adjust_max_active+0x88/0xa7 Dec 5 09:43:12 Tachikoma kernel: btrfs_work_helper+0x114/0x2a5 Dec 5 09:43:12 Tachikoma kernel: process_one_work+0x1ab/0x295 Dec 5 09:43:12 Tachikoma kernel: worker_thread+0x18b/0x244 Dec 5 09:43:12 Tachikoma kernel: ? rescuer_thread+0x281/0x281 Dec 5 09:43:12 Tachikoma kernel: kthread+0xe7/0xef Dec 5 09:43:12 Tachikoma kernel: ? kthread_complete_and_exit+0x1b/0x1b Dec 5 09:43:12 Tachikoma kernel: ret_from_fork+0x22/0x30 Dec 5 09:43:12 Tachikoma kernel: </TASK> any help is appreciated