Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

risenfromthedust

Members

  • Joined

  • Last visited

View Profile Find content

  1. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    So it happened again 4 days ago. That was less than 24 hours after the parity check completed and on par with how it was behaving (ie a shutdown about once every 2-3 days, required the hard reboot). I took that opportunity to try one more thing. I reseated the gpu and also changed the slot in the server it was mounted in. Thus far we are 4 days later and no issue. Im hoping it was either bad pci port or just needing to be reseated. Either way we are better off for now, and Ive been stable thus far. If I dont repost assume that was the issue. Otherwise if it happens again, I will update the post again. But either way, thank you for helping confirm it was the gpu causing my issues.

  2. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    That is how it looked to me too. I have updated the driver as stated in the previous message. I hope that increases my stability. If not, when it becomes unresponsive next, Ill remove the gpu and reboot and see where that gets me. For now parity check is running.

  3. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    So heres where I am. I had to let the parity check from the last forced reboot finish before i could do anything. After our previous messages, i did update the nvidia driver. I was on 560.xxx and updated to 565.xxx, but it would require a reboot. The check would have finished at some point yesterday but i was busy so the server just sat like normal. Well this morning, it happened again. Best I can tell from my grafana install is around 4:34 am. However this time was different. It wasnt completely unresponsive. The iDRAC monitor still didnt have any output on the screen, but a web browser could still get to the login page. Login would stall and just sit, but at least the login page was responsive. Still required a hard reboot though. Looking at this newest syslog, at 4am there was a null pointer reference and then it started spitting out a whole lot of kernel info: I see nvidia-smi at the top of these errors so maybe that points to the gpu driver being the issue. Maybe not. I dont know, thats why Im here. I did a hard reboot and the system is back up and doing another parity check, so itll be a day or so before i can try anything that requires rebooting. But I did confirm that the gpu driver update did take because of the restart. Hopefully that helps me. Heres the syslog starting at just after midnight this morning: Nov 17 00:00:02 unraid apcupsd[18774]: apcupsd 3.14.14 (31 May 2016) slackware startup succeeded Nov 17 00:00:02 unraid apcupsd[18774]: NIS server startup succeeded Nov 17 00:00:16 unraid Plugin Auto Update: community.applications.plg version 2024.11.16 does not meet age requirements to update - 0 days old Nov 17 00:00:17 unraid Plugin Auto Update: unassigned.devices.plg version 2024.11.16 does not meet age requirements to update - 0 days old Nov 17 00:00:18 unraid Plugin Auto Update: Checking for language updates Nov 17 00:00:18 unraid Plugin Auto Update: Community Applications Plugin Auto Update finished Nov 17 00:00:29 unraid Docker Auto Update: Stopping jellyfin Nov 17 00:00:47 unraid kernel: docker0: port 15(vethd367dc1) entered disabled state Nov 17 00:00:47 unraid kernel: vetha6d589f: renamed from eth0 Nov 17 00:00:47 unraid kernel: docker0: port 15(vethd367dc1) entered disabled state Nov 17 00:00:47 unraid kernel: device vethd367dc1 left promiscuous mode Nov 17 00:00:47 unraid kernel: docker0: port 15(vethd367dc1) entered disabled state Nov 17 00:00:48 unraid Docker Auto Update: Stopping jackett Nov 17 00:00:57 unraid kernel: docker0: port 21(veth31e7aef) entered disabled state Nov 17 00:00:57 unraid kernel: vethf3f49a4: renamed from eth0 Nov 17 00:00:57 unraid kernel: docker0: port 21(veth31e7aef) entered disabled state Nov 17 00:00:57 unraid kernel: device veth31e7aef left promiscuous mode Nov 17 00:00:57 unraid kernel: docker0: port 21(veth31e7aef) entered disabled state Nov 17 00:00:58 unraid Docker Auto Update: Installing Updates for jellyfin jackett Nov 17 00:02:51 unraid Docker Auto Update: Restarting jellyfin Nov 17 00:02:51 unraid kernel: docker0: port 15(veth52d3ea0) entered blocking state Nov 17 00:02:51 unraid kernel: docker0: port 15(veth52d3ea0) entered disabled state Nov 17 00:02:51 unraid kernel: device veth52d3ea0 entered promiscuous mode Nov 17 00:02:56 unraid kernel: eth0: renamed from vethcfe337c Nov 17 00:02:56 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth52d3ea0: link becomes ready Nov 17 00:02:56 unraid kernel: docker0: port 15(veth52d3ea0) entered blocking state Nov 17 00:02:56 unraid kernel: docker0: port 15(veth52d3ea0) entered forwarding state Nov 17 00:03:04 unraid Docker Auto Update: Restarting jackett Nov 17 00:03:04 unraid kernel: docker0: port 21(veth32ed69c) entered blocking state Nov 17 00:03:04 unraid kernel: docker0: port 21(veth32ed69c) entered disabled state Nov 17 00:03:04 unraid kernel: device veth32ed69c entered promiscuous mode Nov 17 00:03:09 unraid kernel: eth0: renamed from veth9b1cb5c Nov 17 00:03:09 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth32ed69c: link becomes ready Nov 17 00:03:09 unraid kernel: docker0: port 21(veth32ed69c) entered blocking state Nov 17 00:03:09 unraid kernel: docker0: port 21(veth32ed69c) entered forwarding state Nov 17 00:03:12 unraid Docker Auto Update: Community Applications Docker Autoupdate finished Nov 17 03:24:30 unraid kernel: NVRM: failed to allocate vmap() page descriptor table! Nov 17 03:24:30 unraid kernel: NVRM: GPU 0000:04:00.0: RmInitAdapter failed! (0x62:0xffff:2468) Nov 17 03:24:30 unraid kernel: NVRM: GPU 0000:04:00.0: rm_init_adapter failed, device minor number 0 Nov 17 04:00:17 unraid kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 Nov 17 04:00:17 unraid kernel: #PF: supervisor read access in kernel mode Nov 17 04:00:17 unraid kernel: #PF: error_code(0x0000) - not-present page Nov 17 04:00:17 unraid kernel: PGD 800000013ab4e067 P4D 800000013ab4e067 PUD 13ab63067 PMD 0 Nov 17 04:00:17 unraid kernel: Oops: 0000 [#1] PREEMPT SMP PTI Nov 17 04:00:17 unraid kernel: CPU: 12 PID: 15798 Comm: nvidia-smi Tainted: P O 6.1.106-Unraid #1 Nov 17 04:00:17 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:00:17 unraid kernel: RIP: 0010:__cpa_process_fault+0x3ea/0x40e Nov 17 04:00:17 unraid kernel: Code: d0 48 c1 e8 0c 49 89 44 24 30 31 db eb 2b 49 8b 7c 24 30 e8 d4 f3 ff ff 84 c0 75 18 49 8b 04 24 48 89 ee 48 c7 c7 d7 1a 0c 82 <48> 8b 10 e8 16 c8 00 00 0f 0b bb f2 ff ff ff 48 83 c4 30 89 d8 5b Nov 17 04:00:17 unraid kernel: RSP: 0018:ffffc90008f3f720 EFLAGS: 00010246 Nov 17 04:00:17 unraid kernel: RAX: 0000000000000000 RBX: ffffc90008f3f890 RCX: 0000000080000000 Nov 17 04:00:17 unraid kernel: RDX: 0000000000001001 RSI: ffff1108714cc000 RDI: ffffffff820c1ad7 Nov 17 04:00:17 unraid kernel: RBP: ffff1108714cc000 R08: ffffc90008f3f7de R09: ffffffff8220a110 Nov 17 04:00:17 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc90008f3f890 Nov 17 04:00:17 unraid kernel: R13: 80000008354d9073 R14: 0000000000000000 R15: 0000000000000000 Nov 17 04:00:17 unraid kernel: FS: 000014f77abd9180(0000) GS:ffff88880f980000(0000) knlGS:0000000000000000 Nov 17 04:00:17 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:00:17 unraid kernel: CR2: 0000000000000000 CR3: 00000005aac68005 CR4: 00000000001706e0 Nov 17 04:00:17 unraid kernel: Call Trace: Nov 17 04:00:17 unraid kernel: <TASK> Nov 17 04:00:17 unraid kernel: ? __die_body+0x1a/0x5c Nov 17 04:00:17 unraid kernel: ? page_fault_oops+0x329/0x376 Nov 17 04:00:17 unraid kernel: ? search_bpf_extables+0x5d/0x68 Nov 17 04:00:17 unraid kernel: ? fixup_exception+0x22/0x24b Nov 17 04:00:17 unraid kernel: ? exc_page_fault+0xfb/0x11d Nov 17 04:00:17 unraid kernel: ? asm_exc_page_fault+0x22/0x30 Nov 17 04:00:17 unraid kernel: ? __cpa_process_fault+0x3ea/0x40e Nov 17 04:00:17 unraid kernel: ? lookup_address_in_pgd_attr+0x15b/0x173 Nov 17 04:00:17 unraid kernel: __change_page_attr_set_clr+0xb7/0x848 Nov 17 04:00:17 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 17 04:00:17 unraid kernel: change_page_attr_set_clr+0x129/0x271 Nov 17 04:00:17 unraid kernel: set_pages_array_wb+0x37/0x72 Nov 17 04:00:17 unraid kernel: nv_set_memory_type+0xd9/0x184 [nvidia] Nov 17 04:00:17 unraid kernel: nv_free_system_pages+0x59/0x170 [nvidia] Nov 17 04:00:17 unraid kernel: nv_free_pages+0x68/0x78 [nvidia] Nov 17 04:00:17 unraid kernel: _nv043747rm+0x98/0x140 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv038497rm+0x27a/0x660 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv038458rm+0x5f/0x130 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv004292rm+0xd/0x20 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv006122rm+0x1e/0xb0 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv018080rm+0x59c/0x680 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv048181rm+0xb3/0xe0 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv049956rm+0xb3/0x180 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv049955rm+0x4ae/0x660 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv048073rm+0xdd/0x190 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv017727rm+0x105/0x220 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv004138rm+0xd/0x20 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv006122rm+0x1e/0xb0 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv049238rm+0xd1/0x210 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv038833rm+0xc2/0xf0 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv038629rm+0x3d/0x80 [nvidia] Nov 17 04:00:17 unraid kernel: ? __kmem_cache_free+0xb0/0xf5 Nov 17 04:00:17 unraid kernel: ? _nv035820rm+0x51/0x150 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv035816rm+0x6d/0xf0 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv027931rm+0x106/0x1e0 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv027932rm+0x574/0x850 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv000702rm+0x2aa/0x340 [nvidia] Nov 17 04:00:17 unraid kernel: ? rm_disable_adapter+0x61/0x130 [nvidia] Nov 17 04:00:17 unraid kernel: ? up_write+0x33/0x5a Nov 17 04:00:17 unraid kernel: ? nv_shutdown_adapter+0x17/0x115 [nvidia] Nov 17 04:00:17 unraid kernel: ? nv_close_device+0xa3/0x123 [nvidia] Nov 17 04:00:17 unraid kernel: ? nvidia_close_callback+0x9e/0x121 [nvidia] Nov 17 04:00:17 unraid kernel: ? nvidia_close+0x19c/0x1ea [nvidia] Nov 17 04:00:17 unraid kernel: ? __fput+0x103/0x1d3 Nov 17 04:00:17 unraid kernel: ? task_work_run+0x6b/0x80 Nov 17 04:00:17 unraid kernel: ? exit_to_user_mode_prepare+0x75/0x112 Nov 17 04:00:17 unraid kernel: ? syscall_exit_to_user_mode+0x18/0x23 Nov 17 04:00:17 unraid kernel: ? do_syscall_64+0x71/0x7b Nov 17 04:00:17 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:00:17 unraid kernel: </TASK> Nov 17 04:00:17 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp nvidia_drm(PO) nvidia_modeset(PO) kvm_intel ipmi_ssif kvm nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd Nov 17 04:00:17 unraid kernel: cryptd mgag200 video rapl drm_shmem_helper intel_cstate drm_kms_helper drm intel_uncore mpt3sas mei_me backlight i2c_algo_bit input_leds mei joydev led_class i2c_core raid_class syscopyarea sysfillrect scsi_transport_sas sysimgblt fb_sys_fops wmi ipmi_si button acpi_power_meter unix [last unloaded: xfrm_algo] Nov 17 04:00:17 unraid kernel: CR2: 0000000000000000 Nov 17 04:00:17 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 17 04:00:17 unraid kernel: RIP: 0010:__cpa_process_fault+0x3ea/0x40e Nov 17 04:00:17 unraid kernel: Code: d0 48 c1 e8 0c 49 89 44 24 30 31 db eb 2b 49 8b 7c 24 30 e8 d4 f3 ff ff 84 c0 75 18 49 8b 04 24 48 89 ee 48 c7 c7 d7 1a 0c 82 <48> 8b 10 e8 16 c8 00 00 0f 0b bb f2 ff ff ff 48 83 c4 30 89 d8 5b Nov 17 04:00:17 unraid kernel: RSP: 0018:ffffc90008f3f720 EFLAGS: 00010246 Nov 17 04:00:17 unraid kernel: RAX: 0000000000000000 RBX: ffffc90008f3f890 RCX: 0000000080000000 Nov 17 04:00:17 unraid kernel: RDX: 0000000000001001 RSI: ffff1108714cc000 RDI: ffffffff820c1ad7 Nov 17 04:00:17 unraid kernel: RBP: ffff1108714cc000 R08: ffffc90008f3f7de R09: ffffffff8220a110 Nov 17 04:00:17 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc90008f3f890 Nov 17 04:00:17 unraid kernel: R13: 80000008354d9073 R14: 0000000000000000 R15: 0000000000000000 Nov 17 04:00:17 unraid kernel: FS: 000014f77abd9180(0000) GS:ffff88880f980000(0000) knlGS:0000000000000000 Nov 17 04:00:17 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:00:17 unraid kernel: CR2: 0000000000000000 CR3: 00000005aac68005 CR4: 00000000001706e0 Nov 17 04:00:17 unraid kernel: note: nvidia-smi[15798] exited with irqs disabled Nov 17 04:00:17 unraid kernel: note: nvidia-smi[15798] exited with preempt_count 1 Nov 17 04:00:17 unraid kernel: general protection fault, probably for non-canonical address 0x100dabbd6d5c00: 0000 [#2] PREEMPT SMP PTI Nov 17 04:00:17 unraid kernel: CPU: 12 PID: 15798 Comm: nvidia-smi Tainted: P D O 6.1.106-Unraid #1 Nov 17 04:00:17 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:00:17 unraid kernel: RIP: 0010:_nv013001rm+0x3c/0x310 [nvidia] Nov 17 04:00:17 unraid kernel: Code: 48 63 47 08 48 01 c2 48 8b 07 48 85 c0 75 1b e9 2b 02 00 00 66 2e 0f 1f 84 00 00 00 00 00 48 8b 48 10 48 85 c9 74 17 48 89 c8 <48> 39 30 77 ef 0f 83 f9 01 00 00 48 8b 48 18 48 85 c9 75 e9 48 89 Nov 17 04:00:17 unraid kernel: RSP: 0018:ffffc90008f3fd38 EFLAGS: 00010006 Nov 17 04:00:17 unraid kernel: RAX: 00100dabbd6d5c00 RBX: ffffffffa191981e RCX: 00100dabbd6d5c00 Nov 17 04:00:17 unraid kernel: RDX: ffffc90008f3fdd0 RSI: 0000000000003db6 RDI: ffffffffa4924718 Nov 17 04:00:17 unraid kernel: RBP: ffff88897b256000 R08: 0000000000000000 R09: ffffc90008f3fdf8 Nov 17 04:00:17 unraid kernel: R10: ffffc90008f3fe90 R11: 0000000000000003 R12: ffffc90008f3fd80 Nov 17 04:00:17 unraid kernel: R13: ffff888133140400 R14: ffff888133140400 R15: ffff8881054bb008 Nov 17 04:00:17 unraid kernel: FS: 0000000000000000(0000) GS:ffff88880f980000(0000) knlGS:0000000000000000 Nov 17 04:00:17 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:00:17 unraid kernel: CR2: 0000000000000000 CR3: 000000000220a004 CR4: 00000000001706e0 Nov 17 04:00:17 unraid kernel: Call Trace: Nov 17 04:00:17 unraid kernel: <TASK> Nov 17 04:00:17 unraid kernel: ? __die_body+0x1a/0x5c Nov 17 04:00:17 unraid kernel: ? die_addr+0x38/0x51 Nov 17 04:00:17 unraid kernel: ? exc_general_protection+0x30f/0x345 Nov 17 04:00:17 unraid kernel: ? asm_exc_general_protection+0x22/0x30 Nov 17 04:00:17 unraid kernel: ? rm_cleanup_file_private+0x5e/0x200 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv013001rm+0x3c/0x310 [nvidia] Nov 17 04:00:17 unraid kernel: ? rm_cleanup_file_private+0x5e/0x200 [nvidia] Nov 17 04:00:17 unraid kernel: ? _nv052801rm+0xd6/0x1a0 [nvidia] Nov 17 04:00:17 unraid kernel: ? rm_cleanup_file_private+0x5e/0x200 [nvidia] Nov 17 04:00:17 unraid kernel: ? nvidia_close+0x9b/0x1ea [nvidia] Nov 17 04:00:17 unraid kernel: ? __fput+0x103/0x1d3 Nov 17 04:00:17 unraid kernel: ? task_work_run+0x6b/0x80 Nov 17 04:00:17 unraid kernel: ? do_exit+0x3b4/0x923 Nov 17 04:00:17 unraid kernel: ? make_task_dead+0x11c/0x11c Nov 17 04:00:17 unraid kernel: ? rewind_stack_and_make_dead+0x17/0x17 Nov 17 04:00:17 unraid kernel: </TASK> Nov 17 04:00:17 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp nvidia_drm(PO) nvidia_modeset(PO) kvm_intel ipmi_ssif kvm nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd Nov 17 04:00:17 unraid kernel: cryptd mgag200 video rapl drm_shmem_helper intel_cstate drm_kms_helper drm intel_uncore mpt3sas mei_me backlight i2c_algo_bit input_leds mei joydev led_class i2c_core raid_class syscopyarea sysfillrect scsi_transport_sas sysimgblt fb_sys_fops wmi ipmi_si button acpi_power_meter unix [last unloaded: xfrm_algo] Nov 17 04:00:17 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 17 04:00:17 unraid kernel: RIP: 0010:__cpa_process_fault+0x3ea/0x40e Nov 17 04:00:17 unraid kernel: Code: d0 48 c1 e8 0c 49 89 44 24 30 31 db eb 2b 49 8b 7c 24 30 e8 d4 f3 ff ff 84 c0 75 18 49 8b 04 24 48 89 ee 48 c7 c7 d7 1a 0c 82 <48> 8b 10 e8 16 c8 00 00 0f 0b bb f2 ff ff ff 48 83 c4 30 89 d8 5b Nov 17 04:00:17 unraid kernel: RSP: 0018:ffffc90008f3f720 EFLAGS: 00010246 Nov 17 04:00:17 unraid kernel: RAX: 0000000000000000 RBX: ffffc90008f3f890 RCX: 0000000080000000 Nov 17 04:00:17 unraid kernel: RDX: 0000000000001001 RSI: ffff1108714cc000 RDI: ffffffff820c1ad7 Nov 17 04:00:17 unraid kernel: RBP: ffff1108714cc000 R08: ffffc90008f3f7de R09: ffffffff8220a110 Nov 17 04:00:17 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc90008f3f890 Nov 17 04:00:17 unraid kernel: R13: 80000008354d9073 R14: 0000000000000000 R15: 0000000000000000 Nov 17 04:00:17 unraid kernel: FS: 0000000000000000(0000) GS:ffff88880f980000(0000) knlGS:0000000000000000 Nov 17 04:00:17 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:00:17 unraid kernel: CR2: 0000000000000000 CR3: 000000000220a004 CR4: 00000000001706e0 Nov 17 04:00:17 unraid kernel: note: nvidia-smi[15798] exited with irqs disabled Nov 17 04:00:17 unraid kernel: note: nvidia-smi[15798] exited with preempt_count 1 Nov 17 04:00:17 unraid kernel: Fixing recursive fault but reboot is needed! Nov 17 04:00:17 unraid kernel: BUG: using smp_processor_id() in preemptible [00000000] code: nvidia-smi/15798 Nov 17 04:00:17 unraid kernel: caller is __schedule+0x2f/0x612 Nov 17 04:00:17 unraid kernel: CPU: 12 PID: 15798 Comm: nvidia-smi Tainted: P D O 6.1.106-Unraid #1 Nov 17 04:00:17 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:00:17 unraid kernel: Call Trace: Nov 17 04:00:17 unraid kernel: <TASK> Nov 17 04:00:17 unraid kernel: dump_stack_lvl+0x44/0x5c Nov 17 04:00:17 unraid kernel: check_preemption_disabled+0xcb/0xe0 Nov 17 04:00:17 unraid kernel: __schedule+0x2f/0x612 Nov 17 04:00:17 unraid kernel: ? _printk+0x58/0x73 Nov 17 04:00:17 unraid kernel: do_task_dead+0x3f/0x41 Nov 17 04:00:17 unraid kernel: make_task_dead+0x114/0x11c Nov 17 04:00:17 unraid kernel: rewind_stack_and_make_dead+0x17/0x17 Nov 17 04:00:17 unraid kernel: RIP: 0033:0x14f77af3e5b1 Nov 17 04:00:17 unraid kernel: Code: Unable to access opcode bytes at 0x14f77af3e587. Nov 17 04:00:17 unraid kernel: RSP: 002b:00007fff3e019ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 Nov 17 04:00:17 unraid kernel: RAX: 0000000000000000 RBX: 00007fff3e01a630 RCX: 000014f77af3e5b1 Nov 17 04:00:17 unraid kernel: RDX: 000000000000ffff RSI: 0000000000000000 RDI: 0000000000000004 Nov 17 04:00:17 unraid kernel: RBP: 00007fff3e01a5a0 R08: 00007fff3e019b40 R09: 00007fff3e019b5c Nov 17 04:00:17 unraid kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014f77abd5ec0 Nov 17 04:00:17 unraid kernel: R13: 000014f77abd6820 R14: 000014f77abd6820 R15: 000014f77abd67c0 Nov 17 04:00:17 unraid kernel: </TASK> Nov 17 04:00:17 unraid kernel: BUG: scheduling while atomic: nvidia-smi/15798/0x00000000 Nov 17 04:00:17 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp nvidia_drm(PO) nvidia_modeset(PO) kvm_intel ipmi_ssif kvm nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd Nov 17 04:00:17 unraid kernel: cryptd mgag200 video rapl drm_shmem_helper intel_cstate drm_kms_helper drm intel_uncore mpt3sas mei_me backlight i2c_algo_bit input_leds mei joydev led_class i2c_core raid_class syscopyarea sysfillrect scsi_transport_sas sysimgblt fb_sys_fops wmi ipmi_si button acpi_power_meter unix [last unloaded: xfrm_algo] Nov 17 04:00:17 unraid kernel: Preemption disabled at: Nov 17 04:00:17 unraid kernel: [<0000000000000000>] 0x0 Nov 17 04:00:17 unraid kernel: CPU: 12 PID: 15798 Comm: nvidia-smi Tainted: P D O 6.1.106-Unraid #1 Nov 17 04:00:17 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:00:17 unraid kernel: Call Trace: Nov 17 04:00:17 unraid kernel: <TASK> Nov 17 04:00:17 unraid kernel: dump_stack_lvl+0x44/0x5c Nov 17 04:00:17 unraid kernel: __schedule_bug+0x9a/0xac Nov 17 04:00:17 unraid kernel: __schedule+0x59/0x612 Nov 17 04:00:17 unraid kernel: ? _printk+0x58/0x73 Nov 17 04:00:17 unraid kernel: do_task_dead+0x3f/0x41 Nov 17 04:00:17 unraid kernel: make_task_dead+0x114/0x11c Nov 17 04:00:17 unraid kernel: rewind_stack_and_make_dead+0x17/0x17 Nov 17 04:00:17 unraid kernel: RIP: 0033:0x14f77af3e5b1 Nov 17 04:00:17 unraid kernel: Code: Unable to access opcode bytes at 0x14f77af3e587. Nov 17 04:00:17 unraid kernel: RSP: 002b:00007fff3e019ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 Nov 17 04:00:17 unraid kernel: RAX: 0000000000000000 RBX: 00007fff3e01a630 RCX: 000014f77af3e5b1 Nov 17 04:00:17 unraid kernel: RDX: 000000000000ffff RSI: 0000000000000000 RDI: 0000000000000004 Nov 17 04:00:17 unraid kernel: RBP: 00007fff3e01a5a0 R08: 00007fff3e019b40 R09: 00007fff3e019b5c Nov 17 04:00:17 unraid kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014f77abd5ec0 Nov 17 04:00:17 unraid kernel: R13: 000014f77abd6820 R14: 000014f77abd6820 R15: 000014f77abd67c0 Nov 17 04:00:17 unraid kernel: </TASK> Nov 17 04:21:14 unraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Nov 17 04:30:05 unraid root: Fix Common Problems Version 2024.11.12 Nov 17 04:30:10 unraid root: Fix Common Problems: Other Warning: Docker application tdarr has moderator comments listed ** Ignored Nov 17 04:30:24 unraid root: Fix Common Problems: Warning: Syslog mirrored to flash Nov 17 04:35:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:35:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=24790 Nov 17 04:35:43 unraid kernel: (detected by 21, t=60002 jiffies, g=130869229, q=3978768 ncpus=32) Nov 17 04:35:43 unraid kernel: Sending NMI from CPU 21 to CPUs 3: Nov 17 04:35:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:35:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:35:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:35:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:35:43 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 04:35:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:35:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:35:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:35:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:35:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:35:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:35:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:35:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:35:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:35:43 unraid kernel: Call Trace: Nov 17 04:35:43 unraid kernel: <NMI> Nov 17 04:35:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:35:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:35:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:35:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:35:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:35:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:35:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:35:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:35:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:35:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:35:43 unraid kernel: </NMI> Nov 17 04:35:43 unraid kernel: <TASK> Nov 17 04:35:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:35:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:35:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:35:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:35:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:35:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:35:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:35:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:35:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:35:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:35:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:35:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:35:43 unraid kernel: </TASK> Nov 17 04:35:43 unraid kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.424 msecs Nov 17 04:38:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:38:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=103310 Nov 17 04:38:43 unraid kernel: (detected by 0, t=240010 jiffies, g=130869229, q=13113275 ncpus=32) Nov 17 04:38:43 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 04:38:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:38:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:38:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:38:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:38:43 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 04:38:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:38:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:38:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:38:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:38:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:38:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:38:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:38:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:38:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:38:43 unraid kernel: Call Trace: Nov 17 04:38:43 unraid kernel: <NMI> Nov 17 04:38:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:38:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:38:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:38:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:38:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:38:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:38:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:38:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:38:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:38:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:38:43 unraid kernel: </NMI> Nov 17 04:38:43 unraid kernel: <TASK> Nov 17 04:38:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:38:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:38:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:38:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:38:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:38:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:38:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:38:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:38:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:38:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:38:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:38:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:38:43 unraid kernel: </TASK> Nov 17 04:41:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:41:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=191566 Nov 17 04:41:43 unraid kernel: (detected by 6, t=420018 jiffies, g=130869229, q=22313940 ncpus=32) Nov 17 04:41:43 unraid kernel: Sending NMI from CPU 6 to CPUs 3: Nov 17 04:41:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:41:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:41:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:41:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:41:43 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 04:41:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:41:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:41:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:41:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:41:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:41:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:41:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:41:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:41:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:41:43 unraid kernel: Call Trace: Nov 17 04:41:43 unraid kernel: <NMI> Nov 17 04:41:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:41:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:41:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:41:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:41:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:41:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:41:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:41:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:41:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:41:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:41:43 unraid kernel: </NMI> Nov 17 04:41:43 unraid kernel: <TASK> Nov 17 04:41:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:41:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:41:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:41:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:41:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:41:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:41:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:41:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:41:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:41:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:41:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:41:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:41:43 unraid kernel: </TASK> Nov 17 04:44:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:44:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=280351 Nov 17 04:44:43 unraid kernel: (detected by 6, t=600026 jiffies, g=130869229, q=31460269 ncpus=32) Nov 17 04:44:43 unraid kernel: Sending NMI from CPU 6 to CPUs 3: Nov 17 04:44:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:44:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:44:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:44:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 04:44:43 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 04:44:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:44:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:44:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:44:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:44:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:44:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:44:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:44:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:44:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:44:43 unraid kernel: Call Trace: Nov 17 04:44:43 unraid kernel: <NMI> Nov 17 04:44:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:44:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:44:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:44:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 04:44:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:44:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:44:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:44:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 04:44:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 04:44:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 04:44:43 unraid kernel: </NMI> Nov 17 04:44:43 unraid kernel: <TASK> Nov 17 04:44:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:44:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:44:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:44:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:44:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:44:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:44:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:44:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:44:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:44:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:44:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:44:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:44:43 unraid kernel: </TASK> Nov 17 04:47:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:47:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=362943 Nov 17 04:47:43 unraid kernel: (detected by 7, t=780034 jiffies, g=130869229, q=31746639 ncpus=32) Nov 17 04:47:43 unraid kernel: Sending NMI from CPU 7 to CPUs 3: Nov 17 04:47:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:47:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:47:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:47:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:47:43 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 04:47:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:47:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:47:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:47:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:47:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:47:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:47:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:47:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:47:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:47:43 unraid kernel: Call Trace: Nov 17 04:47:43 unraid kernel: <NMI> Nov 17 04:47:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:47:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:47:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:47:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:47:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:47:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:47:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:47:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:47:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:47:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:47:43 unraid kernel: </NMI> Nov 17 04:47:43 unraid kernel: <TASK> Nov 17 04:47:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:47:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:47:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:47:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:47:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:47:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:47:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:47:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:47:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:47:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:47:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:47:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:47:43 unraid kernel: </TASK> Nov 17 04:50:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:50:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=442915 Nov 17 04:50:43 unraid kernel: (detected by 21, t=960042 jiffies, g=130869229, q=31763640 ncpus=32) Nov 17 04:50:43 unraid kernel: Sending NMI from CPU 21 to CPUs 3: Nov 17 04:50:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:50:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:50:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:50:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:50:43 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 04:50:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:50:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:50:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:50:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:50:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:50:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:50:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:50:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:50:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:50:43 unraid kernel: Call Trace: Nov 17 04:50:43 unraid kernel: <NMI> Nov 17 04:50:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:50:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:50:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:50:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:50:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:50:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:50:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:50:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:50:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:50:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:50:43 unraid kernel: </NMI> Nov 17 04:50:43 unraid kernel: <TASK> Nov 17 04:50:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:50:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:50:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:50:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:50:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:50:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:50:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:50:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:50:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:50:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:50:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:50:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:50:43 unraid kernel: </TASK> Nov 17 04:53:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:53:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=525966 Nov 17 04:53:43 unraid kernel: (detected by 11, t=1140050 jiffies, g=130869229, q=31805055 ncpus=32) Nov 17 04:53:43 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 04:53:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:53:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:53:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:53:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:53:43 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 04:53:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:53:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:53:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:53:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:53:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:53:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:53:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:53:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:53:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:53:43 unraid kernel: Call Trace: Nov 17 04:53:43 unraid kernel: <NMI> Nov 17 04:53:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:53:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:53:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:53:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:53:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:53:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:53:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:53:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:53:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:53:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 04:53:43 unraid kernel: </NMI> Nov 17 04:53:43 unraid kernel: <TASK> Nov 17 04:53:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:53:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:53:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:53:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:53:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:53:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:53:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:53:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:53:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:53:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:53:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:53:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:53:43 unraid kernel: </TASK> Nov 17 04:56:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:56:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=613405 Nov 17 04:56:43 unraid kernel: (detected by 13, t=1320058 jiffies, g=130869229, q=31827757 ncpus=32) Nov 17 04:56:43 unraid kernel: Sending NMI from CPU 13 to CPUs 3: Nov 17 04:56:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:56:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:56:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:56:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:56:43 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 04:56:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:56:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:56:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:56:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:56:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:56:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:56:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:56:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:56:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:56:43 unraid kernel: Call Trace: Nov 17 04:56:43 unraid kernel: <NMI> Nov 17 04:56:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:56:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:56:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:56:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:56:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:56:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:56:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:56:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:56:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:56:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 04:56:43 unraid kernel: </NMI> Nov 17 04:56:43 unraid kernel: <TASK> Nov 17 04:56:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:56:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:56:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:56:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:56:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:56:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:56:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:56:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:56:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:56:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:56:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:56:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:56:43 unraid kernel: </TASK> Nov 17 04:59:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 04:59:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=702829 Nov 17 04:59:43 unraid kernel: (detected by 29, t=1500066 jiffies, g=130869229, q=31860969 ncpus=32) Nov 17 04:59:43 unraid kernel: Sending NMI from CPU 29 to CPUs 3: Nov 17 04:59:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 04:59:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 04:59:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 04:59:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:59:43 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 04:59:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 04:59:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 04:59:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 04:59:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 04:59:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 04:59:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 04:59:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 04:59:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 04:59:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 04:59:43 unraid kernel: Call Trace: Nov 17 04:59:43 unraid kernel: <NMI> Nov 17 04:59:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 04:59:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 04:59:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 04:59:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:59:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 04:59:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 04:59:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 04:59:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:59:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:59:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 04:59:43 unraid kernel: </NMI> Nov 17 04:59:43 unraid kernel: <TASK> Nov 17 04:59:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 04:59:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 04:59:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 04:59:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 04:59:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:59:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 04:59:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 04:59:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 04:59:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 04:59:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 04:59:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 04:59:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 04:59:43 unraid kernel: </TASK> Nov 17 05:02:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:02:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=785465 Nov 17 05:02:43 unraid kernel: (detected by 6, t=1680074 jiffies, g=130869229, q=31900861 ncpus=32) Nov 17 05:02:43 unraid kernel: Sending NMI from CPU 6 to CPUs 3: Nov 17 05:02:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:02:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:02:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:02:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:02:43 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 05:02:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:02:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:02:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:02:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:02:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:02:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:02:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:02:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:02:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:02:43 unraid kernel: Call Trace: Nov 17 05:02:43 unraid kernel: <NMI> Nov 17 05:02:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:02:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:02:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:02:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:02:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:02:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:02:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:02:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:02:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:02:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:02:43 unraid kernel: </NMI> Nov 17 05:02:43 unraid kernel: <TASK> Nov 17 05:02:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:02:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:02:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:02:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:02:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:02:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:02:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:02:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:02:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:02:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:02:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:02:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:02:43 unraid kernel: </TASK> Nov 17 05:05:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:05:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=865327 Nov 17 05:05:43 unraid kernel: (detected by 9, t=1860082 jiffies, g=130869229, q=31920443 ncpus=32) Nov 17 05:05:43 unraid kernel: Sending NMI from CPU 9 to CPUs 3: Nov 17 05:05:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:05:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:05:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:05:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:05:43 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 05:05:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:05:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:05:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:05:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:05:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:05:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:05:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:05:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:05:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:05:43 unraid kernel: Call Trace: Nov 17 05:05:43 unraid kernel: <NMI> Nov 17 05:05:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:05:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:05:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:05:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:05:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:05:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:05:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:05:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:05:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:05:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:05:43 unraid kernel: </NMI> Nov 17 05:05:43 unraid kernel: <TASK> Nov 17 05:05:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:05:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:05:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:05:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:05:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:05:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:05:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:05:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:05:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:05:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:05:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:05:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:05:43 unraid kernel: </TASK> Nov 17 05:08:43 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:08:43 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=946648 Nov 17 05:08:43 unraid kernel: (detected by 30, t=2040090 jiffies, g=130869229, q=31960190 ncpus=32) Nov 17 05:08:43 unraid kernel: Sending NMI from CPU 30 to CPUs 3: Nov 17 05:08:43 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:08:43 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:08:43 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:08:43 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:08:43 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:08:43 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:08:43 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:08:43 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:08:43 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:08:43 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:08:43 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:08:43 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:08:43 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:08:43 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:08:43 unraid kernel: Call Trace: Nov 17 05:08:43 unraid kernel: <NMI> Nov 17 05:08:43 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:08:43 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:08:43 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:08:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:08:43 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:08:43 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:08:43 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:08:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:08:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:08:43 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:08:43 unraid kernel: </NMI> Nov 17 05:08:43 unraid kernel: <TASK> Nov 17 05:08:43 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:08:43 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:08:43 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:08:43 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:08:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:08:43 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:08:43 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:08:43 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:08:43 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:08:43 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:08:43 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:08:43 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:08:43 unraid kernel: </TASK> Nov 17 05:11:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:11:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1026793 Nov 17 05:11:44 unraid kernel: (detected by 13, t=2220098 jiffies, g=130869229, q=31983475 ncpus=32) Nov 17 05:11:44 unraid kernel: Sending NMI from CPU 13 to CPUs 3: Nov 17 05:11:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:11:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:11:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:11:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:11:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:11:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:11:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:11:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:11:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:11:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:11:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:11:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:11:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:11:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:11:44 unraid kernel: Call Trace: Nov 17 05:11:44 unraid kernel: <NMI> Nov 17 05:11:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:11:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:11:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:11:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:11:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:11:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:11:44 unraid kernel: </NMI> Nov 17 05:11:44 unraid kernel: <TASK> Nov 17 05:11:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:11:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:11:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:11:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:11:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:11:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:11:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:11:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:11:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:11:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:11:44 unraid kernel: </TASK> Nov 17 05:14:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:14:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1114680 Nov 17 05:14:44 unraid kernel: (detected by 0, t=2400106 jiffies, g=130869229, q=32020394 ncpus=32) Nov 17 05:14:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 05:14:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:14:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:14:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:14:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:14:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 05:14:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:14:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:14:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:14:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:14:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:14:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:14:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:14:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:14:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:14:44 unraid kernel: Call Trace: Nov 17 05:14:44 unraid kernel: <NMI> Nov 17 05:14:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:14:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:14:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:14:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:14:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:14:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:14:44 unraid kernel: </NMI> Nov 17 05:14:44 unraid kernel: <TASK> Nov 17 05:14:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:14:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:14:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:14:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:14:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:14:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:14:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:14:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:14:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:14:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:14:44 unraid kernel: </TASK> Nov 17 05:17:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:17:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1197933 Nov 17 05:17:44 unraid kernel: (detected by 29, t=2580114 jiffies, g=130869229, q=32055763 ncpus=32) Nov 17 05:17:44 unraid kernel: Sending NMI from CPU 29 to CPUs 3: Nov 17 05:17:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:17:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:17:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:17:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:17:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 05:17:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:17:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:17:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:17:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:17:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:17:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:17:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:17:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:17:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:17:44 unraid kernel: Call Trace: Nov 17 05:17:44 unraid kernel: <NMI> Nov 17 05:17:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:17:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:17:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:17:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:17:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:17:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:17:44 unraid kernel: </NMI> Nov 17 05:17:44 unraid kernel: <TASK> Nov 17 05:17:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:17:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:17:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:17:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:17:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:17:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:17:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:17:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:17:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:17:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:17:44 unraid kernel: </TASK> Nov 17 05:20:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:20:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1277914 Nov 17 05:20:44 unraid kernel: (detected by 18, t=2760122 jiffies, g=130869229, q=32078914 ncpus=32) Nov 17 05:20:44 unraid kernel: Sending NMI from CPU 18 to CPUs 3: Nov 17 05:20:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:20:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:20:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:20:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:20:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 05:20:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:20:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:20:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:20:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:20:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:20:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:20:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:20:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:20:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:20:44 unraid kernel: Call Trace: Nov 17 05:20:44 unraid kernel: <NMI> Nov 17 05:20:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:20:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:20:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:20:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:20:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:20:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:20:44 unraid kernel: </NMI> Nov 17 05:20:44 unraid kernel: <TASK> Nov 17 05:20:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:20:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:20:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:20:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:20:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:20:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:20:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:20:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:20:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:20:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:20:44 unraid kernel: </TASK> Nov 17 05:23:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:23:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1358597 Nov 17 05:23:44 unraid kernel: (detected by 18, t=2940130 jiffies, g=130869229, q=32118820 ncpus=32) Nov 17 05:23:44 unraid kernel: Sending NMI from CPU 18 to CPUs 3: Nov 17 05:23:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:23:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:23:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:23:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:23:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:23:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:23:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:23:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:23:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:23:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:23:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:23:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:23:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:23:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:23:44 unraid kernel: Call Trace: Nov 17 05:23:44 unraid kernel: <NMI> Nov 17 05:23:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:23:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:23:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:23:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:23:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:23:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:23:44 unraid kernel: </NMI> Nov 17 05:23:44 unraid kernel: <TASK> Nov 17 05:23:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:23:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:23:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:23:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:23:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:23:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:23:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:23:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:23:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:23:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:23:44 unraid kernel: </TASK> Nov 17 05:26:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:26:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1431161 Nov 17 05:26:44 unraid kernel: (detected by 17, t=3120138 jiffies, g=130869229, q=32139177 ncpus=32) Nov 17 05:26:44 unraid kernel: Sending NMI from CPU 17 to CPUs 3: Nov 17 05:26:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:26:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:26:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:26:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:26:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 05:26:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:26:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:26:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:26:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:26:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:26:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:26:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:26:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:26:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:26:44 unraid kernel: Call Trace: Nov 17 05:26:44 unraid kernel: <NMI> Nov 17 05:26:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:26:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:26:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:26:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:26:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:26:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:26:44 unraid kernel: </NMI> Nov 17 05:26:44 unraid kernel: <TASK> Nov 17 05:26:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:26:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:26:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:26:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:26:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:26:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:26:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:26:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:26:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:26:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:26:44 unraid kernel: </TASK> Nov 17 05:29:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:29:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1506708 Nov 17 05:29:44 unraid kernel: (detected by 6, t=3300146 jiffies, g=130869229, q=32173129 ncpus=32) Nov 17 05:29:44 unraid kernel: Sending NMI from CPU 6 to CPUs 3: Nov 17 05:29:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:29:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:29:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:29:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:29:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 05:29:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:29:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:29:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:29:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:29:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:29:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:29:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:29:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:29:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:29:44 unraid kernel: Call Trace: Nov 17 05:29:44 unraid kernel: <NMI> Nov 17 05:29:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:29:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:29:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:29:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:29:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:29:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:29:44 unraid kernel: </NMI> Nov 17 05:29:44 unraid kernel: <TASK> Nov 17 05:29:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:29:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:29:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:29:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:29:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:29:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:29:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:29:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:29:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:29:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:29:44 unraid kernel: </TASK> Nov 17 05:32:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:32:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1594832 Nov 17 05:32:44 unraid kernel: (detected by 0, t=3480154 jiffies, g=130869229, q=32207553 ncpus=32) Nov 17 05:32:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 05:32:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:32:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:32:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:32:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:32:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 05:32:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:32:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:32:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:32:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:32:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:32:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:32:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:32:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:32:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:32:44 unraid kernel: Call Trace: Nov 17 05:32:44 unraid kernel: <NMI> Nov 17 05:32:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:32:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:32:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:32:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:32:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:32:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 05:32:44 unraid kernel: </NMI> Nov 17 05:32:44 unraid kernel: <TASK> Nov 17 05:32:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:32:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:32:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:32:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:32:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:32:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:32:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:32:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:32:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:32:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:32:44 unraid kernel: </TASK> Nov 17 05:35:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:35:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1670990 Nov 17 05:35:44 unraid kernel: (detected by 6, t=3660162 jiffies, g=130869229, q=32230454 ncpus=32) Nov 17 05:35:44 unraid kernel: Sending NMI from CPU 6 to CPUs 3: Nov 17 05:35:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:35:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:35:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:35:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:35:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 05:35:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:35:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:35:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:35:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:35:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:35:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:35:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:35:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:35:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:35:44 unraid kernel: Call Trace: Nov 17 05:35:44 unraid kernel: <NMI> Nov 17 05:35:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:35:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:35:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:35:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:35:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:35:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:35:44 unraid kernel: </NMI> Nov 17 05:35:44 unraid kernel: <TASK> Nov 17 05:35:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:35:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:35:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:35:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:35:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:35:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:35:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:35:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:35:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:35:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:35:44 unraid kernel: </TASK> Nov 17 05:38:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:38:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1741277 Nov 17 05:38:44 unraid kernel: (detected by 1, t=3840170 jiffies, g=130869229, q=32265180 ncpus=32) Nov 17 05:38:44 unraid kernel: Sending NMI from CPU 1 to CPUs 3: Nov 17 05:38:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:38:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:38:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:38:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:38:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 05:38:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:38:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:38:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:38:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:38:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:38:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:38:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:38:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:38:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:38:44 unraid kernel: Call Trace: Nov 17 05:38:44 unraid kernel: <NMI> Nov 17 05:38:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:38:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:38:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:38:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:38:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:38:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:38:44 unraid kernel: </NMI> Nov 17 05:38:44 unraid kernel: <TASK> Nov 17 05:38:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:38:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:38:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:38:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:38:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:38:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:38:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:38:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:38:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:38:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:38:44 unraid kernel: </TASK> Nov 17 05:41:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:41:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1812369 Nov 17 05:41:44 unraid kernel: (detected by 11, t=4020178 jiffies, g=130869229, q=32286868 ncpus=32) Nov 17 05:41:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 05:41:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:41:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:41:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:41:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:41:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:41:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:41:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:41:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:41:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:41:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:41:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:41:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:41:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:41:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:41:44 unraid kernel: Call Trace: Nov 17 05:41:44 unraid kernel: <NMI> Nov 17 05:41:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:41:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:41:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:41:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:41:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:41:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:41:44 unraid kernel: </NMI> Nov 17 05:41:44 unraid kernel: <TASK> Nov 17 05:41:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:41:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:41:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:41:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:41:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:41:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:41:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:41:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:41:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:41:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:41:44 unraid kernel: </TASK> Nov 17 05:44:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:44:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1883329 Nov 17 05:44:44 unraid kernel: (detected by 16, t=4200186 jiffies, g=130869229, q=32322693 ncpus=32) Nov 17 05:44:44 unraid kernel: Sending NMI from CPU 16 to CPUs 3: Nov 17 05:44:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:44:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:44:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:44:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:44:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:44:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:44:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:44:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:44:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:44:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:44:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:44:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:44:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:44:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:44:44 unraid kernel: Call Trace: Nov 17 05:44:44 unraid kernel: <NMI> Nov 17 05:44:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:44:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:44:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:44:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:44:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:44:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:44:44 unraid kernel: </NMI> Nov 17 05:44:44 unraid kernel: <TASK> Nov 17 05:44:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:44:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:44:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:44:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:44:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:44:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:44:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:44:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:44:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:44:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:44:44 unraid kernel: </TASK> Nov 17 05:47:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:47:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=1962908 Nov 17 05:47:44 unraid kernel: (detected by 0, t=4380194 jiffies, g=130869229, q=32359772 ncpus=32) Nov 17 05:47:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 05:47:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:47:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:47:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:47:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:47:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 05:47:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:47:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:47:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:47:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:47:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:47:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:47:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:47:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:47:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:47:44 unraid kernel: Call Trace: Nov 17 05:47:44 unraid kernel: <NMI> Nov 17 05:47:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:47:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:47:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:47:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:47:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:47:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:47:44 unraid kernel: </NMI> Nov 17 05:47:44 unraid kernel: <TASK> Nov 17 05:47:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:47:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:47:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:47:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:47:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:47:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:47:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:47:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:47:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:47:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:47:44 unraid kernel: </TASK> Nov 17 05:50:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:50:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2050050 Nov 17 05:50:44 unraid kernel: (detected by 20, t=4560202 jiffies, g=130869229, q=32381085 ncpus=32) Nov 17 05:50:44 unraid kernel: Sending NMI from CPU 20 to CPUs 3: Nov 17 05:50:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:50:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:50:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:50:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:50:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 05:50:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:50:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:50:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:50:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:50:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:50:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:50:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:50:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:50:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:50:44 unraid kernel: Call Trace: Nov 17 05:50:44 unraid kernel: <NMI> Nov 17 05:50:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:50:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:50:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:50:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:50:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:50:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 05:50:44 unraid kernel: </NMI> Nov 17 05:50:44 unraid kernel: <TASK> Nov 17 05:50:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:50:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:50:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:50:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:50:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:50:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:50:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:50:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:50:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:50:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:50:44 unraid kernel: </TASK> Nov 17 05:53:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:53:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2119223 Nov 17 05:53:44 unraid kernel: (detected by 12, t=4740210 jiffies, g=130869229, q=32418940 ncpus=32) Nov 17 05:53:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 05:53:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:53:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:53:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:53:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:53:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:53:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:53:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:53:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:53:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:53:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:53:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:53:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:53:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:53:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:53:44 unraid kernel: Call Trace: Nov 17 05:53:44 unraid kernel: <NMI> Nov 17 05:53:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:53:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:53:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:53:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:53:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:53:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:53:44 unraid kernel: </NMI> Nov 17 05:53:44 unraid kernel: <TASK> Nov 17 05:53:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:53:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:53:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:53:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:53:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:53:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:53:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:53:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:53:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:53:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:53:44 unraid kernel: </TASK> Nov 17 05:56:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:56:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2189735 Nov 17 05:56:44 unraid kernel: (detected by 19, t=4920218 jiffies, g=130869229, q=32439314 ncpus=32) Nov 17 05:56:44 unraid kernel: Sending NMI from CPU 19 to CPUs 3: Nov 17 05:56:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:56:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:56:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:56:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:56:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 05:56:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:56:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:56:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:56:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:56:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:56:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:56:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:56:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:56:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:56:44 unraid kernel: Call Trace: Nov 17 05:56:44 unraid kernel: <NMI> Nov 17 05:56:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:56:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:56:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:56:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:56:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:56:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 05:56:44 unraid kernel: </NMI> Nov 17 05:56:44 unraid kernel: <TASK> Nov 17 05:56:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:56:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:56:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:56:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:56:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:56:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:56:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:56:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:56:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:56:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:56:44 unraid kernel: </TASK> Nov 17 05:56:44 unraid kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.816 msecs Nov 17 05:59:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 05:59:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2260574 Nov 17 05:59:44 unraid kernel: (detected by 25, t=5100226 jiffies, g=130869229, q=32486418 ncpus=32) Nov 17 05:59:44 unraid kernel: Sending NMI from CPU 25 to CPUs 3: Nov 17 05:59:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 05:59:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 05:59:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 05:59:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:59:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 05:59:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 05:59:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 05:59:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 05:59:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 05:59:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 05:59:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 05:59:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 05:59:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 05:59:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 05:59:44 unraid kernel: Call Trace: Nov 17 05:59:44 unraid kernel: <NMI> Nov 17 05:59:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 05:59:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 05:59:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 05:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:59:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 05:59:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 05:59:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 05:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 05:59:44 unraid kernel: </NMI> Nov 17 05:59:44 unraid kernel: <TASK> Nov 17 05:59:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 05:59:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 05:59:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 05:59:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 05:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:59:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 05:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 05:59:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 05:59:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 05:59:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 05:59:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 05:59:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 05:59:44 unraid kernel: </TASK> Nov 17 06:02:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:02:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2330393 Nov 17 06:02:44 unraid kernel: (detected by 2, t=5280234 jiffies, g=130869229, q=32521013 ncpus=32) Nov 17 06:02:44 unraid kernel: Sending NMI from CPU 2 to CPUs 3: Nov 17 06:02:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:02:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:02:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:02:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:02:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:02:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:02:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:02:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:02:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:02:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:02:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:02:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:02:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:02:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:02:44 unraid kernel: Call Trace: Nov 17 06:02:44 unraid kernel: <NMI> Nov 17 06:02:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:02:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:02:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:02:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:02:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:02:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:02:44 unraid kernel: </NMI> Nov 17 06:02:44 unraid kernel: <TASK> Nov 17 06:02:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:02:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:02:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:02:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:02:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:02:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:02:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:02:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:02:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:02:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:02:44 unraid kernel: </TASK> Nov 17 06:05:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:05:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2415726 Nov 17 06:05:44 unraid kernel: (detected by 13, t=5460242 jiffies, g=130869229, q=32539901 ncpus=32) Nov 17 06:05:44 unraid kernel: Sending NMI from CPU 13 to CPUs 3: Nov 17 06:05:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:05:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:05:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:05:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:05:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:05:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:05:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:05:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:05:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:05:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:05:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:05:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:05:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:05:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:05:44 unraid kernel: Call Trace: Nov 17 06:05:44 unraid kernel: <NMI> Nov 17 06:05:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:05:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:05:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:05:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:05:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:05:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:05:44 unraid kernel: </NMI> Nov 17 06:05:44 unraid kernel: <TASK> Nov 17 06:05:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:05:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:05:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:05:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:05:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:05:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:05:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:05:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:05:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:05:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:05:44 unraid kernel: </TASK> Nov 17 06:08:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:08:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2496328 Nov 17 06:08:44 unraid kernel: (detected by 4, t=5640250 jiffies, g=130869229, q=32573242 ncpus=32) Nov 17 06:08:44 unraid kernel: Sending NMI from CPU 4 to CPUs 3: Nov 17 06:08:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:08:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:08:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:08:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:08:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:08:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:08:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:08:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:08:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:08:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:08:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:08:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:08:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:08:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:08:44 unraid kernel: Call Trace: Nov 17 06:08:44 unraid kernel: <NMI> Nov 17 06:08:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:08:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:08:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:08:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:08:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:08:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:08:44 unraid kernel: </NMI> Nov 17 06:08:44 unraid kernel: <TASK> Nov 17 06:08:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:08:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:08:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:08:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:08:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:08:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:08:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:08:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:08:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:08:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:08:44 unraid kernel: </TASK> Nov 17 06:11:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:11:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2567460 Nov 17 06:11:44 unraid kernel: (detected by 21, t=5820258 jiffies, g=130869229, q=32595006 ncpus=32) Nov 17 06:11:44 unraid kernel: Sending NMI from CPU 21 to CPUs 3: Nov 17 06:11:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:11:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:11:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:11:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:11:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 06:11:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:11:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:11:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:11:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:11:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:11:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:11:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:11:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:11:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:11:44 unraid kernel: Call Trace: Nov 17 06:11:44 unraid kernel: <NMI> Nov 17 06:11:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:11:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:11:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:11:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:11:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:11:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:11:44 unraid kernel: </NMI> Nov 17 06:11:44 unraid kernel: <TASK> Nov 17 06:11:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:11:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:11:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:11:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:11:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:11:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:11:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:11:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:11:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:11:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:11:44 unraid kernel: </TASK> Nov 17 06:14:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:14:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2639568 Nov 17 06:14:44 unraid kernel: (detected by 23, t=6000266 jiffies, g=130869229, q=32629143 ncpus=32) Nov 17 06:14:44 unraid kernel: Sending NMI from CPU 23 to CPUs 3: Nov 17 06:14:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:14:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:14:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:14:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:14:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:14:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:14:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:14:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:14:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:14:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:14:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:14:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:14:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:14:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:14:44 unraid kernel: Call Trace: Nov 17 06:14:44 unraid kernel: <NMI> Nov 17 06:14:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:14:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:14:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:14:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:14:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:14:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:14:44 unraid kernel: </NMI> Nov 17 06:14:44 unraid kernel: <TASK> Nov 17 06:14:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:14:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:14:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:14:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:14:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:14:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:14:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:14:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:14:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:14:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:14:44 unraid kernel: </TASK> Nov 17 06:17:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:17:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2711268 Nov 17 06:17:44 unraid kernel: (detected by 5, t=6180274 jiffies, g=130869229, q=32663828 ncpus=32) Nov 17 06:17:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 06:17:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:17:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:17:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:17:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:17:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:17:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:17:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:17:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:17:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:17:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:17:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:17:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:17:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:17:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:17:44 unraid kernel: Call Trace: Nov 17 06:17:44 unraid kernel: <NMI> Nov 17 06:17:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:17:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:17:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:17:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:17:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:17:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:17:44 unraid kernel: </NMI> Nov 17 06:17:44 unraid kernel: <TASK> Nov 17 06:17:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:17:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:17:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:17:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:17:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:17:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:17:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:17:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:17:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:17:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:17:44 unraid kernel: </TASK> Nov 17 06:20:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:20:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2787067 Nov 17 06:20:44 unraid kernel: (detected by 11, t=6360279 jiffies, g=130869229, q=32683191 ncpus=32) Nov 17 06:20:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 06:20:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:20:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:20:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:20:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:20:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:20:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:20:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:20:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:20:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:20:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:20:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:20:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:20:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:20:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:20:44 unraid kernel: Call Trace: Nov 17 06:20:44 unraid kernel: <NMI> Nov 17 06:20:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:20:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:20:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:20:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:20:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:20:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:20:44 unraid kernel: </NMI> Nov 17 06:20:44 unraid kernel: <TASK> Nov 17 06:20:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:20:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:20:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:20:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:20:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:20:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:20:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:20:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:20:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:20:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:20:44 unraid kernel: </TASK> Nov 17 06:23:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:23:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2873746 Nov 17 06:23:44 unraid kernel: (detected by 9, t=6540287 jiffies, g=130869229, q=32718055 ncpus=32) Nov 17 06:23:44 unraid kernel: Sending NMI from CPU 9 to CPUs 3: Nov 17 06:23:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:23:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:23:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:23:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:23:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:23:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:23:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:23:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:23:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:23:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:23:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:23:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:23:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:23:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:23:44 unraid kernel: Call Trace: Nov 17 06:23:44 unraid kernel: <NMI> Nov 17 06:23:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:23:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:23:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:23:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:23:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:23:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:23:44 unraid kernel: </NMI> Nov 17 06:23:44 unraid kernel: <TASK> Nov 17 06:23:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:23:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:23:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:23:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:23:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:23:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:23:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:23:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:23:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:23:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:23:44 unraid kernel: </TASK> Nov 17 06:26:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:26:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=2944716 Nov 17 06:26:44 unraid kernel: (detected by 28, t=6720295 jiffies, g=130869229, q=32736715 ncpus=32) Nov 17 06:26:44 unraid kernel: Sending NMI from CPU 28 to CPUs 3: Nov 17 06:26:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:26:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:26:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:26:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:26:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:26:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:26:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:26:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:26:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:26:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:26:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:26:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:26:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:26:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:26:44 unraid kernel: Call Trace: Nov 17 06:26:44 unraid kernel: <NMI> Nov 17 06:26:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:26:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:26:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:26:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:26:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:26:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:26:44 unraid kernel: </NMI> Nov 17 06:26:44 unraid kernel: <TASK> Nov 17 06:26:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:26:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:26:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:26:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:26:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:26:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:26:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:26:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:26:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:26:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:26:44 unraid kernel: </TASK> Nov 17 06:29:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:29:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3017105 Nov 17 06:29:44 unraid kernel: (detected by 17, t=6900303 jiffies, g=130869229, q=32771901 ncpus=32) Nov 17 06:29:44 unraid kernel: Sending NMI from CPU 17 to CPUs 3: Nov 17 06:29:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:29:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:29:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:29:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:29:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 06:29:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:29:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:29:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:29:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:29:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:29:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:29:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:29:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:29:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:29:44 unraid kernel: Call Trace: Nov 17 06:29:44 unraid kernel: <NMI> Nov 17 06:29:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:29:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:29:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:29:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:29:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:29:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 06:29:44 unraid kernel: </NMI> Nov 17 06:29:44 unraid kernel: <TASK> Nov 17 06:29:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:29:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:29:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:29:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:29:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:29:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:29:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:29:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:29:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:29:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:29:44 unraid kernel: </TASK> Nov 17 06:32:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:32:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3087478 Nov 17 06:32:44 unraid kernel: (detected by 22, t=7080311 jiffies, g=130869229, q=32804135 ncpus=32) Nov 17 06:32:44 unraid kernel: Sending NMI from CPU 22 to CPUs 3: Nov 17 06:32:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:32:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:32:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:32:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:32:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:32:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:32:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:32:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:32:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:32:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:32:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:32:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:32:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:32:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:32:44 unraid kernel: Call Trace: Nov 17 06:32:44 unraid kernel: <NMI> Nov 17 06:32:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:32:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:32:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:32:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:32:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:32:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:32:44 unraid kernel: </NMI> Nov 17 06:32:44 unraid kernel: <TASK> Nov 17 06:32:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:32:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:32:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:32:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:32:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:32:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:32:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:32:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:32:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:32:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:32:44 unraid kernel: </TASK> Nov 17 06:35:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:35:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3159972 Nov 17 06:35:44 unraid kernel: (detected by 20, t=7260319 jiffies, g=130869229, q=32822981 ncpus=32) Nov 17 06:35:44 unraid kernel: Sending NMI from CPU 20 to CPUs 3: Nov 17 06:35:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:35:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:35:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:35:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:35:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:35:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:35:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:35:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:35:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:35:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:35:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:35:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:35:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:35:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:35:44 unraid kernel: Call Trace: Nov 17 06:35:44 unraid kernel: <NMI> Nov 17 06:35:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:35:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:35:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:35:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:35:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:35:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:35:44 unraid kernel: </NMI> Nov 17 06:35:44 unraid kernel: <TASK> Nov 17 06:35:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:35:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:35:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:35:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:35:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:35:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:35:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:35:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:35:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:35:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:35:44 unraid kernel: </TASK> Nov 17 06:38:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:38:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3241394 Nov 17 06:38:44 unraid kernel: (detected by 13, t=7440327 jiffies, g=130869229, q=32860077 ncpus=32) Nov 17 06:38:44 unraid kernel: Sending NMI from CPU 13 to CPUs 3: Nov 17 06:38:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:38:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:38:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:38:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 06:38:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 06:38:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:38:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:38:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:38:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:38:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:38:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:38:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:38:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:38:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:38:44 unraid kernel: Call Trace: Nov 17 06:38:44 unraid kernel: <NMI> Nov 17 06:38:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:38:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:38:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 06:38:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:38:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:38:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 06:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 06:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 06:38:44 unraid kernel: </NMI> Nov 17 06:38:44 unraid kernel: <TASK> Nov 17 06:38:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:38:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:38:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:38:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:38:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:38:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:38:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:38:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:38:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:38:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:38:44 unraid kernel: </TASK> Nov 17 06:41:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:41:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3325981 Nov 17 06:41:44 unraid kernel: (detected by 30, t=7620335 jiffies, g=130869229, q=32876810 ncpus=32) Nov 17 06:41:44 unraid kernel: Sending NMI from CPU 30 to CPUs 3: Nov 17 06:41:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:41:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:41:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:41:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:41:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:41:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:41:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:41:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:41:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:41:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:41:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:41:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:41:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:41:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:41:44 unraid kernel: Call Trace: Nov 17 06:41:44 unraid kernel: <NMI> Nov 17 06:41:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:41:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:41:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:41:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:41:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:41:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:41:44 unraid kernel: </NMI> Nov 17 06:41:44 unraid kernel: <TASK> Nov 17 06:41:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:41:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:41:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:41:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:41:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:41:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:41:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:41:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:41:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:41:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:41:44 unraid kernel: </TASK> Nov 17 06:44:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:44:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3398740 Nov 17 06:44:44 unraid kernel: (detected by 8, t=7800343 jiffies, g=130869229, q=32911426 ncpus=32) Nov 17 06:44:44 unraid kernel: Sending NMI from CPU 8 to CPUs 3: Nov 17 06:44:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:44:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:44:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:44:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:44:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:44:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:44:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:44:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:44:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:44:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:44:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:44:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:44:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:44:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:44:44 unraid kernel: Call Trace: Nov 17 06:44:44 unraid kernel: <NMI> Nov 17 06:44:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:44:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:44:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:44:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:44:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:44:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:44:44 unraid kernel: </NMI> Nov 17 06:44:44 unraid kernel: <TASK> Nov 17 06:44:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:44:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:44:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:44:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:44:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:44:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:44:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:44:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:44:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:44:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:44:44 unraid kernel: </TASK> Nov 17 06:47:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:47:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3470931 Nov 17 06:47:44 unraid kernel: (detected by 23, t=7980351 jiffies, g=130869229, q=32944547 ncpus=32) Nov 17 06:47:44 unraid kernel: Sending NMI from CPU 23 to CPUs 3: Nov 17 06:47:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:47:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:47:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:47:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:47:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:47:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:47:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:47:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:47:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:47:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:47:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:47:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:47:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:47:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:47:44 unraid kernel: Call Trace: Nov 17 06:47:44 unraid kernel: <NMI> Nov 17 06:47:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:47:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:47:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:47:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:47:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:47:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:47:44 unraid kernel: </NMI> Nov 17 06:47:44 unraid kernel: <TASK> Nov 17 06:47:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:47:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:47:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:47:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:47:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:47:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:47:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:47:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:47:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:47:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:47:44 unraid kernel: </TASK> Nov 17 06:50:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:50:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3544023 Nov 17 06:50:44 unraid kernel: (detected by 9, t=8160359 jiffies, g=130869229, q=32963899 ncpus=32) Nov 17 06:50:44 unraid kernel: Sending NMI from CPU 9 to CPUs 3: Nov 17 06:50:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:50:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:50:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:50:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:50:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:50:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:50:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:50:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:50:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:50:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:50:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:50:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:50:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:50:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:50:44 unraid kernel: Call Trace: Nov 17 06:50:44 unraid kernel: <NMI> Nov 17 06:50:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:50:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:50:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:50:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:50:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:50:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:50:44 unraid kernel: </NMI> Nov 17 06:50:44 unraid kernel: <TASK> Nov 17 06:50:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:50:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:50:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:50:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:50:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:50:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:50:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:50:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:50:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:50:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:50:44 unraid kernel: </TASK> Nov 17 06:53:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:53:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3615801 Nov 17 06:53:44 unraid kernel: (detected by 11, t=8340367 jiffies, g=130869229, q=32998763 ncpus=32) Nov 17 06:53:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 06:53:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:53:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:53:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:53:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:53:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 06:53:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:53:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:53:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:53:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:53:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:53:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:53:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:53:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:53:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:53:44 unraid kernel: Call Trace: Nov 17 06:53:44 unraid kernel: <NMI> Nov 17 06:53:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:53:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:53:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:53:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:53:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:53:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 06:53:44 unraid kernel: </NMI> Nov 17 06:53:44 unraid kernel: <TASK> Nov 17 06:53:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:53:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:53:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:53:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:53:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:53:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:53:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:53:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:53:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:53:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:53:44 unraid kernel: </TASK> Nov 17 06:56:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:56:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3703953 Nov 17 06:56:44 unraid kernel: (detected by 29, t=8520375 jiffies, g=130869229, q=33016378 ncpus=32) Nov 17 06:56:44 unraid kernel: Sending NMI from CPU 29 to CPUs 3: Nov 17 06:56:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:56:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:56:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:56:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:56:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:56:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:56:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:56:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:56:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:56:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:56:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:56:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:56:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:56:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:56:44 unraid kernel: Call Trace: Nov 17 06:56:44 unraid kernel: <NMI> Nov 17 06:56:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:56:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:56:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:56:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:56:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:56:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:56:44 unraid kernel: </NMI> Nov 17 06:56:44 unraid kernel: <TASK> Nov 17 06:56:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:56:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:56:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:56:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:56:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:56:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:56:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:56:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:56:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:56:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:56:44 unraid kernel: </TASK> Nov 17 06:59:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 06:59:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3781157 Nov 17 06:59:44 unraid kernel: (detected by 12, t=8700383 jiffies, g=130869229, q=33049059 ncpus=32) Nov 17 06:59:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 06:59:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 06:59:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 06:59:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 06:59:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:59:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 06:59:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 06:59:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 06:59:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 06:59:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 06:59:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 06:59:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 06:59:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 06:59:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 06:59:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 06:59:44 unraid kernel: Call Trace: Nov 17 06:59:44 unraid kernel: <NMI> Nov 17 06:59:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 06:59:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 06:59:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 06:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:59:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 06:59:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 06:59:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 06:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 06:59:44 unraid kernel: </NMI> Nov 17 06:59:44 unraid kernel: <TASK> Nov 17 06:59:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 06:59:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 06:59:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 06:59:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 06:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:59:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 06:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 06:59:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 06:59:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 06:59:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 06:59:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 06:59:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 06:59:44 unraid kernel: </TASK> Nov 17 07:02:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:02:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3850001 Nov 17 07:02:44 unraid kernel: (detected by 12, t=8880391 jiffies, g=130869229, q=33086619 ncpus=32) Nov 17 07:02:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 07:02:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:02:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:02:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:02:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:02:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 07:02:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:02:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:02:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:02:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:02:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:02:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:02:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:02:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:02:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:02:44 unraid kernel: Call Trace: Nov 17 07:02:44 unraid kernel: <NMI> Nov 17 07:02:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:02:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:02:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:02:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:02:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:02:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:02:44 unraid kernel: </NMI> Nov 17 07:02:44 unraid kernel: <TASK> Nov 17 07:02:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:02:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:02:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:02:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:02:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:02:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:02:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:02:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:02:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:02:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:02:44 unraid kernel: </TASK> Nov 17 07:05:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:05:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3921372 Nov 17 07:05:44 unraid kernel: (detected by 12, t=9060399 jiffies, g=130869229, q=33102335 ncpus=32) Nov 17 07:05:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 07:05:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:05:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:05:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:05:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:05:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:05:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:05:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:05:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:05:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:05:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:05:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:05:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:05:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:05:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:05:44 unraid kernel: Call Trace: Nov 17 07:05:44 unraid kernel: <NMI> Nov 17 07:05:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:05:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:05:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:05:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:05:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:05:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:05:44 unraid kernel: </NMI> Nov 17 07:05:44 unraid kernel: <TASK> Nov 17 07:05:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:05:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:05:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:05:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:05:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:05:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:05:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:05:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:05:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:05:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:05:44 unraid kernel: </TASK> Nov 17 07:08:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:08:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=3992593 Nov 17 07:08:44 unraid kernel: (detected by 11, t=9240407 jiffies, g=130869229, q=33135047 ncpus=32) Nov 17 07:08:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 07:08:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:08:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:08:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:08:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:08:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 07:08:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:08:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:08:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:08:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:08:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:08:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:08:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:08:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:08:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:08:44 unraid kernel: Call Trace: Nov 17 07:08:44 unraid kernel: <NMI> Nov 17 07:08:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:08:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:08:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:08:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:08:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:08:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:08:44 unraid kernel: </NMI> Nov 17 07:08:44 unraid kernel: <TASK> Nov 17 07:08:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:08:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:08:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:08:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:08:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:08:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:08:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:08:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:08:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:08:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:08:44 unraid kernel: </TASK> Nov 17 07:11:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:11:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4071645 Nov 17 07:11:44 unraid kernel: (detected by 19, t=9420415 jiffies, g=130869229, q=33156143 ncpus=32) Nov 17 07:11:44 unraid kernel: Sending NMI from CPU 19 to CPUs 3: Nov 17 07:11:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:11:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:11:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:11:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:11:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 07:11:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:11:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:11:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:11:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:11:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:11:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:11:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:11:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:11:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:11:44 unraid kernel: Call Trace: Nov 17 07:11:44 unraid kernel: <NMI> Nov 17 07:11:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:11:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:11:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:11:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:11:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:11:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:11:44 unraid kernel: </NMI> Nov 17 07:11:44 unraid kernel: <TASK> Nov 17 07:11:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:11:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:11:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:11:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:11:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:11:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:11:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:11:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:11:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:11:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:11:44 unraid kernel: </TASK> Nov 17 07:11:44 unraid kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 3.037 msecs Nov 17 07:14:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:14:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4159704 Nov 17 07:14:44 unraid kernel: (detected by 5, t=9600424 jiffies, g=130869229, q=33189141 ncpus=32) Nov 17 07:14:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 07:14:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:14:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:14:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:14:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:14:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 07:14:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:14:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:14:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:14:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:14:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:14:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:14:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:14:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:14:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:14:44 unraid kernel: Call Trace: Nov 17 07:14:44 unraid kernel: <NMI> Nov 17 07:14:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:14:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:14:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:14:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:14:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:14:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:14:44 unraid kernel: </NMI> Nov 17 07:14:44 unraid kernel: <TASK> Nov 17 07:14:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:14:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:14:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:14:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:14:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:14:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:14:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:14:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:14:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:14:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:14:44 unraid kernel: </TASK> Nov 17 07:17:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:17:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4230698 Nov 17 07:17:44 unraid kernel: (detected by 1, t=9780429 jiffies, g=130869229, q=33221264 ncpus=32) Nov 17 07:17:44 unraid kernel: Sending NMI from CPU 1 to CPUs 3: Nov 17 07:17:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:17:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:17:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:17:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:17:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:17:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:17:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:17:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:17:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:17:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:17:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:17:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:17:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:17:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:17:44 unraid kernel: Call Trace: Nov 17 07:17:44 unraid kernel: <NMI> Nov 17 07:17:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:17:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:17:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:17:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:17:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:17:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:17:44 unraid kernel: </NMI> Nov 17 07:17:44 unraid kernel: <TASK> Nov 17 07:17:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:17:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:17:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:17:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:17:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:17:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:17:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:17:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:17:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:17:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:17:44 unraid kernel: </TASK> Nov 17 07:20:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:20:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4300230 Nov 17 07:20:44 unraid kernel: (detected by 17, t=9960437 jiffies, g=130869229, q=33243195 ncpus=32) Nov 17 07:20:44 unraid kernel: Sending NMI from CPU 17 to CPUs 3: Nov 17 07:20:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:20:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:20:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:20:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:20:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 07:20:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:20:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:20:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:20:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:20:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:20:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:20:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:20:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:20:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:20:44 unraid kernel: Call Trace: Nov 17 07:20:44 unraid kernel: <NMI> Nov 17 07:20:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:20:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:20:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:20:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:20:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:20:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:20:44 unraid kernel: </NMI> Nov 17 07:20:44 unraid kernel: <TASK> Nov 17 07:20:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:20:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:20:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:20:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:20:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:20:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:20:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:20:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:20:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:20:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:20:44 unraid kernel: </TASK> Nov 17 07:23:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:23:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4372067 Nov 17 07:23:44 unraid kernel: (detected by 9, t=10140445 jiffies, g=130869229, q=33277488 ncpus=32) Nov 17 07:23:44 unraid kernel: Sending NMI from CPU 9 to CPUs 3: Nov 17 07:23:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:23:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:23:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:23:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:23:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:23:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:23:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:23:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:23:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:23:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:23:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:23:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:23:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:23:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:23:44 unraid kernel: Call Trace: Nov 17 07:23:44 unraid kernel: <NMI> Nov 17 07:23:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:23:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:23:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:23:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:23:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:23:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:23:44 unraid kernel: </NMI> Nov 17 07:23:44 unraid kernel: <TASK> Nov 17 07:23:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:23:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:23:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:23:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:23:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:23:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:23:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:23:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:23:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:23:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:23:44 unraid kernel: </TASK> Nov 17 07:26:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:26:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4445679 Nov 17 07:26:44 unraid kernel: (detected by 8, t=10320453 jiffies, g=130869229, q=33295490 ncpus=32) Nov 17 07:26:44 unraid kernel: Sending NMI from CPU 8 to CPUs 3: Nov 17 07:26:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:26:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:26:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:26:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:26:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:26:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:26:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:26:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:26:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:26:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:26:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:26:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:26:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:26:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:26:44 unraid kernel: Call Trace: Nov 17 07:26:44 unraid kernel: <NMI> Nov 17 07:26:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:26:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:26:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:26:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:26:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:26:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:26:44 unraid kernel: </NMI> Nov 17 07:26:44 unraid kernel: <TASK> Nov 17 07:26:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:26:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:26:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:26:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:26:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:26:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:26:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:26:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:26:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:26:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:26:44 unraid kernel: </TASK> Nov 17 07:29:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:29:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4532127 Nov 17 07:29:44 unraid kernel: (detected by 11, t=10500461 jiffies, g=130869229, q=33328169 ncpus=32) Nov 17 07:29:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 07:29:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:29:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:29:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:29:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:29:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:29:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:29:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:29:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:29:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:29:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:29:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:29:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:29:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:29:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:29:44 unraid kernel: Call Trace: Nov 17 07:29:44 unraid kernel: <NMI> Nov 17 07:29:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:29:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:29:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:29:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:29:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:29:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:29:44 unraid kernel: </NMI> Nov 17 07:29:44 unraid kernel: <TASK> Nov 17 07:29:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:29:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:29:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:29:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:29:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:29:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:29:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:29:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:29:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:29:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:29:44 unraid kernel: </TASK> Nov 17 07:32:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:32:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4613206 Nov 17 07:32:44 unraid kernel: (detected by 12, t=10680469 jiffies, g=130869229, q=33361049 ncpus=32) Nov 17 07:32:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 07:32:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:32:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:32:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:32:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:32:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:32:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:32:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:32:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:32:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:32:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:32:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:32:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:32:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:32:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:32:44 unraid kernel: Call Trace: Nov 17 07:32:44 unraid kernel: <NMI> Nov 17 07:32:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:32:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:32:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:32:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:32:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:32:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:32:44 unraid kernel: </NMI> Nov 17 07:32:44 unraid kernel: <TASK> Nov 17 07:32:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:32:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:32:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:32:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:32:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:32:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:32:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:32:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:32:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:32:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:32:44 unraid kernel: </TASK> Nov 17 07:35:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:35:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4684839 Nov 17 07:35:44 unraid kernel: (detected by 19, t=10860477 jiffies, g=130869229, q=33381795 ncpus=32) Nov 17 07:35:44 unraid kernel: Sending NMI from CPU 19 to CPUs 3: Nov 17 07:35:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:35:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:35:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:35:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:35:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 07:35:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:35:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:35:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:35:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:35:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:35:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:35:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:35:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:35:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:35:44 unraid kernel: Call Trace: Nov 17 07:35:44 unraid kernel: <NMI> Nov 17 07:35:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:35:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:35:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:35:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:35:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:35:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:35:44 unraid kernel: </NMI> Nov 17 07:35:44 unraid kernel: <TASK> Nov 17 07:35:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:35:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:35:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:35:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:35:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:35:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:35:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:35:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:35:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:35:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:35:44 unraid kernel: </TASK> Nov 17 07:38:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:38:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4757089 Nov 17 07:38:44 unraid kernel: (detected by 4, t=11040485 jiffies, g=130869229, q=33414981 ncpus=32) Nov 17 07:38:44 unraid kernel: Sending NMI from CPU 4 to CPUs 3: Nov 17 07:38:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:38:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:38:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:38:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:38:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 07:38:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:38:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:38:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:38:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:38:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:38:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:38:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:38:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:38:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:38:44 unraid kernel: Call Trace: Nov 17 07:38:44 unraid kernel: <NMI> Nov 17 07:38:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:38:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:38:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:38:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:38:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:38:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:38:44 unraid kernel: </NMI> Nov 17 07:38:44 unraid kernel: <TASK> Nov 17 07:38:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:38:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:38:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:38:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:38:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:38:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:38:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:38:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:38:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:38:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:38:44 unraid kernel: </TASK> Nov 17 07:41:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:41:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4829603 Nov 17 07:41:44 unraid kernel: (detected by 13, t=11220493 jiffies, g=130869229, q=33435425 ncpus=32) Nov 17 07:41:44 unraid kernel: Sending NMI from CPU 13 to CPUs 3: Nov 17 07:41:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:41:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:41:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:41:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:41:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 07:41:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:41:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:41:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:41:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:41:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:41:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:41:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:41:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:41:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:41:44 unraid kernel: Call Trace: Nov 17 07:41:44 unraid kernel: <NMI> Nov 17 07:41:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:41:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:41:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:41:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:41:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:41:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 07:41:44 unraid kernel: </NMI> Nov 17 07:41:44 unraid kernel: <TASK> Nov 17 07:41:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:41:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:41:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:41:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:41:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:41:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:41:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:41:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:41:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:41:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:41:44 unraid kernel: </TASK> Nov 17 07:44:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:44:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4902897 Nov 17 07:44:44 unraid kernel: (detected by 0, t=11400501 jiffies, g=130869229, q=33468673 ncpus=32) Nov 17 07:44:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 07:44:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:44:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:44:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:44:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:44:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 07:44:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:44:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:44:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:44:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:44:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:44:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:44:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:44:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:44:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:44:44 unraid kernel: Call Trace: Nov 17 07:44:44 unraid kernel: <NMI> Nov 17 07:44:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:44:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:44:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:44:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:44:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:44:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:44:44 unraid kernel: </NMI> Nov 17 07:44:44 unraid kernel: <TASK> Nov 17 07:44:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:44:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:44:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:44:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:44:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:44:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:44:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:44:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:44:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:44:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:44:44 unraid kernel: </TASK> Nov 17 07:47:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:47:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=4991367 Nov 17 07:47:44 unraid kernel: (detected by 11, t=11580509 jiffies, g=130869229, q=33510906 ncpus=32) Nov 17 07:47:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 07:47:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:47:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:47:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:47:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:47:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 07:47:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:47:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:47:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:47:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:47:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:47:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:47:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:47:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:47:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:47:44 unraid kernel: Call Trace: Nov 17 07:47:44 unraid kernel: <NMI> Nov 17 07:47:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:47:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:47:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:47:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:47:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:47:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:47:44 unraid kernel: </NMI> Nov 17 07:47:44 unraid kernel: <TASK> Nov 17 07:47:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:47:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:47:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:47:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:47:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:47:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:47:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:47:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:47:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:47:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:47:44 unraid kernel: </TASK> Nov 17 07:50:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:50:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5066272 Nov 17 07:50:44 unraid kernel: (detected by 28, t=11760517 jiffies, g=130869229, q=33527493 ncpus=32) Nov 17 07:50:44 unraid kernel: Sending NMI from CPU 28 to CPUs 3: Nov 17 07:50:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:50:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:50:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:50:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:50:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 07:50:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:50:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:50:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:50:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:50:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:50:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:50:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:50:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:50:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:50:44 unraid kernel: Call Trace: Nov 17 07:50:44 unraid kernel: <NMI> Nov 17 07:50:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:50:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:50:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:50:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:50:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:50:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 07:50:44 unraid kernel: </NMI> Nov 17 07:50:44 unraid kernel: <TASK> Nov 17 07:50:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:50:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:50:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:50:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:50:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:50:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:50:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:50:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:50:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:50:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:50:44 unraid kernel: </TASK> Nov 17 07:53:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:53:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5137214 Nov 17 07:53:44 unraid kernel: (detected by 14, t=11940525 jiffies, g=130869229, q=33559293 ncpus=32) Nov 17 07:53:44 unraid kernel: Sending NMI from CPU 14 to CPUs 3: Nov 17 07:53:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:53:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:53:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:53:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:53:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 07:53:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:53:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:53:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:53:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:53:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:53:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:53:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:53:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:53:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:53:44 unraid kernel: Call Trace: Nov 17 07:53:44 unraid kernel: <NMI> Nov 17 07:53:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:53:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:53:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:53:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:53:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:53:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:53:44 unraid kernel: </NMI> Nov 17 07:53:44 unraid kernel: <TASK> Nov 17 07:53:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:53:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:53:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:53:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:53:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:53:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:53:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:53:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:53:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:53:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:53:44 unraid kernel: </TASK> Nov 17 07:56:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:56:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5208980 Nov 17 07:56:44 unraid kernel: (detected by 21, t=12120533 jiffies, g=130869229, q=33581829 ncpus=32) Nov 17 07:56:44 unraid kernel: Sending NMI from CPU 21 to CPUs 3: Nov 17 07:56:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:56:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:56:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:56:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:56:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 07:56:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:56:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:56:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:56:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:56:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:56:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:56:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:56:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:56:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:56:44 unraid kernel: Call Trace: Nov 17 07:56:44 unraid kernel: <NMI> Nov 17 07:56:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:56:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:56:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:56:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:56:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:56:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 07:56:44 unraid kernel: </NMI> Nov 17 07:56:44 unraid kernel: <TASK> Nov 17 07:56:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:56:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:56:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:56:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:56:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:56:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:56:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:56:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:56:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:56:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:56:44 unraid kernel: </TASK> Nov 17 07:59:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 07:59:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5278978 Nov 17 07:59:44 unraid kernel: (detected by 8, t=12300541 jiffies, g=130869229, q=33611912 ncpus=32) Nov 17 07:59:44 unraid kernel: Sending NMI from CPU 8 to CPUs 3: Nov 17 07:59:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 07:59:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 07:59:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 07:59:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:59:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 07:59:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 07:59:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 07:59:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 07:59:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 07:59:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 07:59:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 07:59:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 07:59:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 07:59:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 07:59:44 unraid kernel: Call Trace: Nov 17 07:59:44 unraid kernel: <NMI> Nov 17 07:59:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 07:59:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 07:59:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 07:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:59:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 07:59:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 07:59:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 07:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 07:59:44 unraid kernel: </NMI> Nov 17 07:59:44 unraid kernel: <TASK> Nov 17 07:59:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 07:59:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 07:59:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 07:59:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 07:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:59:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 07:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 07:59:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 07:59:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 07:59:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 07:59:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 07:59:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 07:59:44 unraid kernel: </TASK> Nov 17 08:02:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:02:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5358996 Nov 17 08:02:44 unraid kernel: (detected by 22, t=12480549 jiffies, g=130869229, q=33646086 ncpus=32) Nov 17 08:02:44 unraid kernel: Sending NMI from CPU 22 to CPUs 3: Nov 17 08:02:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:02:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:02:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:02:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:02:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 08:02:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:02:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:02:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:02:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:02:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:02:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:02:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:02:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:02:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:02:44 unraid kernel: Call Trace: Nov 17 08:02:44 unraid kernel: <NMI> Nov 17 08:02:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:02:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:02:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:02:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:02:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:02:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:02:44 unraid kernel: </NMI> Nov 17 08:02:44 unraid kernel: <TASK> Nov 17 08:02:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:02:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:02:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:02:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:02:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:02:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:02:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:02:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:02:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:02:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:02:44 unraid kernel: </TASK> Nov 17 08:05:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:05:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5444172 Nov 17 08:05:44 unraid kernel: (detected by 19, t=12660557 jiffies, g=130869229, q=33665551 ncpus=32) Nov 17 08:05:44 unraid kernel: Sending NMI from CPU 19 to CPUs 3: Nov 17 08:05:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:05:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:05:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:05:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:05:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:05:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:05:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:05:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:05:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:05:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:05:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:05:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:05:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:05:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:05:44 unraid kernel: Call Trace: Nov 17 08:05:44 unraid kernel: <NMI> Nov 17 08:05:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:05:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:05:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:05:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:05:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:05:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:05:44 unraid kernel: </NMI> Nov 17 08:05:44 unraid kernel: <TASK> Nov 17 08:05:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:05:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:05:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:05:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:05:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:05:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:05:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:05:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:05:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:05:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:05:44 unraid kernel: </TASK> Nov 17 08:05:44 unraid kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 3.286 msecs Nov 17 08:08:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:08:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5516431 Nov 17 08:08:44 unraid kernel: (detected by 20, t=12840566 jiffies, g=130869229, q=33699642 ncpus=32) Nov 17 08:08:44 unraid kernel: Sending NMI from CPU 20 to CPUs 3: Nov 17 08:08:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:08:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:08:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:08:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:08:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:08:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:08:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:08:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:08:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:08:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:08:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:08:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:08:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:08:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:08:44 unraid kernel: Call Trace: Nov 17 08:08:44 unraid kernel: <NMI> Nov 17 08:08:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:08:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:08:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:08:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:08:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:08:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:08:44 unraid kernel: </NMI> Nov 17 08:08:44 unraid kernel: <TASK> Nov 17 08:08:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:08:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:08:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:08:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:08:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:08:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:08:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:08:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:08:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:08:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:08:44 unraid kernel: </TASK> Nov 17 08:11:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:11:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5588434 Nov 17 08:11:44 unraid kernel: (detected by 5, t=13020574 jiffies, g=130869229, q=33717407 ncpus=32) Nov 17 08:11:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 08:11:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:11:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:11:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:11:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:11:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 08:11:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:11:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:11:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:11:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:11:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:11:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:11:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:11:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:11:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:11:44 unraid kernel: Call Trace: Nov 17 08:11:44 unraid kernel: <NMI> Nov 17 08:11:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:11:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:11:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:11:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:11:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:11:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:11:44 unraid kernel: </NMI> Nov 17 08:11:44 unraid kernel: <TASK> Nov 17 08:11:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:11:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:11:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:11:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:11:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:11:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:11:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:11:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:11:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:11:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:11:44 unraid kernel: </TASK> Nov 17 08:14:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:14:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5660164 Nov 17 08:14:44 unraid kernel: (detected by 18, t=13200579 jiffies, g=130869229, q=33750315 ncpus=32) Nov 17 08:14:44 unraid kernel: Sending NMI from CPU 18 to CPUs 3: Nov 17 08:14:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:14:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:14:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:14:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:14:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:14:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:14:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:14:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:14:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:14:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:14:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:14:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:14:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:14:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:14:44 unraid kernel: Call Trace: Nov 17 08:14:44 unraid kernel: <NMI> Nov 17 08:14:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:14:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:14:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:14:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:14:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:14:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:14:44 unraid kernel: </NMI> Nov 17 08:14:44 unraid kernel: <TASK> Nov 17 08:14:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:14:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:14:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:14:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:14:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:14:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:14:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:14:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:14:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:14:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:14:44 unraid kernel: </TASK> Nov 17 08:17:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:17:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5731183 Nov 17 08:17:44 unraid kernel: (detected by 30, t=13380587 jiffies, g=130869229, q=33783519 ncpus=32) Nov 17 08:17:44 unraid kernel: Sending NMI from CPU 30 to CPUs 3: Nov 17 08:17:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:17:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:17:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:17:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:17:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 08:17:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:17:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:17:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:17:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:17:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:17:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:17:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:17:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:17:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:17:44 unraid kernel: Call Trace: Nov 17 08:17:44 unraid kernel: <NMI> Nov 17 08:17:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:17:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:17:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:17:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:17:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:17:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:17:44 unraid kernel: </NMI> Nov 17 08:17:44 unraid kernel: <TASK> Nov 17 08:17:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:17:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:17:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:17:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:17:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:17:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:17:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:17:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:17:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:17:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:17:44 unraid kernel: </TASK> Nov 17 08:20:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:20:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5818689 Nov 17 08:20:44 unraid kernel: (detected by 22, t=13560595 jiffies, g=130869229, q=33803440 ncpus=32) Nov 17 08:20:44 unraid kernel: Sending NMI from CPU 22 to CPUs 3: Nov 17 08:20:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:20:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:20:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:20:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:20:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 08:20:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:20:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:20:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:20:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:20:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:20:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:20:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:20:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:20:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:20:44 unraid kernel: Call Trace: Nov 17 08:20:44 unraid kernel: <NMI> Nov 17 08:20:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:20:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:20:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:20:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:20:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:20:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:20:44 unraid kernel: </NMI> Nov 17 08:20:44 unraid kernel: <TASK> Nov 17 08:20:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:20:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:20:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:20:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:20:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:20:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:20:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:20:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:20:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:20:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:20:44 unraid kernel: </TASK> Nov 17 08:23:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:23:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5897637 Nov 17 08:23:44 unraid kernel: (detected by 16, t=13740603 jiffies, g=130869229, q=33838673 ncpus=32) Nov 17 08:23:44 unraid kernel: Sending NMI from CPU 16 to CPUs 3: Nov 17 08:23:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:23:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:23:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:23:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:23:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:23:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:23:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:23:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:23:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:23:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:23:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:23:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:23:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:23:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:23:44 unraid kernel: Call Trace: Nov 17 08:23:44 unraid kernel: <NMI> Nov 17 08:23:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:23:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:23:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:23:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:23:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:23:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:23:44 unraid kernel: </NMI> Nov 17 08:23:44 unraid kernel: <TASK> Nov 17 08:23:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:23:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:23:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:23:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:23:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:23:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:23:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:23:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:23:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:23:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:23:44 unraid kernel: </TASK> Nov 17 08:26:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:26:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=5970946 Nov 17 08:26:44 unraid kernel: (detected by 24, t=13920611 jiffies, g=130869229, q=33856045 ncpus=32) Nov 17 08:26:44 unraid kernel: Sending NMI from CPU 24 to CPUs 3: Nov 17 08:26:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:26:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:26:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:26:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:26:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 08:26:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:26:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:26:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:26:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:26:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:26:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:26:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:26:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:26:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:26:44 unraid kernel: Call Trace: Nov 17 08:26:44 unraid kernel: <NMI> Nov 17 08:26:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:26:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:26:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:26:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:26:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:26:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:26:44 unraid kernel: </NMI> Nov 17 08:26:44 unraid kernel: <TASK> Nov 17 08:26:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:26:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:26:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:26:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:26:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:26:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:26:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:26:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:26:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:26:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:26:44 unraid kernel: </TASK> Nov 17 08:29:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:29:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6044191 Nov 17 08:29:44 unraid kernel: (detected by 5, t=14100619 jiffies, g=130869229, q=33888133 ncpus=32) Nov 17 08:29:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 08:29:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:29:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:29:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:29:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:29:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 08:29:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:29:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:29:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:29:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:29:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:29:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:29:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:29:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:29:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:29:44 unraid kernel: Call Trace: Nov 17 08:29:44 unraid kernel: <NMI> Nov 17 08:29:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:29:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:29:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:29:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:29:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:29:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:29:44 unraid kernel: </NMI> Nov 17 08:29:44 unraid kernel: <TASK> Nov 17 08:29:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:29:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:29:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:29:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:29:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:29:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:29:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:29:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:29:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:29:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:29:44 unraid kernel: </TASK> Nov 17 08:32:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:32:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6117669 Nov 17 08:32:44 unraid kernel: (detected by 14, t=14280624 jiffies, g=130869229, q=33922740 ncpus=32) Nov 17 08:32:44 unraid kernel: Sending NMI from CPU 14 to CPUs 3: Nov 17 08:32:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:32:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:32:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:32:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:32:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 08:32:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:32:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:32:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:32:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:32:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:32:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:32:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:32:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:32:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:32:44 unraid kernel: Call Trace: Nov 17 08:32:44 unraid kernel: <NMI> Nov 17 08:32:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:32:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:32:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:32:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:32:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:32:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 08:32:44 unraid kernel: </NMI> Nov 17 08:32:44 unraid kernel: <TASK> Nov 17 08:32:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:32:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:32:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:32:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:32:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:32:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:32:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:32:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:32:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:32:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:32:44 unraid kernel: </TASK> Nov 17 08:35:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:35:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6196174 Nov 17 08:35:44 unraid kernel: (detected by 7, t=14460632 jiffies, g=130869229, q=33940166 ncpus=32) Nov 17 08:35:44 unraid kernel: Sending NMI from CPU 7 to CPUs 3: Nov 17 08:35:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:35:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:35:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:35:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:35:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:35:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:35:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:35:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:35:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:35:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:35:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:35:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:35:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:35:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:35:44 unraid kernel: Call Trace: Nov 17 08:35:44 unraid kernel: <NMI> Nov 17 08:35:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:35:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:35:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:35:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:35:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:35:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:35:44 unraid kernel: </NMI> Nov 17 08:35:44 unraid kernel: <TASK> Nov 17 08:35:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:35:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:35:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:35:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:35:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:35:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:35:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:35:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:35:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:35:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:35:44 unraid kernel: </TASK> Nov 17 08:38:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:38:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6285539 Nov 17 08:38:44 unraid kernel: (detected by 19, t=14640640 jiffies, g=130869229, q=33980930 ncpus=32) Nov 17 08:38:44 unraid kernel: Sending NMI from CPU 19 to CPUs 3: Nov 17 08:38:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:38:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:38:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:38:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:38:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:38:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:38:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:38:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:38:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:38:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:38:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:38:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:38:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:38:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:38:44 unraid kernel: Call Trace: Nov 17 08:38:44 unraid kernel: <NMI> Nov 17 08:38:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:38:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:38:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:38:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:38:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:38:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:38:44 unraid kernel: </NMI> Nov 17 08:38:44 unraid kernel: <TASK> Nov 17 08:38:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:38:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:38:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:38:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:38:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:38:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:38:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:38:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:38:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:38:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:38:44 unraid kernel: </TASK> Nov 17 08:41:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:41:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6356653 Nov 17 08:41:44 unraid kernel: (detected by 0, t=14820649 jiffies, g=130869229, q=34000784 ncpus=32) Nov 17 08:41:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 08:41:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:41:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:41:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:41:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:41:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:41:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:41:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:41:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:41:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:41:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:41:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:41:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:41:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:41:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:41:44 unraid kernel: Call Trace: Nov 17 08:41:44 unraid kernel: <NMI> Nov 17 08:41:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:41:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:41:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:41:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:41:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:41:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:41:44 unraid kernel: </NMI> Nov 17 08:41:44 unraid kernel: <TASK> Nov 17 08:41:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:41:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:41:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:41:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:41:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:41:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:41:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:41:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:41:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:41:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:41:44 unraid kernel: </TASK> Nov 17 08:44:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:44:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6428293 Nov 17 08:44:44 unraid kernel: (detected by 5, t=15000657 jiffies, g=130869229, q=34034076 ncpus=32) Nov 17 08:44:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 08:44:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:44:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:44:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:44:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:44:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 08:44:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:44:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:44:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:44:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:44:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:44:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:44:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:44:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:44:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:44:44 unraid kernel: Call Trace: Nov 17 08:44:44 unraid kernel: <NMI> Nov 17 08:44:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:44:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:44:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:44:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:44:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:44:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:44:44 unraid kernel: </NMI> Nov 17 08:44:44 unraid kernel: <TASK> Nov 17 08:44:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:44:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:44:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:44:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:44:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:44:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:44:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:44:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:44:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:44:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:44:44 unraid kernel: </TASK> Nov 17 08:47:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:47:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6502274 Nov 17 08:47:44 unraid kernel: (detected by 12, t=15180662 jiffies, g=130869229, q=34069052 ncpus=32) Nov 17 08:47:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 08:47:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:47:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:47:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:47:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:47:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:47:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:47:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:47:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:47:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:47:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:47:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:47:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:47:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:47:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:47:44 unraid kernel: Call Trace: Nov 17 08:47:44 unraid kernel: <NMI> Nov 17 08:47:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:47:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:47:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:47:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:47:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:47:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:47:44 unraid kernel: </NMI> Nov 17 08:47:44 unraid kernel: <TASK> Nov 17 08:47:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:47:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:47:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:47:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:47:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:47:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:47:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:47:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:47:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:47:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:47:44 unraid kernel: </TASK> Nov 17 08:50:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:50:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6573006 Nov 17 08:50:44 unraid kernel: (detected by 11, t=15360670 jiffies, g=130869229, q=34088270 ncpus=32) Nov 17 08:50:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 08:50:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:50:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:50:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:50:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:50:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:50:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:50:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:50:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:50:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:50:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:50:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:50:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:50:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:50:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:50:44 unraid kernel: Call Trace: Nov 17 08:50:44 unraid kernel: <NMI> Nov 17 08:50:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:50:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:50:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:50:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:50:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:50:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:50:44 unraid kernel: </NMI> Nov 17 08:50:44 unraid kernel: <TASK> Nov 17 08:50:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:50:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:50:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:50:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:50:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:50:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:50:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:50:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:50:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:50:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:50:44 unraid kernel: </TASK> Nov 17 08:53:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:53:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6656804 Nov 17 08:53:44 unraid kernel: (detected by 0, t=15540678 jiffies, g=130869229, q=34119792 ncpus=32) Nov 17 08:53:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 08:53:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:53:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:53:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:53:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:53:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 08:53:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:53:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:53:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:53:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:53:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:53:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:53:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:53:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:53:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:53:44 unraid kernel: Call Trace: Nov 17 08:53:44 unraid kernel: <NMI> Nov 17 08:53:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:53:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:53:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:53:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:53:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:53:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 08:53:44 unraid kernel: </NMI> Nov 17 08:53:44 unraid kernel: <TASK> Nov 17 08:53:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:53:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:53:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:53:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:53:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:53:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:53:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:53:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:53:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:53:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:53:44 unraid kernel: </TASK> Nov 17 08:56:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:56:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6737273 Nov 17 08:56:44 unraid kernel: (detected by 12, t=15720686 jiffies, g=130869229, q=34140678 ncpus=32) Nov 17 08:56:44 unraid kernel: Sending NMI from CPU 12 to CPUs 3: Nov 17 08:56:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:56:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:56:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:56:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:56:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 08:56:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:56:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:56:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:56:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:56:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:56:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:56:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:56:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:56:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:56:44 unraid kernel: Call Trace: Nov 17 08:56:44 unraid kernel: <NMI> Nov 17 08:56:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:56:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:56:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:56:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:56:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:56:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 08:56:44 unraid kernel: </NMI> Nov 17 08:56:44 unraid kernel: <TASK> Nov 17 08:56:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:56:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:56:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:56:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:56:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:56:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:56:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:56:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:56:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:56:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:56:44 unraid kernel: </TASK> Nov 17 08:59:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 08:59:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6808807 Nov 17 08:59:44 unraid kernel: (detected by 8, t=15900694 jiffies, g=130869229, q=34173368 ncpus=32) Nov 17 08:59:44 unraid kernel: Sending NMI from CPU 8 to CPUs 3: Nov 17 08:59:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 08:59:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 08:59:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 08:59:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:59:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 08:59:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 08:59:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 08:59:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 08:59:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 08:59:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 08:59:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 08:59:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 08:59:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 08:59:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 08:59:44 unraid kernel: Call Trace: Nov 17 08:59:44 unraid kernel: <NMI> Nov 17 08:59:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 08:59:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 08:59:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 08:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:59:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 08:59:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 08:59:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 08:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 08:59:44 unraid kernel: </NMI> Nov 17 08:59:44 unraid kernel: <TASK> Nov 17 08:59:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 08:59:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 08:59:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 08:59:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 08:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:59:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 08:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 08:59:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 08:59:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 08:59:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 08:59:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 08:59:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 08:59:44 unraid kernel: </TASK> Nov 17 09:02:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:02:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6883760 Nov 17 09:02:44 unraid kernel: (detected by 16, t=16080702 jiffies, g=130869229, q=34194248 ncpus=32) Nov 17 09:02:44 unraid kernel: Sending NMI from CPU 16 to CPUs 3: Nov 17 09:02:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:02:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:02:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:02:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:02:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:02:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:02:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:02:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:02:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:02:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:02:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:02:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:02:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:02:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:02:44 unraid kernel: Call Trace: Nov 17 09:02:44 unraid kernel: <NMI> Nov 17 09:02:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:02:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:02:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:02:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:02:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:02:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:02:44 unraid kernel: </NMI> Nov 17 09:02:44 unraid kernel: <TASK> Nov 17 09:02:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:02:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:02:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:02:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:02:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:02:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:02:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:02:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:02:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:02:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:02:44 unraid kernel: </TASK> Nov 17 09:05:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:05:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=6955180 Nov 17 09:05:44 unraid kernel: (detected by 5, t=16260710 jiffies, g=130869229, q=34227649 ncpus=32) Nov 17 09:05:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 09:05:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:05:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:05:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:05:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:05:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:05:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:05:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:05:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:05:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:05:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:05:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:05:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:05:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:05:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:05:44 unraid kernel: Call Trace: Nov 17 09:05:44 unraid kernel: <NMI> Nov 17 09:05:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:05:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:05:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:05:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:05:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:05:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:05:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:05:44 unraid kernel: </NMI> Nov 17 09:05:44 unraid kernel: <TASK> Nov 17 09:05:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:05:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:05:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:05:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:05:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:05:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:05:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:05:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:05:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:05:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:05:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:05:44 unraid kernel: </TASK> Nov 17 09:08:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:08:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7026654 Nov 17 09:08:44 unraid kernel: (detected by 23, t=16440715 jiffies, g=130869229, q=34247348 ncpus=32) Nov 17 09:08:44 unraid kernel: Sending NMI from CPU 23 to CPUs 3: Nov 17 09:08:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:08:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:08:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:08:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:08:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:08:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:08:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:08:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:08:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:08:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:08:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:08:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:08:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:08:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:08:44 unraid kernel: Call Trace: Nov 17 09:08:44 unraid kernel: <NMI> Nov 17 09:08:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:08:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:08:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:08:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:08:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:08:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:08:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:08:44 unraid kernel: </NMI> Nov 17 09:08:44 unraid kernel: <TASK> Nov 17 09:08:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:08:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:08:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:08:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:08:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:08:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:08:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:08:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:08:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:08:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:08:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:08:44 unraid kernel: </TASK> Nov 17 09:11:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:11:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7115717 Nov 17 09:11:44 unraid kernel: (detected by 0, t=16620723 jiffies, g=130869229, q=34279458 ncpus=32) Nov 17 09:11:44 unraid kernel: Sending NMI from CPU 0 to CPUs 3: Nov 17 09:11:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:11:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:11:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:11:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:11:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:11:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:11:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:11:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:11:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:11:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:11:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:11:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:11:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:11:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:11:44 unraid kernel: Call Trace: Nov 17 09:11:44 unraid kernel: <NMI> Nov 17 09:11:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:11:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:11:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:11:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:11:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:11:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:11:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:11:44 unraid kernel: </NMI> Nov 17 09:11:44 unraid kernel: <TASK> Nov 17 09:11:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:11:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:11:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:11:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:11:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:11:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:11:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:11:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:11:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:11:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:11:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:11:44 unraid kernel: </TASK> Nov 17 09:14:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:14:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7191965 Nov 17 09:14:44 unraid kernel: (detected by 11, t=16800731 jiffies, g=130869229, q=34312316 ncpus=32) Nov 17 09:14:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 09:14:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:14:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:14:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:14:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:14:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:14:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:14:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:14:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:14:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:14:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:14:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:14:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:14:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:14:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:14:44 unraid kernel: Call Trace: Nov 17 09:14:44 unraid kernel: <NMI> Nov 17 09:14:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:14:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:14:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:14:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:14:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:14:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:14:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:14:44 unraid kernel: </NMI> Nov 17 09:14:44 unraid kernel: <TASK> Nov 17 09:14:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:14:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:14:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:14:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:14:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:14:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:14:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:14:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:14:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:14:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:14:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:14:44 unraid kernel: </TASK> Nov 17 09:17:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:17:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7262037 Nov 17 09:17:44 unraid kernel: (detected by 2, t=16980739 jiffies, g=130869229, q=34329748 ncpus=32) Nov 17 09:17:44 unraid kernel: Sending NMI from CPU 2 to CPUs 3: Nov 17 09:17:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:17:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:17:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:17:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:17:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 09:17:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:17:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:17:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:17:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:17:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:17:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:17:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:17:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:17:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:17:44 unraid kernel: Call Trace: Nov 17 09:17:44 unraid kernel: <NMI> Nov 17 09:17:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:17:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:17:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:17:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:17:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:17:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:17:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:17:44 unraid kernel: </NMI> Nov 17 09:17:44 unraid kernel: <TASK> Nov 17 09:17:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:17:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:17:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:17:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:17:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:17:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:17:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:17:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:17:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:17:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:17:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:17:44 unraid kernel: </TASK> Nov 17 09:20:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:20:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7332482 Nov 17 09:20:44 unraid kernel: (detected by 25, t=17160747 jiffies, g=130869229, q=34365716 ncpus=32) Nov 17 09:20:44 unraid kernel: Sending NMI from CPU 25 to CPUs 3: Nov 17 09:20:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:20:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:20:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:20:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:20:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:20:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:20:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:20:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:20:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:20:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:20:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:20:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:20:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:20:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:20:44 unraid kernel: Call Trace: Nov 17 09:20:44 unraid kernel: <NMI> Nov 17 09:20:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:20:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:20:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:20:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:20:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:20:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:20:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:20:44 unraid kernel: </NMI> Nov 17 09:20:44 unraid kernel: <TASK> Nov 17 09:20:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:20:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:20:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:20:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:20:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:20:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:20:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:20:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:20:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:20:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:20:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:20:44 unraid kernel: </TASK> Nov 17 09:23:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:23:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7401956 Nov 17 09:23:44 unraid kernel: (detected by 20, t=17340755 jiffies, g=130869229, q=34385193 ncpus=32) Nov 17 09:23:44 unraid kernel: Sending NMI from CPU 20 to CPUs 3: Nov 17 09:23:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:23:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:23:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:23:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:23:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:23:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:23:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:23:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:23:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:23:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:23:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:23:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:23:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:23:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:23:44 unraid kernel: Call Trace: Nov 17 09:23:44 unraid kernel: <NMI> Nov 17 09:23:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:23:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:23:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:23:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:23:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:23:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:23:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:23:44 unraid kernel: </NMI> Nov 17 09:23:44 unraid kernel: <TASK> Nov 17 09:23:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:23:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:23:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:23:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:23:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:23:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:23:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:23:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:23:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:23:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:23:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:23:44 unraid kernel: </TASK> Nov 17 09:26:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:26:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7481317 Nov 17 09:26:44 unraid kernel: (detected by 16, t=17520763 jiffies, g=130869229, q=34417689 ncpus=32) Nov 17 09:26:44 unraid kernel: Sending NMI from CPU 16 to CPUs 3: Nov 17 09:26:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:26:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:26:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:26:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:26:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:26:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:26:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:26:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:26:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:26:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:26:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:26:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:26:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:26:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:26:44 unraid kernel: Call Trace: Nov 17 09:26:44 unraid kernel: <NMI> Nov 17 09:26:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:26:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:26:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:26:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:26:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:26:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:26:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:26:44 unraid kernel: </NMI> Nov 17 09:26:44 unraid kernel: <TASK> Nov 17 09:26:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:26:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:26:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:26:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:26:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:26:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:26:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:26:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:26:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:26:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:26:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:26:44 unraid kernel: </TASK> Nov 17 09:29:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:29:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7569464 Nov 17 09:29:44 unraid kernel: (detected by 4, t=17700771 jiffies, g=130869229, q=34449718 ncpus=32) Nov 17 09:29:44 unraid kernel: Sending NMI from CPU 4 to CPUs 3: Nov 17 09:29:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:29:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:29:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:29:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:29:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 09:29:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:29:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:29:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:29:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:29:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:29:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:29:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:29:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:29:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:29:44 unraid kernel: Call Trace: Nov 17 09:29:44 unraid kernel: <NMI> Nov 17 09:29:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:29:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:29:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:29:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:29:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:29:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:29:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:29:44 unraid kernel: </NMI> Nov 17 09:29:44 unraid kernel: <TASK> Nov 17 09:29:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:29:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:29:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:29:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:29:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:29:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:29:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:29:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:29:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:29:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:29:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:29:44 unraid kernel: </TASK> Nov 17 09:32:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:32:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7638610 Nov 17 09:32:44 unraid kernel: (detected by 8, t=17880779 jiffies, g=130869229, q=34469420 ncpus=32) Nov 17 09:32:44 unraid kernel: Sending NMI from CPU 8 to CPUs 3: Nov 17 09:32:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:32:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:32:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:32:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:32:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:32:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:32:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:32:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:32:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:32:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:32:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:32:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:32:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:32:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:32:44 unraid kernel: Call Trace: Nov 17 09:32:44 unraid kernel: <NMI> Nov 17 09:32:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:32:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:32:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:32:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:32:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:32:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:32:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:32:44 unraid kernel: </NMI> Nov 17 09:32:44 unraid kernel: <TASK> Nov 17 09:32:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:32:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:32:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:32:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:32:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:32:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:32:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:32:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:32:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:32:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:32:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:32:44 unraid kernel: </TASK> Nov 17 09:35:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:35:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7708188 Nov 17 09:35:44 unraid kernel: (detected by 6, t=18060787 jiffies, g=130869229, q=34501143 ncpus=32) Nov 17 09:35:44 unraid kernel: Sending NMI from CPU 6 to CPUs 3: Nov 17 09:35:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:35:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:35:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:35:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:35:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:35:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:35:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:35:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:35:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:35:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:35:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:35:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:35:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:35:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:35:44 unraid kernel: Call Trace: Nov 17 09:35:44 unraid kernel: <NMI> Nov 17 09:35:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:35:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:35:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:35:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:35:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:35:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:35:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:35:44 unraid kernel: </NMI> Nov 17 09:35:44 unraid kernel: <TASK> Nov 17 09:35:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:35:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:35:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:35:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:35:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:35:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:35:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:35:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:35:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:35:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:35:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:35:44 unraid kernel: </TASK> Nov 17 09:38:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:38:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7781159 Nov 17 09:38:44 unraid kernel: (detected by 7, t=18240795 jiffies, g=130869229, q=34522514 ncpus=32) Nov 17 09:38:44 unraid kernel: Sending NMI from CPU 7 to CPUs 3: Nov 17 09:38:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:38:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:38:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:38:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:38:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 09:38:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:38:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:38:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:38:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:38:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:38:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:38:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:38:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:38:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:38:44 unraid kernel: Call Trace: Nov 17 09:38:44 unraid kernel: <NMI> Nov 17 09:38:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:38:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:38:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:38:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:38:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:38:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:38:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:38:44 unraid kernel: </NMI> Nov 17 09:38:44 unraid kernel: <TASK> Nov 17 09:38:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:38:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:38:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:38:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:38:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:38:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:38:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:38:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:38:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:38:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:38:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:38:44 unraid kernel: </TASK> Nov 17 09:41:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:41:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7852510 Nov 17 09:41:44 unraid kernel: (detected by 15, t=18420803 jiffies, g=130869229, q=34554513 ncpus=32) Nov 17 09:41:44 unraid kernel: Sending NMI from CPU 15 to CPUs 3: Nov 17 09:41:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:41:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:41:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:41:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:41:44 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 17 09:41:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:41:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:41:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:41:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:41:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:41:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:41:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:41:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:41:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:41:44 unraid kernel: Call Trace: Nov 17 09:41:44 unraid kernel: <NMI> Nov 17 09:41:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:41:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:41:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:41:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:41:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:41:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:41:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 17 09:41:44 unraid kernel: </NMI> Nov 17 09:41:44 unraid kernel: <TASK> Nov 17 09:41:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:41:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:41:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:41:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:41:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:41:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:41:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:41:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:41:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:41:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:41:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:41:44 unraid kernel: </TASK> Nov 17 09:44:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:44:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=7937875 Nov 17 09:44:44 unraid kernel: (detected by 11, t=18600811 jiffies, g=130869229, q=34586720 ncpus=32) Nov 17 09:44:44 unraid kernel: Sending NMI from CPU 11 to CPUs 3: Nov 17 09:44:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:44:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:44:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:44:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:44:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:44:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:44:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:44:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:44:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:44:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:44:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:44:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:44:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:44:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:44:44 unraid kernel: Call Trace: Nov 17 09:44:44 unraid kernel: <NMI> Nov 17 09:44:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:44:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:44:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:44:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:44:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:44:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:44:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:44:44 unraid kernel: </NMI> Nov 17 09:44:44 unraid kernel: <TASK> Nov 17 09:44:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:44:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:44:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:44:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:44:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:44:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:44:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:44:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:44:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:44:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:44:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:44:44 unraid kernel: </TASK> Nov 17 09:47:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:47:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=8018116 Nov 17 09:47:44 unraid kernel: (detected by 30, t=18780819 jiffies, g=130869229, q=34609434 ncpus=32) Nov 17 09:47:44 unraid kernel: Sending NMI from CPU 30 to CPUs 3: Nov 17 09:47:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:47:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:47:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:47:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 09:47:44 unraid kernel: Code: 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 <f3> 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 Nov 17 09:47:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:47:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:47:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:47:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:47:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:47:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:47:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:47:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:47:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:47:44 unraid kernel: Call Trace: Nov 17 09:47:44 unraid kernel: <NMI> Nov 17 09:47:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:47:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:47:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 09:47:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:47:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:47:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 09:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 09:47:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x84/0x1cf Nov 17 09:47:44 unraid kernel: </NMI> Nov 17 09:47:44 unraid kernel: <TASK> Nov 17 09:47:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:47:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:47:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:47:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:47:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:47:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:47:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:47:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:47:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:47:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:47:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:47:44 unraid kernel: </TASK> Nov 17 09:50:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:50:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=8086573 Nov 17 09:50:44 unraid kernel: (detected by 15, t=18960827 jiffies, g=130869229, q=34639644 ncpus=32) Nov 17 09:50:44 unraid kernel: Sending NMI from CPU 15 to CPUs 3: Nov 17 09:50:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:50:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:50:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:50:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:50:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:50:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:50:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:50:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:50:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:50:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:50:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:50:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:50:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:50:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:50:44 unraid kernel: Call Trace: Nov 17 09:50:44 unraid kernel: <NMI> Nov 17 09:50:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:50:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:50:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:50:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:50:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:50:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:50:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:50:44 unraid kernel: </NMI> Nov 17 09:50:44 unraid kernel: <TASK> Nov 17 09:50:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:50:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:50:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:50:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:50:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:50:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:50:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:50:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:50:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:50:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:50:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:50:44 unraid kernel: </TASK> Nov 17 09:53:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:53:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=8157045 Nov 17 09:53:44 unraid kernel: (detected by 5, t=19140835 jiffies, g=130869229, q=34659757 ncpus=32) Nov 17 09:53:44 unraid kernel: Sending NMI from CPU 5 to CPUs 3: Nov 17 09:53:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:53:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:53:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:53:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:53:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:53:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:53:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:53:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:53:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:53:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:53:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:53:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:53:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:53:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:53:44 unraid kernel: Call Trace: Nov 17 09:53:44 unraid kernel: <NMI> Nov 17 09:53:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:53:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:53:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:53:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:53:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:53:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:53:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:53:44 unraid kernel: </NMI> Nov 17 09:53:44 unraid kernel: <TASK> Nov 17 09:53:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:53:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:53:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:53:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:53:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:53:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:53:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:53:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:53:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:53:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:53:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:53:44 unraid kernel: </TASK> Nov 17 09:56:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:56:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=8225396 Nov 17 09:56:44 unraid kernel: (detected by 17, t=19320840 jiffies, g=130869229, q=34691514 ncpus=32) Nov 17 09:56:44 unraid kernel: Sending NMI from CPU 17 to CPUs 3: Nov 17 09:56:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:56:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:56:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:56:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:56:44 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 17 09:56:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:56:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:56:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:56:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:56:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:56:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:56:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:56:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:56:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:56:44 unraid kernel: Call Trace: Nov 17 09:56:44 unraid kernel: <NMI> Nov 17 09:56:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:56:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:56:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:56:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:56:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:56:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:56:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 17 09:56:44 unraid kernel: </NMI> Nov 17 09:56:44 unraid kernel: <TASK> Nov 17 09:56:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:56:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:56:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:56:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:56:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:56:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:56:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:56:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:56:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:56:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:56:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:56:44 unraid kernel: </TASK> Nov 17 09:58:13 unraid webGUI: Successful login user root from 10.10.10.68 Nov 17 09:59:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 09:59:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=8298982 Nov 17 09:59:44 unraid kernel: (detected by 30, t=19500848 jiffies, g=130869229, q=34728086 ncpus=32) Nov 17 09:59:44 unraid kernel: Sending NMI from CPU 30 to CPUs 3: Nov 17 09:59:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 09:59:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 09:59:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 09:59:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:59:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 09:59:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 09:59:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 09:59:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 09:59:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 09:59:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 09:59:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 09:59:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 09:59:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 09:59:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 09:59:44 unraid kernel: Call Trace: Nov 17 09:59:44 unraid kernel: <NMI> Nov 17 09:59:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 09:59:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 09:59:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 09:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:59:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 09:59:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 09:59:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 09:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:59:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 09:59:44 unraid kernel: </NMI> Nov 17 09:59:44 unraid kernel: <TASK> Nov 17 09:59:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 09:59:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 09:59:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 09:59:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 09:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:59:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 09:59:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 09:59:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 09:59:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 09:59:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 09:59:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 09:59:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 09:59:44 unraid kernel: </TASK> Nov 17 10:01:50 unraid webGUI: Successful login user root from 10.10.10.99 Nov 17 10:02:33 unraid root: ACPI action down is not defined Nov 17 10:02:33 unraid root: ACPI action down is not defined Nov 17 10:02:37 unraid root: ACPI action down is not defined Nov 17 10:02:37 unraid root: ACPI action up is not defined Nov 17 10:02:37 unraid root: ACPI action down is not defined Nov 17 10:02:37 unraid root: ACPI action up is not defined Nov 17 10:02:38 unraid root: ACPI action down is not defined Nov 17 10:02:38 unraid root: ACPI action up is not defined Nov 17 10:02:38 unraid root: ACPI action down is not defined Nov 17 10:02:38 unraid root: ACPI action up is not defined Nov 17 10:02:38 unraid root: ACPI action down is not defined Nov 17 10:02:38 unraid root: ACPI action down is not defined Nov 17 10:02:38 unraid root: ACPI action right is not defined Nov 17 10:02:38 unraid root: ACPI action down is not defined Nov 17 10:02:38 unraid root: ACPI action left is not defined Nov 17 10:02:38 unraid root: ACPI action up is not defined Nov 17 10:02:38 unraid root: ACPI action left is not defined Nov 17 10:02:39 unraid root: ACPI action right is not defined Nov 17 10:02:39 unraid root: ACPI action left is not defined Nov 17 10:02:44 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 17 10:02:44 unraid kernel: rcu: 3-...0: (1 GPs behind) idle=5984/1/0x4000000000000000 softirq=65785589/65785600 fqs=8387983 Nov 17 10:02:44 unraid kernel: (detected by 19, t=19680856 jiffies, g=130869229, q=34758210 ncpus=32) Nov 17 10:02:44 unraid kernel: Sending NMI from CPU 19 to CPUs 3: Nov 17 10:02:44 unraid kernel: NMI backtrace for cpu 3 Nov 17 10:02:44 unraid kernel: CPU: 3 PID: 15282 Comm: nvidia-containe Tainted: P D W O 6.1.106-Unraid #1 Nov 17 10:02:44 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 17 10:02:44 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 10:02:44 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 17 10:02:44 unraid kernel: RSP: 0018:ffffc900273d7d38 EFLAGS: 00000002 Nov 17 10:02:44 unraid kernel: RAX: 0000000000000101 RBX: ffff88883851f888 RCX: fffffffdda3e82fd Nov 17 10:02:44 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88883851f888 Nov 17 10:02:44 unraid kernel: RBP: ffff88897799b000 R08: 0000000000000000 R09: ffffc900273d7e38 Nov 17 10:02:44 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900273d7dc0 Nov 17 10:02:44 unraid kernel: R13: ffff888c80cf7f00 R14: 0000000000000283 R15: 000000000000001f Nov 17 10:02:44 unraid kernel: FS: 0000153a36cbbb80(0000) GS:ffff88900f840000(0000) knlGS:0000000000000000 Nov 17 10:02:44 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 17 10:02:44 unraid kernel: CR2: 00001539eb64c820 CR3: 0000000a8c976005 CR4: 00000000001706e0 Nov 17 10:02:44 unraid kernel: Call Trace: Nov 17 10:02:44 unraid kernel: <NMI> Nov 17 10:02:44 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 17 10:02:44 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 17 10:02:44 unraid kernel: ? nmi_handle+0x57/0x131 Nov 17 10:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 10:02:44 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 17 10:02:44 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 17 10:02:44 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 17 10:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 10:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 10:02:44 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 17 10:02:44 unraid kernel: </NMI> Nov 17 10:02:44 unraid kernel: <TASK> Nov 17 10:02:44 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 17 10:02:44 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 17 10:02:44 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 17 10:02:44 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 17 10:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 10:02:44 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 17 10:02:44 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 17 10:02:44 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 17 10:02:44 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 17 10:02:44 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 17 10:02:44 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 17 10:02:44 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 17 10:02:44 unraid kernel: </TASK> Nov 17 10:03:08 unraid root: ACPI action down is not defined Nov 17 10:03:08 unraid root: ACPI action right is not defined Nov 17 10:03:08 unraid root: ACPI action left is not defined Nov 17 10:03:08 unraid root: ACPI action up is not defined Nov 17 10:03:08 unraid root: ACPI action right is not defined Nov 17 10:03:08 unraid root: ACPI action left is not defined Nov 17 10:03:08 unraid root: ACPI action up is not defined Nov 17 10:03:08 unraid root: ACPI action right is not defined Nov 17 10:03:08 unraid root: ACPI action left is not defined Nov 17 10:03:08 unraid root: ACPI action right is not defined

  4. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    while i look into that, theres nothing else that looks like it could be a cause? no other posibilities? i ask because i dont know why a physical button would fail after rarely ever being used and this server sitting in a closet. I use the IDRAC functions far more often than the physical buttons. This server has been stable for me for about 2 years now, till this. from what i can find online the ear that the power button sits on is connected via ribbon cable, but based on a quick google search, this doesnt seem to be a common issue.

  5. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    reply about the power button, its a r720xd dell server. how would i check or remove that?

  6. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    its definitely not the power button, as it happens when no one is home or randomly in the night as well. I was worried about it being the gpu. Can i try rolling back to an old driver? I really would like to retain use of the gpu as it handles all my trans/en coding. Or is the gpu being installed the issue? I can see if there was a gpu driver update within the past few months

  7. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust replied to risenfromthedust's topic in General Support
    This is the syslog excerpt for just the day of the shutdown. Hopefully its enough: Nov 14 00:00:04 unraid Docker Auto Update: Community Applications Docker Autoupdate running Nov 14 00:00:04 unraid Docker Auto Update: Checking for available updates Nov 14 00:00:13 unraid Plugin Auto Update: Auto Updating fix.common.problems.plg Nov 14 00:00:14 unraid root: plugin: creating: /boot/config/plugins/fix.common.problems/fix.common.problems-2024.11.12-x86_64-1.txz - downloading from URL https://raw.githubusercontent.com/Squidly271/fix.common.problems/master/archive/fix.common.problems-2024.11.12-x86_64-1.txz Nov 14 00:00:15 unraid root: plugin: checking: /boot/config/plugins/fix.common.problems/fix.common.problems-2024.11.12-x86_64-1.txz - MD5 Nov 14 00:00:15 unraid root: plugin: running: upgradepkg --install-new /boot/config/plugins/fix.common.problems/fix.common.problems-2024.11.12-x86_64-1.txz Nov 14 00:00:15 unraid root: plugin: running: anonymous Nov 14 00:00:16 unraid root: plugin: fix.common.problems.plg updated Nov 14 00:00:17 unraid Plugin Auto Update: Auto Updating unassigned.devices.preclear.plg Nov 14 00:00:20 unraid root: plugin: running: anonymous Nov 14 00:00:20 unraid root: plugin: creating: /boot/config/plugins/unassigned.devices.preclear/unassigned.devices.preclear-2024.11.12.tgz - downloading from URL https://raw.githubusercontent.com/dlandon/unassigned.devices/master/unassigned.devices.preclear-2024.11.12.tgz Nov 14 00:00:20 unraid root: plugin: checking: /boot/config/plugins/unassigned.devices.preclear/unassigned.devices.preclear-2024.11.12.tgz - MD5 Nov 14 00:00:20 unraid root: plugin: checking: /boot/config/plugins/unassigned.devices.preclear/tmux-3.1b-x86_64-1.txz - MD5 Nov 14 00:00:20 unraid root: plugin: skipping: /boot/config/plugins/unassigned.devices.preclear/tmux-3.1b-x86_64-1.txz already exists Nov 14 00:00:20 unraid root: plugin: running: upgradepkg --install-new /boot/config/plugins/unassigned.devices.preclear/tmux-3.1b-x86_64-1.txz Nov 14 00:00:20 unraid root: plugin: checking: /boot/config/plugins/unassigned.devices.preclear/tmux-3.3a-x86_64-1.txz - MD5 Nov 14 00:00:20 unraid root: plugin: skipping: /boot/config/plugins/unassigned.devices.preclear/tmux-3.3a-x86_64-1.txz already exists Nov 14 00:00:20 unraid root: plugin: running: upgradepkg --install-new /boot/config/plugins/unassigned.devices.preclear/tmux-3.3a-x86_64-1.txz Nov 14 00:00:20 unraid root: plugin: checking: /boot/config/plugins/unassigned.devices.preclear/utempter-1.2.1-x86_64-1.txz - MD5 Nov 14 00:00:20 unraid root: plugin: skipping: /boot/config/plugins/unassigned.devices.preclear/utempter-1.2.1-x86_64-1.txz already exists Nov 14 00:00:20 unraid root: plugin: running: upgradepkg --install-new /boot/config/plugins/unassigned.devices.preclear/utempter-1.2.1-x86_64-1.txz Nov 14 00:00:20 unraid root: plugin: running: anonymous Nov 14 00:00:20 unraid root: plugin: running: anonymous Nov 14 00:00:21 unraid root: plugin: unassigned.devices.preclear.plg updated Nov 14 00:00:22 unraid Plugin Auto Update: Checking for language updates Nov 14 00:00:22 unraid Plugin Auto Update: Community Applications Plugin Auto Update finished Nov 14 00:01:26 unraid Docker Auto Update: Stopping jackett Nov 14 00:02:13 unraid kernel: docker0: port 16(veth473b9e8) entered disabled state Nov 14 00:02:13 unraid kernel: vethf14fb46: renamed from eth0 Nov 14 00:02:18 unraid kernel: docker0: port 16(veth473b9e8) entered disabled state Nov 14 00:02:18 unraid kernel: device veth473b9e8 left promiscuous mode Nov 14 00:02:18 unraid kernel: docker0: port 16(veth473b9e8) entered disabled state Nov 14 00:02:27 unraid Docker Auto Update: Installing Updates for calibre jackett Nov 14 00:18:14 unraid Docker Auto Update: Restarting jackett Nov 14 00:18:14 unraid kernel: docker0: port 16(veth03e5e54) entered blocking state Nov 14 00:18:14 unraid kernel: docker0: port 16(veth03e5e54) entered disabled state Nov 14 00:18:14 unraid kernel: device veth03e5e54 entered promiscuous mode Nov 14 00:18:24 unraid kernel: eth0: renamed from veth7c1e445 Nov 14 00:18:24 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth03e5e54: link becomes ready Nov 14 00:18:24 unraid kernel: docker0: port 16(veth03e5e54) entered blocking state Nov 14 00:18:24 unraid kernel: docker0: port 16(veth03e5e54) entered forwarding state Nov 14 00:18:26 unraid Docker Auto Update: Community Applications Docker Autoupdate finished Nov 14 00:21:28 unraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Nov 14 04:29:12 unraid kernel: docker0: port 8(vethcc486dd) entered disabled state Nov 14 04:29:12 unraid kernel: vethc4c5963: renamed from eth0 Nov 14 04:29:14 unraid kernel: docker0: port 8(vethcc486dd) entered disabled state Nov 14 04:29:14 unraid kernel: device vethcc486dd left promiscuous mode Nov 14 04:29:14 unraid kernel: docker0: port 8(vethcc486dd) entered disabled state Nov 14 04:30:46 unraid kernel: docker0: port 8(veth84f2010) entered blocking state Nov 14 04:30:46 unraid kernel: docker0: port 8(veth84f2010) entered disabled state Nov 14 04:30:46 unraid kernel: device veth84f2010 entered promiscuous mode Nov 14 04:30:54 unraid kernel: eth0: renamed from veth55ca6b1 Nov 14 04:30:54 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth84f2010: link becomes ready Nov 14 04:30:54 unraid kernel: docker0: port 8(veth84f2010) entered blocking state Nov 14 04:30:54 unraid kernel: docker0: port 8(veth84f2010) entered forwarding state Nov 14 04:31:08 unraid kernel: docker0: port 15(veth41f5fd2) entered disabled state Nov 14 04:31:08 unraid kernel: vethea384a4: renamed from eth0 Nov 14 04:31:09 unraid kernel: docker0: port 15(veth41f5fd2) entered disabled state Nov 14 04:31:09 unraid kernel: device veth41f5fd2 left promiscuous mode Nov 14 04:31:09 unraid kernel: docker0: port 15(veth41f5fd2) entered disabled state Nov 14 04:34:23 unraid kernel: docker0: port 15(vethdbba25f) entered blocking state Nov 14 04:34:23 unraid kernel: docker0: port 15(vethdbba25f) entered disabled state Nov 14 04:34:23 unraid kernel: device vethdbba25f entered promiscuous mode Nov 14 04:34:30 unraid kernel: eth0: renamed from vethab0e228 Nov 14 04:34:30 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethdbba25f: link becomes ready Nov 14 04:34:30 unraid kernel: docker0: port 15(vethdbba25f) entered blocking state Nov 14 04:34:30 unraid kernel: docker0: port 15(vethdbba25f) entered forwarding state Nov 14 04:34:50 unraid kernel: docker0: port 24(veth652d796) entered disabled state Nov 14 04:34:50 unraid kernel: veth8531ea5: renamed from eth0 Nov 14 04:34:51 unraid kernel: docker0: port 24(veth652d796) entered disabled state Nov 14 04:34:51 unraid kernel: device veth652d796 left promiscuous mode Nov 14 04:34:51 unraid kernel: docker0: port 24(veth652d796) entered disabled state Nov 14 04:34:53 unraid kernel: docker0: port 24(veth610eb11) entered blocking state Nov 14 04:34:53 unraid kernel: docker0: port 24(veth610eb11) entered disabled state Nov 14 04:34:53 unraid kernel: device veth610eb11 entered promiscuous mode Nov 14 04:35:00 unraid kernel: eth0: renamed from veth974e8cd Nov 14 04:35:00 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth610eb11: link becomes ready Nov 14 04:35:00 unraid kernel: docker0: port 24(veth610eb11) entered blocking state Nov 14 04:35:00 unraid kernel: docker0: port 24(veth610eb11) entered forwarding state Nov 14 04:35:08 unraid kernel: docker0: port 5(veth48ff367) entered disabled state Nov 14 04:35:08 unraid kernel: veth728e2af: renamed from eth0 Nov 14 04:35:09 unraid kernel: docker0: port 5(veth48ff367) entered disabled state Nov 14 04:35:09 unraid kernel: device veth48ff367 left promiscuous mode Nov 14 04:35:09 unraid kernel: docker0: port 5(veth48ff367) entered disabled state Nov 14 04:35:12 unraid kernel: docker0: port 5(vethf973fbd) entered blocking state Nov 14 04:35:12 unraid kernel: docker0: port 5(vethf973fbd) entered disabled state Nov 14 04:35:12 unraid kernel: device vethf973fbd entered promiscuous mode Nov 14 04:35:15 unraid kernel: eth0: renamed from veth6c9e669 Nov 14 04:35:15 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf973fbd: link becomes ready Nov 14 04:35:15 unraid kernel: docker0: port 5(vethf973fbd) entered blocking state Nov 14 04:35:15 unraid kernel: docker0: port 5(vethf973fbd) entered forwarding state Nov 14 04:35:31 unraid kernel: docker0: port 6(vethbc32ac9) entered disabled state Nov 14 04:35:31 unraid kernel: veth7a960de: renamed from eth0 Nov 14 04:35:32 unraid kernel: docker0: port 6(vethbc32ac9) entered disabled state Nov 14 04:35:32 unraid kernel: device vethbc32ac9 left promiscuous mode Nov 14 04:35:32 unraid kernel: docker0: port 6(vethbc32ac9) entered disabled state Nov 14 04:35:41 unraid kernel: docker0: port 6(veth440cd5b) entered blocking state Nov 14 04:35:41 unraid kernel: docker0: port 6(veth440cd5b) entered disabled state Nov 14 04:35:41 unraid kernel: device veth440cd5b entered promiscuous mode Nov 14 04:35:46 unraid kernel: eth0: renamed from veth57ae533 Nov 14 04:35:46 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth440cd5b: link becomes ready Nov 14 04:35:46 unraid kernel: docker0: port 6(veth440cd5b) entered blocking state Nov 14 04:35:46 unraid kernel: docker0: port 6(veth440cd5b) entered forwarding state Nov 14 04:36:01 unraid kernel: docker0: port 27(veth7bb35c8) entered disabled state Nov 14 04:36:01 unraid kernel: vetheb6a4cc: renamed from eth0 Nov 14 04:36:02 unraid kernel: docker0: port 27(veth7bb35c8) entered disabled state Nov 14 04:36:02 unraid kernel: device veth7bb35c8 left promiscuous mode Nov 14 04:36:02 unraid kernel: docker0: port 27(veth7bb35c8) entered disabled state Nov 14 04:36:54 unraid kernel: docker0: port 27(veth052ffe0) entered blocking state Nov 14 04:36:54 unraid kernel: docker0: port 27(veth052ffe0) entered disabled state Nov 14 04:36:54 unraid kernel: device veth052ffe0 entered promiscuous mode Nov 14 04:36:57 unraid kernel: eth0: renamed from veth6e17fc5 Nov 14 04:36:57 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth052ffe0: link becomes ready Nov 14 04:36:57 unraid kernel: docker0: port 27(veth052ffe0) entered blocking state Nov 14 04:36:57 unraid kernel: docker0: port 27(veth052ffe0) entered forwarding state Nov 14 04:38:20 unraid kernel: docker0: port 23(vetha5273c6) entered disabled state Nov 14 04:38:20 unraid kernel: veth4bb366e: renamed from eth0 Nov 14 04:38:21 unraid kernel: docker0: port 23(vetha5273c6) entered disabled state Nov 14 04:38:21 unraid kernel: device vetha5273c6 left promiscuous mode Nov 14 04:38:21 unraid kernel: docker0: port 23(vetha5273c6) entered disabled state Nov 14 04:38:22 unraid kernel: docker0: port 23(veth48f9bba) entered blocking state Nov 14 04:38:22 unraid kernel: docker0: port 23(veth48f9bba) entered disabled state Nov 14 04:38:22 unraid kernel: device veth48f9bba entered promiscuous mode Nov 14 04:38:26 unraid kernel: eth0: renamed from veth95e7a3d Nov 14 04:38:26 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth48f9bba: link becomes ready Nov 14 04:38:26 unraid kernel: docker0: port 23(veth48f9bba) entered blocking state Nov 14 04:38:26 unraid kernel: docker0: port 23(veth48f9bba) entered forwarding state Nov 14 04:39:09 unraid kernel: docker0: port 22(veth162eaa4) entered disabled state Nov 14 04:39:09 unraid kernel: vethcacf8ba: renamed from eth0 Nov 14 04:39:10 unraid kernel: docker0: port 22(veth162eaa4) entered disabled state Nov 14 04:39:10 unraid kernel: device veth162eaa4 left promiscuous mode Nov 14 04:39:10 unraid kernel: docker0: port 22(veth162eaa4) entered disabled state Nov 14 04:45:43 unraid kernel: docker0: port 22(vethf77343a) entered blocking state Nov 14 04:45:43 unraid kernel: docker0: port 22(vethf77343a) entered disabled state Nov 14 04:45:43 unraid kernel: device vethf77343a entered promiscuous mode Nov 14 04:45:47 unraid kernel: eth0: renamed from veth6bb58ec Nov 14 04:45:47 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf77343a: link becomes ready Nov 14 04:45:47 unraid kernel: docker0: port 22(vethf77343a) entered blocking state Nov 14 04:45:47 unraid kernel: docker0: port 22(vethf77343a) entered forwarding state Nov 14 04:46:03 unraid kernel: docker0: port 21(vethe35a60e) entered disabled state Nov 14 04:46:03 unraid kernel: veth70d2ad1: renamed from eth0 Nov 14 04:46:03 unraid kernel: docker0: port 21(vethe35a60e) entered disabled state Nov 14 04:46:03 unraid kernel: device vethe35a60e left promiscuous mode Nov 14 04:46:03 unraid kernel: docker0: port 21(vethe35a60e) entered disabled state Nov 14 04:47:35 unraid kernel: docker0: port 21(veth25a6c3c) entered blocking state Nov 14 04:47:35 unraid kernel: docker0: port 21(veth25a6c3c) entered disabled state Nov 14 04:47:35 unraid kernel: device veth25a6c3c entered promiscuous mode Nov 14 04:47:53 unraid kernel: eth0: renamed from veth43ad681 Nov 14 04:47:53 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth25a6c3c: link becomes ready Nov 14 04:47:53 unraid kernel: docker0: port 21(veth25a6c3c) entered blocking state Nov 14 04:47:53 unraid kernel: docker0: port 21(veth25a6c3c) entered forwarding state Nov 14 04:48:11 unraid kernel: docker0: port 20(vethb403c1b) entered disabled state Nov 14 04:48:11 unraid kernel: veth34f6649: renamed from eth0 Nov 14 04:48:11 unraid kernel: docker0: port 20(vethb403c1b) entered disabled state Nov 14 04:48:11 unraid kernel: device vethb403c1b left promiscuous mode Nov 14 04:48:11 unraid kernel: docker0: port 20(vethb403c1b) entered disabled state Nov 14 04:55:27 unraid kernel: docker0: port 20(veth3eec58b) entered blocking state Nov 14 04:55:27 unraid kernel: docker0: port 20(veth3eec58b) entered disabled state Nov 14 04:55:27 unraid kernel: device veth3eec58b entered promiscuous mode Nov 14 04:55:33 unraid kernel: eth0: renamed from vethc182fc0 Nov 14 04:55:33 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth3eec58b: link becomes ready Nov 14 04:55:33 unraid kernel: docker0: port 20(veth3eec58b) entered blocking state Nov 14 04:55:33 unraid kernel: docker0: port 20(veth3eec58b) entered forwarding state Nov 14 04:55:40 unraid kernel: docker0: port 12(veth1be01a7) entered disabled state Nov 14 04:55:40 unraid kernel: vethd089615: renamed from eth0 Nov 14 04:55:40 unraid kernel: docker0: port 12(veth1be01a7) entered disabled state Nov 14 04:55:40 unraid kernel: device veth1be01a7 left promiscuous mode Nov 14 04:55:40 unraid kernel: docker0: port 12(veth1be01a7) entered disabled state Nov 14 04:55:45 unraid kernel: docker0: port 12(veth313043b) entered blocking state Nov 14 04:55:45 unraid kernel: docker0: port 12(veth313043b) entered disabled state Nov 14 04:55:45 unraid kernel: device veth313043b entered promiscuous mode Nov 14 04:55:49 unraid kernel: eth0: renamed from veth75ae438 Nov 14 04:55:49 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth313043b: link becomes ready Nov 14 04:55:49 unraid kernel: docker0: port 12(veth313043b) entered blocking state Nov 14 04:55:49 unraid kernel: docker0: port 12(veth313043b) entered forwarding state Nov 14 04:56:00 unraid kernel: docker0: port 26(veth759645f) entered disabled state Nov 14 04:56:00 unraid kernel: veth0ed4b07: renamed from eth0 Nov 14 04:56:01 unraid kernel: docker0: port 26(veth759645f) entered disabled state Nov 14 04:56:01 unraid kernel: device veth759645f left promiscuous mode Nov 14 04:56:01 unraid kernel: docker0: port 26(veth759645f) entered disabled state Nov 14 05:02:32 unraid kernel: docker0: port 26(vethaf6f3ce) entered blocking state Nov 14 05:02:32 unraid kernel: docker0: port 26(vethaf6f3ce) entered disabled state Nov 14 05:02:32 unraid kernel: device vethaf6f3ce entered promiscuous mode Nov 14 05:02:34 unraid kernel: eth0: renamed from veth06128f2 Nov 14 05:02:34 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethaf6f3ce: link becomes ready Nov 14 05:02:34 unraid kernel: docker0: port 26(vethaf6f3ce) entered blocking state Nov 14 05:02:34 unraid kernel: docker0: port 26(vethaf6f3ce) entered forwarding state Nov 14 05:02:39 unraid kernel: docker0: port 18(veth635360c) entered disabled state Nov 14 05:02:39 unraid kernel: veth325d156: renamed from eth0 Nov 14 05:02:40 unraid kernel: docker0: port 18(veth635360c) entered disabled state Nov 14 05:02:40 unraid kernel: device veth635360c left promiscuous mode Nov 14 05:02:40 unraid kernel: docker0: port 18(veth635360c) entered disabled state Nov 14 05:02:41 unraid kernel: docker0: port 18(veth9056cf6) entered blocking state Nov 14 05:02:41 unraid kernel: docker0: port 18(veth9056cf6) entered disabled state Nov 14 05:02:41 unraid kernel: device veth9056cf6 entered promiscuous mode Nov 14 05:02:45 unraid kernel: eth0: renamed from vethb635d06 Nov 14 05:02:45 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9056cf6: link becomes ready Nov 14 05:02:45 unraid kernel: docker0: port 18(veth9056cf6) entered blocking state Nov 14 05:02:45 unraid kernel: docker0: port 18(veth9056cf6) entered forwarding state Nov 14 05:02:54 unraid kernel: docker0: port 25(vethbc3ebff) entered disabled state Nov 14 05:02:54 unraid kernel: veth226a35d: renamed from eth0 Nov 14 05:02:54 unraid kernel: docker0: port 25(vethbc3ebff) entered disabled state Nov 14 05:02:54 unraid kernel: device vethbc3ebff left promiscuous mode Nov 14 05:02:54 unraid kernel: docker0: port 25(vethbc3ebff) entered disabled state Nov 14 05:08:25 unraid kernel: docker0: port 25(veth68213ed) entered blocking state Nov 14 05:08:25 unraid kernel: docker0: port 25(veth68213ed) entered disabled state Nov 14 05:08:25 unraid kernel: device veth68213ed entered promiscuous mode Nov 14 05:08:34 unraid kernel: eth0: renamed from veth127126f Nov 14 05:08:34 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth68213ed: link becomes ready Nov 14 05:08:34 unraid kernel: docker0: port 25(veth68213ed) entered blocking state Nov 14 05:08:34 unraid kernel: docker0: port 25(veth68213ed) entered forwarding state Nov 14 05:08:48 unraid kernel: docker0: port 1(vethe62623b) entered disabled state Nov 14 05:08:48 unraid kernel: veth9f71e8c: renamed from eth0 Nov 14 05:08:49 unraid kernel: docker0: port 1(vethe62623b) entered disabled state Nov 14 05:08:49 unraid kernel: device vethe62623b left promiscuous mode Nov 14 05:08:49 unraid kernel: docker0: port 1(vethe62623b) entered disabled state Nov 14 05:11:37 unraid kernel: docker0: port 1(vethda9af30) entered blocking state Nov 14 05:11:37 unraid kernel: docker0: port 1(vethda9af30) entered disabled state Nov 14 05:11:37 unraid kernel: device vethda9af30 entered promiscuous mode Nov 14 05:11:46 unraid kernel: eth0: renamed from veth4ee6dce Nov 14 05:11:46 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethda9af30: link becomes ready Nov 14 05:11:46 unraid kernel: docker0: port 1(vethda9af30) entered blocking state Nov 14 05:11:46 unraid kernel: docker0: port 1(vethda9af30) entered forwarding state Nov 14 05:12:14 unraid kernel: docker0: port 19(veth2f9048a) entered disabled state Nov 14 05:12:14 unraid kernel: veth1f7e35a: renamed from eth0 Nov 14 05:12:15 unraid kernel: docker0: port 19(veth2f9048a) entered disabled state Nov 14 05:12:15 unraid kernel: device veth2f9048a left promiscuous mode Nov 14 05:12:15 unraid kernel: docker0: port 19(veth2f9048a) entered disabled state Nov 14 06:04:53 unraid kernel: docker0: port 19(veth51a207c) entered blocking state Nov 14 06:04:53 unraid kernel: docker0: port 19(veth51a207c) entered disabled state Nov 14 06:04:53 unraid kernel: device veth51a207c entered promiscuous mode Nov 14 06:05:04 unraid kernel: eth0: renamed from vethc1c0576 Nov 14 06:05:04 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth51a207c: link becomes ready Nov 14 06:05:04 unraid kernel: docker0: port 19(veth51a207c) entered blocking state Nov 14 06:05:04 unraid kernel: docker0: port 19(veth51a207c) entered forwarding state Nov 14 06:05:28 unraid kernel: docker0: port 16(veth03e5e54) entered disabled state Nov 14 06:05:28 unraid kernel: veth7c1e445: renamed from eth0 Nov 14 06:05:28 unraid kernel: docker0: port 16(veth03e5e54) entered disabled state Nov 14 06:05:28 unraid kernel: device veth03e5e54 left promiscuous mode Nov 14 06:05:28 unraid kernel: docker0: port 16(veth03e5e54) entered disabled state Nov 14 06:05:30 unraid kernel: docker0: port 16(veth63c9a24) entered blocking state Nov 14 06:05:30 unraid kernel: docker0: port 16(veth63c9a24) entered disabled state Nov 14 06:05:30 unraid kernel: device veth63c9a24 entered promiscuous mode Nov 14 06:05:34 unraid kernel: eth0: renamed from veth3c5a857 Nov 14 06:05:34 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth63c9a24: link becomes ready Nov 14 06:05:34 unraid kernel: docker0: port 16(veth63c9a24) entered blocking state Nov 14 06:05:34 unraid kernel: docker0: port 16(veth63c9a24) entered forwarding state Nov 14 06:05:39 unraid kernel: docker0: port 13(veth38e8052) entered disabled state Nov 14 06:05:39 unraid kernel: veth2ce2b74: renamed from eth0 Nov 14 06:05:40 unraid kernel: docker0: port 13(veth38e8052) entered disabled state Nov 14 06:05:40 unraid kernel: device veth38e8052 left promiscuous mode Nov 14 06:05:40 unraid kernel: docker0: port 13(veth38e8052) entered disabled state Nov 14 06:51:37 unraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Nov 14 06:58:06 unraid kernel: docker0: port 13(vethe91938e) entered blocking state Nov 14 06:58:06 unraid kernel: docker0: port 13(vethe91938e) entered disabled state Nov 14 06:58:06 unraid kernel: device vethe91938e entered promiscuous mode Nov 14 06:58:18 unraid kernel: eth0: renamed from veth79a7e09 Nov 14 06:58:18 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethe91938e: link becomes ready Nov 14 06:58:18 unraid kernel: docker0: port 13(vethe91938e) entered blocking state Nov 14 06:58:18 unraid kernel: docker0: port 13(vethe91938e) entered forwarding state Nov 14 06:58:33 unraid kernel: docker0: port 11(veth6272a95) entered disabled state Nov 14 06:58:33 unraid kernel: vethe351889: renamed from eth0 Nov 14 06:58:34 unraid kernel: docker0: port 11(veth6272a95) entered disabled state Nov 14 06:58:34 unraid kernel: device veth6272a95 left promiscuous mode Nov 14 06:58:34 unraid kernel: docker0: port 11(veth6272a95) entered disabled state Nov 14 06:58:48 unraid kernel: docker0: port 11(vethc21540f) entered blocking state Nov 14 06:58:48 unraid kernel: docker0: port 11(vethc21540f) entered disabled state Nov 14 06:58:48 unraid kernel: device vethc21540f entered promiscuous mode Nov 14 06:58:50 unraid kernel: eth0: renamed from veth4a3ca3e Nov 14 06:58:50 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethc21540f: link becomes ready Nov 14 06:58:50 unraid kernel: docker0: port 11(vethc21540f) entered blocking state Nov 14 06:58:50 unraid kernel: docker0: port 11(vethc21540f) entered forwarding state Nov 14 06:59:07 unraid kernel: docker0: port 14(veth5a7b790) entered disabled state Nov 14 06:59:07 unraid kernel: veth5a69678: renamed from eth0 Nov 14 06:59:08 unraid kernel: docker0: port 14(veth5a7b790) entered disabled state Nov 14 06:59:08 unraid kernel: device veth5a7b790 left promiscuous mode Nov 14 06:59:08 unraid kernel: docker0: port 14(veth5a7b790) entered disabled state Nov 14 06:59:12 unraid kernel: docker0: port 14(vethee8c60b) entered blocking state Nov 14 06:59:12 unraid kernel: docker0: port 14(vethee8c60b) entered disabled state Nov 14 06:59:12 unraid kernel: device vethee8c60b entered promiscuous mode Nov 14 06:59:14 unraid kernel: eth0: renamed from veth2274b36 Nov 14 06:59:14 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethee8c60b: link becomes ready Nov 14 06:59:14 unraid kernel: docker0: port 14(vethee8c60b) entered blocking state Nov 14 06:59:14 unraid kernel: docker0: port 14(vethee8c60b) entered forwarding state Nov 14 06:59:30 unraid kernel: docker0: port 10(veth788bb31) entered disabled state Nov 14 06:59:30 unraid kernel: veth78e2a28: renamed from eth0 Nov 14 06:59:31 unraid kernel: docker0: port 10(veth788bb31) entered disabled state Nov 14 06:59:31 unraid kernel: device veth788bb31 left promiscuous mode Nov 14 06:59:31 unraid kernel: docker0: port 10(veth788bb31) entered disabled state Nov 14 06:59:32 unraid kernel: docker0: port 10(veth7fd7396) entered blocking state Nov 14 06:59:32 unraid kernel: docker0: port 10(veth7fd7396) entered disabled state Nov 14 06:59:32 unraid kernel: device veth7fd7396 entered promiscuous mode Nov 14 06:59:35 unraid kernel: eth0: renamed from veth66689d0 Nov 14 06:59:35 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7fd7396: link becomes ready Nov 14 06:59:35 unraid kernel: docker0: port 10(veth7fd7396) entered blocking state Nov 14 06:59:35 unraid kernel: docker0: port 10(veth7fd7396) entered forwarding state Nov 14 07:02:58 unraid kernel: docker0: port 17(vethfd0128a) entered disabled state Nov 14 07:02:58 unraid kernel: veth2f44ae9: renamed from eth0 Nov 14 07:02:59 unraid kernel: docker0: port 17(vethfd0128a) entered disabled state Nov 14 07:02:59 unraid kernel: device vethfd0128a left promiscuous mode Nov 14 07:02:59 unraid kernel: docker0: port 17(vethfd0128a) entered disabled state Nov 14 07:03:07 unraid kernel: docker0: port 17(veth301344f) entered blocking state Nov 14 07:03:07 unraid kernel: docker0: port 17(veth301344f) entered disabled state Nov 14 07:03:07 unraid kernel: device veth301344f entered promiscuous mode Nov 14 07:03:09 unraid kernel: eth0: renamed from veth3402db0 Nov 14 07:03:09 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth301344f: link becomes ready Nov 14 07:03:09 unraid kernel: docker0: port 17(veth301344f) entered blocking state Nov 14 07:03:09 unraid kernel: docker0: port 17(veth301344f) entered forwarding state Nov 14 07:03:14 unraid kernel: docker0: port 9(veth5356c50) entered disabled state Nov 14 07:03:14 unraid kernel: vethccd17db: renamed from eth0 Nov 14 07:03:15 unraid kernel: docker0: port 9(veth5356c50) entered disabled state Nov 14 07:03:15 unraid kernel: device veth5356c50 left promiscuous mode Nov 14 07:03:15 unraid kernel: docker0: port 9(veth5356c50) entered disabled state Nov 14 07:05:02 unraid kernel: docker0: port 9(veth0bd2da8) entered blocking state Nov 14 07:05:02 unraid kernel: docker0: port 9(veth0bd2da8) entered disabled state Nov 14 07:05:02 unraid kernel: device veth0bd2da8 entered promiscuous mode Nov 14 07:05:04 unraid kernel: eth0: renamed from vetha9f76fb Nov 14 07:05:04 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0bd2da8: link becomes ready Nov 14 07:05:04 unraid kernel: docker0: port 9(veth0bd2da8) entered blocking state Nov 14 07:05:04 unraid kernel: docker0: port 9(veth0bd2da8) entered forwarding state Nov 14 07:07:12 unraid kernel: docker0: port 3(veth0a56286) entered disabled state Nov 14 07:07:12 unraid kernel: vethfbf9a58: renamed from eth0 Nov 14 07:07:12 unraid kernel: docker0: port 3(veth0a56286) entered disabled state Nov 14 07:07:12 unraid kernel: device veth0a56286 left promiscuous mode Nov 14 07:07:12 unraid kernel: docker0: port 3(veth0a56286) entered disabled state Nov 14 07:07:17 unraid kernel: docker0: port 3(veth097b828) entered blocking state Nov 14 07:07:17 unraid kernel: docker0: port 3(veth097b828) entered disabled state Nov 14 07:07:17 unraid kernel: device veth097b828 entered promiscuous mode Nov 14 07:07:23 unraid kernel: eth0: renamed from veth899215e Nov 14 07:07:23 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth097b828: link becomes ready Nov 14 07:07:23 unraid kernel: docker0: port 3(veth097b828) entered blocking state Nov 14 07:07:23 unraid kernel: docker0: port 3(veth097b828) entered forwarding state Nov 14 07:07:36 unraid kernel: docker0: port 2(veth2002b9b) entered disabled state Nov 14 07:07:36 unraid kernel: veth5200e88: renamed from eth0 Nov 14 07:07:37 unraid kernel: docker0: port 2(veth2002b9b) entered disabled state Nov 14 07:07:37 unraid kernel: device veth2002b9b left promiscuous mode Nov 14 07:07:37 unraid kernel: docker0: port 2(veth2002b9b) entered disabled state Nov 14 07:07:38 unraid kernel: docker0: port 2(veth0b89298) entered blocking state Nov 14 07:07:38 unraid kernel: docker0: port 2(veth0b89298) entered disabled state Nov 14 07:07:38 unraid kernel: device veth0b89298 entered promiscuous mode Nov 14 07:07:40 unraid kernel: eth0: renamed from vethafdbcf2 Nov 14 07:07:40 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0b89298: link becomes ready Nov 14 07:07:40 unraid kernel: docker0: port 2(veth0b89298) entered blocking state Nov 14 07:07:40 unraid kernel: docker0: port 2(veth0b89298) entered forwarding state Nov 14 07:07:45 unraid kernel: docker0: port 4(vetha3eb46a) entered disabled state Nov 14 07:07:45 unraid kernel: veth60057f5: renamed from eth0 Nov 14 07:07:46 unraid kernel: docker0: port 4(vetha3eb46a) entered disabled state Nov 14 07:07:46 unraid kernel: device vetha3eb46a left promiscuous mode Nov 14 07:07:46 unraid kernel: docker0: port 4(vetha3eb46a) entered disabled state Nov 14 07:07:51 unraid kernel: docker0: port 4(veth7bb1d3b) entered blocking state Nov 14 07:07:51 unraid kernel: docker0: port 4(veth7bb1d3b) entered disabled state Nov 14 07:07:51 unraid kernel: device veth7bb1d3b entered promiscuous mode Nov 14 07:07:56 unraid kernel: eth0: renamed from veth702d09c Nov 14 07:07:56 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7bb1d3b: link becomes ready Nov 14 07:07:56 unraid kernel: docker0: port 4(veth7bb1d3b) entered blocking state Nov 14 07:07:56 unraid kernel: docker0: port 4(veth7bb1d3b) entered forwarding state Nov 14 07:08:22 unraid kernel: docker0: port 7(veth884d0f9) entered disabled state Nov 14 07:08:22 unraid kernel: veth7aab50d: renamed from eth0 Nov 14 07:08:22 unraid kernel: docker0: port 7(veth884d0f9) entered disabled state Nov 14 07:08:23 unraid kernel: device veth884d0f9 left promiscuous mode Nov 14 07:08:23 unraid kernel: docker0: port 7(veth884d0f9) entered disabled state Nov 14 07:08:25 unraid kernel: docker0: port 7(veth5c61dc4) entered blocking state Nov 14 07:08:25 unraid kernel: docker0: port 7(veth5c61dc4) entered disabled state Nov 14 07:08:25 unraid kernel: device veth5c61dc4 entered promiscuous mode Nov 14 07:08:27 unraid kernel: eth0: renamed from veth907af96 Nov 14 07:08:27 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5c61dc4: link becomes ready Nov 14 07:08:27 unraid kernel: docker0: port 7(veth5c61dc4) entered blocking state Nov 14 07:08:27 unraid kernel: docker0: port 7(veth5c61dc4) entered forwarding state Nov 14 07:09:34 unraid kernel: TCP: request_sock_TCP: Possible SYN flooding on port 33137. Sending cookies. Check SNMP counters. Nov 14 08:12:15 unraid kernel: NVRM: failed to allocate vmap() page descriptor table! Nov 14 08:12:15 unraid kernel: NVRM: GPU 0000:04:00.0: RmInitAdapter failed! (0x62:0xffff:2468) Nov 14 08:12:15 unraid kernel: NVRM: GPU 0000:04:00.0: rm_init_adapter failed, device minor number 0 Nov 14 09:02:00 unraid kernel: NVRM: failed to allocate vmap() page descriptor table! Nov 14 09:02:00 unraid kernel: NVRM: GPU 0000:04:00.0: RmInitAdapter failed! (0x62:0xffff:2468) Nov 14 09:02:00 unraid kernel: NVRM: GPU 0000:04:00.0: rm_init_adapter failed, device minor number 0 Nov 14 11:54:31 unraid sshd-session[42616]: Received disconnect from 10.10.10.9 port 41810:11: disconnected by user Nov 14 11:54:31 unraid sshd-session[42616]: Disconnected from user root 10.10.10.9 port 41810 Nov 14 11:54:31 unraid sshd-session[42532]: pam_unix(sshd:session): session closed for user root Nov 14 13:25:32 unraid kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 Nov 14 13:25:32 unraid kernel: #PF: supervisor read access in kernel mode Nov 14 13:25:32 unraid kernel: #PF: error_code(0x0000) - not-present page Nov 14 13:25:32 unraid kernel: PGD 80000009247ab067 P4D 80000009247ab067 PUD 94e3ce067 PMD 0 Nov 14 13:25:32 unraid kernel: Oops: 0000 [#1] PREEMPT SMP PTI Nov 14 13:25:32 unraid kernel: CPU: 13 PID: 39147 Comm: nvidia-smi Tainted: P O 6.1.106-Unraid #1 Nov 14 13:25:32 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:25:32 unraid kernel: RIP: 0010:__cpa_process_fault+0x3ea/0x40e Nov 14 13:25:32 unraid kernel: Code: d0 48 c1 e8 0c 49 89 44 24 30 31 db eb 2b 49 8b 7c 24 30 e8 d4 f3 ff ff 84 c0 75 18 49 8b 04 24 48 89 ee 48 c7 c7 d7 1a 0c 82 <48> 8b 10 e8 16 c8 00 00 0f 0b bb f2 ff ff ff 48 83 c4 30 89 d8 5b Nov 14 13:25:32 unraid kernel: RSP: 0018:ffffc9002408f720 EFLAGS: 00010246 Nov 14 13:25:32 unraid kernel: RAX: 0000000000000000 RBX: ffffc9002408f890 RCX: 0000000080000000 Nov 14 13:25:32 unraid kernel: RDX: 0000000000001001 RSI: ffff1101761ff000 RDI: ffffffff820c1ad7 Nov 14 13:25:32 unraid kernel: RBP: ffff1101761ff000 R08: ffffc9002408f7de R09: ffffffff8220a110 Nov 14 13:25:32 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9002408f890 Nov 14 13:25:32 unraid kernel: R13: 800000016cfe3073 R14: 0000000000000000 R15: 0000000000000000 Nov 14 13:25:32 unraid kernel: FS: 000014c718e2e180(0000) GS:ffff88900f980000(0000) knlGS:0000000000000000 Nov 14 13:25:32 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:25:32 unraid kernel: CR2: 0000000000000000 CR3: 0000000af6824004 CR4: 00000000001706e0 Nov 14 13:25:32 unraid kernel: Call Trace: Nov 14 13:25:32 unraid kernel: <TASK> Nov 14 13:25:32 unraid kernel: ? __die_body+0x1a/0x5c Nov 14 13:25:32 unraid kernel: ? page_fault_oops+0x329/0x376 Nov 14 13:25:32 unraid kernel: ? search_bpf_extables+0x5d/0x68 Nov 14 13:25:32 unraid kernel: ? fixup_exception+0x22/0x24b Nov 14 13:25:32 unraid kernel: ? exc_page_fault+0xfb/0x11d Nov 14 13:25:32 unraid kernel: ? asm_exc_page_fault+0x22/0x30 Nov 14 13:25:32 unraid kernel: ? __cpa_process_fault+0x3ea/0x40e Nov 14 13:25:32 unraid kernel: ? lookup_address_in_pgd_attr+0x15b/0x173 Nov 14 13:25:32 unraid kernel: __change_page_attr_set_clr+0xb7/0x848 Nov 14 13:25:32 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:25:32 unraid kernel: change_page_attr_set_clr+0x129/0x271 Nov 14 13:25:32 unraid kernel: set_pages_array_wb+0x37/0x72 Nov 14 13:25:32 unraid kernel: nv_set_memory_type+0xd9/0x184 [nvidia] Nov 14 13:25:32 unraid kernel: nv_free_system_pages+0x59/0x170 [nvidia] Nov 14 13:25:32 unraid kernel: nv_free_pages+0x68/0x78 [nvidia] Nov 14 13:25:32 unraid kernel: _nv043747rm+0x98/0x140 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv038497rm+0x27a/0x660 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv038458rm+0x5f/0x130 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv004292rm+0xd/0x20 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv006122rm+0x1e/0xb0 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv018080rm+0x59c/0x680 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv048181rm+0xb3/0xe0 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv049956rm+0xb3/0x180 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv049955rm+0x4ae/0x660 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv048073rm+0xdd/0x190 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv017727rm+0x105/0x220 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv004138rm+0xd/0x20 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv006122rm+0x1e/0xb0 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv049238rm+0xd1/0x210 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv038833rm+0xc2/0xf0 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv038629rm+0x3d/0x80 [nvidia] Nov 14 13:25:32 unraid kernel: ? __kmem_cache_free+0xb0/0xf5 Nov 14 13:25:32 unraid kernel: ? _nv035820rm+0x51/0x150 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv035816rm+0x6d/0xf0 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv027931rm+0x106/0x1e0 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv027932rm+0x574/0x850 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv000702rm+0x2aa/0x340 [nvidia] Nov 14 13:25:32 unraid kernel: ? rm_disable_adapter+0x61/0x130 [nvidia] Nov 14 13:25:32 unraid kernel: ? up_write+0x33/0x5a Nov 14 13:25:32 unraid kernel: ? nv_shutdown_adapter+0x17/0x115 [nvidia] Nov 14 13:25:32 unraid kernel: ? nv_close_device+0xa3/0x123 [nvidia] Nov 14 13:25:32 unraid kernel: ? nvidia_close_callback+0x9e/0x121 [nvidia] Nov 14 13:25:32 unraid kernel: ? nvidia_close+0x19c/0x1ea [nvidia] Nov 14 13:25:32 unraid kernel: ? __fput+0x103/0x1d3 Nov 14 13:25:32 unraid kernel: ? task_work_run+0x6b/0x80 Nov 14 13:25:32 unraid kernel: ? exit_to_user_mode_prepare+0x75/0x112 Nov 14 13:25:32 unraid kernel: ? syscall_exit_to_user_mode+0x18/0x23 Nov 14 13:25:32 unraid kernel: ? do_syscall_64+0x71/0x7b Nov 14 13:25:32 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:25:32 unraid kernel: </TASK> Nov 14 13:25:32 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 mgag200 sha1_ssse3 video Nov 14 13:25:32 unraid kernel: drm_shmem_helper aesni_intel drm_kms_helper crypto_simd cryptd rapl intel_cstate drm intel_uncore mpt3sas mei_me i2c_algo_bit backlight input_leds joydev led_class mei i2c_core syscopyarea raid_class sysfillrect sysimgblt scsi_transport_sas fb_sys_fops wmi ipmi_si acpi_power_meter button unix [last unloaded: xfrm_algo] Nov 14 13:25:32 unraid kernel: CR2: 0000000000000000 Nov 14 13:25:32 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 14 13:25:32 unraid kernel: RIP: 0010:__cpa_process_fault+0x3ea/0x40e Nov 14 13:25:32 unraid kernel: Code: d0 48 c1 e8 0c 49 89 44 24 30 31 db eb 2b 49 8b 7c 24 30 e8 d4 f3 ff ff 84 c0 75 18 49 8b 04 24 48 89 ee 48 c7 c7 d7 1a 0c 82 <48> 8b 10 e8 16 c8 00 00 0f 0b bb f2 ff ff ff 48 83 c4 30 89 d8 5b Nov 14 13:25:32 unraid kernel: RSP: 0018:ffffc9002408f720 EFLAGS: 00010246 Nov 14 13:25:32 unraid kernel: RAX: 0000000000000000 RBX: ffffc9002408f890 RCX: 0000000080000000 Nov 14 13:25:32 unraid kernel: RDX: 0000000000001001 RSI: ffff1101761ff000 RDI: ffffffff820c1ad7 Nov 14 13:25:32 unraid kernel: RBP: ffff1101761ff000 R08: ffffc9002408f7de R09: ffffffff8220a110 Nov 14 13:25:32 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9002408f890 Nov 14 13:25:32 unraid kernel: R13: 800000016cfe3073 R14: 0000000000000000 R15: 0000000000000000 Nov 14 13:25:32 unraid kernel: FS: 000014c718e2e180(0000) GS:ffff88900f980000(0000) knlGS:0000000000000000 Nov 14 13:25:32 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:25:32 unraid kernel: CR2: 0000000000000000 CR3: 0000000af6824004 CR4: 00000000001706e0 Nov 14 13:25:32 unraid kernel: note: nvidia-smi[39147] exited with irqs disabled Nov 14 13:25:32 unraid kernel: note: nvidia-smi[39147] exited with preempt_count 1 Nov 14 13:25:32 unraid kernel: general protection fault, probably for non-canonical address 0x102fbbc0e51940: 0000 [#2] PREEMPT SMP PTI Nov 14 13:25:32 unraid kernel: CPU: 13 PID: 39147 Comm: nvidia-smi Tainted: P D O 6.1.106-Unraid #1 Nov 14 13:25:32 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:25:32 unraid kernel: RIP: 0010:_nv013001rm+0x3c/0x310 [nvidia] Nov 14 13:25:32 unraid kernel: Code: 48 63 47 08 48 01 c2 48 8b 07 48 85 c0 75 1b e9 2b 02 00 00 66 2e 0f 1f 84 00 00 00 00 00 48 8b 48 10 48 85 c9 74 17 48 89 c8 <48> 39 30 77 ef 0f 83 f9 01 00 00 48 8b 48 18 48 85 c9 75 e9 48 89 Nov 14 13:25:32 unraid kernel: RSP: 0018:ffffc9002408fd38 EFLAGS: 00010002 Nov 14 13:25:32 unraid kernel: RAX: 00102fbbc0e51940 RBX: ffffffffa1b2581e RCX: 00102fbbc0e51940 Nov 14 13:25:32 unraid kernel: RDX: ffffc9002408fdd0 RSI: 00000000000098eb RDI: ffffffffa4b30718 Nov 14 13:25:32 unraid kernel: RBP: ffff888857553000 R08: 0000000000000000 R09: ffffc9002408fdf8 Nov 14 13:25:32 unraid kernel: R10: ffffc9002408fe90 R11: 0000000000000003 R12: ffffc9002408fd80 Nov 14 13:25:32 unraid kernel: R13: ffff888235c38c00 R14: ffff888235c38c00 R15: ffff888106764808 Nov 14 13:25:32 unraid kernel: FS: 0000000000000000(0000) GS:ffff88900f980000(0000) knlGS:0000000000000000 Nov 14 13:25:32 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:25:32 unraid kernel: CR2: 0000000000000000 CR3: 000000000220a003 CR4: 00000000001706e0 Nov 14 13:25:32 unraid kernel: Call Trace: Nov 14 13:25:32 unraid kernel: <TASK> Nov 14 13:25:32 unraid kernel: ? __die_body+0x1a/0x5c Nov 14 13:25:32 unraid kernel: ? die_addr+0x38/0x51 Nov 14 13:25:32 unraid kernel: ? exc_general_protection+0x30f/0x345 Nov 14 13:25:32 unraid kernel: ? asm_exc_general_protection+0x22/0x30 Nov 14 13:25:32 unraid kernel: ? rm_cleanup_file_private+0x5e/0x200 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv013001rm+0x3c/0x310 [nvidia] Nov 14 13:25:32 unraid kernel: ? rm_cleanup_file_private+0x5e/0x200 [nvidia] Nov 14 13:25:32 unraid kernel: ? _nv052801rm+0xd6/0x1a0 [nvidia] Nov 14 13:25:32 unraid kernel: ? rm_cleanup_file_private+0x5e/0x200 [nvidia] Nov 14 13:25:32 unraid kernel: ? nvidia_close+0x9b/0x1ea [nvidia] Nov 14 13:25:32 unraid kernel: ? __fput+0x103/0x1d3 Nov 14 13:25:32 unraid kernel: ? task_work_run+0x6b/0x80 Nov 14 13:25:32 unraid kernel: ? do_exit+0x3b4/0x923 Nov 14 13:25:32 unraid kernel: ? make_task_dead+0x11c/0x11c Nov 14 13:25:32 unraid kernel: ? rewind_stack_and_make_dead+0x17/0x17 Nov 14 13:25:32 unraid kernel: </TASK> Nov 14 13:25:32 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 mgag200 sha1_ssse3 video Nov 14 13:25:32 unraid kernel: drm_shmem_helper aesni_intel drm_kms_helper crypto_simd cryptd rapl intel_cstate drm intel_uncore mpt3sas mei_me i2c_algo_bit backlight input_leds joydev led_class mei i2c_core syscopyarea raid_class sysfillrect sysimgblt scsi_transport_sas fb_sys_fops wmi ipmi_si acpi_power_meter button unix [last unloaded: xfrm_algo] Nov 14 13:25:32 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 14 13:25:32 unraid kernel: RIP: 0010:__cpa_process_fault+0x3ea/0x40e Nov 14 13:25:32 unraid kernel: Code: d0 48 c1 e8 0c 49 89 44 24 30 31 db eb 2b 49 8b 7c 24 30 e8 d4 f3 ff ff 84 c0 75 18 49 8b 04 24 48 89 ee 48 c7 c7 d7 1a 0c 82 <48> 8b 10 e8 16 c8 00 00 0f 0b bb f2 ff ff ff 48 83 c4 30 89 d8 5b Nov 14 13:25:32 unraid kernel: RSP: 0018:ffffc9002408f720 EFLAGS: 00010246 Nov 14 13:25:32 unraid kernel: RAX: 0000000000000000 RBX: ffffc9002408f890 RCX: 0000000080000000 Nov 14 13:25:32 unraid kernel: RDX: 0000000000001001 RSI: ffff1101761ff000 RDI: ffffffff820c1ad7 Nov 14 13:25:32 unraid kernel: RBP: ffff1101761ff000 R08: ffffc9002408f7de R09: ffffffff8220a110 Nov 14 13:25:32 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9002408f890 Nov 14 13:25:32 unraid kernel: R13: 800000016cfe3073 R14: 0000000000000000 R15: 0000000000000000 Nov 14 13:25:32 unraid kernel: FS: 0000000000000000(0000) GS:ffff88900f980000(0000) knlGS:0000000000000000 Nov 14 13:25:32 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:25:32 unraid kernel: CR2: 0000000000000000 CR3: 000000000220a003 CR4: 00000000001706e0 Nov 14 13:25:32 unraid kernel: note: nvidia-smi[39147] exited with irqs disabled Nov 14 13:25:32 unraid kernel: note: nvidia-smi[39147] exited with preempt_count 1 Nov 14 13:25:32 unraid kernel: Fixing recursive fault but reboot is needed! Nov 14 13:25:32 unraid kernel: BUG: using smp_processor_id() in preemptible [00000000] code: nvidia-smi/39147 Nov 14 13:25:32 unraid kernel: caller is __schedule+0x2f/0x612 Nov 14 13:25:32 unraid kernel: CPU: 13 PID: 39147 Comm: nvidia-smi Tainted: P D O 6.1.106-Unraid #1 Nov 14 13:25:32 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:25:32 unraid kernel: Call Trace: Nov 14 13:25:32 unraid kernel: <TASK> Nov 14 13:25:32 unraid kernel: dump_stack_lvl+0x44/0x5c Nov 14 13:25:32 unraid kernel: check_preemption_disabled+0xcb/0xe0 Nov 14 13:25:32 unraid kernel: __schedule+0x2f/0x612 Nov 14 13:25:32 unraid kernel: ? _printk+0x58/0x73 Nov 14 13:25:32 unraid kernel: do_task_dead+0x3f/0x41 Nov 14 13:25:32 unraid kernel: make_task_dead+0x114/0x11c Nov 14 13:25:32 unraid kernel: rewind_stack_and_make_dead+0x17/0x17 Nov 14 13:25:32 unraid kernel: RIP: 0033:0x14c7191935b1 Nov 14 13:25:32 unraid kernel: Code: Unable to access opcode bytes at 0x14c719193587. Nov 14 13:25:32 unraid kernel: RSP: 002b:00007fffd1acb798 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 Nov 14 13:25:32 unraid kernel: RAX: 0000000000000000 RBX: 00007fffd1acc300 RCX: 000014c7191935b1 Nov 14 13:25:32 unraid kernel: RDX: 000000000000ffff RSI: 0000000000000000 RDI: 0000000000000004 Nov 14 13:25:32 unraid kernel: RBP: 00007fffd1acc270 R08: 00007fffd1acb810 R09: 00007fffd1acb82c Nov 14 13:25:32 unraid kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014c718e2aec0 Nov 14 13:25:32 unraid kernel: R13: 000014c718e2b820 R14: 000014c718e2b820 R15: 000014c718e2b7c0 Nov 14 13:25:32 unraid kernel: </TASK> Nov 14 13:25:32 unraid kernel: BUG: scheduling while atomic: nvidia-smi/39147/0x00000000 Nov 14 13:25:32 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 mgag200 sha1_ssse3 video Nov 14 13:25:32 unraid kernel: drm_shmem_helper aesni_intel drm_kms_helper crypto_simd cryptd rapl intel_cstate drm intel_uncore mpt3sas mei_me i2c_algo_bit backlight input_leds joydev led_class mei i2c_core syscopyarea raid_class sysfillrect sysimgblt scsi_transport_sas fb_sys_fops wmi ipmi_si acpi_power_meter button unix [last unloaded: xfrm_algo] Nov 14 13:25:32 unraid kernel: Preemption disabled at: Nov 14 13:25:32 unraid kernel: [<0000000000000000>] 0x0 Nov 14 13:25:32 unraid kernel: CPU: 13 PID: 39147 Comm: nvidia-smi Tainted: P D O 6.1.106-Unraid #1 Nov 14 13:25:32 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:25:32 unraid kernel: Call Trace: Nov 14 13:25:32 unraid kernel: <TASK> Nov 14 13:25:32 unraid kernel: dump_stack_lvl+0x44/0x5c Nov 14 13:25:32 unraid kernel: __schedule_bug+0x9a/0xac Nov 14 13:25:32 unraid kernel: __schedule+0x59/0x612 Nov 14 13:25:32 unraid kernel: ? _printk+0x58/0x73 Nov 14 13:25:32 unraid kernel: do_task_dead+0x3f/0x41 Nov 14 13:25:32 unraid kernel: make_task_dead+0x114/0x11c Nov 14 13:25:32 unraid kernel: rewind_stack_and_make_dead+0x17/0x17 Nov 14 13:25:32 unraid kernel: RIP: 0033:0x14c7191935b1 Nov 14 13:25:32 unraid kernel: Code: Unable to access opcode bytes at 0x14c719193587. Nov 14 13:25:32 unraid kernel: RSP: 002b:00007fffd1acb798 EFLAGS: 00000246 ORIG_RAX: 0000000000000003 Nov 14 13:25:32 unraid kernel: RAX: 0000000000000000 RBX: 00007fffd1acc300 RCX: 000014c7191935b1 Nov 14 13:25:32 unraid kernel: RDX: 000000000000ffff RSI: 0000000000000000 RDI: 0000000000000004 Nov 14 13:25:32 unraid kernel: RBP: 00007fffd1acc270 R08: 00007fffd1acb810 R09: 00007fffd1acb82c Nov 14 13:25:32 unraid kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014c718e2aec0 Nov 14 13:25:32 unraid kernel: R13: 000014c718e2b820 R14: 000014c718e2b820 R15: 000014c718e2b7c0 Nov 14 13:25:32 unraid kernel: </TASK> Nov 14 13:27:18 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 14 13:27:18 unraid kernel: rcu: 14-...0: (1 GPs behind) idle=981c/1/0x4000000000000000 softirq=63315393/63315394 fqs=26589 Nov 14 13:27:18 unraid kernel: (detected by 20, t=60002 jiffies, g=156290629, q=3611433 ncpus=32) Nov 14 13:27:18 unraid kernel: Sending NMI from CPU 20 to CPUs 14: Nov 14 13:27:18 unraid kernel: NMI backtrace for cpu 14 Nov 14 13:27:18 unraid kernel: CPU: 14 PID: 41201 Comm: nvidia-smi Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:27:18 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:27:18 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 14 13:27:18 unraid kernel: Code: 0f ba 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a <8b> 03 84 c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea Nov 14 13:27:18 unraid kernel: RSP: 0018:ffffc9000aabfd38 EFLAGS: 00000002 Nov 14 13:27:18 unraid kernel: RAX: 0000000000000101 RBX: ffff888105784488 RCX: fffffffdda3e82fd Nov 14 13:27:18 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105784488 Nov 14 13:27:18 unraid kernel: RBP: ffff88885dad6000 R08: 0000000000000000 R09: ffffc9000aabfe38 Nov 14 13:27:18 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000aabfdc0 Nov 14 13:27:18 unraid kernel: R13: ffff88813ab87960 R14: 0000000000000287 R15: 000000000000001f Nov 14 13:27:18 unraid kernel: FS: 000015104f8c41c0(0000) GS:ffff88880f9c0000(0000) knlGS:0000000000000000 Nov 14 13:27:18 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:27:18 unraid kernel: CR2: 000015104f84c820 CR3: 0000000862624001 CR4: 00000000001706e0 Nov 14 13:27:18 unraid kernel: Call Trace: Nov 14 13:27:18 unraid kernel: <NMI> Nov 14 13:27:18 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 14 13:27:18 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 14 13:27:18 unraid kernel: ? nmi_handle+0x57/0x131 Nov 14 13:27:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 14 13:27:18 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 14 13:27:18 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 14 13:27:18 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 14 13:27:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 14 13:27:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 14 13:27:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x7e/0x1cf Nov 14 13:27:18 unraid kernel: </NMI> Nov 14 13:27:18 unraid kernel: <TASK> Nov 14 13:27:18 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 14 13:27:18 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 14 13:27:18 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 14 13:27:18 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 14 13:27:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:27:18 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 14 13:27:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:27:18 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 14 13:27:18 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:27:18 unraid kernel: ? do_fcntl+0x19a/0x569 Nov 14 13:27:18 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 14 13:27:18 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 14 13:27:18 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 14 13:27:18 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:27:18 unraid kernel: </TASK> Nov 14 13:27:18 unraid kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.060 msecs Nov 14 13:27:38 unraid kernel: ------------[ cut here ]------------ Nov 14 13:27:38 unraid kernel: NETDEV WATCHDOG: eth0 (igb): transmit queue 1 timed out Nov 14 13:27:38 unraid kernel: WARNING: CPU: 10 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x14e/0x1c0 Nov 14 13:27:38 unraid kernel: Modules linked in: xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xt_addrtype br_netfilter nvidia_uvm(PO) xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls ixgbe xfrm_algo mdio igb nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 mgag200 sha1_ssse3 video Nov 14 13:27:38 unraid kernel: drm_shmem_helper aesni_intel drm_kms_helper crypto_simd cryptd rapl intel_cstate drm intel_uncore mpt3sas mei_me i2c_algo_bit backlight input_leds joydev led_class mei i2c_core syscopyarea raid_class sysfillrect sysimgblt scsi_transport_sas fb_sys_fops wmi ipmi_si acpi_power_meter button unix [last unloaded: xfrm_algo] Nov 14 13:27:38 unraid kernel: CPU: 10 PID: 0 Comm: swapper/10 Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:27:38 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:27:38 unraid kernel: RIP: 0010:dev_watchdog+0x14e/0x1c0 Nov 14 13:27:38 unraid kernel: Code: 40 c5 00 00 75 26 48 89 ef c6 05 cc 40 c5 00 01 e8 36 22 fc ff 44 89 f1 48 89 ee 48 c7 c7 fa a1 15 82 48 89 c2 e8 94 2e 93 ff <0f> 0b 48 89 ef e8 32 fb ff ff 48 8b 83 88 fc ff ff 48 89 ef 44 89 Nov 14 13:27:38 unraid kernel: RSP: 0018:ffffc90006688ea8 EFLAGS: 00010282 Nov 14 13:27:38 unraid kernel: RAX: 0000000000000000 RBX: ffff888150140448 RCX: 0000000000000027 Nov 14 13:27:38 unraid kernel: RDX: 0000000000000104 RSI: ffffffff820d99e1 RDI: 00000000ffffffff Nov 14 13:27:38 unraid kernel: RBP: ffff888150140000 R08: 000000000000000b R09: 0000000000000031 Nov 14 13:27:38 unraid kernel: R10: 0000000000000000 R11: ffff88902ff9bc87 R12: 0000000000000001 Nov 14 13:27:38 unraid kernel: R13: ffff88815014039c R14: 0000000000000001 R15: 0000000000000008 Nov 14 13:27:38 unraid kernel: FS: 0000000000000000(0000) GS:ffff88880f940000(0000) knlGS:0000000000000000 Nov 14 13:27:38 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:27:38 unraid kernel: CR2: 00000b7da87b6000 CR3: 000000000220a004 CR4: 00000000001706e0 Nov 14 13:27:38 unraid kernel: Call Trace: Nov 14 13:27:38 unraid kernel: <IRQ> Nov 14 13:27:38 unraid kernel: ? __warn+0xab/0x122 Nov 14 13:27:38 unraid kernel: ? report_bug+0x109/0x17e Nov 14 13:27:38 unraid kernel: ? dev_watchdog+0x14e/0x1c0 Nov 14 13:27:38 unraid kernel: ? handle_bug+0x41/0x6f Nov 14 13:27:38 unraid kernel: ? exc_invalid_op+0x13/0x60 Nov 14 13:27:38 unraid kernel: ? asm_exc_invalid_op+0x16/0x20 Nov 14 13:27:38 unraid kernel: ? dev_watchdog+0x14e/0x1c0 Nov 14 13:27:38 unraid kernel: ? dev_watchdog+0x14e/0x1c0 Nov 14 13:27:38 unraid kernel: ? psched_ppscfg_precompute+0x57/0x57 Nov 14 13:27:38 unraid kernel: ? psched_ppscfg_precompute+0x57/0x57 Nov 14 13:27:38 unraid kernel: call_timer_fn+0x6f/0x10d Nov 14 13:27:38 unraid kernel: __run_timers+0x144/0x184 Nov 14 13:27:38 unraid kernel: ? hrtimer_forward+0x82/0x92 Nov 14 13:27:38 unraid kernel: ? timerqueue_add+0x6a/0x76 Nov 14 13:27:38 unraid kernel: ? enqueue_hrtimer+0x7f/0x8a Nov 14 13:27:38 unraid kernel: ? __hrtimer_next_event_base+0x27/0x81 Nov 14 13:27:38 unraid kernel: run_timer_softirq+0x2b/0x43 Nov 14 13:27:38 unraid kernel: handle_softirqs+0x12c/0x271 Nov 14 13:27:38 unraid kernel: __irq_exit_rcu+0x5e/0xb8 Nov 14 13:27:38 unraid kernel: sysvec_apic_timer_interrupt+0x85/0xa6 Nov 14 13:27:38 unraid kernel: </IRQ> Nov 14 13:27:38 unraid kernel: <TASK> Nov 14 13:27:38 unraid kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20 Nov 14 13:27:38 unraid kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202 Nov 14 13:27:38 unraid kernel: Code: 86 c1 9f ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 0a 7b a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29 Nov 14 13:27:38 unraid kernel: RSP: 0018:ffffc90004347e98 EFLAGS: 00000246 Nov 14 13:27:38 unraid kernel: RAX: ffff88880f940000 RBX: ffff88880f977738 RCX: 0000000000000000 Nov 14 13:27:38 unraid kernel: RDX: 0000eddc13c9e4b6 RSI: ffffffff820d99e1 RDI: ffffffff820d9eea Nov 14 13:27:38 unraid kernel: RBP: 0000000000000004 R08: 0000000000000002 R09: 0000000000000002 Nov 14 13:27:38 unraid kernel: R10: 0000000000000020 R11: 00000000000002f4 R12: 0000000000000004 Nov 14 13:27:38 unraid kernel: R13: ffffffff82321760 R14: 0000eddc13c9e4b6 R15: 0000000000000000 Nov 14 13:27:38 unraid kernel: ? cpuidle_enter_state+0xf7/0x202 Nov 14 13:27:38 unraid kernel: cpuidle_enter+0x2a/0x38 Nov 14 13:27:38 unraid kernel: do_idle+0x18d/0x1fb Nov 14 13:27:38 unraid kernel: cpu_startup_entry+0x2a/0x2c Nov 14 13:27:38 unraid kernel: start_secondary+0x101/0x101 Nov 14 13:27:38 unraid kernel: secondary_startup_64_no_verify+0xce/0xdb Nov 14 13:27:38 unraid kernel: </TASK> Nov 14 13:27:38 unraid kernel: ---[ end trace 0000000000000000 ]--- Nov 14 13:27:38 unraid kernel: igb 0000:07:00.0 eth0: Reset adapter Nov 14 13:27:40 unraid kernel: igb 0000:07:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Nov 14 13:30:18 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 14 13:30:18 unraid kernel: rcu: 14-...0: (1 GPs behind) idle=981c/1/0x4000000000000000 softirq=63315393/63315394 fqs=108157 Nov 14 13:30:18 unraid kernel: (detected by 21, t=240010 jiffies, g=156290629, q=12770423 ncpus=32) Nov 14 13:30:18 unraid kernel: Sending NMI from CPU 21 to CPUs 14: Nov 14 13:30:18 unraid kernel: NMI backtrace for cpu 14 Nov 14 13:30:18 unraid kernel: CPU: 14 PID: 41201 Comm: nvidia-smi Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:30:18 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:30:18 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:30:18 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 14 13:30:18 unraid kernel: RSP: 0018:ffffc9000aabfd38 EFLAGS: 00000002 Nov 14 13:30:18 unraid kernel: RAX: 0000000000000101 RBX: ffff888105784488 RCX: fffffffdda3e82fd Nov 14 13:30:18 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105784488 Nov 14 13:30:18 unraid kernel: RBP: ffff88885dad6000 R08: 0000000000000000 R09: ffffc9000aabfe38 Nov 14 13:30:18 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000aabfdc0 Nov 14 13:30:18 unraid kernel: R13: ffff88813ab87960 R14: 0000000000000287 R15: 000000000000001f Nov 14 13:30:18 unraid kernel: FS: 000015104f8c41c0(0000) GS:ffff88880f9c0000(0000) knlGS:0000000000000000 Nov 14 13:30:18 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:30:18 unraid kernel: CR2: 000015104f84c820 CR3: 0000000862624001 CR4: 00000000001706e0 Nov 14 13:30:18 unraid kernel: Call Trace: Nov 14 13:30:18 unraid kernel: <NMI> Nov 14 13:30:18 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 14 13:30:18 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 14 13:30:18 unraid kernel: ? nmi_handle+0x57/0x131 Nov 14 13:30:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:30:18 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 14 13:30:18 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 14 13:30:18 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 14 13:30:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:30:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:30:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:30:18 unraid kernel: </NMI> Nov 14 13:30:18 unraid kernel: <TASK> Nov 14 13:30:18 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 14 13:30:18 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 14 13:30:18 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 14 13:30:18 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 14 13:30:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:30:18 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 14 13:30:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:30:18 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 14 13:30:18 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:30:18 unraid kernel: ? do_fcntl+0x19a/0x569 Nov 14 13:30:18 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 14 13:30:18 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 14 13:30:18 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 14 13:30:18 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:30:18 unraid kernel: </TASK> Nov 14 13:30:18 unraid kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 2.102 msecs Nov 14 13:33:18 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 14 13:33:18 unraid kernel: rcu: 14-...0: (1 GPs behind) idle=981c/1/0x4000000000000000 softirq=63315393/63315394 fqs=193527 Nov 14 13:33:18 unraid kernel: (detected by 18, t=420018 jiffies, g=156290629, q=21713157 ncpus=32) Nov 14 13:33:18 unraid kernel: Sending NMI from CPU 18 to CPUs 14: Nov 14 13:33:18 unraid kernel: NMI backtrace for cpu 14 Nov 14 13:33:18 unraid kernel: CPU: 14 PID: 41201 Comm: nvidia-smi Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:33:18 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:33:18 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:33:18 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 14 13:33:18 unraid kernel: RSP: 0018:ffffc9000aabfd38 EFLAGS: 00000002 Nov 14 13:33:18 unraid kernel: RAX: 0000000000000101 RBX: ffff888105784488 RCX: fffffffdda3e82fd Nov 14 13:33:18 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105784488 Nov 14 13:33:18 unraid kernel: RBP: ffff88885dad6000 R08: 0000000000000000 R09: ffffc9000aabfe38 Nov 14 13:33:18 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000aabfdc0 Nov 14 13:33:18 unraid kernel: R13: ffff88813ab87960 R14: 0000000000000287 R15: 000000000000001f Nov 14 13:33:18 unraid kernel: FS: 000015104f8c41c0(0000) GS:ffff88880f9c0000(0000) knlGS:0000000000000000 Nov 14 13:33:18 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:33:18 unraid kernel: CR2: 000015104f84c820 CR3: 0000000862624001 CR4: 00000000001706e0 Nov 14 13:33:18 unraid kernel: Call Trace: Nov 14 13:33:18 unraid kernel: <NMI> Nov 14 13:33:18 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 14 13:33:18 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 14 13:33:18 unraid kernel: ? nmi_handle+0x57/0x131 Nov 14 13:33:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:33:18 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 14 13:33:18 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 14 13:33:18 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 14 13:33:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:33:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:33:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:33:18 unraid kernel: </NMI> Nov 14 13:33:18 unraid kernel: <TASK> Nov 14 13:33:18 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 14 13:33:18 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 14 13:33:18 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 14 13:33:18 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 14 13:33:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:33:18 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 14 13:33:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:33:18 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 14 13:33:18 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:33:18 unraid kernel: ? do_fcntl+0x19a/0x569 Nov 14 13:33:18 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 14 13:33:18 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 14 13:33:18 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 14 13:33:18 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:33:18 unraid kernel: </TASK> Nov 14 13:36:18 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 14 13:36:18 unraid kernel: rcu: 14-...0: (1 GPs behind) idle=981c/1/0x4000000000000000 softirq=63315393/63315394 fqs=279411 Nov 14 13:36:18 unraid kernel: (detected by 13, t=600026 jiffies, g=156290629, q=30782558 ncpus=32) Nov 14 13:36:18 unraid kernel: Sending NMI from CPU 13 to CPUs 14: Nov 14 13:36:18 unraid kernel: NMI backtrace for cpu 14 Nov 14 13:36:18 unraid kernel: CPU: 14 PID: 41201 Comm: nvidia-smi Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:36:18 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:36:18 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:36:18 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 14 13:36:18 unraid kernel: RSP: 0018:ffffc9000aabfd38 EFLAGS: 00000002 Nov 14 13:36:18 unraid kernel: RAX: 0000000000000101 RBX: ffff888105784488 RCX: fffffffdda3e82fd Nov 14 13:36:18 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105784488 Nov 14 13:36:18 unraid kernel: RBP: ffff88885dad6000 R08: 0000000000000000 R09: ffffc9000aabfe38 Nov 14 13:36:18 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000aabfdc0 Nov 14 13:36:18 unraid kernel: R13: ffff88813ab87960 R14: 0000000000000287 R15: 000000000000001f Nov 14 13:36:18 unraid kernel: FS: 000015104f8c41c0(0000) GS:ffff88880f9c0000(0000) knlGS:0000000000000000 Nov 14 13:36:18 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:36:18 unraid kernel: CR2: 000015104f84c820 CR3: 0000000862624001 CR4: 00000000001706e0 Nov 14 13:36:18 unraid kernel: Call Trace: Nov 14 13:36:18 unraid kernel: <NMI> Nov 14 13:36:18 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 14 13:36:18 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 14 13:36:18 unraid kernel: ? nmi_handle+0x57/0x131 Nov 14 13:36:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:36:18 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 14 13:36:18 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 14 13:36:18 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 14 13:36:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:36:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:36:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:36:18 unraid kernel: </NMI> Nov 14 13:36:18 unraid kernel: <TASK> Nov 14 13:36:18 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 14 13:36:18 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 14 13:36:18 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 14 13:36:18 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 14 13:36:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:36:18 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 14 13:36:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:36:18 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 14 13:36:18 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:36:18 unraid kernel: ? do_fcntl+0x19a/0x569 Nov 14 13:36:18 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 14 13:36:18 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 14 13:36:18 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 14 13:36:18 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:36:18 unraid kernel: </TASK> Nov 14 13:36:58 unraid nginx: 2024/11/14 13:36:58 [error] 8403#8403: *1461024 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.10.10.175, server: , request: "GET /plugins/gpustat/gpustatusmulti.php?gpus={%2204:00.0%22:{%22id%22:%2204:00.0%22,%22model%22:%22NVIDIA%20RTX%20A4000%22,%22vendor%22:%22nvidia%22,%22guid%22:%22GPU-2ce633a4-69cd-f2d3-9ab1-245a7ee339ea%22,%22panel%22:1}} HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "10.10.10.10", referrer: "http://10.10.10.10/Dashboard" Nov 14 13:39:18 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 14 13:39:18 unraid kernel: rcu: 14-...0: (1 GPs behind) idle=981c/1/0x4000000000000000 softirq=63315393/63315394 fqs=358909 Nov 14 13:39:18 unraid kernel: (detected by 18, t=780034 jiffies, g=156290629, q=39727575 ncpus=32) Nov 14 13:39:18 unraid kernel: Sending NMI from CPU 18 to CPUs 14: Nov 14 13:39:18 unraid kernel: NMI backtrace for cpu 14 Nov 14 13:39:18 unraid kernel: CPU: 14 PID: 41201 Comm: nvidia-smi Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:39:18 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:39:18 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x80/0x1cf Nov 14 13:39:18 unraid kernel: Code: 2b 08 8b 03 0f 92 c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 <84> c0 74 04 f3 90 eb f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff Nov 14 13:39:18 unraid kernel: RSP: 0018:ffffc9000aabfd38 EFLAGS: 00000002 Nov 14 13:39:18 unraid kernel: RAX: 0000000000000101 RBX: ffff888105784488 RCX: fffffffdda3e82fd Nov 14 13:39:18 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105784488 Nov 14 13:39:18 unraid kernel: RBP: ffff88885dad6000 R08: 0000000000000000 R09: ffffc9000aabfe38 Nov 14 13:39:18 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000aabfdc0 Nov 14 13:39:18 unraid kernel: R13: ffff88813ab87960 R14: 0000000000000287 R15: 000000000000001f Nov 14 13:39:18 unraid kernel: FS: 000015104f8c41c0(0000) GS:ffff88880f9c0000(0000) knlGS:0000000000000000 Nov 14 13:39:18 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:39:18 unraid kernel: CR2: 000015104f84c820 CR3: 0000000862624001 CR4: 00000000001706e0 Nov 14 13:39:18 unraid kernel: Call Trace: Nov 14 13:39:18 unraid kernel: <NMI> Nov 14 13:39:18 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 14 13:39:18 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 14 13:39:18 unraid kernel: ? nmi_handle+0x57/0x131 Nov 14 13:39:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 14 13:39:18 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 14 13:39:18 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 14 13:39:18 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 14 13:39:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 14 13:39:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 14 13:39:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x80/0x1cf Nov 14 13:39:18 unraid kernel: </NMI> Nov 14 13:39:18 unraid kernel: <TASK> Nov 14 13:39:18 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 14 13:39:18 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 14 13:39:18 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 14 13:39:18 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 14 13:39:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:39:18 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 14 13:39:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:39:18 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 14 13:39:18 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:39:18 unraid kernel: ? do_fcntl+0x19a/0x569 Nov 14 13:39:18 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 14 13:39:18 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 14 13:39:18 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 14 13:39:18 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:39:18 unraid kernel: </TASK> Nov 14 13:42:18 unraid kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Nov 14 13:42:18 unraid kernel: rcu: 14-...0: (1 GPs behind) idle=981c/1/0x4000000000000000 softirq=63315393/63315394 fqs=436062 Nov 14 13:42:18 unraid kernel: (detected by 7, t=960042 jiffies, g=156290629, q=49119696 ncpus=32) Nov 14 13:42:18 unraid kernel: Sending NMI from CPU 7 to CPUs 14: Nov 14 13:42:18 unraid kernel: NMI backtrace for cpu 14 Nov 14 13:42:18 unraid kernel: CPU: 14 PID: 41201 Comm: nvidia-smi Tainted: P D W O 6.1.106-Unraid #1 Nov 14 13:42:18 unraid kernel: Hardware name: Dell Inc. PowerEdge R720xd/0X3D66, BIOS 2.9.0 12/06/2019 Nov 14 13:42:18 unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:42:18 unraid kernel: Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 ea 3b ff ff 49 c7 c4 80 ed Nov 14 13:42:18 unraid kernel: RSP: 0018:ffffc9000aabfd38 EFLAGS: 00000002 Nov 14 13:42:18 unraid kernel: RAX: 0000000000000101 RBX: ffff888105784488 RCX: fffffffdda3e82fd Nov 14 13:42:18 unraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888105784488 Nov 14 13:42:18 unraid kernel: RBP: ffff88885dad6000 R08: 0000000000000000 R09: ffffc9000aabfe38 Nov 14 13:42:18 unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000aabfdc0 Nov 14 13:42:18 unraid kernel: R13: ffff88813ab87960 R14: 0000000000000287 R15: 000000000000001f Nov 14 13:42:18 unraid kernel: FS: 000015104f8c41c0(0000) GS:ffff88880f9c0000(0000) knlGS:0000000000000000 Nov 14 13:42:18 unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 14 13:42:18 unraid kernel: CR2: 000015104f84c820 CR3: 0000000862624001 CR4: 00000000001706e0 Nov 14 13:42:18 unraid kernel: Call Trace: Nov 14 13:42:18 unraid kernel: <NMI> Nov 14 13:42:18 unraid kernel: ? nmi_cpu_backtrace+0xd3/0x104 Nov 14 13:42:18 unraid kernel: ? nmi_cpu_backtrace_handler+0xd/0x15 Nov 14 13:42:18 unraid kernel: ? nmi_handle+0x57/0x131 Nov 14 13:42:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:42:18 unraid kernel: ? default_do_nmi+0x66/0x15b Nov 14 13:42:18 unraid kernel: ? exc_nmi+0xbf/0x11c Nov 14 13:42:18 unraid kernel: ? end_repeat_nmi+0x16/0x67 Nov 14 13:42:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:42:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:42:18 unraid kernel: ? native_queued_spin_lock_slowpath+0x86/0x1cf Nov 14 13:42:18 unraid kernel: </NMI> Nov 14 13:42:18 unraid kernel: <TASK> Nov 14 13:42:18 unraid kernel: do_raw_spin_lock+0x14/0x1a Nov 14 13:42:18 unraid kernel: _raw_spin_lock_irqsave+0x2c/0x37 Nov 14 13:42:18 unraid kernel: os_acquire_spinlock+0xe/0x1d [nvidia] Nov 14 13:42:18 unraid kernel: _nv046702rm+0x10/0x20 [nvidia] Nov 14 13:42:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:42:18 unraid kernel: ? _nv052801rm+0xc3/0x1a0 [nvidia] Nov 14 13:42:18 unraid kernel: ? rm_perform_version_check+0x37/0x150 [nvidia] Nov 14 13:42:18 unraid kernel: ? nvidia_unlocked_ioctl+0x4b1/0x6c2 [nvidia] Nov 14 13:42:18 unraid kernel: ? _raw_spin_unlock+0x14/0x29 Nov 14 13:42:18 unraid kernel: ? do_fcntl+0x19a/0x569 Nov 14 13:42:18 unraid kernel: ? vfs_ioctl+0x1e/0x2f Nov 14 13:42:18 unraid kernel: ? __do_sys_ioctl+0x52/0x78 Nov 14 13:42:18 unraid kernel: ? do_syscall_64+0x65/0x7b Nov 14 13:42:18 unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Nov 14 13:42:18 unraid kernel: </TASK> Nov 14 13:43:12 unraid shutdown[23788]: shutting down for system halt

  8. Unraid experiencing random shutdowns - cant diagnose

    risenfromthedust posted a topic in General Support
    My main unraid server has been experiencing random shutdowns for the past few weeks. Without any known symtoms, the server would randomly become unavailable. At first i thought something was triggering a shutdown, like my ups not having power, but i dont think thats it. I have shut down my server during extended power outages, and the network closet goes quiet. But this leaves the server unavailable but the fans spinning in a low or standby state. I have tried checking physical connections, and swapping primary and secondary psus (on different ups) a couple times. I was worried that my power and/or ups were failing to give power and it was triggering either a shutdown or hard power off. But again, when its unavailable the fans are still running. This requires me to either hit the power switch, or go into IDRAC and trigger a cold boot (restart). Then the jets kick off and it reboots. This will get the server running, and usually trigger a parity check. Most of the time the check finishes (takes roughly 24 hours) but once or twice it hasnt. Luckily I havent lost data yet, but Id like to figure this out before I do. I tried digging into the logs but as they tend to reset when the server boots, i wasnt getting enough data. I turned on the syslog and i got logs up until it seems to go down now. It just happened again 20 minutes prior to me typing this. I really need help with this one. Am I ok to attach my syslog or will it have too much private data (IPs and such)? Is there a better log? What else is needed for help?

  9. Unraid startup failure - cannot get to GUI

    risenfromthedust replied to risenfromthedust's topic in General Support
    Recreating the docker img seems to have resolved my issues. While still in safemode, I started the array, went to Settings>Docker, turned Docker off, deleted the 'vDisk', then turned Docker back on. Then I rebooted back to regular mode. From there I reinstalled my apps from 'Previous apps' and all appears to be working. Thanks

  10. Unraid startup failure - cannot get to GUI

    risenfromthedust replied to risenfromthedust's topic in General Support
    I rebooted using 'powerdown -r' and via iDRAC, chose to boot into safemode. Now I can get into the webGUI and click around. Of course the array isnt started and no plungins are available. But the basics are working. What do I need to do from here to hopefully not lose my data and fix my server?
  11. risenfromthedust started following Unraid startup failure - cannot get to GUI

  12. Unraid startup failure - cannot get to GUI

    risenfromthedust posted a topic in General Support
    Everything was known working last night. This morning, I noticed a couple of my docker apps werent responding so I logged on my server. On the Dashboard, I saw that several docker apps had question mark logos, instead of their respective logos. When I tried to go to the Docker page, I got a message that docker failed to start. I then went to Settings>Docker to see if i could restart the service. I changed the yes to no and back, and then hit apply. The page reloaded, but still said service was stopped. Then I went to tools to look at the syslog. I saw the following error at the bottom: Jan 24 08:06:11 unraid kernel: lo_write_bvec: 29 callbacks suppressed Jan 24 08:06:11 unraid kernel: loop: Write error at byte offset 7246594048, length 4096. Jan 24 08:06:11 unraid kernel: blk_print_req_error: 29 callbacks suppressed Jan 24 08:06:11 unraid kernel: I/O error, dev loop2, sector 14153504 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2 Jan 24 08:06:11 unraid kernel: btrfs_dev_stat_inc_and_print: 29 callbacks suppressed Jan 24 08:06:11 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 904, rd 0, flush 0, corrupt 0, gen 0 Jan 24 08:06:11 unraid kernel: loop: Write error at byte offset 7515029504, length 4096. Jan 24 08:06:11 unraid kernel: I/O error, dev loop2, sector 14677792 op 0x1:(WRITE) flags 0x1800 phys_seg 1 prio class 2 Jan 24 08:06:11 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 905, rd 0, flush 0, corrupt 0, gen 0 Jan 24 08:06:12 unraid kernel: loop: Write error at byte offset 3662774272, length 4096. Jan 24 08:06:12 unraid kernel: I/O error, dev loop2, sector 7153856 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2 Jan 24 08:06:12 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 906, rd 0, flush 0, corrupt 0, gen 0 Jan 24 08:06:12 unraid kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2494: errno=-5 IO failure (Error while writing out transaction) Jan 24 08:06:12 unraid kernel: BTRFS info (device loop2: state E): forced readonly Jan 24 08:06:12 unraid kernel: BTRFS warning (device loop2: state E): Skipping commit of aborted transaction. Jan 24 08:06:12 unraid kernel: BTRFS: error (device loop2: state EA) in cleanup_transaction:1992: errno=-5 IO failure My next idea was the 'the old standby' - turn it off and on. I went to 'Main' and told it to reboot. Note: at this point I could still navigate around the webGUI, the array was still showing green and no errors on any drives. After reboot, the login page showed and I attempted to login. The login page hung and never loaded. Eventually I got this: When I tried to ssh into the box, I got the login, but after typing in pw, it never dumped me at the prompt. Luckily I'm running on an old Dell r720xd, so I logged into iDRAC, and pulled up the console. I was able to login again, and again I wasnt dumped at the prompt. Frustrated I hit <CTRL> +C and that dumped me at the prompt. Something caught my eye. Normally when I ssh into the box, the username@servername part before the command prompt is usually green, but it stayed the black and white prompt colors, so already I knew somethings amiss. I went to /var/log/syslog and found more of the error above. I also saw my failure to load the 'Main' page after logging in: Jan 24 08:40:08 unraid nginx: 2024/01/24 08:40:08 [error] 8620#8620: *3494 upstream timed out (110: Connection timed out) while reading response header from upstream, client: [MY LAPTOP IP], server: , request: "GET /Main HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "[MY SERVER IP]" I figured I was at a loss. I was seeing errors in my logs and I had no way to mess with the GUI. I knew I needed help, so I googled how to get a diagnostics file from command line and ran the command ("diagnostics" is the command, in case someone sees this). When I went to the location of the file, I saw that the system had auto generated two more, likely from failures at reboot. I am including all 3 just in case. The latest file is the one I generated. unraid-diagnostics-20240124-0743.zip unraid-diagnostics-20240124-0759.zip unraid-diagnostics-20240124-0855.zip

Account

Navigation

Search

Search

Configure browser push notifications
Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.