I have recently been having more system hangs. I have always had them to an extent, but they would happen so sporadically (nothing for 6-9 months, then 3 at once).

I have usually fixed it by getting a new Unraid boot USB. I have been using SanDisk ultras for a long time and have probably gone through 4 or 5 in about 3 years. I have tried the Samsung metal USB that SpaceInvader (I think) recommended but that didn't seem to last long at all. I have also in the last 6 months started using a USB extender for the boot USB, and that seems to keep temps down on the drive itself.

Any time I have hangs, I have run a parity check, extended SMART checks on the drives, and a memtest (takes forever). I do have a 13th Gen Intel which I know can have issues, I have updated my bios with the microcode patches, so not sure what else I can do there.

About a month ago, I had a system hang with all my dockers running and 2 VMs. It took a long time to get up and running, including replacing the USB. I have always struggled to diagnose the issue other than it being a corrupted/dying boot USB, as due to having to perform a hard reset and having long periods between the issue, I haven't been able to capture a system log (mirror to flash wearing out boot USB over long periods). This system hang corrupted my VM in some way, and I had to rebuild, the dockers weren't so bad, I just had to add them all again.

However, today, I went to start up my Windows VM (I have started putting it in hibernate when not using it to avoid it getting corrupted on a system hang), and it didn't finish booting. The web UI was not responding but SSH-ing in still worked. I have managed to view the system logs and generate diagnostics.

The main thing in the syslog that stood out to me is at the bottom, but I don't really know what it means. Something tainted.

I would really appreciate it if someone was able to take a look at this and advise on somethings I can test or settings I can change.

Jan 20 19:41:55 Tower kernel: ------------[ cut here ]------------
Jan 20 19:41:55 Tower kernel: kernel BUG at arch/x86/kvm/mmu/tdp_mmu.c:475!
Jan 20 19:41:55 Tower kernel: Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Jan 20 19:41:55 Tower kernel: CPU: 6 UID: 0 PID: 1985113 Comm: CPU 2/KVM Tainted: P           O       6.12.54-Unraid #1
Jan 20 19:41:55 Tower kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Jan 20 19:41:55 Tower kernel: Hardware name: ASUS System Product Name/ROG STRIX Z790-F GAMING WIFI, BIOS 3107 11/28/2025
Jan 20 19:41:55 Tower kernel: RIP: 0010:handle_changed_spte+0x11a/0x470 [kvm]
Jan 20 19:41:55 Tower kernel: Code: 80 e2 01 74 27 80 7c 24 2e 00 74 20 48 8b 54 24 10 45 89 e1 49 89 d8 48 89 e9 8b 74 24 28 48 c7 c7 b6 1b ae a0 e8 06 f3 10 e1 <0f> 0b 48 39 dd 0f 84 34 03 00 00 48 8b 74 24 10 49 89 d8 48 89 e9
Jan 20 19:41:55 Tower kernel: RSP: 0018:ffffc90035d9bba0 EFLAGS: 00010246
Jan 20 19:41:55 Tower kernel: RAX: 00000000000000c8 RBX: 8000000000000000 RCX: 0000000000000002
Jan 20 19:41:55 Tower kernel: RDX: 0000000000000000 RSI: ffffffff82365ea5 RDI: 00000000ffffffff
Jan 20 19:41:55 Tower kernel: RBP: 86000001de862b77 R08: 0000000000000000 R09: 0000000000000000
Jan 20 19:41:55 Tower kernel: R10: 00000000000000c8 R11: 0000000000000000 R12: 0000000000000001
Jan 20 19:41:55 Tower kernel: R13: ffffc9000f6b5000 R14: 0000000000000001 R15: 0000000000000001
Jan 20 19:41:55 Tower kernel: FS:  0000147a31dff6c0(0000) GS:ffff88903f180000(0000) knlGS:0000000000000000
Jan 20 19:41:55 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 20 19:41:55 Tower kernel: CR2: 0000147a1c169000 CR3: 000000012b3f6001 CR4: 0000000000772ef0
Jan 20 19:41:55 Tower kernel: PKRU: 55555554
Jan 20 19:41:55 Tower kernel: Call Trace:
Jan 20 19:41:55 Tower kernel: <TASK>
Jan 20 19:41:55 Tower kernel: ? rcu_read_unlock_sched+0xa/0x20 [kvm]
Jan 20 19:41:55 Tower kernel: tdp_mmu_set_spte_atomic+0x3c/0x50 [kvm]
Jan 20 19:41:55 Tower kernel: __tdp_mmu_zap_root+0xa6/0xe0 [kvm]
Jan 20 19:41:55 Tower kernel: ? kvm_kick_many_cpus+0x18/0x40 [kvm]
Jan 20 19:41:55 Tower kernel: tdp_mmu_zap_root+0x35/0x80 [kvm]
Jan 20 19:41:55 Tower kernel: kvm_tdp_mmu_zap_invalidated_roots+0x64/0x80 [kvm]
Jan 20 19:41:55 Tower kernel: kvm_set_memslot+0x10f/0x3a0 [kvm]
Jan 20 19:41:55 Tower kernel: kvm_set_memory_region+0x23/0x40 [kvm]
Jan 20 19:41:55 Tower kernel: kvm_vm_ioctl+0x755/0x10b0 [kvm]
Jan 20 19:41:55 Tower kernel: ? ttwu_queue_wakelist+0xaa/0xe0
Jan 20 19:41:55 Tower kernel: ? rawspin_unlock_irqrestore+0x1f/0x40
Jan 20 19:41:55 Tower kernel: ? class_preempt_destructor.isra.0+0xf/0x30
Jan 20 19:41:55 Tower kernel: ? try_to_wake_up+0x2f2/0x320
Jan 20 19:41:55 Tower kernel: vfs_ioctl+0x20/0x40
Jan 20 19:41:55 Tower kernel: __do_sys_ioctl+0x53/0x70
Jan 20 19:41:55 Tower kernel: do_syscall_64+0x68/0xe0
Jan 20 19:41:55 Tower kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Jan 20 19:41:55 Tower kernel: RIP: 0033:0x148038ccd5b8
Jan 20 19:41:55 Tower kernel: Code: 00 00 48 8d 44 24 08 48 89 54 24 e0 48 89 44 24 c0 48 8d 44 24 d0 48 89 44 24 c8 b8 10 00 00 00 c7 44 24 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 07 89 d0 c3 0f 1f 40 00 48 8b 15 f9 d7 0d
Jan 20 19:41:55 Tower kernel: RSP: 002b:0000147a31dfe358 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jan 20 19:41:55 Tower kernel: RAX: ffffffffffffffda RBX: 000000004020ae46 RCX: 0000148038ccd5b8
Jan 20 19:41:55 Tower kernel: RDX: 0000147a31dfe4b0 RSI: 000000004020ae46 RDI: 0000000000000012
Jan 20 19:41:55 Tower kernel: RBP: 0000147a31dfe4b0 R08: 0000000000000006 R09: 0000000000000000
Jan 20 19:41:55 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000564881588300
Jan 20 19:41:55 Tower kernel: R13: 0000000000100000 R14: 00005648815883b8 R15: 0000000000100000
Jan 20 19:41:55 Tower kernel: </TASK>
Jan 20 19:41:55 Tower kernel: Modules linked in: nf_tables wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_mark nf_conntrack_netlink veth xt_CHECKSUM xt_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan br_netfilter xt_conntrack xt_MASQUERADE nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype md_mod zfs(PO) i915 spl(O) intel_gtt ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc bonding tls iwlmvm mac80211 intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp libarc4 kvm_intel xe kvm drm_gpuvm drm_exec gpu_sched iwlwifi drm_buddy drm_ttm_helper ttm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit sha512_ssse3 drm_suballoc_helper sha256_ssse3 sha1_ssse3 drm_display_helper aesni_intel btusb crypto_simd btrtl cryptd btbcm
Jan 20 19:41:55 Tower kernel: drm_kms_helper btintel rapl intel_cstate cfg80211 drm mei_hdcp bluetooth mei_pxp wmi_bmof joydev input_leds intel_uncore igc i2c_i801 mei_me agpgart i2c_smbus led_class tpm_crb mei i2c_core rfkill tpm_tis tpm_tis_core vmd video thermal fan tpm wmi libaescfb ecdh_generic backlight ecc acpi_tad acpi_pad button
Jan 20 19:41:55 Tower kernel: ---[ end trace 0000000000000000 ]---
Jan 20 19:41:55 Tower kernel: RIP: 0010:handle_changed_spte+0x11a/0x470 [kvm]
Jan 20 19:41:55 Tower kernel: Code: 80 e2 01 74 27 80 7c 24 2e 00 74 20 48 8b 54 24 10 45 89 e1 49 89 d8 48 89 e9 8b 74 24 28 48 c7 c7 b6 1b ae a0 e8 06 f3 10 e1 <0f> 0b 48 39 dd 0f 84 34 03 00 00 48 8b 74 24 10 49 89 d8 48 89 e9
Jan 20 19:41:55 Tower kernel: RSP: 0018:ffffc90035d9bba0 EFLAGS: 00010246
Jan 20 19:41:55 Tower kernel: RAX: 00000000000000c8 RBX: 8000000000000000 RCX: 0000000000000002
Jan 20 19:41:55 Tower kernel: RDX: 0000000000000000 RSI: ffffffff82365ea5 RDI: 00000000ffffffff
Jan 20 19:41:55 Tower kernel: RBP: 86000001de862b77 R08: 0000000000000000 R09: 0000000000000000
Jan 20 19:41:55 Tower kernel: R10: 00000000000000c8 R11: 0000000000000000 R12: 0000000000000001
Jan 20 19:41:55 Tower kernel: R13: ffffc9000f6b5000 R14: 0000000000000001 R15: 0000000000000001
Jan 20 19:41:55 Tower kernel: FS:  0000147a31dff6c0(0000) GS:ffff88903f180000(0000) knlGS:0000000000000000
Jan 20 19:41:55 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 20 19:41:55 Tower kernel: CR2: 0000147a1c169000 CR3: 000000012b3f6001 CR4: 0000000000772ef0
Jan 20 19:41:55 Tower kernel: PKRU: 55555554
Jan 20 19:41:55 Tower kernel: note: CPU 2/KVM[1985113] exited with preempt_count 1

tower-diagnostics-20260120-1958.zip

Crash is related to kvm, but can't see whether it's a hardware or software issue.

If you reboot and try again starting the same VM, do you get the same error?

19 hours ago, stuart576 said:

I haven't been able to capture a system log (mirror to flash wearing out boot USB over long periods).

Syslog server can also be set to save the persistent syslog to a share.

19 hours ago, stuart576 said:

I do have a 13th Gen Intel which I know can have issues

Yep, and it's the 13900K, the most affected model, so that can well be the issue.

Thanks for the reply.

I have reboot the VM several times now, and rebooted the server too. I did have an issue with the iGPU not being set to multimonitor, I think this reset when I did a BIOS update after the crash last month, but I don't think that is the cause of this issue. Only because I saw the kernel bug / tainted message on the crash last month on the main display of the server whenever I tried to reboot (not in safe mode). A new boot USB, new VM and rebuild of dockers got me up and running again in the end.

You reminded me that since my crash around a month ago, I have set the syslog to save to a local share. I think I will have to wait for more data.

Having realised that the 13th Gen Intel cpus are an issue, I have been considering new hardware. This would reduce the performance requirements of the new unraid server somewhat, as I wouldn't run a Windows VM for daily use. I am finding it difficult to decide on hardware though, as there is a lot to choose from. I'd like to focus on reliability considering my current issues, and secondly RAM is expensive right now. I've considered a second hand super micro, due to the lower price, but they come with their own problems, often no quicksync, high power and noise etc.

1 hour ago, stuart576 said:

I saw the kernel bug / tainted message

FYI, this is normal, since the out-of-tree zfs module is installed; the kernel will always be "tainted", so that by itself doesn't mean much.