I am now on 6.12.10, and am now experiencing shutdowns around every 24-36 hours.  One recent trace is attached.  My logs show frequent issues with the r8169 driver as well (see the last line below).  I removed a second network card and am only using the on-board ethernet port now.

 

The docker and VM services are running, but there are no dockers or VM instances running.

 

Traces all include this: "RIP: 0010:dev_watchdog+0x14e/0x1c0"

 

Any ideas?

 

Jul 12 11:17:23 BadAxe2 kernel: ------------[ cut here ]------------
Jul 12 11:17:23 BadAxe2 kernel: NETDEV WATCHDOG: eth0 (r8169): transmit queue 0 timed out
Jul 12 11:17:23 BadAxe2 kernel: WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x14e/0x1c0
Jul 12 11:17:23 BadAxe2 kernel: Modules linked in: reiserfs cmac cifs asn1_decoder cifs_arc4 cifs_md4 dns_resolver xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid corefreqk(O) ip6table_filter ip6_tables iptable_filter ip_tables x_tables af_packet 8021q garp mrp bridge stp llc r8169 realtek intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp i915 kvm_intel kvm iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 drm_kms_helper aesni_intel
Jul 12 11:17:23 BadAxe2 kernel: crypto_simd mei_hdcp mei_pxp drm cryptd mpt3sas intel_gtt rapl agpgart wmi_bmof i2c_i801 intel_cstate mei_me i2c_smbus ahci raid_class mxm_wmi i2c_core syscopyarea sysfillrect mei libahci scsi_transport_sas intel_uncore sysimgblt fb_sys_fops thermal fan video backlight wmi button acpi_pad unix [last unloaded: realtek]
Jul 12 11:17:23 BadAxe2 kernel: CPU: 6 PID: 0 Comm: swapper/6 Tainted: P           O       6.1.79-Unraid #1
Jul 12 11:17:23 BadAxe2 kernel: Hardware name: ASUS All Series/H97-PLUS, BIOS 2504 05/20/2015
Jul 12 11:17:23 BadAxe2 kernel: RIP: 0010:dev_watchdog+0x14e/0x1c0
Jul 12 11:17:23 BadAxe2 kernel: Code: 86 c5 00 00 75 26 48 89 ef c6 05 26 86 c5 00 01 e8 11 23 fc ff 44 89 f1 48 89 ee 48 c7 c7 bc 8e 15 82 48 89 c2 e8 cf 54 93 ff <0f> 0b 48 89 ef e8 32 fb ff ff 48 8b 83 88 fc ff ff 48 89 ef 44 89
Jul 12 11:17:23 BadAxe2 kernel: RSP: 0018:ffffc90000230ea8 EFLAGS: 00010282
Jul 12 11:17:23 BadAxe2 kernel: RAX: 0000000000000000 RBX: ffff888105ec4448 RCX: 0000000000000027
Jul 12 11:17:23 BadAxe2 kernel: RDX: 0000000000000104 RSI: ffffffff820d8b42 RDI: 00000000ffffffff
Jul 12 11:17:23 BadAxe2 kernel: RBP: ffff888105ec4000 R08: 0000000000000000 R09: ffffffff82245f70
Jul 12 11:17:23 BadAxe2 kernel: R10: 00007fffffffffff R11: ffffffff82965079 R12: 0000000000000000
Jul 12 11:17:23 BadAxe2 kernel: R13: ffff888105ec439c R14: 0000000000000000 R15: 0000000000000001
Jul 12 11:17:23 BadAxe2 kernel: FS:  0000000000000000(0000) GS:ffff8887fff80000(0000) knlGS:0000000000000000
Jul 12 11:17:23 BadAxe2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 12 11:17:23 BadAxe2 kernel: CR2: 000014c27a97c484 CR3: 000000000220a004 CR4: 00000000001706e0
Jul 12 11:17:23 BadAxe2 kernel: Call Trace:
Jul 12 11:17:23 BadAxe2 kernel: <IRQ>
Jul 12 11:17:23 BadAxe2 kernel: ? __warn+0xab/0x122
Jul 12 11:17:23 BadAxe2 kernel: ? report_bug+0x109/0x17e
Jul 12 11:17:23 BadAxe2 kernel: ? dev_watchdog+0x14e/0x1c0
Jul 12 11:17:23 BadAxe2 kernel: ? handle_bug+0x41/0x6f
Jul 12 11:17:23 BadAxe2 kernel: ? exc_invalid_op+0x13/0x60
Jul 12 11:17:23 BadAxe2 kernel: ? asm_exc_invalid_op+0x16/0x20
Jul 12 11:17:23 BadAxe2 kernel: ? dev_watchdog+0x14e/0x1c0
Jul 12 11:17:23 BadAxe2 kernel: ? dev_watchdog+0x14e/0x1c0
Jul 12 11:17:23 BadAxe2 kernel: ? psched_ppscfg_precompute+0x57/0x57
Jul 12 11:17:23 BadAxe2 kernel: ? psched_ppscfg_precompute+0x57/0x57
Jul 12 11:17:23 BadAxe2 kernel: call_timer_fn+0x6f/0x10d
Jul 12 11:17:23 BadAxe2 kernel: __run_timers+0x144/0x184
Jul 12 11:17:23 BadAxe2 kernel: ? timerqueue_add+0x6a/0x76
Jul 12 11:17:23 BadAxe2 kernel: ? enqueue_hrtimer+0x7f/0x8a
Jul 12 11:17:23 BadAxe2 kernel: ? __hrtimer_next_event_base+0x27/0x81
Jul 12 11:17:23 BadAxe2 kernel: run_timer_softirq+0x2b/0x43
Jul 12 11:17:23 BadAxe2 kernel: __do_softirq+0x129/0x288
Jul 12 11:17:23 BadAxe2 kernel: __irq_exit_rcu+0x5e/0xb8
Jul 12 11:17:23 BadAxe2 kernel: sysvec_apic_timer_interrupt+0x85/0xa6
Jul 12 11:17:23 BadAxe2 kernel: </IRQ>
Jul 12 11:17:23 BadAxe2 kernel: <TASK>
Jul 12 11:17:23 BadAxe2 kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20
Jul 12 11:17:23 BadAxe2 kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202
Jul 12 11:17:23 BadAxe2 kernel: Code: ed e3 9f ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 4e a1 a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29
Jul 12 11:17:23 BadAxe2 kernel: RSP: 0018:ffffc900000e7e98 EFLAGS: 00000246
Jul 12 11:17:23 BadAxe2 kernel: RAX: ffff8887fff80000 RBX: ffff8887fffb6f00 RCX: 0000000000000000
Jul 12 11:17:23 BadAxe2 kernel: RDX: 00000aa582d6a2aa RSI: ffffffff820d8b42 RDI: ffffffff820d904b
Jul 12 11:17:23 BadAxe2 kernel: RBP: 0000000000000001 R08: 0000000000000002 R09: 0000000000000002
Jul 12 11:17:23 BadAxe2 kernel: R10: 0000000000000020 R11: 0000000000000013 R12: 0000000000000001
Jul 12 11:17:23 BadAxe2 kernel: R13: ffffffff823206e0 R14: 00000aa582d6a2aa R15: 0000000000000000
Jul 12 11:17:23 BadAxe2 kernel: ? cpuidle_enter_state+0xf7/0x202
Jul 12 11:17:23 BadAxe2 kernel: cpuidle_enter+0x2a/0x38
Jul 12 11:17:23 BadAxe2 kernel: do_idle+0x18d/0x1fb
Jul 12 11:17:23 BadAxe2 kernel: cpu_startup_entry+0x2a/0x2c
Jul 12 11:17:23 BadAxe2 kernel: start_secondary+0x101/0x101
Jul 12 11:17:23 BadAxe2 kernel: secondary_startup_64_no_verify+0xce/0xdb
Jul 12 11:17:23 BadAxe2 kernel: </TASK>
Jul 12 11:17:23 BadAxe2 kernel: ---[ end trace 0000000000000000 ]---
Jul 12 11:17:23 BadAxe2 kernel: r8169 0000:03:00.0: can't disable ASPM; OS doesn't have ASPM control

 

19 minutes ago, tcharron said:

NETDEV WATCHDOG: eth0 (r8169): transmit queue 0 timed out

First line also indicates that the call trace is related to the NIC, try a different one if possible.

23 minutes ago, JorgeB said:

First line also indicates that the call trace is related to the NIC, try a different one if possible.

Yeah... I removed the PCIe NIC but am now only using the motherboard NIC.

I run the machine headless so haven't tried to check what is configurable in the firmware.  I might be able to change the power settings (ASPM), but will need to physically move the machine to hook it to a monitor and keyboard.  It's an old build so I honestly felt I'd be 'safe' not being able to easily connect it to a monitor!  guess not.

I'll also try updating the bios.  Looks like I can upgrade from a May 2015 to a March 2016 one.  Release notes say "Improve system stability".  I doubt it'll make a difference but there's nothing to lose!

 Hardware name: ASUS All Series/H97-PLUS, BIOS 2504 05/20/2015

 

Looks like you got a RTL8111GR, you could try the realtek package from the community apps

On 7/12/2024 at 3:20 PM, Mainfrezzer said:

Looks like you got a RTL8111GR, you could try the realtek package from the community apps

Thanks.  I have installed it.  I had been looking for a way to roll back the driver to r6168 but didn't realize that there were plugins to do it!

 

However, I'm not sure now if the 8169 driver is the problem or not.  I got a ton of errors from it that did not cause a shutdown.  Some of my reading on that driver is that it does have random behavour related to fragmented packets. https://bugzilla.kernel.org/show_bug.cgi?id=209839 .  Either way, going back to the r8168 driver will rule the r8169 driver out.

 

I've also enabled ASPM support and updated my BIOS.  Time will tell if any of this helps.

 

Thanks again.

Tim

Edited July 14, 20241 yr by tcharron
typos

So it's been stable now.

I updated the bios and enable ASPM, but I think the key was to rollback the driver to r8168 from r8169.