G'Day Guys, 

I'm at a loss.  The system was running ok.  I've noticed the system crashing for a few months.  However I did not have a screen and the system log was not going to a server. 

 

The system is Kernal trapping reasonable regularly.  It was once a week, however the last 2 where over the weekend.

The system is an Intel i9.  There isn't any overclocking modification. (Ignore the system name as Ryzen1. It used to be a Ryzen many years ago). 

 

The log also shows an ACPI error, however I have the latest BIOS version for my ASUS HERO motherboard. So I don't know why this error is also appearing.

 

The last entry in the syslog before the crash is:

Oct 21 12:56:02 Ryzen1 kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184)

Oct 21 12:56:02 Ryzen1 kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477)

Oct 21 12:56:02 Ryzen1 kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529)

Oct 21 12:56:02 Ryzen1 kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184)

Oct 21 12:56:02 Ryzen1 kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477)

Oct 21 12:56:02 Ryzen1 kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529)

Oct 21 20:11:27 Ryzen1 webGUI: Successful login user root from 192.168.200.203

Oct 21 20:12:46 Ryzen1 monitor: Stop running nchan processes

Oct 22 03:30:01 Ryzen1 Plugin Auto Update: Checking for available plugin updates

Oct 22 03:30:05 Ryzen1 Plugin Auto Update: Community Applications Plugin Auto Update finished

Oct 22 04:00:01 Ryzen1 Docker Auto Update: Community Applications Docker Autoupdate running

Oct 22 04:00:01 Ryzen1 Docker Auto Update: Checking for available updates

Oct 22 04:00:46 Ryzen1 Docker Auto Update: Found update for swag. Not set to autoupdate

Oct 22 04:00:46 Ryzen1 Docker Auto Update: Found update for unifi-controller. Not set to autoupdate

Oct 22 04:00:46 Ryzen1 Docker Auto Update: No updates will be installed

Oct 22 05:42:50 Ryzen1 kernel: traps: node[18612] trap int3 ip:1e75f12 sp:1532988ac400 error:0 in node[400000+4d69000]

Oct 22 20:24:46 Ryzen1 kernel: ------------[ cut here ]------------

Oct 22 20:24:46 Ryzen1 kernel: !entity_is_task(se)

Oct 22 20:24:46 Ryzen1 kernel: WARNING: CPU: 1 PID: 28769 at kernel/sched/sched.h:1404 task_of+0x2a/0x36

Oct 22 20:24:46 Ryzen1 kernel: Modules linked in: xt_connmark xt_comment iptable_raw wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha xt_mark veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap macvlan xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nvidia_uvm(PO) xfs nfsd auth_rpcgss oid_registry lockd grace sunrpc md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag i915 drm_buddy i2c_algo_bit ttm drm_display_helper intel_gtt agpgart ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc bonding tls nvidia_drm(PO) nvidia_modeset(PO) intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel

Oct 22 20:24:46 Ryzen1 kernel: nvidia(PO) kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 drm_kms_helper btusb aesni_intel btrtl btbcm btintel crypto_simd bluetooth drm cryptd mei_pxp syscopyarea mei_hdcp i2c_i801 rapl ecdh_generic input_leds mpt3sas joydev led_class ecc nvme intel_cstate sysfillrect wmi_bmof i2c_smbus ahci mei_me raid_class tpm_crb sysimgblt video intel_uncore thunderbolt nvme_core igc i2c_core mei libahci scsi_transport_sas vmd thermal fb_sys_fops fan tpm_tis tpm_tis_core wmi tpm intel_pmc_core backlight acpi_tad acpi_pad button unix

Oct 23 08:03:18 Ryzen1 emhttpd: shcmd (45): /etc/rc.d/rc.php-fpm start

 

I've added images from the last 2 panics, the diagnosics from the system and the full system log from off the syslog server.,

 

I'm hoping you can help me.  

 

Kind Regards

Peter

 

ryzen1-diagnostics-20231023-0830.zip intel1-syslog-192.168.200.202-20231022-2121.zip

Edited October 22, 20232 yr by peter williams

There are multiple call traces suggesting a hardware problem, there are also macvaln related call traces, so start by changing the docker network to ipvlan, but likely that's not the only issue.

G'Day, 

 

OK.  Thank you.  I will change to ipvlan and see if this helps.

 

Kind Regards

Peter

G'Day,  Well that did not help.  

I awoke today to find the system again has kenerla trapped.  

It was around 7 days since rebooting after making the ipvlan change.

If it is a hardware, where should I start?

 

The remote syslog has little in it leading up to the kernal trap.  Maybe the image has more of a clue.  I'm not sure what it is telling me however.

 

Oct 25 18:39:26 Ryzen1 monitor: Stop running nchan processes

Oct 26 03:43:10 Ryzen1 kernel: TCP: request_sock_TCP: Possible SYN flooding on port 46801. Sending cookies. Check SNMP counters.

Oct 26 04:50:01 Ryzen1 rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="1726" x-info="https://www.rsyslog.com"] rsyslogd was HUPed

Oct 26 17:17:06 Ryzen1 webGUI: Successful login user root from 192.168.200.203

Oct 26 17:20:00 Ryzen1 monitor: Stop running nchan processes

Oct 26 17:23:37 Ryzen1 webGUI: Successful login user root from 192.168.200.100

Oct 26 17:31:53 Ryzen1 kernel: eth0: renamed from vethbfa21c8

Oct 26 18:10:11 Ryzen1 kernel: veth081721c: renamed from eth0

Oct 26 18:10:12 Ryzen1 kernel: eth0: renamed from vethd535b0e

Oct 26 18:11:02 Ryzen1 kernel: veth8e2d71f: renamed from eth0

Oct 26 18:11:03 Ryzen1 kernel: eth0: renamed from veth8e54022

Oct 26 18:14:15 Ryzen1 flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update

Oct 26 18:23:15 Ryzen1 flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update

Oct 26 18:30:29 Ryzen1 webGUI: Successful login user root from 192.168.200.203

Oct 26 18:37:26 Ryzen1 kernel: eth0: renamed from veth9621a80

Oct 26 19:28:04 Ryzen1 monitor: Stop running nchan processes

Oct 26 19:29:42 Ryzen1 webGUI: Successful login user root from 192.168.200.100

Oct 26 20:30:08 Ryzen1 monitor: Stop running nchan processes

Oct 27 09:00:56 Ryzen1 webGUI: Successful login user root from 192.168.200.100

Oct 27 09:12:42 Ryzen1 monitor: Stop running nchan processes

Oct 27 18:02:34 Ryzen1 webGUI: Successful login user root from 192.168.200.203

Oct 27 18:02:45 Ryzen1 monitor: Stop running nchan processes

Oct 29 10:09:26 Ryzen1 emhttpd: shcmd (45): /etc/rc.d/rc.php-fpm start

Oct 29 10:09:26 Ryzen1 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0 try https://www.rsyslog.com/e/2359 ]

Oct 29 10:09:26 Ryzen1 root: Starting php-fpm done

Oct 29 10:09:26 Ryzen1 emhttpd: shcmd (46): /etc/rc.d/rc.unraid-api install

Oct 29 10:09:26 Ryzen1 root: unraid-api installed

Oct 29 10:09:26 Ryzen1 emhttpd: shcmd (47): /etc/rc.d/rc.nginx start

Oct 29 10:09:26 Ryzen1 root: Starting Nginx server daemon...

Oct 29 10:09:27 Ryzen1 root: Starting [email protected]

Oct 29 10:09:27 Ryzen1 root: Watch Setup on Config Path: /boot/config/plugins/dynamix.my.servers/myservers.cfg

Oct 29 10:09:27 Ryzen1 emhttpd: shcmd (48): /etc/rc.d/rc.flash_backup start

Oct 29 10:09:27 Ryzen1 flash_backup: flush: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update

Oct 29 10:09:27 Ryzen1 flash_backup: start watching for file changes

Oct 29 10:09:28 Ryzen1 unraid-api[8336]: ✔️ UNRAID API started successfully!

 

Not really unexpected as mentioned, start by running memtest, or try with just two sticks of RAM, if the same try the other two, that will basically rule out bad RAM or a board stability issue with all 4 DIMMs populated.

OK.  Thank you.  I'll try that. 

I'll also order a new Motherboard, since the log still indicates that "kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184)" error in the syslog.

That error is usually harmless, other than the log spam it creates, look for a BIOS update.