Hey folks - hope that maybe someone can help me figure out why my server keeps hard locking.  It seems that this is happening every 2-ish days.  

 

Console is gone.  Things like numlock/capslock light are non functional.  Total hard lock.

 

I pulled these logs after a reboot - crossing my fingers that it contains something useful.

 

 

san-diagnostics-20221104-0846.zip

4 minutes ago, TheSnotRocket said:

I pulled these logs after a reboot

Logs after a reboot do not usually contain anything useful as anything that was recorded prior to the reboot is lost.  You should setup the syslog server which will allow recorded information to survive reboots. 

 

After configuring the syslog server, post diagnostics again after you experience another lockup.

6 minutes ago, Hoopster said:

After configuring the syslog server, post diagnostics again after you experience another lockup.

Also post the output of the syslog server as it will not be included in the diagnostics file.

Thanks... I'll get syslog going and report back on my next hard lock.

 

2 minutes ago, ChatNoir said:

Also post the output of the syslog server as it will not be included in the diagnostics file.

Yes, that information should not have been omitted. 

These are the last 40 lines I got from it:

 

11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: FS:  00001523c1723700(0000) GS:ffff889fffbc0000(0000) knlGS:0000000000000000
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: R13: 00001511d0edcf70 R14: ffffffff826e7480 R15: 0000000000000000
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: R10: 59a545b015aa135a R11: 429f1261555797f6 R12: ffffc9000043c80c
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RBP: ffffc9000043c7f8 R08: 579e89f86a7cb354 R09: 8b602b5426b4b34a
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RDX: 00001511d0edd000 RSI: 06ec3a26cb1f79a8 RDI: 22b7e2ecece96bc6
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RAX: ffff8881cbe88806 RBX: ffff8881c2d13b00 RCX: a83086ca42e7d68b
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RSP: 0018:ffffc9000043c730 EFLAGS: 00010202
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: Code: 4c 89 f7 e8 2f f8 ff ff 48 8b 15 66 6a 00 00 89 c0 48 8d 04 c2 4c 8b 28 4d 85 ed 74 2a 49 81 ed 90 00 00 00 eb 21 8a 44 24 46 <41> 38 45 46 74 21 49 8b 95 90 00 00 00 48 85 d2 0f 84 53 ff ff ff
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RIP: 0010:nf_nat_setup_info+0x142/0x7b1 [nf_nat]
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: ---[ end trace 0000000000000000 ]---
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: CR2: 00001511d0edcfb6
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: input_leds intel_uncore mpt3sas joydev led_class i2c_core nvme_core libahci raid_class syscopyarea sysfillrect scsi_transport_sas sysimgblt fb_sys_fops wmi acpi_ipmi ipmi_si acpi_power_meter acpi_pad button unix [last unloaded: mlx4_core]
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: Modules linked in: nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth macvlan xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs md_mod ipmi_devintf iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc bonding tls ipv6 mlx4_en mlx4_core igb nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) ast drm_vram_helper drm_ttm_helper ttm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper aesni_intel crypto_simd cryptd rapl drm i2c_i801 intel_cstate nvme agpgart backlight i2c_smbus i2c_algo_bit ahci
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: </TASK>
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: R13: 00001523ac7993a0 R14: 0000000000000010 R15: 00001523ac0832e0
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: R10: 0000000000000020 R11: 0000000000000293 R12: 0000555a5a69c2b0
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RBP: 00001523c1720a20 R08: 0000000000000000 R09: 000000000000000b
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RDX: 0000000000000010 RSI: 00001523ac7993a0 RDI: 0000000000000017
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RAX: ffffffffffffffda RBX: 00001523c1720c48 RCX: 00001523c5eee59b
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RSP: 002b:00001523c17209d0 EFLAGS: 00000293 ORIG_RAX: 000000000000002a
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: Code: 83 ec 18 89 54 24 0c 48 89 34 24 89 7c 24 08 e8 bb fa ff ff 8b 54 24 0c 48 8b 34 24 41 89 c0 8b 7c 24 08 b8 2a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2f 44 89 c7 89 44 24 08 e8 f1 fa ff ff 8b 44
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: RIP: 0033:0x1523c5eee59b
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: do_syscall_64+0x6b/0x81
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: __x64_sys_connect+0x14/0x1b
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: __sys_connect+0x68/0xa7
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: inet_stream_connect+0x39/0x52
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: ? preempt_latency_start+0x2b/0x46
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: ? percpu_counter_add_batch+0x85/0xa2
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: __inet_stream_connect+0x113/0x311
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: tcp_v4_connect+0x3e3/0x43e
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: tcp_connect+0x7e4/0x86f
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: __tcp_transmit_skb+0x823/0x893
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: __ip_queue_xmit+0x2fd/0x344
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: ip_finish_output2+0x37d/0x3b0
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: __local_bh_enable_ip+0x4c/0x6b
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: <TASK>
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: </IRQ>
11-04-2022    20:49:24    Kernel.Warning    10.0.0.33    Nov  4 20:49:23 SAN kernel: do_softirq+0x7f/0xab

 

 

Attached syslog

 

SyslogCatchAll-2022-11-04.txt

Edited November 5, 20223 yr by TheSnotRocket

Try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

1 hour ago, JorgeB said:

Try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

Change made.