Random disconnects from network

[ensnare]

Thought I finally got my Unraid server stable overnight, only to wake up to find it disconnected from the network. I was able to log in through the local console, but running network commands (ifconfig, ip a, etc...) locked it up. I saw this repeated over and over in the syslog:

 

Apr 18 06:15:38 NYCMedia kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141
Apr 18 06:15:38 NYCMedia kernel: Code: ff 45 84 f6 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 7a 8d bb ff fb 66 0f 1f 44 00 00 <48> 2b 2c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cd
Apr 18 06:15:38 NYCMedia kernel: RSP: 0018:ffffc90000257e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd7
Apr 18 06:15:38 NYCMedia kernel: RAX: ffff88bf4cc9fac0 RBX: ffff88bf3dc78c00 RCX: 000000000000001f
Apr 18 06:15:38 NYCMedia kernel: RDX: 0000000000000000 RSI: 000000002c3889ce RDI: 0000000000000000
Apr 18 06:15:38 NYCMedia kernel: RBP: 00002be1be42a70e R08: 00002be1be42a70e R09: 0000000000003a17
Apr 18 06:15:38 NYCMedia kernel: R10: 000000003b3913f0 R11: 071c71c71c71c71c R12: 0000000000000002
Apr 18 06:15:38 NYCMedia kernel: R13: ffffffff81e5e1e0 R14: 0000000000000000 R15: ffffffff81e5e2b8
Apr 18 06:15:38 NYCMedia kernel: ? cpuidle_enter_state+0xbf/0x141
Apr 18 06:15:38 NYCMedia kernel: do_idle+0x17e/0x1fc
Apr 18 06:15:38 NYCMedia kernel: cpu_startup_entry+0x6a/0x6c
Apr 18 06:15:38 NYCMedia kernel: start_secondary+0x197/0x1b2
Apr 18 06:15:38 NYCMedia kernel: secondary_startup_64+0xa4/0xb0
Apr 18 06:16:18 NYCMedia kernel: rcu: INFO: rcu_bh self-detected stall on CPU
Apr 18 06:16:18 NYCMedia kernel: rcu:   98-....: (4681870 ticks this GP) idle=b8e/1/0x4000000000000002 softirq=3032606/3063928 fqs=1054603
Apr 18 06:16:18 NYCMedia kernel: rcu:    (t=4380077 jiffies g=-483 q=10)
Apr 18 06:16:18 NYCMedia kernel: Sending NMI from CPU 98 to CPUs 2:
Apr 18 06:16:18 NYCMedia kernel: NMI backtrace for cpu 2
Apr 18 06:16:18 NYCMedia kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P        W  O      4.19.107-Unraid #1
Apr 18 06:16:18 NYCMedia kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 0902 03/17/2020
Apr 18 06:16:18 NYCMedia kernel: RIP: 0010:nf_conntrack_tuple_taken+0x88/0x221
Apr 18 06:16:18 NYCMedia kernel: Code: 49 c7 c6 f0 ff ff ff 48 8b 18 f6 c3 01 0f 85 8a 01 00 00 0f b6 43 37 4c 89 f7 48 89 c1 48 6b c0 38 48 29 c7 48 01 df 48 39 ef <0f> 84 65 01 00 00 48 8b 05 c8 ed 88 00 8b 97 88 00 00 00 29 c2 85
Apr 18 06:16:18 NYCMedia kernel: RSP: 0018:ffff88bf4cc83a00 EFLAGS: 00000202
Apr 18 06:16:18 NYCMedia kernel: RAX: 0000000000000038 RBX: ffff8882898ecb88 RCX: 0000000000000001
Apr 18 06:16:18 NYCMedia kernel: RDX: 000000000295e873 RSI: 0000000000000000 RDI: ffff8882898ecb40
Apr 18 06:16:18 NYCMedia kernel: RBP: ffff8882898ec3c0 R08: 0000000000000003 R09: ffffffff81c8aa80
Apr 18 06:16:18 NYCMedia kernel: R10: 0000000000000001 R11: ffff88bec7ca7118 R12: ffff88bf4cc83a40
Apr 18 06:16:18 NYCMedia kernel: R13: ffffffff81e91080 R14: fffffffffffffff0 R15: 00000000000021fd
Apr 18 06:16:18 NYCMedia kernel: FS:  0000000000000000(0000) GS:ffff88bf4cc80000(0000) knlGS:0000000000000000
Apr 18 06:16:18 NYCMedia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 18 06:16:18 NYCMedia kernel: CR2: 00001462e9d9b000 CR3: 0000000005e0a000 CR4: 0000000000340ee0
Apr 18 06:16:18 NYCMedia kernel: Call Trace:
Apr 18 06:16:18 NYCMedia kernel: <IRQ>
Apr 18 06:16:18 NYCMedia kernel: nf_nat_used_tuple+0x2e/0x49 [nf_nat]
Apr 18 06:16:18 NYCMedia kernel: nf_nat_setup_info+0x5fd/0x666 [nf_nat]
Apr 18 06:16:18 NYCMedia kernel: nf_nat_alloc_null_binding+0x71/0x88 [nf_nat]
Apr 18 06:16:18 NYCMedia kernel: nf_nat_inet_fn+0x9f/0x1b9 [nf_nat]
Apr 18 06:16:18 NYCMedia kernel: ? br_handle_local_finish+0xe/0xe
Apr 18 06:16:18 NYCMedia kernel: nf_nat_ipv4_in+0x1e/0x62 [nf_nat_ipv4]
Apr 18 06:16:18 NYCMedia kernel: nf_hook_slow+0x3a/0x90
Apr 18 06:16:18 NYCMedia kernel: br_nf_pre_routing+0x303/0x343
Apr 18 06:16:18 NYCMedia kernel: ? br_nf_forward_ip+0x362/0x362
Apr 18 06:16:18 NYCMedia kernel: nf_hook_slow+0x3a/0x90
Apr 18 06:16:18 NYCMedia kernel: br_handle_frame+0x27e/0x2bd
Apr 18 06:16:18 NYCMedia kernel: ? br_pass_frame_up+0x14a/0x14a
Apr 18 06:16:18 NYCMedia kernel: __netif_receive_skb_core+0x4a7/0x7b1
Apr 18 06:16:18 NYCMedia kernel: ? udp_gro_receive+0x4b/0x136
Apr 18 06:16:18 NYCMedia kernel: __netif_receive_skb_one_core+0x35/0x6f
Apr 18 06:16:18 NYCMedia kernel: netif_receive_skb_internal+0x79/0x94
Apr 18 06:16:18 NYCMedia kernel: napi_gro_receive+0x44/0x7b
Apr 18 06:16:18 NYCMedia kernel: ixgbe_poll+0xb97/0xce4 [ixgbe]
Apr 18 06:16:18 NYCMedia kernel: net_rx_action+0x107/0x26c
Apr 18 06:16:18 NYCMedia kernel: __do_softirq+0xc9/0x1d7
Apr 18 06:16:18 NYCMedia kernel: irq_exit+0x5e/0x9d
Apr 18 06:16:18 NYCMedia kernel: do_IRQ+0xb2/0xd0
Apr 18 06:16:18 NYCMedia kernel: common_interrupt+0xf/0xf
Apr 18 06:16:18 NYCMedia kernel: </IRQ>
Apr 18 06:16:18 NYCMedia kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141
Apr 18 06:16:18 NYCMedia kernel: Code: ff 45 84 f6 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 7a 8d bb ff fb 66 0f 1f 44 00 00 <48> 2b 2c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cd
Apr 18 06:16:18 NYCMedia kernel: RSP: 0018:ffffc90000257e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd7
Apr 18 06:16:18 NYCMedia kernel: RAX: ffff88bf4cc9fac0 RBX: ffff88bf3dc78c00 RCX: 000000000000001f
Apr 18 06:16:18 NYCMedia kernel: RDX: 0000000000000000 RSI: 000000002c3889ce RDI: 0000000000000000
Apr 18 06:16:18 NYCMedia kernel: RBP: 00002be1be42a70e R08: 00002be1be42a70e R09: 0000000000003a17
Apr 18 06:16:18 NYCMedia kernel: R10: 000000003b3913f0 R11: 071c71c71c71c71c R12: 0000000000000002
Apr 18 06:16:18 NYCMedia kernel: R13: ffffffff81e5e1e0 R14: 0000000000000000 R15: ffffffff81e5e2b8
Apr 18 06:16:18 NYCMedia kernel: ? cpuidle_enter_state+0xbf/0x141
Apr 18 06:16:18 NYCMedia kernel: do_idle+0x17e/0x1fc
Apr 18 06:16:18 NYCMedia kernel: cpu_startup_entry+0x6a/0x6c
Apr 18 06:16:18 NYCMedia kernel: start_secondary+0x197/0x1b2
Apr 18 06:16:18 NYCMedia kernel: secondary_startup_64+0xa4/0xb0
Apr 18 06:16:18 NYCMedia kernel: Sending NMI from CPU 98 to CPUs 54:
Apr 18 06:16:18 NYCMedia kernel: NMI backtrace for cpu 54
Apr 18 06:16:18 NYCMedia kernel: CPU: 54 PID: 76534 Comm: EmbyServer Tainted: P        W  O      4.19.107-Unraid #1
Apr 18 06:16:18 NYCMedia kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 0902 03/17/2020
Apr 18 06:16:18 NYCMedia kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x6b/0x171
Apr 18 06:16:18 NYCMedia kernel: Code: 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02 00 65 48 03 15 80 6a f8
Apr 18 06:16:18 NYCMedia kernel: RSP: 0018:ffffc9000d0a3948 EFLAGS: 00000202
Apr 18 06:16:18 NYCMedia kernel: RAX: 0000000000740101 RBX: 0000000000000000 RCX: 000000009e0002bb
Apr 18 06:16:18 NYCMedia kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff81e08df4
Apr 18 06:16:18 NYCMedia kernel: RBP: ffffffff81e08df4 R08: 00000000bec9ebc5 R09: ffffffff81c8aa80
Apr 18 06:16:18 NYCMedia kernel: R10: ffffc9000d0a3ec0 R11: ffffc9000d0a3c58 R12: 00000000000003fa
Apr 18 06:16:18 NYCMedia kernel: R13: ffffffff81e095e8 R14: 0000000000000000 R15: ffff8881c5b697e0
Apr 18 06:16:18 NYCMedia kernel: FS:  00001529c3b1f700(0000) GS:ffff88bf4d980000(0000) knlGS:0000000000000000
Apr 18 06:16:18 NYCMedia kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 18 06:16:18 NYCMedia kernel: CR2: 000014aff8618dd0 CR3: 0000003e9a832000 CR4: 0000000000340ee0

Diagnostics attached. Any idea what's going on?

 

 

nycmedia-diagnostics-20200418-0755.zip

[trurl]

Not related, but why have you allocated 100G to docker image? Have you had problems filling it?

 

12 minutes ago, ensnare said:

running network commands (ifconfig, ip a, etc...) locked it up.

Maybe just a coincidence, since your diagnostics includes the results of ifconfig and ethtool and they look OK.

 

Have you done memtest?

[ensnare]

4 minutes ago, trurl said:

Not related, but why have you allocated 100G to docker image? Have you had problems filling it?

 

Maybe just a coincidence, since your diagnostics includes the results of ifconfig and ethtool and they look OK.

 

Have you done memtest?

I was building a docker image with a lot of built-in packages and got an error that space was running low, so I increased the size.

[trurl]

Usually growing Docker image is due to an application writing to a path that isn't mapped. Very rare to need more than 20G and never much more. If it is growing something is set up wrong.