ensnare Posted April 18, 2020 Share Posted April 18, 2020 Thought I finally got my Unraid server stable overnight, only to wake up to find it disconnected from the network. I was able to log in through the local console, but running network commands (ifconfig, ip a, etc...) locked it up. I saw this repeated over and over in the syslog: Apr 18 06:15:38 NYCMedia kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141 Apr 18 06:15:38 NYCMedia kernel: Code: ff 45 84 f6 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 7a 8d bb ff fb 66 0f 1f 44 00 00 <48> 2b 2c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cd Apr 18 06:15:38 NYCMedia kernel: RSP: 0018:ffffc90000257e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd7 Apr 18 06:15:38 NYCMedia kernel: RAX: ffff88bf4cc9fac0 RBX: ffff88bf3dc78c00 RCX: 000000000000001f Apr 18 06:15:38 NYCMedia kernel: RDX: 0000000000000000 RSI: 000000002c3889ce RDI: 0000000000000000 Apr 18 06:15:38 NYCMedia kernel: RBP: 00002be1be42a70e R08: 00002be1be42a70e R09: 0000000000003a17 Apr 18 06:15:38 NYCMedia kernel: R10: 000000003b3913f0 R11: 071c71c71c71c71c R12: 0000000000000002 Apr 18 06:15:38 NYCMedia kernel: R13: ffffffff81e5e1e0 R14: 0000000000000000 R15: ffffffff81e5e2b8 Apr 18 06:15:38 NYCMedia kernel: ? cpuidle_enter_state+0xbf/0x141 Apr 18 06:15:38 NYCMedia kernel: do_idle+0x17e/0x1fc Apr 18 06:15:38 NYCMedia kernel: cpu_startup_entry+0x6a/0x6c Apr 18 06:15:38 NYCMedia kernel: start_secondary+0x197/0x1b2 Apr 18 06:15:38 NYCMedia kernel: secondary_startup_64+0xa4/0xb0 Apr 18 06:16:18 NYCMedia kernel: rcu: INFO: rcu_bh self-detected stall on CPU Apr 18 06:16:18 NYCMedia kernel: rcu: 98-....: (4681870 ticks this GP) idle=b8e/1/0x4000000000000002 softirq=3032606/3063928 fqs=1054603 Apr 18 06:16:18 NYCMedia kernel: rcu: (t=4380077 jiffies g=-483 q=10) Apr 18 06:16:18 NYCMedia kernel: Sending NMI from CPU 98 to CPUs 2: Apr 18 06:16:18 NYCMedia kernel: NMI backtrace for cpu 2 Apr 18 06:16:18 NYCMedia kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P W O 4.19.107-Unraid #1 Apr 18 06:16:18 NYCMedia kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 0902 03/17/2020 Apr 18 06:16:18 NYCMedia kernel: RIP: 0010:nf_conntrack_tuple_taken+0x88/0x221 Apr 18 06:16:18 NYCMedia kernel: Code: 49 c7 c6 f0 ff ff ff 48 8b 18 f6 c3 01 0f 85 8a 01 00 00 0f b6 43 37 4c 89 f7 48 89 c1 48 6b c0 38 48 29 c7 48 01 df 48 39 ef <0f> 84 65 01 00 00 48 8b 05 c8 ed 88 00 8b 97 88 00 00 00 29 c2 85 Apr 18 06:16:18 NYCMedia kernel: RSP: 0018:ffff88bf4cc83a00 EFLAGS: 00000202 Apr 18 06:16:18 NYCMedia kernel: RAX: 0000000000000038 RBX: ffff8882898ecb88 RCX: 0000000000000001 Apr 18 06:16:18 NYCMedia kernel: RDX: 000000000295e873 RSI: 0000000000000000 RDI: ffff8882898ecb40 Apr 18 06:16:18 NYCMedia kernel: RBP: ffff8882898ec3c0 R08: 0000000000000003 R09: ffffffff81c8aa80 Apr 18 06:16:18 NYCMedia kernel: R10: 0000000000000001 R11: ffff88bec7ca7118 R12: ffff88bf4cc83a40 Apr 18 06:16:18 NYCMedia kernel: R13: ffffffff81e91080 R14: fffffffffffffff0 R15: 00000000000021fd Apr 18 06:16:18 NYCMedia kernel: FS: 0000000000000000(0000) GS:ffff88bf4cc80000(0000) knlGS:0000000000000000 Apr 18 06:16:18 NYCMedia kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 18 06:16:18 NYCMedia kernel: CR2: 00001462e9d9b000 CR3: 0000000005e0a000 CR4: 0000000000340ee0 Apr 18 06:16:18 NYCMedia kernel: Call Trace: Apr 18 06:16:18 NYCMedia kernel: <IRQ> Apr 18 06:16:18 NYCMedia kernel: nf_nat_used_tuple+0x2e/0x49 [nf_nat] Apr 18 06:16:18 NYCMedia kernel: nf_nat_setup_info+0x5fd/0x666 [nf_nat] Apr 18 06:16:18 NYCMedia kernel: nf_nat_alloc_null_binding+0x71/0x88 [nf_nat] Apr 18 06:16:18 NYCMedia kernel: nf_nat_inet_fn+0x9f/0x1b9 [nf_nat] Apr 18 06:16:18 NYCMedia kernel: ? br_handle_local_finish+0xe/0xe Apr 18 06:16:18 NYCMedia kernel: nf_nat_ipv4_in+0x1e/0x62 [nf_nat_ipv4] Apr 18 06:16:18 NYCMedia kernel: nf_hook_slow+0x3a/0x90 Apr 18 06:16:18 NYCMedia kernel: br_nf_pre_routing+0x303/0x343 Apr 18 06:16:18 NYCMedia kernel: ? br_nf_forward_ip+0x362/0x362 Apr 18 06:16:18 NYCMedia kernel: nf_hook_slow+0x3a/0x90 Apr 18 06:16:18 NYCMedia kernel: br_handle_frame+0x27e/0x2bd Apr 18 06:16:18 NYCMedia kernel: ? br_pass_frame_up+0x14a/0x14a Apr 18 06:16:18 NYCMedia kernel: __netif_receive_skb_core+0x4a7/0x7b1 Apr 18 06:16:18 NYCMedia kernel: ? udp_gro_receive+0x4b/0x136 Apr 18 06:16:18 NYCMedia kernel: __netif_receive_skb_one_core+0x35/0x6f Apr 18 06:16:18 NYCMedia kernel: netif_receive_skb_internal+0x79/0x94 Apr 18 06:16:18 NYCMedia kernel: napi_gro_receive+0x44/0x7b Apr 18 06:16:18 NYCMedia kernel: ixgbe_poll+0xb97/0xce4 [ixgbe] Apr 18 06:16:18 NYCMedia kernel: net_rx_action+0x107/0x26c Apr 18 06:16:18 NYCMedia kernel: __do_softirq+0xc9/0x1d7 Apr 18 06:16:18 NYCMedia kernel: irq_exit+0x5e/0x9d Apr 18 06:16:18 NYCMedia kernel: do_IRQ+0xb2/0xd0 Apr 18 06:16:18 NYCMedia kernel: common_interrupt+0xf/0xf Apr 18 06:16:18 NYCMedia kernel: </IRQ> Apr 18 06:16:18 NYCMedia kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141 Apr 18 06:16:18 NYCMedia kernel: Code: ff 45 84 f6 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 7a 8d bb ff fb 66 0f 1f 44 00 00 <48> 2b 2c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cd Apr 18 06:16:18 NYCMedia kernel: RSP: 0018:ffffc90000257e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd7 Apr 18 06:16:18 NYCMedia kernel: RAX: ffff88bf4cc9fac0 RBX: ffff88bf3dc78c00 RCX: 000000000000001f Apr 18 06:16:18 NYCMedia kernel: RDX: 0000000000000000 RSI: 000000002c3889ce RDI: 0000000000000000 Apr 18 06:16:18 NYCMedia kernel: RBP: 00002be1be42a70e R08: 00002be1be42a70e R09: 0000000000003a17 Apr 18 06:16:18 NYCMedia kernel: R10: 000000003b3913f0 R11: 071c71c71c71c71c R12: 0000000000000002 Apr 18 06:16:18 NYCMedia kernel: R13: ffffffff81e5e1e0 R14: 0000000000000000 R15: ffffffff81e5e2b8 Apr 18 06:16:18 NYCMedia kernel: ? cpuidle_enter_state+0xbf/0x141 Apr 18 06:16:18 NYCMedia kernel: do_idle+0x17e/0x1fc Apr 18 06:16:18 NYCMedia kernel: cpu_startup_entry+0x6a/0x6c Apr 18 06:16:18 NYCMedia kernel: start_secondary+0x197/0x1b2 Apr 18 06:16:18 NYCMedia kernel: secondary_startup_64+0xa4/0xb0 Apr 18 06:16:18 NYCMedia kernel: Sending NMI from CPU 98 to CPUs 54: Apr 18 06:16:18 NYCMedia kernel: NMI backtrace for cpu 54 Apr 18 06:16:18 NYCMedia kernel: CPU: 54 PID: 76534 Comm: EmbyServer Tainted: P W O 4.19.107-Unraid #1 Apr 18 06:16:18 NYCMedia kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH II EXTREME ALPHA, BIOS 0902 03/17/2020 Apr 18 06:16:18 NYCMedia kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x6b/0x171 Apr 18 06:16:18 NYCMedia kernel: Code: 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02 00 65 48 03 15 80 6a f8 Apr 18 06:16:18 NYCMedia kernel: RSP: 0018:ffffc9000d0a3948 EFLAGS: 00000202 Apr 18 06:16:18 NYCMedia kernel: RAX: 0000000000740101 RBX: 0000000000000000 RCX: 000000009e0002bb Apr 18 06:16:18 NYCMedia kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff81e08df4 Apr 18 06:16:18 NYCMedia kernel: RBP: ffffffff81e08df4 R08: 00000000bec9ebc5 R09: ffffffff81c8aa80 Apr 18 06:16:18 NYCMedia kernel: R10: ffffc9000d0a3ec0 R11: ffffc9000d0a3c58 R12: 00000000000003fa Apr 18 06:16:18 NYCMedia kernel: R13: ffffffff81e095e8 R14: 0000000000000000 R15: ffff8881c5b697e0 Apr 18 06:16:18 NYCMedia kernel: FS: 00001529c3b1f700(0000) GS:ffff88bf4d980000(0000) knlGS:0000000000000000 Apr 18 06:16:18 NYCMedia kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 18 06:16:18 NYCMedia kernel: CR2: 000014aff8618dd0 CR3: 0000003e9a832000 CR4: 0000000000340ee0 Diagnostics attached. Any idea what's going on? nycmedia-diagnostics-20200418-0755.zip Quote Link to comment
trurl Posted April 18, 2020 Share Posted April 18, 2020 Not related, but why have you allocated 100G to docker image? Have you had problems filling it? 12 minutes ago, ensnare said: running network commands (ifconfig, ip a, etc...) locked it up. Maybe just a coincidence, since your diagnostics includes the results of ifconfig and ethtool and they look OK. Have you done memtest? Quote Link to comment
ensnare Posted April 18, 2020 Author Share Posted April 18, 2020 4 minutes ago, trurl said: Not related, but why have you allocated 100G to docker image? Have you had problems filling it? Maybe just a coincidence, since your diagnostics includes the results of ifconfig and ethtool and they look OK. Have you done memtest? I was building a docker image with a lot of built-in packages and got an error that space was running low, so I increased the size. Quote Link to comment
trurl Posted April 18, 2020 Share Posted April 18, 2020 Usually growing Docker image is due to an application writing to a path that isn't mapped. Very rare to need more than 20G and never much more. If it is growing something is set up wrong. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.