noties

Members
 View Profile  See their activity

  • Posts

    47

  • Joined

  • Last visited

 Content Type 

Report Comments posted by noties

    Unraid OS version 6.9.0-rc2 available

    in Prereleases

    Posted

    On 2/24/2021 at 11:38 AM, jbartlett said:

    Note that the version of memtest built into UNRAID is a much older version that doesn't have the newer testing capabilities of the current version located on memtest's website. This is no fault of unraid, the people behind memtest won't allow anything newer to be installed by 3rd parties.

    I ran full memtest with newest version of memtest (9.0) and all memory passed testing.  I'm now thinking this is related to my docker networking, user defined networks, and the specific ethernet hardware I have.  I think it is related to the folliowing:  

     

     

    I have since moved my containers off of the Unraid IP physical port and moved them to another physical port.  I have not had crashes or errors in the last 24 hours, but will report back after a longer period of time.  I believe this to be my issue.

     

    Unraid OS version 6.9.0-rc2 available

    in Prereleases

    Posted

    On 2/20/2021 at 9:54 AM, trurl said:

    Have you done memtest?

     

    Yes, ran a memtest on each of the RAM sticks individually.  I was having BTRFS issues with my previous RAM and that sparked me to test the memory.  I've not seen BTRFS errors since swapping out the RAM and the RAM passed memtest.

     

    I changed a few things all at once and this seems to be biting me.  I swapped motherboards, RAM, and upgraded to 6.9rc2.  I had zero problems with my previous mobo, RAM and 6.8.3.

    Unraid OS version 6.9.0-rc2 available

    in Prereleases

    Posted

    Following up.  I ended up having a hard crash of my Unraid server a couple hours later after that message.  This time, I got the following message in syslog when it crashed.  Crashed, as in, dropped network connectivity, refused connections to any container, and no input on the physical console.

     

    Hoping this is right forum to be posting this.  Diagnostics attached.

     

      

    



HARD CRASH



Feb 19 15:00:02 teraserver rpcbind[3654]: connect from 192.168.4.7 to getport/addr(555555555)
Feb 19 15:02:16 teraserver rpcbind[11177]: connect from 192.168.4.7 to getport/addr(555555555)
Feb 19 15:07:33 teraserver ool www[22782]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/var/lib/docker' '-r'
Feb 19 15:07:33 teraserver kernel: BTRFS info (device loop2): scrub: started on devid 1
Feb 19 15:07:57 teraserver kernel: BTRFS info (device loop2): scrub: finished on devid 1 with status: 0
Feb 19 15:08:05 teraserver kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
Feb 19 15:08:32 teraserver emhttpd: shcmd (235): /usr/local/sbin/mover &> /dev/null &
Feb 19 15:13:06 teraserver kernel: general protection fault, probably for non-canonical address 0x315a4c61f4cef4d8: 0000 [#1] SMP PTI
Feb 19 15:13:06 teraserver kernel: CPU: 15 PID: 0 Comm: swapper/15 Tainted: P        W  O      5.10.1-Unraid #1
Feb 19 15:13:06 teraserver kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X79 Extreme4-M, BIOS P3.40 03/10/2018
Feb 19 15:13:06 teraserver kernel: RIP: 0010:nf_conntrack_udp_packet+0x119/0x219
Feb 19 15:13:06 teraserver kernel: Code: 81 48 8b 73 20 49 c7 c0 9b 6d dc 81 b9 11 00 00 00 e8 f0 9f 11 00 83 c8 ff e9 ea 00 00 00 48 8b 85 b8 00 00 00 48 85 c0 74 1b <0f> b6 50 06 84 d2 74 13 48 01 d0 74 0e 48 8b 00 48 85 c0 74 06 48
Feb 19 15:13:06 teraserver kernel: RSP: 0018:ffffc90000494b70 EFLAGS: 00010206
Feb 19 15:13:06 teraserver kernel: RAX: 315a4c61f4cef4d8 RBX: ffffc90000494c68 RCX: 0000000000000011
Feb 19 15:13:06 teraserver kernel: RDX: 0000000000000014 RSI: 0000000000000000 RDI: ffff888596154200
Feb 19 15:13:06 teraserver kernel: RBP: ffff888115eae3c0 R08: ffff88819c1a47d0 R09: ffff888115eae3c0
Feb 19 15:13:06 teraserver kernel: R10: ffffffff8210da40 R11: ffff888115eae3c0 R12: 0000000000000002
Feb 19 15:13:06 teraserver kernel: R13: 00000000000000b0 R14: 0000000000000014 R15: ffff888115eae3c0
Feb 19 15:13:06 teraserver kernel: FS:  0000000000000000(0000) GS:ffff888627bc0000(0000) knlGS:0000000000000000
Feb 19 15:13:06 teraserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 19 15:13:06 teraserver kernel: CR2: 000014e0b6b8fbd0 CR3: 000000000200c002 CR4: 00000000001706e0
Feb 19 15:13:06 teraserver kernel: Call Trace:
Feb 19 15:13:06 teraserver kernel: <IRQ>
Feb 19 15:13:06 teraserver kernel: nf_conntrack_in+0x2bc/0x3b7
Feb 19 15:13:06 teraserver kernel: ? br_handle_frame_finish+0x351/0x351
Feb 19 15:13:06 teraserver kernel: nf_hook_slow+0x39/0x8e
Feb 19 15:13:06 teraserver kernel: ? br_nf_forward_finish+0xd0/0xd0
Feb 19 15:13:06 teraserver kernel: NF_HOOK+0xb7/0xf7
Feb 19 15:13:06 teraserver kernel: ? br_nf_forward_finish+0xd0/0xd0
Feb 19 15:13:06 teraserver kernel: br_nf_pre_routing+0x229/0x239
Feb 19 15:13:06 teraserver kernel: ? br_nf_forward_finish+0xd0/0xd0
Feb 19 15:13:06 teraserver kernel: br_handle_frame+0x25e/0x2a6
Feb 19 15:13:06 teraserver kernel: ? br_pass_frame_up+0xda/0xda
Feb 19 15:13:06 teraserver kernel: __netif_receive_skb_core+0x335/0x4e7
Feb 19 15:13:06 teraserver kernel: ? inet_gro_receive+0x252/0x264
Feb 19 15:13:06 teraserver kernel: __netif_receive_skb_list_core+0x78/0x104
Feb 19 15:13:06 teraserver kernel: netif_receive_skb_list_internal+0x1bf/0x1f2
Feb 19 15:13:06 teraserver kernel: gro_normal_list+0x1d/0x39
Feb 19 15:13:06 teraserver kernel: napi_complete_done+0x79/0x104
Feb 19 15:13:06 teraserver kernel: tg3_poll_msix+0xb3/0x124 [tg3]
Feb 19 15:13:06 teraserver kernel: net_rx_action+0xf4/0x29d
Feb 19 15:13:06 teraserver kernel: __do_softirq+0xc4/0x1c2
Feb 19 15:13:06 teraserver kernel: asm_call_irq_on_stack+0x12/0x20
Feb 19 15:13:06 teraserver kernel: </IRQ>
Feb 19 15:13:06 teraserver kernel: do_softirq_own_stack+0x2c/0x39
Feb 19 15:13:06 teraserver kernel: __irq_exit_rcu+0x45/0x80
Feb 19 15:13:06 teraserver kernel: common_interrupt+0x119/0x12e
Feb 19 15:13:06 teraserver kernel: asm_common_interrupt+0x1e/0x40
Feb 19 15:13:06 teraserver kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Feb 19 15:13:06 teraserver kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5
Feb 19 15:13:06 teraserver kernel: RSP: 0018:ffffc900000e7ea0 EFLAGS: 00000246
Feb 19 15:13:06 teraserver kernel: RAX: ffff888627be2300 RBX: 0000000000000004 RCX: 000000000000001f
Feb 19 15:13:06 teraserver kernel: RDX: 0000000000000000 RSI: 000000002f6898f3 RDI: 0000000000000000
Feb 19 15:13:06 teraserver kernel: RBP: ffffe8fffefc8200 R08: 0000032cd47e275f R09: 0000000000000000
Feb 19 15:13:06 teraserver kernel: R10: 0000000000002bd2 R11: 071c71c71c71c71c R12: 0000032cd47e275f
Feb 19 15:13:06 teraserver kernel: R13: ffffffff820c7e80 R14: 0000000000000004 R15: 0000000000000000
Feb 19 15:13:06 teraserver kernel: cpuidle_enter_state+0x101/0x1c4
Feb 19 15:13:06 teraserver kernel: cpuidle_enter+0x25/0x31
Feb 19 15:13:06 teraserver kernel: do_idle+0x1a1/0x20f
Feb 19 15:13:06 teraserver kernel: cpu_startup_entry+0x18/0x1a
Feb 19 15:13:06 teraserver kernel: secondary_startup_64_no_verify+0xb0/0xbb
Feb 19 15:13:06 teraserver kernel: Modules linked in: veth xt_nat xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost vhost_iotlb tap macvlan xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea sysfillrect nvidia_uvm(PO) sysimgblt fb_sys_fops nvidia(PO) nct6775 hwmon_vid r8169 realtek tg3 x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd mxm_wmi cryptd glue_helper rapl i2c_i801 mpt3sas intel_cstate i2c_smbus i2c_core raid_class intel_uncore ahci wmi scsi_transport_sas libahci button [last unloaded: realtek]
Feb 19 15:13:06 teraserver kernel: ---[ end trace 53855679afd353bd ]---
Feb 19 15:13:06 teraserver kernel: RIP: 0010:nf_conntrack_udp_packet+0x119/0x219
Feb 19 15:13:06 teraserver kernel: Code: 81 48 8b 73 20 49 c7 c0 9b 6d dc 81 b9 11 00 00 00 e8 f0 9f 11 00 83 c8 ff e9 ea 00 00 00 48 8b 85 b8 00 00 00 48 85 c0 74 1b <0f> b6 50 06 84 d2 74 13 48 01 d0 74 0e 48 8b 00 48 85 c0 74 06 48
Feb 19 15:13:06 teraserver kernel: RSP: 0018:ffffc90000494b70 EFLAGS: 00010206
Feb 19 15:13:06 teraserver kernel: RAX: 315a4c61f4cef4d8 RBX: ffffc90000494c68 RCX: 0000000000000011
Feb 19 15:13:06 teraserver kernel: RDX: 0000000000000014 RSI: 0000000000000000 RDI: ffff888596154200
Feb 19 15:13:06 teraserver kernel: RBP: ffff888115eae3c0 R08: ffff88819c1a47d0 R09: ffff888115eae3c0
Feb 19 15:13:06 teraserver kernel: R10: ffffffff8210da40 R11: ffff888115eae3c0 R12: 0000000000000002
Feb 19 15:13:06 teraserver kernel: R13: 00000000000000b0 R14: 0000000000000014 R15: ffff888115eae3c0
Feb 19 15:13:06 teraserver kernel: FS:  0000000000000000(0000) GS:ffff888627bc0000(0000) knlGS:0000000000000000

     

    teraserver-diagnostics-20210220-0829.zip

    Unraid OS version 6.9.0-rc2 available

    in Prereleases

    Posted

    Got the following message in syslog on RC2.  second time I've seen it and I don't notice anything wrong with my server, but thought I would post to see if someone knows something about it.  Looks like something related to the macvlan process maybe?

      

    Feb 19 13:23:57 teraserver kernel: ------------[ cut here ]------------
Feb 19 13:23:57 teraserver kernel: WARNING: CPU: 13 PID: 23974 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x99/0x1e1
Feb 19 13:23:57 teraserver kernel: Modules linked in: vhost_net tun vhost vhost_iotlb tap kvm_intel kvm xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle ip6table_filter ip6_tables md_mod dm_crypt dm_mod dax veth macvlan xt_nat xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs nfsd lockd grace sunrpc nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea nvidia_uvm(PO) sysfillrect sysimgblt fb_sys_fops nvidia(PO) nct6775 hwmon_vid r8169 realtek tg3 x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd mxm_wmi mpt3sas glue_helper rapl i2c_i801 intel_cstate raid_class i2c_smbus intel_uncore i2c_core input_leds ahci led_class scsi_transport_sas wmi libahci button [last unloaded: kvm]
Feb 19 13:23:57 teraserver kernel: CPU: 13 PID: 23974 Comm: kworker/13:1 Tainted: P        W  O      5.10.1-Unraid #1
Feb 19 13:23:57 teraserver kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X79 Extreme4-M, BIOS P3.40 03/10/2018
Feb 19 13:23:57 teraserver kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Feb 19 13:23:57 teraserver kernel: RIP: 0010:__nf_conntrack_confirm+0x99/0x1e1
Feb 19 13:23:57 teraserver kernel: Code: e4 e3 ff ff 8b 54 24 14 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 54 e1 ff ff 84 c0 75 b8 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 89 de ff ff e8 af e0 ff ff e9 1f 01
Feb 19 13:23:57 teraserver kernel: RSP: 0018:ffffc9000043cd38 EFLAGS: 00010202
Feb 19 13:23:57 teraserver kernel: RAX: 0000000000000188 RBX: 0000000000000478 RCX: 000000007f7d0276
Feb 19 13:23:57 teraserver kernel: RDX: 0000000000000000 RSI: 00000000000000f0 RDI: ffffffff82009c00
Feb 19 13:23:57 teraserver kernel: RBP: ffff888271c808c0 R08: 00000000fa10a875 R09: ffff8881a92aeaa0
Feb 19 13:23:57 teraserver kernel: R10: 0000000000000158 R11: ffff888125843200 R12: 00000000000000f0
Feb 19 13:23:57 teraserver kernel: R13: ffffffff8210da40 R14: 0000000000000478 R15: ffff888271c808cc
Feb 19 13:23:57 teraserver kernel: FS:  0000000000000000(0000) GS:ffff888627b40000(0000) knlGS:0000000000000000
Feb 19 13:23:57 teraserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 19 13:23:57 teraserver kernel: CR2: 00003c2889e6a008 CR3: 000000000200c001 CR4: 00000000001706e0
Feb 19 13:23:57 teraserver kernel: Call Trace:
Feb 19 13:23:57 teraserver kernel: <IRQ>
Feb 19 13:23:57 teraserver kernel: nf_conntrack_confirm+0x2f/0x36
Feb 19 13:23:57 teraserver kernel: nf_hook_slow+0x39/0x8e
Feb 19 13:23:57 teraserver kernel: nf_hook.constprop.0+0xb1/0xd8
Feb 19 13:23:57 teraserver kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe
Feb 19 13:23:57 teraserver kernel: ip_local_deliver+0x49/0x75
Feb 19 13:23:57 teraserver kernel: ip_sabotage_in+0x43/0x4d
Feb 19 13:23:57 teraserver kernel: nf_hook_slow+0x39/0x8e
Feb 19 13:23:57 teraserver kernel: nf_hook.constprop.0+0xb1/0xd8
Feb 19 13:23:57 teraserver kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50
Feb 19 13:23:57 teraserver kernel: ip_rcv+0x41/0x61
Feb 19 13:23:57 teraserver kernel: __netif_receive_skb_one_core+0x74/0x95
Feb 19 13:23:57 teraserver kernel: process_backlog+0xa3/0x13b
Feb 19 13:23:57 teraserver kernel: net_rx_action+0xf4/0x29d
Feb 19 13:23:57 teraserver kernel: __do_softirq+0xc4/0x1c2
Feb 19 13:23:57 teraserver kernel: asm_call_irq_on_stack+0x12/0x20
Feb 19 13:23:57 teraserver kernel: </IRQ>
Feb 19 13:23:57 teraserver kernel: do_softirq_own_stack+0x2c/0x39
Feb 19 13:23:57 teraserver kernel: do_softirq+0x3a/0x44
Feb 19 13:23:57 teraserver kernel: netif_rx_ni+0x1c/0x22
Feb 19 13:23:57 teraserver kernel: macvlan_broadcast+0x10e/0x13c [macvlan]
Feb 19 13:23:57 teraserver kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan]
Feb 19 13:23:57 teraserver kernel: process_one_work+0x13c/0x1d5
Feb 19 13:23:57 teraserver kernel: worker_thread+0x18b/0x22f
Feb 19 13:23:57 teraserver kernel: ? process_scheduled_works+0x27/0x27
Feb 19 13:23:57 teraserver kernel: kthread+0xe5/0xea
Feb 19 13:23:57 teraserver kernel: ? kthread_unpark+0x52/0x52
Feb 19 13:23:57 teraserver kernel: ret_from_fork+0x22/0x30
Feb 19 13:23:57 teraserver kernel: ---[ end trace 388b86a70436712d ]---