• [6.6.0] Kernel Panics

    Robert Long

    By Robert Long

      • Retest Urgent

    Hey Guys,

     

    Have upgraded to 6.6.0 recently and ever since I've been getting kernel panics around every 12 hours or so, as expected requires a hard reset to get it back up and going again.

     

    Doesn't seem hardware related as it was running fine on the previous stable version.

     

    Diagnostics attached

     

    Thanks!

    tower-diagnostics-20180926-1022.zip

    Go to reports Stable Releases

    User Feedback

    Recommended Comments

    EdwinKlemann

    I have a simmular problem, ever since 6.60 i also get kernel panics ( it just doesn't take 12 hours to get them) . I reverted back to 6.5.3 for now and it's running fine again. I tried 6.6.1 without success. eg. same issue. Do I need to update again in order to attach a diagnostic file ? or can i generate one while running 6.5.3. My config hasn't changed yet. Can I maybe increase the console resolution (so more text is visible) in order to see more kernel panic output ? 

    Link to comment
    Robert Long

    Yeh, from what I'm seeing, it looks like it's most likely an issue with the new Kernel in the 6.6.0+ releases, I'm not sure if that bugfix has made it to GA but I'm sure once it does the guys at Lime Tech will jump on it.

     

    I guess it's just a waiting game for the time being.

    Link to comment
    EdwinKlemann

    2x Onboard Broadcom NexExtreme II

    [14e4:164c]06:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

    [14e4:164c]03:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5708 Gigabit Ethernet (rev 12)

    1x Dual Intel Gigabit Controller PCIe

    [8086:105e]10:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (rev 06)

    [8086:105e]10:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (rev 06)

    1x Quad Intel Gigabit Controller PCIe

    [8086:10bc]26:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

    [8086:10bc]26:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

    [8086:10bc]27:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

    [8086:10bc]27:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

    All interfaces Bonded in Active-Backup. 

    Not all interfaces connected (the Adapters were installed in the server when i got it. too much trouble to remove them ;-) )

    1x Broadcom, 2x Intel connected but has been and still works flawless in 6.5.3. 

     

    Link to comment
    limetech

    There is not enough info in the screen shots to give a clue about what is failing.

    Please open a terminal window and tail the syslog:

     

    tail -f /var/log/syslog

     

    Then when server hits this issue you can select/copy/paste the entire window contents.

     

    Also, never hurts to run in 'Safe Mode' to rule out plugin issue.

    Link to comment
    EdwinKlemann

    login as: root
    [email protected]'s password:
    Linux 4.18.14-unRAID.
    root@Tower:~# tail -f /var/log/syslog
    Oct 18 11:32:25 Tower kernel: docker0: port 22(veth333a06f) entered disabled state
    Oct 18 11:32:25 Tower kernel: device veth333a06f entered promiscuous mode
    Oct 18 11:32:25 Tower kernel: IPv6: ADDRCONF(NETDEV_UP): veth333a06f: link is not ready
    Oct 18 11:32:25 Tower kernel: docker0: port 22(veth333a06f) entered blocking state
    Oct 18 11:32:25 Tower kernel: docker0: port 22(veth333a06f) entered forwarding state
    Oct 18 11:32:25 Tower kernel: docker0: port 22(veth333a06f) entered disabled state
    Oct 18 11:32:30 Tower kernel: eth0: renamed from veth85f0673
    Oct 18 11:32:30 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth333a06f: link becomes ready
    Oct 18 11:32:30 Tower kernel: docker0: port 22(veth333a06f) entered blocking state
    Oct 18 11:32:30 Tower kernel: docker0: port 22(veth333a06f) entered forwarding state
    Oct 18 11:32:32 Tower rc.docker: radarr: started succesfully!
    Oct 18 11:32:32 Tower avahi-daemon[17279]: Joining mDNS multicast group on interface veth333a06f.IPv6 with address fe80::38f9:e0ff:fe17:4264.
    Oct 18 11:32:32 Tower avahi-daemon[17279]: New relevant interface veth333a06f.IPv6 for mDNS.
    Oct 18 11:32:32 Tower avahi-daemon[17279]: Registering new address record for fe80::38f9:e0ff:fe17:4264 on veth333a06f.*.
    Oct 18 11:32:33 Tower kernel: docker0: port 23(vethd74dfe6) entered blocking state
    Oct 18 11:32:33 Tower kernel: docker0: port 23(vethd74dfe6) entered disabled state
    Oct 18 11:32:33 Tower kernel: device vethd74dfe6 entered promiscuous mode
    Oct 18 11:32:33 Tower kernel: IPv6: ADDRCONF(NETDEV_UP): vethd74dfe6: link is not ready
    Oct 18 11:32:33 Tower kernel: docker0: port 23(vethd74dfe6) entered blocking state
    Oct 18 11:32:33 Tower kernel: docker0: port 23(vethd74dfe6) entered forwarding state
    Oct 18 11:32:33 Tower kernel: docker0: port 23(vethd74dfe6) entered disabled state
    Oct 18 11:32:44 Tower kernel: eth0: renamed from vethab027f7
    Oct 18 11:32:44 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethd74dfe6: link becomes ready
    Oct 18 11:32:44 Tower kernel: docker0: port 23(vethd74dfe6) entered blocking state
    Oct 18 11:32:44 Tower kernel: docker0: port 23(vethd74dfe6) entered forwarding state
    Oct 18 11:32:46 Tower avahi-daemon[17279]: Joining mDNS multicast group on interface vethd74dfe6.IPv6 with address fe80::dc44:7cff:fecd:e7aa.
    Oct 18 11:32:46 Tower avahi-daemon[17279]: New relevant interface vethd74dfe6.IPv6 for mDNS.
    Oct 18 11:32:46 Tower avahi-daemon[17279]: Registering new address record for fe80::dc44:7cff:fecd:e7aa on vethd74dfe6.*.
    Oct 18 11:32:46 Tower rc.docker: Lidarr: started succesfully!
    Oct 18 11:32:47 Tower kernel: docker0: port 24(veth5b4d02a) entered blocking state
    Oct 18 11:32:47 Tower kernel: docker0: port 24(veth5b4d02a) entered disabled state
    Oct 18 11:32:47 Tower kernel: device veth5b4d02a entered promiscuous mode
    Oct 18 11:32:47 Tower kernel: IPv6: ADDRCONF(NETDEV_UP): veth5b4d02a: link is not ready
    Oct 18 11:32:47 Tower kernel: docker0: port 24(veth5b4d02a) entered blocking state
    Oct 18 11:32:47 Tower kernel: docker0: port 24(veth5b4d02a) entered forwarding state
    Oct 18 11:32:47 Tower kernel: docker0: port 24(veth5b4d02a) entered disabled state
    Oct 18 11:32:53 Tower kernel: eth0: renamed from veth81d2ef1
    Oct 18 11:32:53 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5b4d02a: link becomes ready
    Oct 18 11:32:53 Tower kernel: docker0: port 24(veth5b4d02a) entered blocking state
    Oct 18 11:32:53 Tower kernel: docker0: port 24(veth5b4d02a) entered forwarding state
    Oct 18 11:32:54 Tower avahi-daemon[17279]: Joining mDNS multicast group on interface veth5b4d02a.IPv6 with address fe80::c4e6:74ff:fe86:b1fe.
    Oct 18 11:32:54 Tower avahi-daemon[17279]: New relevant interface veth5b4d02a.IPv6 for mDNS.
    Oct 18 11:32:54 Tower avahi-daemon[17279]: Registering new address record for fe80::c4e6:74ff:fe86:b1fe on veth5b4d02a.*.
    Oct 18 11:32:55 Tower rc.docker: sonarr: started succesfully!
    Oct 18 11:32:56 Tower kernel: docker0: port 25(veth35ef07a) entered blocking state
    Oct 18 11:32:56 Tower kernel: docker0: port 25(veth35ef07a) entered disabled state
    Oct 18 11:32:56 Tower kernel: device veth35ef07a entered promiscuous mode
    Oct 18 11:32:56 Tower kernel: IPv6: ADDRCONF(NETDEV_UP): veth35ef07a: link is not ready
    Oct 18 11:32:56 Tower kernel: docker0: port 25(veth35ef07a) entered blocking state
    Oct 18 11:32:56 Tower kernel: docker0: port 25(veth35ef07a) entered forwarding state
    Oct 18 11:32:56 Tower kernel: docker0: port 25(veth35ef07a) entered disabled state
    Oct 18 11:32:58 Tower dhcpcd[1814]: br0: fe80::3a10:d5ff:fe22:31db is unreachable, expiring it
    Oct 18 11:33:02 Tower kernel: eth0: renamed from vethbc86383
    Oct 18 11:33:02 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth35ef07a: link becomes ready
    Oct 18 11:33:02 Tower kernel: docker0: port 25(veth35ef07a) entered blocking state
    Oct 18 11:33:02 Tower kernel: docker0: port 25(veth35ef07a) entered forwarding state
    Oct 18 11:33:04 Tower rc.docker: ubooquity: started succesfully!
    Oct 18 11:33:04 Tower avahi-daemon[17279]: Joining mDNS multicast group on interface veth35ef07a.IPv6 with address fe80::1477:a3ff:fe0e:788d.
    Oct 18 11:33:04 Tower avahi-daemon[17279]: New relevant interface veth35ef07a.IPv6 for mDNS.
    Oct 18 11:33:04 Tower avahi-daemon[17279]: Registering new address record for fe80::1477:a3ff:fe0e:788d on veth35ef07a.*.
    Oct 18 11:33:45 Tower dhcpcd[1814]: br0: DHCPv6 REPLY: in stateless mode
    Oct 18 11:37:00 Tower root: Fix Common Problems Version 2018.10.02
    Oct 18 11:38:16 Tower kernel: general protection fault: 0000 [#1] SMP PTI
    Oct 18 11:38:16 Tower kernel: CPU: 6 PID: 41 Comm: ksoftirqd/6 Not tainted 4.18.14-unRAID #1
    Oct 18 11:38:16 Tower kernel: Hardware name: IBM IBM System x3650 -[7979CBG]-/System Planar, BIOS -[GGE149AUS-1.19]- 02/11/2011
    Oct 18 11:38:16 Tower kernel: RIP: 0010:__srcu_read_unlock+0x9/0x17
    Oct 18 11:38:16 Tower kernel: Code: 5d c3 8b 87 c0 06 00 00 48 8b 97 e8 06 00 00 83 e0 01 48 63 c8 65 48 ff 04 ca f0 83 44 24 fc 00 c3 f0 83 44 24 fc 00 48 63 f6 <48> 8b 87 e8 06 00 00 65 48 ff 44 f0 10 c3 8b 87 c0 06 00 00 c3 55
    Oct 18 11:38:16 Tower kernel: RSP: 0018:ffffc900063bbd68 EFLAGS: 00010246
    Oct 18 11:38:16 Tower kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
    Oct 18 11:38:16 Tower kernel: RDX: 00000000000f4240 RSI: 0000000000000000 RDI: 070011ac294d1158
    Oct 18 11:38:16 Tower kernel: RBP: ffff8804c6486200 R08: 0000000000000000 R09: 0000000000000000
    Oct 18 11:38:16 Tower kernel: R10: 00000000000007b2 R11: ffff880b4fde0c80 R12: 0000000000000246
    Oct 18 11:38:16 Tower kernel: R13: 0000000000000000 R14: ffff880b18b49840 R15: dead000000000100
    Oct 18 11:38:16 Tower kernel: FS:  0000000000000000(0000) GS:ffff880b4fd80000(0000) knlGS:0000000000000000
    Oct 18 11:38:16 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Oct 18 11:38:16 Tower kernel: CR2: 000001e2dd518028 CR3: 000000054ca74000 CR4: 00000000000026e0
    Oct 18 11:38:16 Tower kernel: Call Trace:
    Oct 18 11:38:16 Tower kernel: deliver_response+0x8a/0xbb
    Oct 18 11:38:16 Tower kernel: deliver_local_response+0x9/0x1f
    Oct 18 11:38:16 Tower kernel: handle_one_recv_msg+0xaa9/0xad3
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x40/0x70
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x34/0x70
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x40/0x70
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x34/0x70
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x40/0x70
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x34/0x70
    Oct 18 11:38:16 Tower kernel: ? __switch_to_asm+0x40/0x70
    Oct 18 11:38:16 Tower kernel: handle_new_recv_msgs+0x94/0x176
    Oct 18 11:38:16 Tower kernel: tasklet_action_common.isra.3+0x63/0xa2
    Oct 18 11:38:16 Tower kernel: __do_softirq+0xce/0x1c8
    Oct 18 11:38:16 Tower kernel: ? smpboot_park_thread+0x25/0x25
    Oct 18 11:38:16 Tower kernel: run_ksoftirqd+0x19/0x2d
    Oct 18 11:38:16 Tower kernel: smpboot_thread_fn+0x134/0x149
    Oct 18 11:38:16 Tower kernel: kthread+0x10b/0x113
    Oct 18 11:38:16 Tower kernel: ? kthread_flush_work_fn+0x9/0x9
    Oct 18 11:38:16 Tower kernel: ret_from_fork+0x35/0x40
    Oct 18 11:38:16 Tower kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net vhost tun tap xt_nat veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat nfsv3 nfs ext4 mbcache jbd2 xfs nfsd lockd grace sunrpc md_mod ipmi_devintf bonding e1000e bnx2 sr_mod cdrom ibmpex coretemp kvm_intel ibmaem i5000_edac kvm i5k_amb aacraid ipmi_ssif i2c_i801 i2c_core ahci libahci ata_piix pcc_cpufreq button ipmi_si acpi_cpufreq [last unloaded: e1000e]
    Oct 18 11:38:16 Tower kernel: ---[ end trace d753477d6817eab2 ]---
    Oct 18 11:38:16 Tower kernel: RIP: 0010:__srcu_read_unlock+0x9/0x17
    Oct 18 11:38:16 Tower kernel: Code: 5d c3 8b 87 c0 06 00 00 48 8b 97 e8 06 00 00 83 e0 01 48 63 c8 65 48 ff 04 ca f0 83 44 24 fc 00 c3 f0 83 44 24 fc 00 48 63 f6 <48> 8b 87 e8 06 00 00 65 48 ff 44 f0 10 c3 8b 87 c0 06 00 00 c3 55
    Oct 18 11:38:16 Tower kernel: RSP: 0018:ffffc900063bbd68 EFLAGS: 00010246
    Oct 18 11:38:16 Tower kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
    Oct 18 11:38:16 Tower kernel: RDX: 00000000000f4240 RSI: 0000000000000000 RDI: 070011ac294d1158
    Oct 18 11:38:16 Tower kernel: RBP: ffff8804c6486200 R08: 0000000000000000 R09: 0000000000000000
    Oct 18 11:38:16 Tower kernel: R10: 00000000000007b2 R11: ffff880b4fde0c80 R12: 0000000000000246
    Oct 18 11:38:16 Tower kernel: R13: 0000000000000000 R14: ffff880b18b49840 R15: dead000000000100
    Oct 18 11:38:16 Tower kernel: FS:  0000000000000000(0000) GS:ffff880b4fd80000(0000) knlGS:0000000000000000
     

     

    does this help ? 

    Didn't try safe mode yet because i have docker containers that are linked to NFS shares on other hosts.  

    downgraded back to 6.5.3 for now

    Link to comment
    limetech

    Yes, thanks.  googling the failure:

     

    kernel: general protection fault: 0000 [#1] SMP PTI

     

    doesn't reveal much, other than other people in the linux community has seen this.

     

    This automated kernel test looks similar:

    https://lkml.org/lkml/2018/2/22/937

     

    Probably this issue will get fixed in a future kernel patch release.

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    ×

  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.

    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.