Skip to content
View in the app

A better way to browse. Learn more.
Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.
Unraid Summer Sale in live: 20% off Starter and Unleashed + buy one, get one 50% off

unRAID crashing with increasing frequency over the last few weeks - "e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:"

dglb99
in General Support

Featured Replies

Hi all,

My unraid server has been crashing and the frequency has continued to increase and at this point I am not sure what to do.  From the persistent syslog, once this error starts it loops forever until I hard reboot the server. 

Sep 18 20:35:38 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:
Sep 18 20:35:38 NASDex kernel:  TDH                  <0>
Sep 18 20:35:38 NASDex kernel:  TDT                  <7>
Sep 18 20:35:38 NASDex kernel:  next_to_use          <7>
Sep 18 20:35:38 NASDex kernel:  next_to_clean        <0>
Sep 18 20:35:38 NASDex kernel: buffer_info[next_to_clean]:
Sep 18 20:35:38 NASDex kernel:  time_stamp           <124b24440>
Sep 18 20:35:38 NASDex kernel:  next_to_watch        <0>
Sep 18 20:35:38 NASDex kernel:  jiffies              <124b26b80>
Sep 18 20:35:38 NASDex kernel:  next_to_watch.status <0>
Sep 18 20:35:38 NASDex kernel: MAC Status             <40080083>
Sep 18 20:35:38 NASDex kernel: PHY Status             <796d>
Sep 18 20:35:38 NASDex kernel: PHY 1000BASE-T Status  <3800>
Sep 18 20:35:38 NASDex kernel: PHY Extended Status    <3000>
Sep 18 20:35:38 NASDex kernel: PCI Status             <10>
Sep 18 20:35:38 NASDex kernel: e1000e 0000:00:1f.6 eth0: Reset adapter unexpectedly
Sep 18 20:35:38 NASDex kernel: br0: port 1(eth0) entered disabled state
Sep 18 20:35:41 NASDex kernel: e1000e 0000:00:1f.6 eth0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Sep 18 20:35:41 NASDex kernel: br0: port 1(eth0) entered blocking state
Sep 18 20:35:41 NASDex kernel: br0: port 1(eth0) entered forwarding state
Sep 18 20:35:44 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:
Sep 18 20:35:44 NASDex kernel:  TDH                  <0>
Sep 18 20:35:44 NASDex kernel:  TDT                  <7>
Sep 18 20:35:44 NASDex kernel:  next_to_use          <7>
Sep 18 20:35:44 NASDex kernel:  next_to_clean        <0>
Sep 18 20:35:44 NASDex kernel: buffer_info[next_to_clean]:
Sep 18 20:35:44 NASDex kernel:  time_stamp           <124b27c40>
Sep 18 20:35:44 NASDex kernel:  next_to_watch        <0>
Sep 18 20:35:44 NASDex kernel:  jiffies              <124b28301>
Sep 18 20:35:44 NASDex kernel:  next_to_watch.status <0>
Sep 18 20:35:44 NASDex kernel: MAC Status             <40080083>
Sep 18 20:35:44 NASDex kernel: PHY Status             <796d>
Sep 18 20:35:44 NASDex kernel: PHY 1000BASE-T Status  <3800>
Sep 18 20:35:44 NASDex kernel: PHY Extended Status    <3000>
Sep 18 20:35:44 NASDex kernel: PCI Status             <10>
Sep 18 20:35:46 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:
Sep 18 20:35:46 NASDex kernel:  TDH                  <0>
Sep 18 20:35:46 NASDex kernel:  TDT                  <7>
Sep 18 20:35:46 NASDex kernel:  next_to_use          <7>
Sep 18 20:35:46 NASDex kernel:  next_to_clean        <0>
Sep 18 20:35:46 NASDex kernel: buffer_info[next_to_clean]:
Sep 18 20:35:46 NASDex kernel:  time_stamp           <124b27c40>
Sep 18 20:35:46 NASDex kernel:  next_to_watch        <0>
Sep 18 20:35:46 NASDex kernel:  jiffies              <124b28ac0>
Sep 18 20:35:46 NASDex kernel:  next_to_watch.status <0>
Sep 18 20:35:46 NASDex kernel: MAC Status             <40080083>
Sep 18 20:35:46 NASDex kernel: PHY Status             <796d>
Sep 18 20:35:46 NASDex kernel: PHY 1000BASE-T Status  <3800>
Sep 18 20:35:46 NASDex kernel: PHY Extended Status    <3000>
Sep 18 20:35:46 NASDex kernel: PCI Status             <10>
Sep 18 20:35:48 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:
Sep 18 20:35:48 NASDex kernel:  TDH                  <0>
Sep 18 20:35:48 NASDex kernel:  TDT                  <7>
Sep 18 20:35:48 NASDex kernel:  next_to_use          <7>
Sep 18 20:35:48 NASDex kernel:  next_to_clean        <0>
Sep 18 20:35:48 NASDex kernel: buffer_info[next_to_clean]:
Sep 18 20:35:48 NASDex kernel:  time_stamp           <124b27c40>
Sep 18 20:35:48 NASDex kernel:  next_to_watch        <0>
Sep 18 20:35:48 NASDex kernel:  jiffies              <124b292c0>
Sep 18 20:35:48 NASDex kernel:  next_to_watch.status <0>
Sep 18 20:35:48 NASDex kernel: MAC Status             <40080083>
Sep 18 20:35:48 NASDex kernel: PHY Status             <796d>
Sep 18 20:35:48 NASDex kernel: PHY 1000BASE-T Status  <3800>
Sep 18 20:35:48 NASDex kernel: PHY Extended Status    <3000>
Sep 18 20:35:48 NASDex kernel: PCI Status             <10>
Sep 18 20:35:50 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:
Sep 18 20:35:50 NASDex kernel:  TDH                  <0>
Sep 18 20:35:50 NASDex kernel:  TDT                  <7>
Sep 18 20:35:50 NASDex kernel:  next_to_use          <7>
Sep 18 20:35:50 NASDex kernel:  next_to_clean        <0>
Sep 18 20:35:50 NASDex kernel: buffer_info[next_to_clean]:
Sep 18 20:35:50 NASDex kernel:  time_stamp           <124b27c40>
Sep 18 20:35:50 NASDex kernel:  next_to_watch        <0>
Sep 18 20:35:50 NASDex kernel:  jiffies              <124b29a80>
Sep 18 20:35:50 NASDex kernel:  next_to_watch.status <0>
Sep 18 20:35:50 NASDex kernel: MAC Status             <40080083>
Sep 18 20:35:50 NASDex kernel: PHY Status             <796d>
Sep 18 20:35:50 NASDex kernel: PHY 1000BASE-T Status  <3800>
Sep 18 20:35:50 NASDex kernel: PHY Extended Status    <3000>
Sep 18 20:35:50 NASDex kernel: PCI Status             <10>
Sep 18 20:35:52 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:
Sep 18 20:35:52 NASDex kernel:  TDH                  <0>
Sep 18 20:35:52 NASDex kernel:  TDT                  <7>
Sep 18 20:35:52 NASDex kernel:  next_to_use          <7>
Sep 18 20:35:52 NASDex kernel:  next_to_clean        <0>
Sep 18 20:35:52 NASDex kernel: buffer_info[next_to_clean]:
Sep 18 20:35:52 NASDex kernel:  time_stamp           <124b27c40>
Sep 18 20:35:52 NASDex kernel:  next_to_watch        <0>
Sep 18 20:35:52 NASDex kernel:  jiffies              <124b2a240>
Sep 18 20:35:52 NASDex kernel:  next_to_watch.status <0>
Sep 18 20:35:52 NASDex kernel: MAC Status             <40080083>
Sep 18 20:35:52 NASDex kernel: PHY Status             <796d>
Sep 18 20:35:52 NASDex kernel: PHY 1000BASE-T Status  <3800>
Sep 18 20:35:52 NASDex kernel: PHY Extended Status    <3000>
Sep 18 20:35:52 NASDex kernel: PCI Status             <10>
Sep 18 20:35:53 NASDex kernel: e1000e 0000:00:1f.6 eth0: Reset adapter unexpectedly
Sep 18 20:35:53 NASDex kernel: br0: port 1(eth0) entered disabled state
Sep 18 20:35:57 NASDex kernel: e1000e 0000:00:1f.6 eth0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Sep 18 20:35:57 NASDex kernel: br0: port 1(eth0) entered blocking state
Sep 18 20:35:57 NASDex kernel: br0: port 1(eth0) entered forwarding state
Sep 18 20:35:59 NASDex kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:

I will be posting the diagnostics soon but it will have to wait until I get home (the server is crashed at the moment).  I was on 6.11.5 for a while and this was happening.  I recently updated to 6.12.4 and it has not gotten any better.  Motherboard is a MSI MAG B460 TOMAHAWK LGA 1200 which has two ethernet ports, 1gb NIC and 12.5gb NIC.  Currently I am tied into just the 1gb NIC.

 

Other people's solutions seem to disable NIC flow control and NIC offloading, but it feels like a band-aid fix to me and I want to address the root problem.

  • Quote

    • Solved by dglb99

    Go to solution
    • Community Expert
    On 9/27/2023 at 4:50 PM, dglb99 said:

    Other people's solutions seem to disable NIC flow control and NIC offloading, but it feels like a band-aid fix to me and I want to address the root problem.

    Root of the problem may be the NIC driver, unless is fixed in the kernel not much else than can be done, so I would try those options, or using a different NIC.

  • Quote
    • Author

    Very well then.  I have disabled flow control and offloading.  Will see how it goes.

  • Quote
    • Author

    Server crashed in about 6 days with flow control and offloading disabled.  I'm looking into getting a new motherboard since I am already out of PCIE slots.

  • Quote
    • 3 weeks later...
    • Author

    Updated the bios hoping the issue would be fixed.  It still crashes every few days.

     

    Spoiler 
    Oct 24 11:18:41 NASDex kernel: ------------[ cut here ]------------
Oct 24 11:18:41 NASDex kernel: NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
Oct 24 11:18:41 NASDex kernel: WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x14e/0x1c0
Oct 24 11:18:41 NASDex kernel: Modules linked in: udp_diag xt_mark wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter xfs md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) i915 drm_buddy i2c_algo_bit ttm drm_display_helper intel_gtt agpgart tcp_diag inet_diag nct6683 ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap bridge stp llc e1000e r8169 realtek intel_rapl_msr intel_rapl_common iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp nvidia_drm(PO) kvm_intel nvidia_modeset(PO) kvm nvidia(PO) crct10dif_pclmul crc32_pclmul
Oct 24 11:18:41 NASDex kernel: crc32c_intel ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd drm_kms_helper cryptd rapl mei_hdcp mei_pxp intel_cstate wmi_bmof intel_wmi_thunderbolt mxm_wmi drm nvme i2c_i801 mpt3sas intel_uncore i2c_smbus nvme_core mei_me i2c_core syscopyarea mei ahci sysfillrect sysimgblt input_leds fb_sys_fops raid_class led_class joydev libahci scsi_transport_sas thermal fan tpm_crb video tpm_tis tpm_tis_core tpm wmi backlight intel_pmc_core acpi_tad acpi_pad button unix [last unloaded: e1000e]
Oct 24 11:18:41 NASDex kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: P           O       6.1.49-Unraid #1
Oct 24 11:18:41 NASDex kernel: Hardware name: Micro-Star International Co., Ltd. MS-7C81/MAG B460 TOMAHAWK (MS-7C81), BIOS 1.50 08/28/2023
Oct 24 11:18:41 NASDex kernel: RIP: 0010:dev_watchdog+0x14e/0x1c0
Oct 24 11:18:41 NASDex kernel: Code: ca c4 00 00 75 26 48 89 ef c6 05 72 ca c4 00 01 e8 6a 23 fb ff 44 89 f1 48 89 ee 48 c7 c7 68 dd 16 82 48 89 c2 e8 56 9d 92 ff <0f> 0b 48 89 ef e8 32 fb ff ff 48 8b 83 88 fc ff ff 48 89 ef 44 89
Oct 24 11:18:41 NASDex kernel: RSP: 0018:ffffc900001d8ea8 EFLAGS: 00010282
Oct 24 11:18:41 NASDex kernel: RAX: 0000000000000000 RBX: ffff8881407a4448 RCX: 0000000000000003
Oct 24 11:18:41 NASDex kernel: RDX: 0000000000000104 RSI: 0000000000000003 RDI: 00000000ffffffff
Oct 24 11:18:41 NASDex kernel: RBP: ffff8881407a4000 R08: 0000000000000000 R09: ffffffff8294e3f0
Oct 24 11:18:41 NASDex kernel: R10: 0000000000000000 R11: ffff888c7efaca1a R12: 0000000000000000
Oct 24 11:18:41 NASDex kernel: R13: ffff8881407a439c R14: 0000000000000000 R15: 0000000000000001
Oct 24 11:18:41 NASDex kernel: FS:  0000000000000000(0000) GS:ffff888c4e840000(0000) knlGS:0000000000000000
Oct 24 11:18:41 NASDex kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 24 11:18:41 NASDex kernel: CR2: 0000147084a0eeb0 CR3: 000000035c2dc004 CR4: 00000000007726e0
Oct 24 11:18:41 NASDex kernel: PKRU: 55555554
Oct 24 11:18:41 NASDex kernel: Call Trace:
Oct 24 11:18:41 NASDex kernel: <IRQ>
Oct 24 11:18:41 NASDex kernel: ? __warn+0xab/0x122
Oct 24 11:18:41 NASDex kernel: ? report_bug+0x109/0x17e
Oct 24 11:18:41 NASDex kernel: ? dev_watchdog+0x14e/0x1c0
Oct 24 11:18:41 NASDex kernel: ? handle_bug+0x41/0x6f
Oct 24 11:18:41 NASDex kernel: ? exc_invalid_op+0x13/0x60
Oct 24 11:18:41 NASDex kernel: ? asm_exc_invalid_op+0x16/0x20
Oct 24 11:18:41 NASDex kernel: ? dev_watchdog+0x14e/0x1c0
Oct 24 11:18:41 NASDex kernel: ? dev_watchdog+0x14e/0x1c0
Oct 24 11:18:41 NASDex kernel: ? psched_ppscfg_precompute+0x57/0x57
Oct 24 11:18:41 NASDex kernel: ? psched_ppscfg_precompute+0x57/0x57
Oct 24 11:18:41 NASDex kernel: call_timer_fn+0x6c/0x10d
Oct 24 11:18:41 NASDex kernel: __run_timers+0x144/0x184
Oct 24 11:18:41 NASDex kernel: run_timer_softirq+0x2b/0x43
Oct 24 11:18:41 NASDex kernel: __do_softirq+0x126/0x288
Oct 24 11:18:41 NASDex kernel: __irq_exit_rcu+0x5e/0xb8
Oct 24 11:18:41 NASDex kernel: sysvec_apic_timer_interrupt+0x85/0xa6
Oct 24 11:18:41 NASDex kernel: </IRQ>
Oct 24 11:18:41 NASDex kernel: <TASK>
Oct 24 11:18:41 NASDex kernel: asm_sysvec_apic_timer_interrupt+0x16/0x20
Oct 24 11:18:41 NASDex kernel: RIP: 0010:cpuidle_enter_state+0x11d/0x202
Oct 24 11:18:41 NASDex kernel: Code: 20 22 a0 ff 45 84 ff 74 1b 9c 58 0f 1f 40 00 0f ba e0 09 73 08 0f 0b fa 0f 1f 44 00 00 31 ff e8 4c e3 a4 ff fb 0f 1f 44 00 00 <45> 85 e4 0f 88 ba 00 00 00 48 8b 04 24 49 63 cc 48 6b d1 68 49 29
Oct 24 11:18:41 NASDex kernel: RSP: 0018:ffffc9000010be98 EFLAGS: 00000246
Oct 24 11:18:41 NASDex kernel: RAX: ffff888c4e840000 RBX: ffff888c4e876460 RCX: 0000000000000000
Oct 24 11:18:41 NASDex kernel: RDX: 00008846fd8e8a41 RSI: ffffffff820ed4af RDI: ffffffff820ed9b8
Oct 24 11:18:41 NASDex kernel: RBP: 0000000000000002 R08: 0000000000000002 R09: 0000000000000002
Oct 24 11:18:41 NASDex kernel: R10: 0000000000000020 R11: 0000000000000053 R12: 0000000000000002
Oct 24 11:18:41 NASDex kernel: R13: ffffffff823205c0 R14: 00008846fd8e8a41 R15: 0000000000000000
Oct 24 11:18:41 NASDex kernel: ? cpuidle_enter_state+0xf7/0x202
Oct 24 11:18:41 NASDex kernel: cpuidle_enter+0x2a/0x38
Oct 24 11:18:41 NASDex kernel: do_idle+0x18d/0x1fb
Oct 24 11:18:41 NASDex kernel: cpu_startup_entry+0x1d/0x1f
Oct 24 11:18:41 NASDex kernel: start_secondary+0x101/0x101
Oct 24 11:18:41 NASDex kernel: secondary_startup_64_no_verify+0xce/0xdb
Oct 24 11:18:41 NASDex kernel: </TASK>
Oct 24 11:18:41 NASDex kernel: ---[ end trace 0000000000000000 ]---

     

     

  • Quote
    • 2 months later...
    • Author
    • Solution

    Updating unraid to 6.12.6 and installing the Realtek NIC Drivers plugin in community applications seems to have resolved these crashes.  Server currently has 2 weeks uptime.

  • Quote
    • Community Expert

    Do you mean the Realtek driver helped with Intel NIC crashes? That seems unlikely, maybe it was just the kernel update? It may include a newer in tree driver.

  • Quote
    • Like 1
    • Author

    Don't know what to you.  The board has a Realtek RTL8125 2.5gb port, and seeing the note about 2.5gb ports causing issues in the changelog for the latest unRAID update I installed the recommended drivers and haven't had issues yet.  Currently at just under 17 days of uptime.

  • Quote
    • 2 weeks later...
    On 1/2/2024 at 3:28 PM, dglb99 said:

    Don't know what to you.  The board has a Realtek RTL8125 2.5gb port, and seeing the note about 2.5gb ports causing issues in the changelog for the latest unRAID update I installed the recommended drivers and haven't had issues yet.  Currently at just under 17 days of uptime.

    But your original post clearly indicated it was a hang on a intel e1000e. I started having the same issue a few years back and ethtool -K eth0 tso off seems to have fixed it. Very curious to understand more what happened here.

  • Quote

    • Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Reply to this topic...
    Go to topic listing

    Account

    Navigation

    Search

    Search

    Configure browser push notifications
    Chrome (Android)
    1. Tap the lock icon next to the address bar.
    2. Tap Permissions → Notifications.
    3. Adjust your preference.
    Chrome (Desktop)
    1. Click the padlock icon in the address bar.
    2. Select Site settings.
    3. Find Notifications and adjust your preference.