• [6.7.x - 6.7.3 RC4] Parity check high CPU usage causes Network Problems


    Videodr0me
    • Solved Annoyance

    UPDATE: Partially solved in 6.8.0 rc6. No loss of network connectivtiy anymore. But parity checks are now painfully slow (see parity speed thread for 6.8RC)

     

    UPDATE: The problem still persists with 6.7.3 RC4, although recovery speed seems faster then with 6.7.3. RC1 and parity check speed improved by 10% (on my slow J1900 with eight 8TB drives and dual parity)

    UPDATE: The problem still persists with 6.7.3 RC1

     

    I first noticed this with 6.7.0 RC6 and verified on RC7 and RC8, but it might be present in all 6.7.0 RCs. If I start a parity check [7 8TB drives and 2 Parity 8Tb drives) it maxes out one CPU core fully (parity calculation takes its toll on the J1900) - this has been normal for quite some versions and is not the main problem, even though it might be possible to optimize parity calculation (making it multithreaded, or getting rid of some redudant operations). The problem is that as of 6.7.0 RC6 (can't confirm or deny for earlier RCs) this leads to not servicing some IRQs on the network hardware. The parity check continues, but naturally the webinterface becomes very unresponsive because it takes some time for the network connection to reestablish itself. It might be new linux kernel as there has been some changes in this regard (maybe in conjunction with my specific hardware). With 6.6.x no such problems existed. The log show this:

     

    Apr 10 08:54:53 Tower emhttpd: req (4): startState=STARTED&file=&cmdCheck=Check&csrf_token=****************
    Apr 10 08:54:53 Tower kernel: mdcmd (89): check nocorrect
    Apr 10 08:54:53 Tower kernel: md: recovery thread: check P Q ...
    Apr 10 08:54:53 Tower kernel: md: using 3072k window, over a total of 7814026532 blocks.
    Apr 10 08:57:08 Tower kernel: ------------[ cut here ]------------
    Apr 10 08:57:08 Tower kernel: NETDEV WATCHDOG: eth0 (tg3): transmit queue 0 timed out
    Apr 10 08:57:08 Tower kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x15f/0x1b7
    Apr 10 08:57:08 Tower kernel: Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 iptable_filter ip_tables nf_nat it87 hwmon_vid xfs nfsd lockd grace sunrpc md_mod tg3 intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel cryptd intel_cstate i2c_i801 i2c_core ahci libahci iosf_mbi video thermal backlight fan button pcc_cpufreq [last unloaded: tg3]
    Apr 10 08:57:08 Tower kernel: CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.33-Unraid #1
    Apr 10 08:57:08 Tower kernel: Hardware name: Insyde AS Series/Type2 - Board Product Name, BIOS V1.37 01/27/2015
    Apr 10 08:57:08 Tower kernel: RIP: 0010:dev_watchdog+0x15f/0x1b7
    Apr 10 08:57:08 Tower kernel: Code: b1 59 97 00 00 75 36 4c 89 ef c6 05 a5 59 97 00 01 e8 4b b9 fd ff 89 e9 4c 89 ee 48 c7 c7 6e 66 d9 81 48 89 c2 e8 2f 1e b2 ff <0f> 0b eb 0f ff c5 48 81 c2 40 01 00 00 39 cd 75 98 eb 13 48 8b 83
    Apr 10 08:57:08 Tower kernel: RSP: 0018:ffff888277b83ea0 EFLAGS: 00010286
    Apr 10 08:57:08 Tower kernel: RAX: 0000000000000000 RBX: ffff888272142438 RCX: 0000000000000007
    Apr 10 08:57:08 Tower kernel: RDX: 0000000000000006 RSI: ffff888277b964e0 RDI: ffff888277b964e0
    Apr 10 08:57:08 Tower kernel: RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000020300
    Apr 10 08:57:08 Tower kernel: R10: 0000000000000472 R11: 0000000000017ee0 R12: ffff88827214241c
    Apr 10 08:57:08 Tower kernel: R13: ffff888272142000 R14: ffff888272105580 R15: 0000000000000003
    Apr 10 08:57:08 Tower kernel: FS:  0000000000000000(0000) GS:ffff888277b80000(0000) knlGS:0000000000000000
    Apr 10 08:57:08 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    Apr 10 08:57:08 Tower kernel: CR2: 000014cd9c640000 CR3: 000000027219a000 CR4: 00000000001006e0
    Apr 10 08:57:08 Tower kernel: Call Trace:
    Apr 10 08:57:08 Tower kernel: <IRQ>
    Apr 10 08:57:08 Tower kernel: call_timer_fn+0x18/0x7b
    Apr 10 08:57:08 Tower kernel: ? qdisc_reset+0xc0/0xc0
    Apr 10 08:57:08 Tower kernel: expire_timers+0x7f/0x8e
    Apr 10 08:57:08 Tower kernel: run_timer_softirq+0x72/0x120
    Apr 10 08:57:08 Tower kernel: ? enqueue_hrtimer.isra.3+0x23/0x27
    Apr 10 08:57:08 Tower kernel: ? __hrtimer_run_queues+0xd7/0x105
    Apr 10 08:57:08 Tower kernel: ? recalibrate_cpu_khz+0x1/0x1
    Apr 10 08:57:08 Tower kernel: ? ktime_get+0x3a/0x8d
    Apr 10 08:57:08 Tower kernel: __do_softirq+0xce/0x1e2
    Apr 10 08:57:08 Tower kernel: irq_exit+0x5e/0x9d
    Apr 10 08:57:08 Tower kernel: smp_apic_timer_interrupt+0x7e/0x91
    Apr 10 08:57:08 Tower kernel: apic_timer_interrupt+0xf/0x20
    Apr 10 08:57:08 Tower kernel: </IRQ>
    Apr 10 08:57:08 Tower kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141
    Apr 10 08:57:08 Tower kernel: Code: ff 45 84 ff 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 03 47 be ff fb 66 0f 1f 44 00 00 <48> 2b 1c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cb
    Apr 10 08:57:08 Tower kernel: RSP: 0018:ffffc90000cefea0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
    Apr 10 08:57:08 Tower kernel: RAX: ffff888277ba0b00 RBX: 00002d13e912c1b9 RCX: 000000000000001f
    Apr 10 08:57:08 Tower kernel: RDX: 00002d13e912c1b9 RSI: 0000000040068e64 RDI: 0000000000000000
    Apr 10 08:57:08 Tower kernel: RBP: ffff888277ba8d10 R08: 0000000000000002 R09: 00000000000203c0
    Apr 10 08:57:08 Tower kernel: R10: 00000000000441fc R11: 00005a6aed31e218 R12: 0000000000000001
    Apr 10 08:57:08 Tower kernel: R13: 0000000000000001 R14: ffffffff81e59e78 R15: 0000000000000000
    Apr 10 08:57:08 Tower kernel: do_idle+0x192/0x20e
    Apr 10 08:57:08 Tower kernel: cpu_startup_entry+0x6a/0x6c
    Apr 10 08:57:08 Tower kernel: start_secondary+0x197/0x1b2
    Apr 10 08:57:08 Tower kernel: secondary_startup_64+0xa4/0xb0
    Apr 10 08:57:08 Tower kernel: ---[ end trace 3dab24a7f901a6f0 ]---
    Apr 10 08:57:08 Tower kernel: tg3 0000:03:00.0 eth0: transmit timed out, resetting
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000000: 0x16b114e4, 0x00100406, 0x02000010, 0x00000010
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000010: 0x9071000c, 0x00000000, 0x9070000c, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000020: 0x00000000, 0x00000000, 0x00000000, 0x96b114e4
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000030: 0xfffff800, 0x00000048, 0x00000000, 0x0000010a
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000040: 0x00000000, 0x00000001, 0xc8035801, 0x16002008
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000050: 0x00000003, 0x00000000, 0x0086a005, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000060: 0x00000000, 0x00000000, 0xf1000298, 0x01f80051
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000070: 0x000010b2, 0xfd10b201, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000080: 0x16b114e4, 0x800000f8, 0x00000000, 0x00000120
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000090: 0x00000000, 0x000001b8, 0x00000000, 0x000001ac
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000000a0: 0x8004ac11, 0x00000002, 0x00000122, 0x00020010
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000000b0: 0x00688d80, 0x00102400, 0x00075c11, 0x10110040
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000000d0: 0x0000001f, 0x00000000, 0x00000000, 0x00000001
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000000f0: 0x00000000, 0x00000000, 0x00000000, 0x57785100
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000110: 0x00000000, 0x00002000, 0x000000a0, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000140: 0x488ac130, 0x000010bf, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000150: 0x16010004, 0x00000000, 0x00078116, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000160: 0x00010002, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000200: 0x00000000, 0x00000001, 0x00000000, 0x00000001
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000210: 0x00000000, 0xa9000000, 0x00000000, 0xc4000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000220: 0x00000000, 0x00000001, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x000001ac
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000280: 0x00000000, 0x00000120, 0x00000000, 0x00000333
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000290: 0x00000000, 0x000001a3, 0x00000000, 0x000002e2
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000300: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000310: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000320: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000330: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000340: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000350: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000360: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000370: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000380: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000390: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000003a0: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000003b0: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000003c0: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000003d0: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000003e0: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000003f0: 0x00000000, 0x000001b8, 0x00000000, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000400: 0x00e04808, 0x00400000, 0x00001000, 0x00000880
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000410: 0x000010bf, 0x488ac130, 0x000010bf, 0x488ac130
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000420: 0x000010bf, 0x488ac130, 0x000010bf, 0x488ac130
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000430: 0x00000000, 0x00000000, 0x0000031d, 0x000005f2
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000112
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000460: 0x0000000e, 0x00002620, 0x01ff0006, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000470: 0xa0002000, 0x00000080, 0x00000000, 0x50000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000500: 0x00000008, 0x00000001, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000590: 0x00900110, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000005c0: 0xd320fa58, 0x018b84de, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000600: 0xffffffff, 0x00f80011, 0x00000000, 0x00001f04
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00004, 0x30000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000670: 0xfab2595a, 0x19671df2, 0xb131d7e7, 0x381e557c
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000680: 0x08d67ba0, 0xb9fa539d, 0x5fa0f59b, 0x1562f543
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000690: 0x97135ee8, 0xfa1ad7df, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000880: 0x00000165, 0x001ddd00, 0x00000000, 0x00000003
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000008f0: 0x007c0001, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x1d880000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000c80: 0x001eb379, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000ce0: 0x7a8853ac, 0x00000001, 0x006604ec, 0x000c062c
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00000cf0: 0x00000000, 0x40000038, 0x84ec0066, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001000: 0x00000002, 0x00000000, 0xa000545f, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001010: 0x01b81b80, 0x0001545f, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000005
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001440: 0x000001b8, 0x000001b8, 0x000001b8, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001450: 0x000001b8, 0x000001b8, 0x000001b8, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001460: 0x000001b8, 0x000001b8, 0x000001b8, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001470: 0x000001b8, 0x000001b8, 0x000001b8, 0x000001b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001800: 0x00000016, 0x00000000, 0x000001b8, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001850: 0x0000001f, 0x00000002, 0x00004360, 0x01b801b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001860: 0x02000000, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002010: 0x00000181, 0x00000001, 0x007bffff, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002100: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002110: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002120: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002130: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002140: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002150: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002160: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002170: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002180: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002190: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000021a0: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000021b0: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000021c0: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000021d0: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000021e0: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000021f0: 0x000cc262, 0x000cc262, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002200: 0x00f93a8a, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002400: 0x00000012, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002450: 0x00000002, 0x507c4000, 0x02001800, 0x00006000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002470: 0x00000000, 0x00000134, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000024c0: 0x00800101, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002800: 0x00000006, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002c00: 0x00000006, 0x00000000, 0x00000000, 0x000001a0
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002c10: 0x00000000, 0x00000000, 0x00000020, 0x0000000c
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00002c20: 0x00020002, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003000: 0x00000006, 0x00000000, 0x00000000, 0x000001a0
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003600: 0x00037c00, 0x00130000, 0x00110000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003610: 0x00170000, 0x00130000, 0x00150000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003620: 0x00000000, 0x00000000, 0x00000000, 0x00032080
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003630: 0x00000000, 0x80008000, 0x00001080, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003640: 0x0000000a, 0x02f00003, 0x00000020, 0x0000004b
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003650: 0x00000171, 0x00000bff, 0x57785100, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003660: 0x00000000, 0x00000000, 0x000400a7, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003670: 0x0000002a, 0xfefffff7, 0x0000000a, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000036a0: 0x00000000, 0x00000000, 0x00000000, 0x00000040
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000036b0: 0x001003c8, 0x07ff07ff, 0x07ff07ff, 0x01000004
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000036c0: 0xffffffff, 0x2338530a, 0x00000002, 0x48c42fe6
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000036d0: 0x0000019d, 0x00000000, 0x00000000, 0x00001274
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000036e0: 0x00000b19, 0x10880819, 0x00580819, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c00: 0x00000302, 0x00000000, 0x00000000, 0x00000048
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c10: 0x00000000, 0x00000035, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c30: 0x00000000, 0x00000000, 0x00000002, 0x72d55000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c50: 0x00000000, 0x0000013a, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003c80: 0x0000015b, 0x00000333, 0x000001a4, 0x00000308
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003cc0: 0x000001b8, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003d00: 0x00000002, 0x71d15000, 0x00000002, 0x7210f000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003d10: 0x00000002, 0x7210e000, 0x00000002, 0x7186f000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004000: 0x00000002, 0x00000000, 0x000acd10, 0x000e2601
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004010: 0x0045f060, 0x0026d012, 0x00800440, 0x00848082
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004020: 0x00000000, 0x00000010, 0x00000010, 0x00000050
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004030: 0x00000000, 0x01086e20, 0x0026d010, 0x00455002
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004040: 0x00460000, 0x0f345e00, 0x00000010, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004400: 0x00000006, 0x00000000, 0x00010000, 0x0000a000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x05417c61
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004450: 0x0002033f, 0x006e006f, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004800: 0x180303fe, 0x00000000, 0x00000000, 0x00000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004810: 0x00000000, 0x00000004, 0x02000acd, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004820: 0x00000060, 0x00000000, 0xf02c0000, 0x507c7400
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004830: 0x00000000, 0x0000010c, 0x0000010c, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004840: 0x00000081, 0x01798b81, 0x000e2200, 0x6036d8aa
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004850: 0xae726877, 0xe59a0522, 0x55545554, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004860: 0x01000060, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004900: 0x00028207, 0x50205408, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004910: 0x00000018, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c10: 0x0000002a, 0x00000000, 0x00000006, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c30: 0x00000000, 0x00570000, 0x00000093, 0x00000093
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c40: 0x00000000, 0x71d15000, 0x00010020, 0x00040020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c50: 0x1a3cc95b, 0x13bfff07, 0x30869333, 0x003bb8b8
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005000: 0x00009800, 0x80008000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x40003c84
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005020: 0x24a50001, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005030: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005040: 0x00000000, 0x00000000, 0x40003274, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005080: 0x00009800, 0x80000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x40003b30
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000050a0: 0x8f87105c, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000050b0: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000050c0: 0x00000000, 0x00000000, 0x40003c7c, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005100: 0x00009800, 0x80000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x400033ec
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005120: 0x3c04dead, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005130: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005140: 0x00000000, 0x00000000, 0x400033ec, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005180: 0x00009800, 0x80008000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x40003c70
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000051a0: 0x00051840, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000051b0: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000051c0: 0x00000000, 0x00000000, 0x400033f4, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005200: 0x00000000, 0x00000000, 0x0800ffc0, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005210: 0x00000001, 0x00000000, 0x00009810, 0x0800ffc0
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005220: 0x0800ffe0, 0x40001fa4, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005230: 0x00000001, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005250: 0x00000000, 0x00009810, 0x00080000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005260: 0xb49a89ab, 0x00000000, 0x0800ffc0, 0x0800ffc0
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005270: 0x00000001, 0x00000000, 0x00009810, 0x0800ffc0
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005280: 0x00009800, 0x80000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x40003b0c
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000052a0: 0x00a09021, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000052b0: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000052c0: 0x00000000, 0x00000000, 0x40003c7c, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x40003b38
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005320: 0x3c02dead, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005330: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005340: 0x00000000, 0x00000000, 0x40003c62, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005380: 0x00009800, 0x80008000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x400033f4
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000053a0: 0x1460fff4, 0x00000000, 0x00000000, 0x40000020
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000053b0: 0x00000000, 0x00000051, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000053c0: 0x00000000, 0x00000000, 0x40003274, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005800: 0x00000000, 0xb8000000, 0x00000000, 0x82000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005810: 0x00000000, 0xa9000000, 0x00000000, 0x00000001
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005820: 0x00000000, 0xde000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005860: 0x00000000, 0x00000000, 0x00000000, 0x0000003c
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005880: 0x00000000, 0x0000015e, 0x00000000, 0x00000333
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005890: 0x00000000, 0x000001a3, 0x00000000, 0x00000308
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005900: 0x00000000, 0x000001b8, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005980: 0x000001b8, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006000: 0x00000082, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006400: 0x00000000, 0x00000000, 0x00010191, 0xc0000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006410: 0x50000016, 0x50000016, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006430: 0x00000000, 0x14e416b1, 0x96b114e4, 0x10020000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006440: 0x0000304b, 0x000002e4, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000064c0: 0x00000004, 0x00000002, 0x00000122, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000064d0: 0x00000000, 0x00008d80, 0x00000000, 0x00d75e11
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000064f0: 0x00000002, 0x00000030, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006500: 0x01e10003, 0x488ac130, 0x000010bf, 0x00000003
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006510: 0x00078116, 0x0005810b, 0x00046105, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006550: 0x00000000, 0x02800000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006800: 0x04130034, 0x20081082, 0x01000018, 0xc8a9f986
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006810: 0xa1000000, 0xffffffff, 0x00000000, 0x000000f0
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006880: 0x77fff020, 0x00000040, 0x081316b1, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006890: 0x00800000, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000068a0: 0x00000000, 0x00010001, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x000068b0: 0x00040000, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00006900: 0x2134a508, 0x00000000, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00007000: 0x20000108, 0x00000000, 0x00000014, 0x00000074
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00007010: 0x03780c10, 0x02008200, 0x000500db, 0xa100a000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00007030: 0x00020000, 0x00000078, 0x001f0000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00007500: 0x00000000, 0x00000000, 0x0000008f, 0x00000082
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0x00007510: 0x00000000, 0x00001024, 0x00000000, 0x00000000
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0: Host status block [00000001:000000b8:(0000:00df:0000):(0000:01b8)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 0: NAPI info [000000b8:000000b8:(01b8:01b8:0100):0000:(004d:0000:0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 1: Host status block [00000001:00000092:(0000:0000:0000):(016e:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 1: NAPI info [00000092:00000092:(0000:0000:0100):016e:(016e:016e:0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 2: Host status block [00000001:000000a9:(0333:0000:0000):(0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 2: NAPI info [000000a9:000000a9:(0000:0000:0100):0333:(0133:0133:0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 3: Host status block [00000001:000000c5:(0000:0000:0000):(0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 3: NAPI info [000000c4:000000c4:(0000:0000:0100):01a3:(01a3:01a3:0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 4: Host status block [00000001:000000df:(0000:0000:0309):(0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: 4: NAPI info [000000df:000000df:(0000:0000:0100):0309:(0109:0109:0000:0000)]
    Apr 10 08:57:11 Tower kernel: tg3 0000:03:00.0 eth0: Link is down
    Apr 10 08:57:11 Tower dhcpcd[1645]: eth0: carrier lost
    Apr 10 08:57:11 Tower avahi-daemon[5570]: Withdrawing address record for 192.168.178.29 on eth0.
    Apr 10 08:57:11 Tower avahi-daemon[5570]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.178.29.
    Apr 10 08:57:11 Tower avahi-daemon[5570]: Interface eth0.IPv4 no longer relevant for mDNS.
    Apr 10 08:57:12 Tower ntpd[1706]: Deleting interface #1 eth0, 192.168.178.29#123, interface stats: received=128, sent=130, dropped=0, active_time=49495 secs
    Apr 10 08:57:12 Tower ntpd[1706]: 195.201.19.162 local addr 192.168.178.29 -> <null>
    Apr 10 08:57:12 Tower ntpd[1706]: Deleting interface #3 eth0, fe80::12bf:48ff:fe8a:c130%12#123, interface stats: received=0, sent=0, dropped=0, active_time=49495 secs
    Apr 10 08:57:14 Tower kernel: tg3 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex
    Apr 10 08:57:14 Tower kernel: tg3 0000:03:00.0 eth0: Flow control is on for TX and on for RX
    Apr 10 08:57:14 Tower kernel: tg3 0000:03:00.0 eth0: EEE is enabled
    Apr 10 08:57:15 Tower dhcpcd[1645]: eth0: deleting route to 192.168.178.0/24
    Apr 10 08:57:15 Tower dhcpcd[1645]: eth0: deleting default route via 192.168.178.1
    Apr 10 08:57:15 Tower dhcpcd[1645]: eth0: carrier acquired
    Apr 10 08:57:15 Tower ntpd[1706]: Listen normally on 4 eth0 [fe80::12bf:48ff:fe8a:c130%12]:123
    Apr 10 08:57:15 Tower ntpd[1706]: new interface(s) found: waking up resolver
    Apr 10 08:57:16 Tower dhcpcd[1645]: eth0: rebinding lease of 192.168.178.29
    Apr 10 08:57:16 Tower dhcpcd[1645]: eth0: probing address 192.168.178.29/24
    Apr 10 08:57:20 Tower dhcpcd[1645]: eth0: leased 192.168.178.29 for 864000 seconds
    Apr 10 08:57:20 Tower dhcpcd[1645]: eth0: adding route to 192.168.178.0/24
    Apr 10 08:57:20 Tower dhcpcd[1645]: eth0: adding default route via 192.168.178.1
    Apr 10 08:57:20 Tower avahi-daemon[5570]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.178.29.
    Apr 10 08:57:20 Tower avahi-daemon[5570]: New relevant interface eth0.IPv4 for mDNS.
    Apr 10 08:57:20 Tower avahi-daemon[5570]: Registering new address record for 192.168.178.29 on eth0.IPv4.
    Apr 10 08:57:21 Tower ntpd[1706]: Listen normally on 5 eth0 192.168.178.29:123
    Apr 10 08:57:21 Tower ntpd[1706]: new interface(s) found: waking up resolver
    Apr 10 08:59:08 Tower emhttpd: req (5): startState=STARTED&file=&csrf_token=****************&cmdNoCheck=Cancel
    Apr 10 08:59:08 Tower kernel: mdcmd (90): nocheck Cancel
    Apr 10 08:59:08 Tower kernel: md: recovery thread: exit status: -4




    User Feedback

    Recommended Comments

    Checking the 4.20 changelog:

     

    commit 59663e42199c93d1d7314d1446f6782fc4b1eb81 Author: Siva Reddy Kallam <[email protected]> Date: Tue Nov 20 10:04:04 2018 +0530

    tg3: Add PHY reset for 5717/5719/5720 in change ring and flow control paths This patch has the fix to avoid PHY lockup with 5717/5719/5720 in change ring and flow control paths. This patch solves the RX hang while doing continuous ring or flow control parameters with heavy traffic from peer. Signed-off-by: Siva Reddy Kallam <[email protected]> Acked-by: Michael Chan <[email protected]> Signed-off-by: David S. Miller <[email protected]>

     

    This might mitigate this somewhat, even though the original bug report this commit is based on exhibited loss of carrier timeouts with heavy network traffic generating the irqs. In my case the irqs are generated by the parity check (either directly or by the SATA devices), so it remains to be seen if that will fix it. Also it might not apply to my exact version of broadcom hardware:

     

    Apr 9 19:11:57 Tower kernel: tg3 0000:03:00.0 eth0: Tigon3 [partno(BCM57781) rev 57785100] (PCI Express) MAC address 10:bf:48:8a:c1:30

    Apr 9 19:11:57 Tower kernel: tg3 0000:03:00.0 eth0: attached PHY is 57765 (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])

    Apr 9 19:11:57 Tower kernel: tg3 0000:03:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]

    Apr 9 19:11:57 Tower kernel: tg3 0000:03:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]

     

    Edited by Videodr0me
    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.