June 30, 201610 yr Author Fixed the CPU usage yet it just happened again. What can I look for? I'm absolutely lost here. I don't know what to try next.
July 20, 20169 yr Author 17 days ago I implemented isolcpus=1,2,3,5,6,7 So far so good. I figure if I post it on here then that's a good way to jinx it (test it). If this fails, i'm going to move my CCTV VM (Blue Iris) to a different box to see if it's the root cause.
July 22, 20169 yr Author aaaaaaaaaaand it happened again. This time I had a monitor attached. (Plugged keyboard in after couldn't ping unRAID box) -Terminal responds fine. -Cannot ping anything on the network but can ping it's own IP fine with returned packets. SYSLOG shows nothing out of the ordinary: Jul 22 18:00:01 unRAID logger: skipping "appdata" Jul 22 18:00:01 unRAID logger: mover finished Jul 22 19:00:01 unRAID logger: mover started Jul 22 19:00:01 unRAID logger: skipping "VDisks" Jul 22 19:00:01 unRAID logger: skipping "appdata" Jul 22 19:00:01 unRAID logger: mover finished Jul 22 19:13:09 unRAID kernel: usb 3-2: new low-speed USB device number 2 using uhci_hcd Jul 22 19:13:09 unRAID kernel: input: Chicony USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb3/3-2/3-2:1.0/0003:04F2:0403.0002/input/input5 Jul 22 19:13:09 unRAID kernel: hid-generic 0003:04F2:0403.0002: input,hidraw1: USB HID v1.11 Keyboard [Chicony USB Keyboard] on usb-0000:00:1a.0-2/input0 Jul 22 19:13:09 unRAID kernel: input: Chicony USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb3/3-2/3-2:1.1/0003:04F2:0403.0003/input/input6 Jul 22 19:13:09 unRAID kernel: hid-generic 0003:04F2:0403.0003: input,hiddev0,hidraw2: USB HID v1.11 Device [Chicony USB Keyboard] on usb-0000:00:1a.0-2/input1 Jul 22 19:13:20 unRAID login[4027]: ROOT LOGIN on '/dev/tty1' I ran /etc/rc.d.rc.inetd restart and everything returned to normal. I'm hoping this helps narrow it down. Thoughts?
July 25, 20169 yr Author root@unRAID:~# dmesg | grep eth2 e1000e 0000:06:00.1 eth2: (PCI Express:2.5GT/s:Width x4) 00:1f:29:57:08:d3 e1000e 0000:06:00.1 eth2: Intel® PRO/1000 Network Connection e1000e 0000:06:00.1 eth2: MAC: 0, PHY: 4, PBA No: D51930-004 device eth2 entered promiscuous mode e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx br0: port 3(eth2) entered listening state br0: port 3(eth2) entered listening state br0: port 3(eth2) entered learning state br0: port 3(eth2) entered forwarding state br0: port 3(eth2) entered disabled state br0: port 3(eth2) entered disabled state e1000e: eth2 NIC Link is Down e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx br0: port 3(eth2) entered listening state br0: port 3(eth2) entered listening state br0: port 3(eth2) entered learning state br0: port 3(eth2) entered forwarding state root@unRAID:~# Is entered disabled state a clue at all? Ned in this thread is showing the exact same symptoms: http://lime-technology.com/forum/index.php?topic=47927.0 I've disabled docker and rebooted. Time to start the clock again.
July 26, 20169 yr Things i would try at this point: kernel option 'pcie_aspm=off'' ethtool -G eth0 rx 4096 tx 4096 ethtool -G eth1 rx 4096 tx 4096 ethtool --offload eth0 gso off tso off sg off gro off ethtool --offload eth1 gso off tso off sg off gro off Tim Edit: Some other network vars i have tuned in my bonded setup using e1000's: sysctl net.core.rmem_max=16777216 sysctl net.core.wmem_max=16777216 sysctl net.ipv4.tcp_rmem='4096 87380 16777216' sysctl net.ipv4.tcp_wmem='4096 65536 16777216' sysctl net.core.netdev_max_backlog=5000 sysctl net.nf_conntrack_max=700000 root@unRAID:~# dmesg | grep eth2 e1000e 0000:06:00.1 eth2: (PCI Express:2.5GT/s:Width x4) 00:1f:29:57:08:d3 e1000e 0000:06:00.1 eth2: Intel® PRO/1000 Network Connection e1000e 0000:06:00.1 eth2: MAC: 0, PHY: 4, PBA No: D51930-004 device eth2 entered promiscuous mode e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx br0: port 3(eth2) entered listening state br0: port 3(eth2) entered listening state br0: port 3(eth2) entered learning state br0: port 3(eth2) entered forwarding state br0: port 3(eth2) entered disabled state br0: port 3(eth2) entered disabled state e1000e: eth2 NIC Link is Down e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx br0: port 3(eth2) entered listening state br0: port 3(eth2) entered listening state br0: port 3(eth2) entered learning state br0: port 3(eth2) entered forwarding state root@unRAID:~# Is entered disabled state a clue at all? Ned in this thread is showing the exact same symptoms: http://lime-technology.com/forum/index.php?topic=47927.0 I've disabled docker and rebooted. Time to start the clock again.
July 26, 20169 yr Things i would try at this point: kernel option 'pcie_aspm=off'' ethtool -G eth0 rx 4096 tx 4096 ethtool -G eth1 rx 4096 tx 4096 ethtool --offload eth0 gso off tso off sg off gro off ethtool --offload eth1 gso off tso off sg off gro off Tim I think the tips and tweaks plugin in CA will handle all that for you
August 12, 20169 yr Author Just doing my jinx test by posting here. root@unRAID:/# uptime 14:42:46 up 15 days, 21:37, 1 user, load average: 2.04, 1.52, 1.37 root@unRAID:/# dmesg |grep pcie_aspm pci 0000:06:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' root@unRAID:/# dmesg |grep eth1 e1000e 0000:06:00.0 eth1: (PCI Express:2.5GT/s:Width x4) 00:1f:29:57:08:d2 e1000e 0000:06:00.0 eth1: Intel® PRO/1000 Network Connection e1000e 0000:06:00.0 eth1: MAC: 0, PHY: 4, PBA No: D51930-004 15 days ago I found that pci 0000:06:00.0 already had ASPM disabled. This is eth1 so I moved my network cable from eth2 to eth1. I also enabled NIC bonding to enable another troubleshooting method.
August 18, 20169 yr Author 22 days strong! If this keeps going it seems it was either: ASPM disabled Disabled Observium Docker Link aggregation enabling
Archived
This topic is now archived and is closed to further replies.