(SOLVED) System randomly locking up

[smikwily]

I thought originally I had some bad memory, but then figured out that I had it installed incorrectly. Everything seemed to be working better, but over the last few days, I've had a few random lockups. I can't get anything to respond at that point. I can't get to the browser based GUI, can't SSH, and even with a local monitor, nothing will respond. 

Any suggestions on where to start? I just did a diagnostics run, but I don't think that is going to do any good now after having to reboot to get it.

X9DRH-iTF w/ 2x Intel Xeon E5-2650V2
2 sets of 2 sticks each 32GB Samsung memory: M386B4G70DM0-YK0 (128 total)
F80 for cache
500GB Unassigned for Plex Caches
2 Parity Drives

32TB Total Storage

I bumped the warning and danger levels up on the F80, as it runs hot and Unraid complains about it from time to time. I found the usual running temps and adjusted them up a touch higher than I had them previously.

Edited September 2, 2020 by smikwily

[smikwily]

Diags attached if that will help. Again - this is when things are running. When it locks up, I can't get to anything without a hard reboot.

smik-tower-diagnostics-20200823-0033.zip

[JorgeB]

2 hours ago, smikwily said:

When it locks up, I can't get to anything without a hard reboot.

Try this:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=781601

 

[smikwily]

7 hours ago, johnnie.black said:

Try this:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=781601

 

Thanks. Just configured this.

[smikwily]

Ok. I was working on things off and on this AM after getting this set up. Just noticed my server was down. Nothing is jumping out at me here, but including everything from when it started until some of the early parts of the powering it back up via hardware reset:
 

Aug 23 11:19:52 SMIK-TOWER rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="21279" x-info="https://www.rsyslog.com"] start
Aug 23 13:44:14 SMIK-TOWER kernel: br0: port 2(vnet0) entered blocking state
Aug 23 13:44:14 SMIK-TOWER kernel: br0: port 2(vnet0) entered disabled state
Aug 23 13:44:14 SMIK-TOWER kernel: device vnet0 entered promiscuous mode
Aug 23 13:44:14 SMIK-TOWER kernel: br0: port 2(vnet0) entered blocking state
Aug 23 13:44:14 SMIK-TOWER kernel: br0: port 2(vnet0) entered forwarding state
Aug 23 13:44:18 SMIK-TOWER kernel: usb 2-1.2: reset high-speed USB device number 4 using ehci-pci
Aug 23 14:13:53 SMIK-TOWER kernel: usb 2-1.2: reset high-speed USB device number 4 using ehci-pci
Aug 23 14:13:56 SMIK-TOWER kernel: usb 2-1.2: reset high-speed USB device number 4 using ehci-pci
Aug 23 14:43:40 SMIK-TOWER kernel: perf: interrupt took too long (2507 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Aug 23 17:58:54 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Aug 23 17:58:54 SMIK-TOWER rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.1908.0 try https://www.rsyslog.com/e/2359 ]
Aug 23 17:58:54 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_UP): br-cc709da2b6c7: link is not ready
Aug 23 17:58:54 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_UP): hassio: link is not ready
Aug 23 17:58:54 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_UP): br-538e29e4d2f7: link is not ready
Aug 23 17:59:43 SMIK-TOWER kernel: docker0: port 1(veth8c6e751) entered blocking state
Aug 23 17:59:43 SMIK-TOWER kernel: docker0: port 1(veth8c6e751) entered disabled state
Aug 23 17:59:43 SMIK-TOWER kernel: device veth8c6e751 entered promiscuous mode
Aug 23 17:59:43 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_UP): veth8c6e751: link is not ready
Aug 23 17:59:45 SMIK-TOWER kernel: eth0: renamed from veth76fd770
Aug 23 17:59:45 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth8c6e751: link becomes ready
Aug 23 17:59:45 SMIK-TOWER kernel: docker0: port 1(veth8c6e751) entered blocking state
Aug 23 17:59:45 SMIK-TOWER kernel: docker0: port 1(veth8c6e751) entered forwarding state
Aug 23 17:59:45 SMIK-TOWER kernel: IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
Aug 23 17:59:49 SMIK-TOWER emhttpd: shcmd (106): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Aug 23 17:59:49 SMIK-TOWER kernel: BTRFS: device fsid 0721a6b0-6d5d-400e-86e2-ee1afdfcc283 devid 1 transid 104 /dev/loop3
Aug 23 17:59:49 SMIK-TOWER kernel: BTRFS info (device loop3): disk space caching is enabled
Aug 23 17:59:49 SMIK-TOWER kernel: BTRFS info (device loop3): has skinny extents
Aug 23 17:59:49 SMIK-TOWER root: Resize '/etc/libvirt' of 'max'
Aug 23 17:59:49 SMIK-TOWER kernel: BTRFS info (device loop3): new size for /dev/loop3 is 1073741824
Aug 23 17:59:49 SMIK-TOWER emhttpd: shcmd (108): /etc/rc.d/rc.libvirt start
Aug 23 17:59:49 SMIK-TOWER root: Starting virtlockd...
Aug 23 17:59:49 SMIK-TOWER root: Starting virtlogd...

 

[JorgeB]

Nothing logged suggests a hardware issue, run the server as a basic NAS for a couple of days, boot in safe mode and disable all dockers/VMs, if it still locks up like that you need to start swapping some hardware.

[smikwily]

Just checking back in to let you know that everything has been running a lot smoother the last two weeks with no lockups. I think I figured it out by accident. I had opened things up to remove some hardware to see if that could be causing some problems and decided to move my USB drive to an internal USB port that was on my motherboard. While doing that, I noticed that I had apparently plugged both sets of 4 pin USB cables for my front ports into the same 9 pin header. I wondered if that could be causing the problem, so I pulled the cables out. Both of them were looking just slightly "melty", so I just removed them, as I wasn't using the front ports anyhow. That alone seems to have helped with the random left field lockups. 

Thanks again for all your help and suggestions!