[Server keeps freezing overnight, with caps lock and scroll lock lights blinking on the keyboard.]

Ok, I had two more freezes. Yesterday's was a bit different - it happened in the afternoon, they keyboard lights didn't blink, and when I rebooted it, it told me that the CPU was over temperature. This made me realize that the fan in the AOI cooler had died. (The pump was still working though.) That might be the root cause of the whole mess. I swapped it with a case fan and rebooted. It kicked off a parity check and seemed to be working when I went to bed. I thought maybe I had fixed the issue.

 

This morning it was back to the same type of crash that had happened before yesterday with the keyboard lights blinking.

 

I'm attaching the syslog and syslog-previous, but my suspicion now is that something is got permanently damaged by the CPU overheating.

 

---------

 

Update: (April 22) maybe it was just overheating. It's been a month and a half, and aside from the one crash that happened the night after replacing the fan, it's been rock solid.

 

I ordered an HBA card, and was waiting for it to arrive so that I could use a different CPU & motherboard with fewer SATA ports. But now that it's arrived, I'm not sure I actually need it. Oh well. I'll probably swap it out anyways.

 

syslog syslog-previous

[Server keeps freezing overnight, with caps lock and scroll lock lights blinking on the keyboard.]

11 hours ago, JorgeB said:

Looks more like a hardware issue, but enable the syslog server and post that after a crash just in case there's something there.

 

Ok, I turned on Mirror syslog to flash, I'll post another update after the next freeze.

 

I see it says a copy of the syslog is stored in the logs folder on the flash drive - what's the right way to retrieve that file? Is it exposed in the web UI or over the network, or should I just plug the flash drive into another computer?

[Server keeps freezing overnight, with caps lock and scroll lock lights blinking on the keyboard.]

I put my server together about 6 years ago and it's been really solid until recently. However, 4 times in the last ~month I've found it completely frozen, not responding to network requests (file shares, docker images, the admin web UI), and not responding to keyboard or mouse inputs at the physical machine. When this happens, the numlock light on the keyboard turns off and the caps lock and scroll lock lights start blinking. It seems to be getting more frequent, with the most recent two freezes happening after only a day or two of uptime.

 

The screen shows whatever was happening at the moment it froze, so I've left `htop` open once, and `dmesg --follow` the next time, but neither have anything too obvious. htop shows shfs using about 10% CPU and transmission using another 15% across two processes; dmesg shows only two recent messages:

 

md: sync done. time=60938sec
md: recovery thread: exit status 0

 

I'm not sure what those mean, but "exit status 0" sounds like "not a crash".

I'm also attaching an anonymized diagnostics bundle.

 

The CPU, MB, and RAM (i7-2600K, Asus P8P67 Pro, & 2x8GB Kingston HyperX Fury DDR3-1866) are all recycled from the desktop PC I built about 15 years ago, so my first thought is that maybe one of them is going out.  But I'd still like to understand what's happening better.

Also, all of 4 freezes have happened in the middle of the night, which makes me think it might be some scheduled thing that's triggering it.

 

[Edit] One other thing that comes to mind is that I switched the cache drive from a SATA SSD to an NVMe SSD a couple of months ago. I initially messed up the file owners when copying everything over to the new SSD, which broke some of my docker images, but I think I have it straightened out now.

Does anyone here have any ideas what the root cause might be?

unraid-diagnostics-20240226-1418.zip

[How to set up unRAID notification using Gmail SMTP settings]

On 5/19/2020 at 10:18 PM, wobblybob said:

This worked great. If for some reason you don't have the option called App Passwords (like I did), according to google it's one of these reasons. I had to enable 2-step verification and then app passwords showed up upon refresh.

 

2-Step Verification is not set up for your account.

2-Step Verification is set up for security keys only.

Your account is through work, school, or other organization.

You’ve turned on Advanced Protection for your account.

In my case, none of these were true - I'm on a personal account and I've had 2FA enabled for years - but it still wouldn't show me the App Passwords option. I poked around for a while and eventually found that the url https://myaccount.google.com/apppasswords works even though it doesn't show up in the security settings.

After that, when I did finally get google to give me an app password, I tried just copy-pasting it to unraid, but that got an authentication error. Eventually I realized that even though the password was really 16-characters, google had put spaces after every 4 characters for readability (or whatever), so when I copy-pasted I got extra spaces that weren't part of the password. I had to paste it into a notepad, delete the spaces, then copy it into unraid.