My server has been acting up for some time now, and recently it has gotten significantly worse. It will now become unresponsive multiple times a day. This can only be corrected by doing a full power cycle of the machine.  I have updated it to the latest patch version (6.12.6), previously on 6.12.4.  I have attached the diagnostics file, as well as the syslog from before and after the latest freeze up.

 

Hardware setup, is 8 spinning disks connected directly to the mobo, while 2 SSDs, are connected via a PCI to sata connector. The SSDs are used as a Cache drive and the otherone for a download drive.

 

 

If some one could point me in the direction of the issue, I would appreaciate it.

 

 

tower-diagnostics-20240103-1445.zip syslog-previous syslog

I'd recommend reading the Known Issues area of the release notes:
  https://docs.unraid.net/unraid-os/release-notes/6.12.6/

Specifically, you'll want to switch from macvlan to ipvlan and consider installing drivers for your Realtek RTL8125B network card

3 hours ago, ljm42 said:

I'd recommend reading the Known Issues area of the release notes:
  https://docs.unraid.net/unraid-os/release-notes/6.12.6/

Specifically, you'll want to switch from macvlan to ipvlan and consider installing drivers for your Realtek RTL8125B network card

Thanks for the reply,

 

I tried changing from macvlan to ip, and it didn't seem to have any affect, as the same issue still occured, so I changed it back.

I highly recommend that you leave ipvlan enabled, or take the steps listed in the release notes to make macvlan safe.

On 1/3/2024 at 11:32 PM, ljm42 said:

I highly recommend that you leave ipvlan enabled, or take the steps listed in the release notes to make macvlan safe.

Hi ljm42,

 

I changed to IP, and rebooted. But within a day it had the same issue again. I have attached the new syslogs and diag. again. Any ideas?

 

Thanks?

tower-diagnostics-20240108-1125.zip syslog-previous(1) syslog(1)

I had just rebooted my UnRaid server and it has become unresponsive in under an hour again.

I see some strange errors logged but not what's causing them, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

I have rebooted again, disabled docker, and am just running a parity check now. Will see how that goes.

 

Was there something in particular that was erroring?

There are some ATA errors, but those should not make the server crash, should still be checked, looks more like a cable issue with the cache SSD, then it's spammed with a lot of these:

 

Jan  6 01:50:30 Tower avahi-daemon[735]: avahi_normalize_name() failed.
Jan  6 01:50:30 Tower avahi-daemon[735]: avahi_key_new() failed.
Jan  6 01:50:30 Tower avahi-daemon[735]: avahi_normalize_name() failed.
Jan  6 01:50:30 Tower avahi-daemon[735]: avahi_key_new() failed.
Jan  6 01:50:30 Tower avahi-daemon[735]: avahi_normalize_name() failed.

 

But not sure what causing them, safe mode will rule out a plugin issue.

 

 

So the server stayed up when there was nothing going on, other then the parity check scanning.  What would be the next steps?

Start the services one by one.

Crashed again.

Had the following Docker services running at the time:

Plex, CrashplanPro, DuckDNS, LuckyBackup, Syncthing, and LubeLogger.

 

It seems like it is when an application is doing a sync/scan of files, it seems to crash.

 

New logs attached.

tower-diagnostics-20240113-2020.zip syslog-previous(2) syslog(2)

Several MCE events logged, so possibly a hardware issue:
 

Jan 12 04:51:54 Tower kernel: mce: [Hardware Error]: Machine check events logged
Jan 12 04:52:48 Tower kernel: mce: [Hardware Error]: Machine check events logged
Jan 12 04:53:28 Tower kernel: mce: [Hardware Error]: Machine check events logged
Jan 12 04:54:15 Tower kernel: mce: [Hardware Error]: Machine check events logged
Jan 12 04:54:31 Tower kernel: mce: [Hardware Error]: Machine check events logged
Jan 12 04:54:46 Tower kernel: mce: [Hardware Error]: Machine check events logged

 

You are also having what look like power/connection issues with the cache device, replace cables.

It seems like the issue only occures if a docker service does a full file scan (ie. if crashplan rescans the files being backed up, or Plex rescans the media on the filesystem).

 

I verified the cache cables are connected well, didn't seem to make a difference.

 

Is there any way to determine what Hardware error is being logged?

55 minutes ago, SpyKiIIer said:

It seems like the issue only occures if a docker service does a full file scan

Maybe a power or temperature issue.

 

55 minutes ago, SpyKiIIer said:

Is there any way to determine what Hardware error is being logged?

I can't see any more details.

Following up on this one, when I tried to copy files of any signifiant size the system would freeze up.  I turned off docker completely, and the system has been solid for copying files off.  Is it possible that something has gone wrong with my docker setup, or an application that has been installed?

 

Is there a way to blow away all the docker setup and appdata, and start docker fresh to see if that is the issue?

Seems unlikely do me that the docker service would make the server crash during a file transfer, but you can try recreating it, just delete the docker image and also delete/rename the appdata folder.