since about monday my server keeps locking up constantly. it seems to take between 5 minutes or over an hour. it starts with some docker containers becoming unresponsive, following some webUI pages like the docker containers section, apps, plugins. The main page seems to work mostly. eventually everything in the web UI will respond with an error 500 and eventually the site will no longer load and become completely unresponsive. I can though still SSH into the server periodically although some commands do not work like shutdown, docker related commands. htop etc. In some instances docker containers will still work, and will continue to work until backend updates, or backups try to run. I also cannot start or stop the containers while they are running and this issue is persisting.

 

Shutting down does not work, I have to force power off. when coming back up everything loads normally minus the parity check messages from the failed power off. I thought it was a parity check causing latency, but after stopping that, the issue persisted. 

ive also stopped all containers and left the server and it seemed to work for a longer period of time before the issue coming up again. 

 

I thought it was a faulty disk, which one in particular that has given me issues was removed. then I thought it was a particular docker container, now im entirely not sure at all. everything works super well until it doesn't.... the only way to recover is a hard power off. 

I will attach my diagnostics file, but here is some server info also:

Board PRO Z790-A MAX WIFI

CPU i7-14700K
GTX 1060

1000W Seasonic PSU

13 Hard drives in various size and age

3 cache SSDs

2 NVME drives in a ZFS RAID 1 for apps

NOTE this build is relatively new, before then I was running a whitebox Xeon build and it ran relatively stable with the same GPU and PSU I am using now

nas-diagnostics-20250124-1459.zip

11 hours ago, jromkes226 said:

CPU i7-14700K

Could be the Intel 13/14 issue, but enable the syslog server and post that after a crash.

21 hours ago, jromkes226 said:

since about monday my server keeps locking up constantly. it seems to take between 5 minutes or over an hour. it starts with some docker containers becoming unresponsive, following some webUI pages like the docker containers section, apps, plugins. The main page seems to work mostly. eventually everything in the web UI will respond with an error 500 and eventually the site will no longer load and become completely unresponsive. I can though still SSH into the server periodically although some commands do not work like shutdown, docker related commands. htop etc. In some instances docker containers will still work, and will continue to work until backend updates, or backups try to run. I also cannot start or stop the containers while they are running and this issue is persisting.

 

Shutting down does not work, I have to force power off. when coming back up everything loads normally minus the parity check messages from the failed power off. I thought it was a parity check causing latency, but after stopping that, the issue persisted. 

ive also stopped all containers and left the server and it seemed to work for a longer period of time before the issue coming up again. 

 

I thought it was a faulty disk, which one in particular that has given me issues was removed. then I thought it was a particular docker container, now im entirely not sure at all. everything works super well until it doesn't.... the only way to recover is a hard power off. 

I will attach my diagnostics file, but here is some server info also:

Board PRO Z790-A MAX WIFI

CPU i7-14700K
GTX 1060

1000W Seasonic PSU

13 Hard drives in various size and age

3 cache SSDs

2 NVME drives in a ZFS RAID 1 for apps

NOTE this build is relatively new, before then I was running a whitebox Xeon build and it ran relatively stable with the same GPU and PSU I am using now

nas-diagnostics-20250124-1459.zip 195.04 kB · 1 download

 

I just tried to update to 7.0 last night myself, wish I never did...  I essentially am having the same issue. however the list of symptoms is unbelievable. I downgraded back to 6.12.10 just to be "more" stable. started with Dev SDA sector issues (usb) to network issues to just flat out crashing hard... the 7.0 update messed something up. this system used to be rock solid would run for 2+ months with no issues. I would end up rebooting due to external reasons. either way... I have had to scan for errors in windows a couple of times and windows would say it is fixed. I only did this because Unraid game me a message for the USB drive being corrupt. it would then seem to run better for a little, then I start having either other issues or just locks up. I myself am out of ideas. I have just done a format, fresh re-install of unraid on the USB and then copied my backup back over to the USB... seeing how this works now. I have been working on my server for MANY hours... hopefully that made sense... brain hurts lol

Edited January 25, 20251 yr by KooKoo102

On 1/25/2025 at 2:40 AM, JorgeB said:

Could be the Intel 13/14 issue, but enable the syslog server and post that after a crash.

 

On 1/25/2025 at 12:21 PM, KooKoo102 said:

 

I just tried to update to 7.0 last night myself, wish I never did...  I essentially am having the same issue. however the list of symptoms is unbelievable. I downgraded back to 6.12.10 just to be "more" stable. started with Dev SDA sector issues (usb) to network issues to just flat out crashing hard... the 7.0 update messed something up. this system used to be rock solid would run for 2+ months with no issues. I would end up rebooting due to external reasons. either way... I have had to scan for errors in windows a couple of times and windows would say it is fixed. I only did this because Unraid game me a message for the USB drive being corrupt. it would then seem to run better for a little, then I start having either other issues or just locks up. I myself am out of ideas. I have just done a format, fresh re-install of unraid on the USB and then copied my backup back over to the USB... seeing how this works now. I have been working on my server for MANY hours... hopefully that made sense... brain hurts lol

yes this has been brutal, 5 hours into my server running and I basically loose all manageability to the server but everything runs still, its ridiculous. 
sadly I dont think I can roll back cause I am using ZFS now for my apps directory

On 1/25/2025 at 2:40 AM, JorgeB said:

Could be the Intel 13/14 issue, but enable the syslog server and post that after a crash.

what is the intel 13/14 issue? still trying to get the syslog working properly, but I will share when it's functioning.
last log I found was related to a out memory issue with nginx

Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [crit] 3858#3858: ngx_slab_alloc() failed: no memory
Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: shpool alloc failed
Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: nchan: Out of shared
memory while allocating message of size 23760. Increase nchan_max_reserved_memory.
Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: *928685 nchan: error
publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: MEMSTORE:00: can't create shared message for channel /devices

I left the server to run over the weekend in safe mode, it worked relatively well but when I went to do some work this morning, same issue. server seems fine but when I go to restart etc. completely unresponsive

 

EDIT: Disregard, found it. now im not getting a full kernel panic / crash like what others are reporting. but nonetheless I will complete a BIOS update and see how it goes. from what I read it mostly affects i9 chips. 

Edited January 27, 20251 yr by jromkes226
More Research

On 1/25/2025 at 2:40 AM, JorgeB said:

Could be the Intel 13/14 issue, but enable the syslog server and post that after a crash.

 

INFO	KERN	2025-01-27 1:45:48 PM	NAS	kernel: wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
INFO	KERN	2025-01-27 1:45:48 PM	NAS	kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
INFO	KERN	2025-01-27 1:58:50 PM	NAS	kernel: TCP: request_sock_TCP: Possible SYN flooding on port 10.20.178.59:42807. Sending cookies.
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: exception Emask 0x10 SAct 0x8000004 SErr 0x400100 action 0x6 frozen
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: irq_stat 0x08000000, interface fatal error
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5: SError: { UnrecovData Handshk }
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: failed command: WRITE FPDMA QUEUED
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: cmd 61/58:10:38:6f:9f/58:00:0e:00:00/40 tag 2 ncq dma 45056 out
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: status: { DRDY }
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: failed command: WRITE FPDMA QUEUED
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: cmd 61/58:d8:e0:6e:9f/00:00:0e:00:00/40 tag 27 ncq dma 45056 out
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel:         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
ERROR	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: status: { DRDY }
INFO	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5: hard resetting link
INFO	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
WARNING	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: supports DRM functions and may not be fully accessible
WARNING	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: supports DRM functions and may not be fully accessible
INFO	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5.00: configured for UDMA/133
INFO	KERN	2025-01-27 2:03:59 PM	NAS	kernel: ata5: EH complete
INFO	KERN	2025-01-27 2:08:59 PM	NAS	kernel: smartctl[123611]: segfault at 0 ip 000014ea2d82078b sp 00007ffd93946ed8 error 4 in libc-2.40.so[14ea2d7b4000+17c000] likely on CPU 3 (core 4, socket 0)
INFO	KERN	2025-01-27 2:08:59 PM	NAS	kernel: Code: 8b 47 28 48 8b 57 10 48 39 c2 73 07 48 89 47 10 48 89 c2 8b 0f 89 ce f7 d6 81 e6 00 0c 00 00 74 0e 48 8b 47 08 48 39 d0 73 1d <0f> b6 00 c3 90 80 e5 f7 48 89 47 08 89 0f 48 8b 4f 30 48 89 4f 28
INFO	KERN	2025-01-27 2:31:46 PM	NAS	kernel: vetha91dc29: renamed from eth0

reading into this, I see two things, some kinda disk error on ATA5? ill take a look at what drive that is and try and figure that out.
and a segfault? 

any ideas are appreciated!

Further to the previous syslog, here is more later

Kernel Trace 2-53PM.txt

Those ATA errors should not be a reason for the server crashing, but check/replace cables for that device.

 

11 hours ago, jromkes226 said:

here is more later

That shows the Unraid driver crashing, and that it's almost always a hardware issue.

7 hours ago, JorgeB said:

Those ATA errors should not be a reason for the server crashing, but check/replace cables for that device.

 

That shows the Unraid driver crashing, and that it's almost always a hardware issue.

I agree, I’ve never seen a drive fail present itself like that, but that particular ssd was quite old with lots of TBW.
 

I also ended up doing a BIOS update. the BIOS wasn’t super old, but about 4 revisions back. A memtest which passed. And a replacement of ATA 5

 

seems to be working fine now (12H solid) I’ll do further testing with that potentially failed drive later. I’ll also close this post in a few days if things continue stable. But my hopes are high. Normally the crash would happen within the hour