jromkes226

I agree, I’ve never seen a drive fail present itself like that, but that particular ssd was quite old with lots of TBW. I also ended up doing a BIOS update. the BIOS wasn’t super old, but about 4 revisions back. A memtest which passed. And a replacement of ATA 5 seems to be working fine now (12H solid) I’ll do further testing with that potentially failed drive later. I’ll also close this post in a few days if things continue stable. But my hopes are high. Normally the crash would happen within the hour

Further to the previous syslog, here is more later Kernel Trace 2-53PM.txt

INFO KERN 2025-01-27 1:45:48 PM NAS kernel: wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information. INFO KERN 2025-01-27 1:45:48 PM NAS kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. INFO KERN 2025-01-27 1:58:50 PM NAS kernel: TCP: request_sock_TCP: Possible SYN flooding on port 10.20.178.59:42807. Sending cookies. ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: exception Emask 0x10 SAct 0x8000004 SErr 0x400100 action 0x6 frozen ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: irq_stat 0x08000000, interface fatal error ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5: SError: { UnrecovData Handshk } ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: failed command: WRITE FPDMA QUEUED ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: cmd 61/58:10:38:6f:9f/58:00:0e:00:00/40 tag 2 ncq dma 45056 out ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error) ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: status: { DRDY } ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: failed command: WRITE FPDMA QUEUED ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: cmd 61/58:d8:e0:6e:9f/00:00:0e:00:00/40 tag 27 ncq dma 45056 out ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error) ERROR KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: status: { DRDY } INFO KERN 2025-01-27 2:03:59 PM NAS kernel: ata5: hard resetting link INFO KERN 2025-01-27 2:03:59 PM NAS kernel: ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300) WARNING KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: supports DRM functions and may not be fully accessible WARNING KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: supports DRM functions and may not be fully accessible INFO KERN 2025-01-27 2:03:59 PM NAS kernel: ata5.00: configured for UDMA/133 INFO KERN 2025-01-27 2:03:59 PM NAS kernel: ata5: EH complete INFO KERN 2025-01-27 2:08:59 PM NAS kernel: smartctl[123611]: segfault at 0 ip 000014ea2d82078b sp 00007ffd93946ed8 error 4 in libc-2.40.so[14ea2d7b4000+17c000] likely on CPU 3 (core 4, socket 0) INFO KERN 2025-01-27 2:08:59 PM NAS kernel: Code: 8b 47 28 48 8b 57 10 48 39 c2 73 07 48 89 47 10 48 89 c2 8b 0f 89 ce f7 d6 81 e6 00 0c 00 00 74 0e 48 8b 47 08 48 39 d0 73 1d <0f> b6 00 c3 90 80 e5 f7 48 89 47 08 89 0f 48 8b 4f 30 48 89 4f 28 INFO KERN 2025-01-27 2:31:46 PM NAS kernel: vetha91dc29: renamed from eth0 reading into this, I see two things, some kinda disk error on ATA5? ill take a look at what drive that is and try and figure that out. and a segfault? any ideas are appreciated!

what is the intel 13/14 issue? still trying to get the syslog working properly, but I will share when it's functioning. last log I found was related to a out memory issue with nginx Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [crit] 3858#3858: ngx_slab_alloc() failed: no memory Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: shpool alloc failed Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: nchan: Out of shared memory while allocating message of size 23760. Increase nchan_max_reserved_memory. Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: *928685 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost" Jan 27 08:37:34 NAS nginx: 2025/01/27 08:37:34 [error] 3858#3858: MEMSTORE:00: can't create shared message for channel /devices I left the server to run over the weekend in safe mode, it worked relatively well but when I went to do some work this morning, same issue. server seems fine but when I go to restart etc. completely unresponsive EDIT: Disregard, found it. now im not getting a full kernel panic / crash like what others are reporting. but nonetheless I will complete a BIOS update and see how it goes. from what I read it mostly affects i9 chips.

yes this has been brutal, 5 hours into my server running and I basically loose all manageability to the server but everything runs still, its ridiculous. sadly I dont think I can roll back cause I am using ZFS now for my apps directory

since about monday my server keeps locking up constantly. it seems to take between 5 minutes or over an hour. it starts with some docker containers becoming unresponsive, following some webUI pages like the docker containers section, apps, plugins. The main page seems to work mostly. eventually everything in the web UI will respond with an error 500 and eventually the site will no longer load and become completely unresponsive. I can though still SSH into the server periodically although some commands do not work like shutdown, docker related commands. htop etc. In some instances docker containers will still work, and will continue to work until backend updates, or backups try to run. I also cannot start or stop the containers while they are running and this issue is persisting. Shutting down does not work, I have to force power off. when coming back up everything loads normally minus the parity check messages from the failed power off. I thought it was a parity check causing latency, but after stopping that, the issue persisted. ive also stopped all containers and left the server and it seemed to work for a longer period of time before the issue coming up again. I thought it was a faulty disk, which one in particular that has given me issues was removed. then I thought it was a particular docker container, now im entirely not sure at all. everything works super well until it doesn't.... the only way to recover is a hard power off. I will attach my diagnostics file, but here is some server info also: Board PRO Z790-A MAX WIFI CPU i7-14700K GTX 1060 1000W Seasonic PSU 13 Hard drives in various size and age 3 cache SSDs 2 NVME drives in a ZFS RAID 1 for apps NOTE this build is relatively new, before then I was running a whitebox Xeon build and it ran relatively stable with the same GPU and PSU I am using now nas-diagnostics-20250124-1459.zip