Bob-omb

Since I am running an Intel Core i9-14900K the issue could theoretically be related to Intel "vmin shift instability issue", but I have had the CPU replaced and updated the microcode to version 0x12F which per September 7th 2025 seems to be the latest.

Looking at the syslog: The GPU crashes happened around 14:43–14:44 on Sept 6 but the system clearly kept running after that — I logged into the WebGUI at 15:44, Docker veth cleanup happened at 16:33, and SMART polls/spindowns are logged right through the evening. At 20:10–20:17 there are still disk SMART and spindown events. The last normal activity before midnight was SMART reads/spindowns around 23:50. On Sept 7, logs resume as if the system was alive until about 04:28, when drives were spun down again. Then… nothing. No shutdown, no kernel panic, no DHCP renew. Just silence.

As I said I have enabled the syslog server: The crash happened again today and this is what the syslog server has logged: <14>Sep 6 00:00:07 nova emhttpd: read SMART /dev/sdh <14>Sep 6 00:00:07 nova emhttpd: read SMART /dev/sdd <14>Sep 6 00:16:57 nova emhttpd: spinning down /dev/sdh <14>Sep 6 00:16:57 nova emhttpd: spinning down /dev/sdd <14>Sep 6 00:59:19 nova emhttpd: read SMART /dev/sdf <14>Sep 6 01:11:17 nova emhttpd: read SMART /dev/sdj <14>Sep 6 01:11:17 nova emhttpd: read SMART /dev/sdd <14>Sep 6 01:14:31 nova emhttpd: spinning down /dev/sdf <14>Sep 6 01:43:56 nova emhttpd: spinning down /dev/sdj <14>Sep 6 01:43:56 nova emhttpd: spinning down /dev/sdd <14>Sep 6 02:04:17 nova emhttpd: read SMART /dev/sdf <14>Sep 6 02:19:28 nova emhttpd: spinning down /dev/sdf <14>Sep 6 02:58:17 nova emhttpd: read SMART /dev/sdj <14>Sep 6 02:58:17 nova emhttpd: read SMART /dev/sdd <14>Sep 6 03:16:05 nova emhttpd: spinning down /dev/sdj <14>Sep 6 03:16:05 nova emhttpd: spinning down /dev/sdd <14>Sep 6 03:44:08 nova emhttpd: read SMART /dev/sdj <14>Sep 6 03:44:08 nova emhttpd: read SMART /dev/sdd <14>Sep 6 04:00:12 nova emhttpd: read SMART /dev/sdh <14>Sep 6 04:04:25 nova emhttpd: spinning down /dev/sdj <14>Sep 6 04:17:45 nova emhttpd: spinning down /dev/sdh <14>Sep 6 04:17:45 nova emhttpd: spinning down /dev/sdd <14>Sep 6 07:02:57 nova emhttpd: read SMART /dev/sdj <14>Sep 6 07:02:57 nova emhttpd: read SMART /dev/sdd <14>Sep 6 07:51:08 nova emhttpd: spinning down /dev/sdj <14>Sep 6 07:51:08 nova emhttpd: spinning down /dev/sdd <14>Sep 6 07:52:40 nova emhttpd: read SMART /dev/sdj <14>Sep 6 07:52:40 nova emhttpd: read SMART /dev/sdd <14>Sep 6 08:00:06 nova emhttpd: read SMART /dev/sdh <14>Sep 6 08:04:00 nova emhttpd: read SMART /dev/sde <14>Sep 6 08:04:54 nova emhttpd: read SMART /dev/sdf <14>Sep 6 08:09:16 nova emhttpd: spinning down /dev/sdj <14>Sep 6 08:17:24 nova emhttpd: spinning down /dev/sdh <14>Sep 6 08:17:24 nova emhttpd: spinning down /dev/sdd <14>Sep 6 08:20:16 nova emhttpd: spinning down /dev/sde <14>Sep 6 08:20:16 nova emhttpd: spinning down /dev/sdf <14>Sep 6 08:38:22 nova emhttpd: read SMART /dev/sdj <14>Sep 6 08:38:22 nova emhttpd: read SMART /dev/sdd <14>Sep 6 08:54:34 nova emhttpd: spinning down /dev/sdj <14>Sep 6 08:54:34 nova emhttpd: spinning down /dev/sdd <14>Sep 6 09:34:33 nova emhttpd: read SMART /dev/sdj <14>Sep 6 09:36:18 nova emhttpd: read SMART /dev/sde <14>Sep 6 09:38:36 nova emhttpd: read SMART /dev/sdd <14>Sep 6 09:51:28 nova emhttpd: spinning down /dev/sde <14>Sep 6 10:02:33 nova emhttpd: spinning down /dev/sdj <14>Sep 6 10:02:33 nova emhttpd: spinning down /dev/sdd <14>Sep 6 10:24:05 nova emhttpd: read SMART /dev/sdj <14>Sep 6 10:24:05 nova emhttpd: read SMART /dev/sdd <14>Sep 6 10:31:52 nova emhttpd: read SMART /dev/sdf <14>Sep 6 10:40:18 nova emhttpd: spinning down /dev/sdj <14>Sep 6 10:40:18 nova emhttpd: spinning down /dev/sdd <14>Sep 6 10:47:37 nova emhttpd: spinning down /dev/sdf <14>Sep 6 11:08:46 nova emhttpd: read SMART /dev/sde <14>Sep 6 11:25:30 nova emhttpd: spinning down /dev/sde <14>Sep 6 11:30:31 nova emhttpd: read SMART /dev/sdj <14>Sep 6 11:45:47 nova emhttpd: spinning down /dev/sdj <14>Sep 6 11:53:51 nova emhttpd: read SMART /dev/sdj <14>Sep 6 11:53:51 nova emhttpd: read SMART /dev/sdd <14>Sep 6 12:00:02 nova emhttpd: read SMART /dev/sdh <14>Sep 6 12:11:26 nova emhttpd: spinning down /dev/sdj <14>Sep 6 12:17:13 nova emhttpd: spinning down /dev/sdh <14>Sep 6 12:17:13 nova emhttpd: spinning down /dev/sdd <14>Sep 6 12:23:56 nova emhttpd: read SMART /dev/sdj <14>Sep 6 12:23:56 nova emhttpd: read SMART /dev/sdd <14>Sep 6 12:56:14 nova emhttpd: spinning down /dev/sdj <14>Sep 6 12:56:14 nova emhttpd: spinning down /dev/sdd <14>Sep 6 14:26:12 nova emhttpd: read SMART /dev/sdj <14>Sep 6 14:26:12 nova emhttpd: read SMART /dev/sdd <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=416, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] Xe device coredump has been created <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: failed to get forcewake for coredump capture <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:58 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:43:58 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:00 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:44:00 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:02 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:44:02 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967181, guc_id=4, flags=0x0 in ffmpeg [1433513] <14>Sep 6 14:44:33 nova emhttpd: spinning down /dev/sdj <14>Sep 6 14:44:33 nova emhttpd: spinning down /dev/sdd <14>Sep 6 14:56:46 nova emhttpd: read SMART /dev/sdj <14>Sep 6 14:56:46 nova emhttpd: read SMART /dev/sdd <14>Sep 6 15:31:02 nova emhttpd: spinning down /dev/sdj <14>Sep 6 15:31:02 nova emhttpd: spinning down /dev/sdd <6>Sep 6 15:44:11 nova kernel: xe 0000:03:00.0: [drm] Xe device coredump has been deleted. <13>Sep 6 15:44:34 nova webgui: Successful login user root from 192.168.1.50 <14>Sep 6 15:47:12 nova emhttpd: read SMART /dev/sdj <14>Sep 6 15:47:23 nova emhttpd: read SMART /dev/sdd <14>Sep 6 16:08:17 nova emhttpd: spinning down /dev/sdj <14>Sep 6 16:16:52 nova emhttpd: spinning down /dev/sdh <14>Sep 6 16:16:52 nova emhttpd: spinning down /dev/sdd <14>Sep 6 16:31:56 nova emhttpd: read SMART /dev/sdj <14>Sep 6 16:31:56 nova emhttpd: read SMART /dev/sdd <14>Sep 6 16:32:07 nova emhttpd: read SMART /dev/sdg <6>Sep 6 16:33:26 nova kernel: docker0: port 5(veth6430636) entered disabled state <6>Sep 6 16:33:26 nova kernel: veth3dfc2d2: renamed from eth0 <6>Sep 6 16:33:26 nova kernel: docker0: port 5(veth6430636) entered disabled state <6>Sep 6 16:33:26 nova kernel: veth6430636 (unregistering): left allmulticast mode <6>Sep 6 16:33:26 nova kernel: veth6430636 (unregistering): left promiscuous mode <6>Sep 6 16:33:26 nova kernel: docker0: port 5(veth6430636) entered disabled state <14>Sep 6 16:54:05 nova emhttpd: spinning down /dev/sdg <6>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: AER: Correctable error message received from 0000:00:06.0 <4>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID) <4>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: device [8086:a74d] error status/mask=00000001/00002000 <4>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: [ 0] RxErr (First) <14>Sep 6 17:49:23 nova emhttpd: spinning down /dev/sdj <14>Sep 6 17:49:23 nova emhttpd: spinning down /dev/sdd <14>Sep 6 18:30:51 nova emhttpd: read SMART /dev/sdj <14>Sep 6 18:30:51 nova emhttpd: read SMART /dev/sdd <14>Sep 6 19:02:53 nova emhttpd: spinning down /dev/sdj <14>Sep 6 19:02:53 nova emhttpd: spinning down /dev/sdd <14>Sep 6 19:39:10 nova emhttpd: read SMART /dev/sde <14>Sep 6 20:00:07 nova emhttpd: read SMART /dev/sdh <14>Sep 6 20:00:07 nova emhttpd: read SMART /dev/sdd <14>Sep 6 20:10:29 nova emhttpd: read SMART /dev/sdf <14>Sep 6 20:14:55 nova emhttpd: spinning down /dev/sde <14>Sep 6 20:17:00 nova emhttpd: spinning down /dev/sdh <14>Sep 6 20:17:00 nova emhttpd: spinning down /dev/sdd <14>Sep 6 20:48:29 nova emhttpd: read SMART /dev/sdj <14>Sep 6 20:48:29 nova emhttpd: read SMART /dev/sdd <14>Sep 6 21:06:47 nova emhttpd: spinning down /dev/sdj <14>Sep 6 21:06:47 nova emhttpd: spinning down /dev/sdd <14>Sep 6 21:36:43 nova emhttpd: read SMART /dev/sde <14>Sep 6 21:51:54 nova emhttpd: spinning down /dev/sde <14>Sep 6 22:01:20 nova emhttpd: spinning down /dev/sdf <14>Sep 6 22:33:07 nova emhttpd: read SMART /dev/sdf <14>Sep 6 22:50:29 nova emhttpd: spinning down /dev/sdf <14>Sep 6 23:31:27 nova emhttpd: read SMART /dev/sdj <14>Sep 6 23:31:27 nova emhttpd: read SMART /dev/sdd <14>Sep 6 23:44:15 nova emhttpd: read SMART /dev/sdf <14>Sep 6 23:47:54 nova emhttpd: spinning down /dev/sdd <14>Sep 6 23:48:44 nova emhttpd: spinning down /dev/sdj <14>Sep 6 23:50:22 nova emhttpd: read SMART /dev/sdj <14>Sep 6 23:50:22 nova emhttpd: read SMART /dev/sdd <14>Sep 6 23:59:27 nova emhttpd: spinning down /dev/sdf <14>Sep 7 00:00:07 nova emhttpd: read SMART /dev/sdh <14>Sep 7 00:07:01 nova emhttpd: spinning down /dev/sdj <14>Sep 7 00:17:18 nova emhttpd: spinning down /dev/sdh <14>Sep 7 00:17:18 nova emhttpd: spinning down /dev/sdd <14>Sep 7 00:37:47 nova emhttpd: read SMART /dev/sdj <14>Sep 7 00:37:47 nova emhttpd: read SMART /dev/sdd <14>Sep 7 00:49:16 nova emhttpd: read SMART /dev/sdf <14>Sep 7 00:55:52 nova emhttpd: spinning down /dev/sdj <14>Sep 7 00:55:52 nova emhttpd: spinning down /dev/sdd <14>Sep 7 01:04:28 nova emhttpd: spinning down /dev/sdf <14>Sep 7 01:54:21 nova emhttpd: read SMART /dev/sdf <14>Sep 7 02:09:30 nova emhttpd: spinning down /dev/sdf <14>Sep 7 02:24:21 nova emhttpd: spinning down /dev/sde <14>Sep 7 02:55:21 nova emhttpd: read SMART /dev/sdj <14>Sep 7 02:55:21 nova emhttpd: read SMART /dev/sdd <14>Sep 7 02:59:13 nova emhttpd: read SMART /dev/sdf <14>Sep 7 03:14:28 nova emhttpd: spinning down /dev/sdf <14>Sep 7 03:19:16 nova emhttpd: spinning down /dev/sdj <14>Sep 7 03:19:16 nova emhttpd: spinning down /dev/sdd <14>Sep 7 03:42:34 nova emhttpd: read SMART /dev/sde <14>Sep 7 03:58:28 nova emhttpd: spinning down /dev/sde <14>Sep 7 04:00:06 nova emhttpd: read SMART /dev/sdh <14>Sep 7 04:00:06 nova emhttpd: read SMART /dev/sdd <14>Sep 7 04:11:46 nova emhttpd: read SMART /dev/sdj <14>Sep 7 04:16:57 nova emhttpd: spinning down /dev/sdh <14>Sep 7 04:28:14 nova emhttpd: spinning down /dev/sdj <14>Sep 7 04:28:14 nova emhttpd: spinning down /dev/sdd <14>Sep 6 12:00:02 nova emhttpd: read SMART /dev/sdh <14>Sep 6 12:11:26 nova emhttpd: spinning down /dev/sdj <14>Sep 6 12:17:13 nova emhttpd: spinning down /dev/sdh <14>Sep 6 12:17:13 nova emhttpd: spinning down /dev/sdd <14>Sep 6 12:23:56 nova emhttpd: read SMART /dev/sdj <14>Sep 6 12:23:56 nova emhttpd: read SMART /dev/sdd <14>Sep 6 12:56:14 nova emhttpd: spinning down /dev/sdj <14>Sep 6 12:56:14 nova emhttpd: spinning down /dev/sdd <14>Sep 6 14:26:12 nova emhttpd: read SMART /dev/sdj <14>Sep 6 14:26:12 nova emhttpd: read SMART /dev/sdd <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=416, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] Xe device coredump has been created <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: failed to get forcewake for coredump capture <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data <6>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:43:57 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:58 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:43:58 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:43:59 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:00 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:44:00 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:44:01 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:02 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:44:02 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=4, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=12 <5>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967182, guc_id=12, flags=0x0 in ffmpeg [1433513] <6>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Engine reset: engine_class=vcs, logical_mask: 0x1, guc_id=4 <5>Sep 6 14:44:03 nova kernel: xe 0000:03:00.0: [drm] GT1: Timedout job: seqno=0, lrc_seqno=4294967181, guc_id=4, flags=0x0 in ffmpeg [1433513] <14>Sep 6 14:44:33 nova emhttpd: spinning down /dev/sdj <14>Sep 6 14:44:33 nova emhttpd: spinning down /dev/sdd <14>Sep 6 14:56:46 nova emhttpd: read SMART /dev/sdj <14>Sep 6 14:56:46 nova emhttpd: read SMART /dev/sdd <14>Sep 6 15:31:02 nova emhttpd: spinning down /dev/sdj <14>Sep 6 15:31:02 nova emhttpd: spinning down /dev/sdd <6>Sep 6 15:44:11 nova kernel: xe 0000:03:00.0: [drm] Xe device coredump has been deleted. <13>Sep 6 15:44:34 nova webgui: Successful login user root from 192.168.1.50 <14>Sep 6 15:47:12 nova emhttpd: read SMART /dev/sdj <14>Sep 6 15:47:23 nova emhttpd: read SMART /dev/sdd <14>Sep 6 16:08:17 nova emhttpd: spinning down /dev/sdj <14>Sep 6 16:16:52 nova emhttpd: spinning down /dev/sdh <14>Sep 6 16:16:52 nova emhttpd: spinning down /dev/sdd <14>Sep 6 16:31:56 nova emhttpd: read SMART /dev/sdj <14>Sep 6 16:31:56 nova emhttpd: read SMART /dev/sdd <14>Sep 6 16:32:07 nova emhttpd: read SMART /dev/sdg <6>Sep 6 16:33:26 nova kernel: docker0: port 5(veth6430636) entered disabled state <6>Sep 6 16:33:26 nova kernel: veth3dfc2d2: renamed from eth0 <6>Sep 6 16:33:26 nova kernel: docker0: port 5(veth6430636) entered disabled state <6>Sep 6 16:33:26 nova kernel: veth6430636 (unregistering): left allmulticast mode <6>Sep 6 16:33:26 nova kernel: veth6430636 (unregistering): left promiscuous mode <6>Sep 6 16:33:26 nova kernel: docker0: port 5(veth6430636) entered disabled state <14>Sep 6 16:54:05 nova emhttpd: spinning down /dev/sdg <6>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: AER: Correctable error message received from 0000:00:06.0 <4>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID) <4>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: device [8086:a74d] error status/mask=00000001/00002000 <4>Sep 6 17:35:08 nova kernel: pcieport 0000:00:06.0: [ 0] RxErr (First) <14>Sep 6 17:49:23 nova emhttpd: spinning down /dev/sdj <14>Sep 6 17:49:23 nova emhttpd: spinning down /dev/sdd <14>Sep 6 18:30:51 nova emhttpd: read SMART /dev/sdj <14>Sep 6 18:30:51 nova emhttpd: read SMART /dev/sdd <14>Sep 6 19:02:53 nova emhttpd: spinning down /dev/sdj <14>Sep 6 19:02:53 nova emhttpd: spinning down /dev/sdd <14>Sep 6 19:39:10 nova emhttpd: read SMART /dev/sde <14>Sep 6 20:00:07 nova emhttpd: read SMART /dev/sdh <14>Sep 6 20:00:07 nova emhttpd: read SMART /dev/sdd <14>Sep 6 20:10:29 nova emhttpd: read SMART /dev/sdf <14>Sep 6 20:14:55 nova emhttpd: spinning down /dev/sde <14>Sep 6 20:17:00 nova emhttpd: spinning down /dev/sdh <14>Sep 6 20:17:00 nova emhttpd: spinning down /dev/sdd <14>Sep 6 20:48:29 nova emhttpd: read SMART /dev/sdj <14>Sep 6 20:48:29 nova emhttpd: read SMART /dev/sdd <14>Sep 6 21:06:47 nova emhttpd: spinning down /dev/sdj <14>Sep 6 21:06:47 nova emhttpd: spinning down /dev/sdd <14>Sep 6 21:36:43 nova emhttpd: read SMART /dev/sde <14>Sep 6 21:51:54 nova emhttpd: spinning down /dev/sde <14>Sep 6 22:01:20 nova emhttpd: spinning down /dev/sdf <14>Sep 6 22:33:07 nova emhttpd: read SMART /dev/sdf <14>Sep 6 22:50:29 nova emhttpd: spinning down /dev/sdf <14>Sep 6 23:31:27 nova emhttpd: read SMART /dev/sdj <14>Sep 6 23:31:27 nova emhttpd: read SMART /dev/sdd <14>Sep 6 23:44:15 nova emhttpd: read SMART /dev/sdf <14>Sep 6 23:47:54 nova emhttpd: spinning down /dev/sdd <14>Sep 6 23:48:44 nova emhttpd: spinning down /dev/sdj <14>Sep 6 23:50:22 nova emhttpd: read SMART /dev/sdj <14>Sep 6 23:50:22 nova emhttpd: read SMART /dev/sdd <14>Sep 6 23:59:27 nova emhttpd: spinning down /dev/sdf <14>Sep 7 00:00:07 nova emhttpd: read SMART /dev/sdh <14>Sep 7 00:07:01 nova emhttpd: spinning down /dev/sdj <14>Sep 7 00:17:18 nova emhttpd: spinning down /dev/sdh <14>Sep 7 00:17:18 nova emhttpd: spinning down /dev/sdd <14>Sep 7 00:37:47 nova emhttpd: read SMART /dev/sdj <14>Sep 7 00:37:47 nova emhttpd: read SMART /dev/sdd <14>Sep 7 00:49:16 nova emhttpd: read SMART /dev/sdf <14>Sep 7 00:55:52 nova emhttpd: spinning down /dev/sdj <14>Sep 7 00:55:52 nova emhttpd: spinning down /dev/sdd <14>Sep 7 01:04:28 nova emhttpd: spinning down /dev/sdf <14>Sep 7 01:54:21 nova emhttpd: read SMART /dev/sdf <14>Sep 7 02:09:30 nova emhttpd: spinning down /dev/sdf <14>Sep 7 02:24:21 nova emhttpd: spinning down /dev/sde <14>Sep 7 02:55:21 nova emhttpd: read SMART /dev/sdj <14>Sep 7 02:55:21 nova emhttpd: read SMART /dev/sdd <14>Sep 7 02:59:13 nova emhttpd: read SMART /dev/sdf <14>Sep 7 03:14:28 nova emhttpd: spinning down /dev/sdf <14>Sep 7 03:19:16 nova emhttpd: spinning down /dev/sdj <14>Sep 7 03:19:16 nova emhttpd: spinning down /dev/sdd <14>Sep 7 03:42:34 nova emhttpd: read SMART /dev/sde <14>Sep 7 03:58:28 nova emhttpd: spinning down /dev/sde <14>Sep 7 04:00:06 nova emhttpd: read SMART /dev/sdh <14>Sep 7 04:00:06 nova emhttpd: read SMART /dev/sdd <14>Sep 7 04:11:46 nova emhttpd: read SMART /dev/sdj <14>Sep 7 04:16:57 nova emhttpd: spinning down /dev/sdh <14>Sep 7 04:28:14 nova emhttpd: spinning down /dev/sdj <14>Sep 7 04:28:14 nova emhttpd: spinning down /dev/sdd

My system has been running stable for about 29 days or so then all of a sudden my Unraid server crashes during night for some reason. Twice. That is; the computer is still hot and running, but the interface is black and I can't access SSH, dockers or the webGUI. It's all inaccessible. Can't even ping the Unraid IP. I have enabled the syslog server but, I don't get anything useful. The last errors I got was at 20:03, but I used the system well past this — till around 22:40. <5>Aug 24 19:48:06 nova kernel: XFS (sdj1): Unmounting Filesystem 72afde36-ad87-4e2a-bd83-259fbdbfbc06 <13>Aug 24 19:48:06 nova unassigned.devices: Successfully unmounted '/dev/sdj1' <13>Aug 24 19:48:58 nova monitor_nchan: Stop running nchan processes <14>Aug 24 20:03:07 nova emhttpd: spinning down /dev/sdj <14>Aug 24 20:03:15 nova emhttpd: sdspin /dev/sdj down: 1I exported a diagnostics but it only includes logs from the last boot, not last night. What can I do?

I am wondering the same.

Indeed. Sorry. But now that I have your attention, do you happen to know how one can apply a custom theme?

Turns out I just had to reset the filter in Syslog Watcher

Update: I am having mounting issues again. Not only that but for the first time ever the entire server completely freezed with the CPU stuck at 99 degrees C. Nothing worked. Docker containers, SSH, GUI everything was unresponsive. I shut the server down through KVM and restarted. The first time I restarted the GUI was unresponsive and would not let me see the drives. It was basically just loading forever. The second time I restarted I got the good old `Unmountable: wrong file system` error again. Everything ran smooth for a two weeks. I had an uptime of 13 days, but now I have the same issue again. I kind of expected that this could happen since the Intel stability issues usually increase over time, but I really hoped for the best. At this point I am fairly certain that the CPU is the culprit since everything was working super smooth for the first days after I switched out the CPU. Although, in theory it might be caused by bad MB or PSU. So now what? Should I RMA again? I mean if this continues I am tempted to just ask for a different CPU or get my money back. nova-diagnostics-20250722-1256.zip

Hmm... I get messages in Wireshark if I filter udp.port == 514

Since Unraid 7.1.4 I am not able to get syslog messages when I have enabled the syslog server. It does not appear to even attempt to send any messages, even when it should. Not sure why. Here is a screenshot from Syslog Watcher which I have used before without issues.

I would really like to use this plugin but I have no idea if it still works. Is it maintained still? Where is the repo. Where is the "how to use it" guide?...

They are in XFS. I tried that plugin once but I could not find any documentation on how to use it so I gave up. Would love to know how I could use it though. CRC errors doesn't count?

Sorry, I am a bit paranoid about getting corrupt data and I am new to Unraid. Should I finish the non-correcting parity check? Or just go straight for the correcting one? How do I know if my data is corrupted?

Hi all, I just ran a non-correcting parity check on my Unraid system and got 26 sync errors. Here's the context: I recently replaced the CPU in the system. There have been a few hard reboots (unclean shutdowns) in the last few weeks. I hadn't run a parity check for several months before this. This check was non-correcting, so no changes were made. All drives have passed SMART tests with no reallocated or pending sectors. I did notice a few drives have ICRC errors in the SMART logs, which I understand may point to transient or cable-related issues. So far, the system seems stable, and no data loss is evident to my knowledge. I'm trying to determine the best path forward. My questions: Is it reasonable to assume these sync errors were caused by the hard reboots or the CPU replacement? Would you recommend running a correcting parity check now? Should I be concerned about the ICRC errors, or is this something to just monitor unless it increases? Any advice would be appreciated!

Got a replacement CPU and everything works like a dream again. I am so happy! 🥳

Yes, that or the motherboard. I have just sent the CPU as an RMA. Let's see how it goes.

Here is what I did: Made a new Unraid flash disk on a Kingston DataTraveler 50 USB 3.1 Gen 1. Connected the flash disk to a USB 2.0 port on the MB. Took out all NVMe disks and GPU. Connected devices are now: IPMI card, flash disk, 6 x 3,5" drives and one 2,5", LAN, power. Loaded Optimized Defaults in the BIOS (although I had to change SlimSAS to SATA mode). Started Unraid and signed up for a 30 day trial. Setup an array with the exact same drive assignment, checked "Parity is already valid" and entered the passphrase. Started array. Array started fine around Jun 16 00:56 (clock not adjusted). I get these errors again: Jun 16 00:58:20 Tower kernel: SQUASHFS error: xz decompression failed, data probably corrupt Jun 16 00:58:20 Tower kernel: SQUASHFS error: Failed to read block 0x1c2a290: -5 Jun 16 00:58:20 Tower kernel: SQUASHFS error: xz decompression failed, data probably corrupt Jun 16 00:58:20 Tower kernel: SQUASHFS error: Failed to read block 0x177274c: -5At Jun 16 01:00:07 I start the array again. Loaded OK. No issues. At Jun 16 01:01:47 I start the array again. Loaded OK. No issues. At Jun 16 01:03:47 I start the array again. Loaded OK. No issues. At Jun 16 01:04:59 I start the array again. This time I get the Unmountable: wrong file system on Disk 1. At Jun 16 01:07:57 I start the array again. This time I get the Unmountable: wrong file system on Disk 3. Summary: Even after making a fourth flash drive (this time a different brand) I still get the xz decompression failed, data probably corrupt error. Disks in the array continues to fail mounting when starting the array although it works sometimes. Hypothesis: Bad CPU. Bad motherboard. Bad Unraid. It would be very helpful to have someone from Lime take a look at the logs before I do an RMA on the motherboard and CPU. Note: The Intel 13th and 14th Gen CPU Stability "Vmin Bug/voltage-shift bug"is mostly visible during gaming and heavy load, but this guy shows evidence that decompression can trigger the instability issue. Perhaps this can explain why we get a SQUASHFS error: xz decompression failed.

I ran the server on one RAM module at a time to see if the problem was related to bad RAM sticks. I switched out one after the other, trying to startup the array. All four RAM modules produced the same problem. This leads me to believe the problem is either the motherboard or the CPU, but I don't have any direct evidence. I am fairly new to Unraid and Linux so I will need some help describing the underlying issue if I am going to do an RMA on the CPU or motherboard. Basically I have no clue what is going on under the hood. Regarding bad flash drive I have tried three of them now and I always get the SQUASHFS error: xz decompression failed error. Although I should say all of the flash drives are the same type (Samsung FIT Plus 3.1).

Thank you so much for taking the time to help out @JorgeB I know you don't have to spend your time helping with this, so I really appreciate it! 🙏 This has been driving me nuts. Well it seems like the drive is detected, it just won't mount: Jun 14 08:44:57 Tower emhttpd: mounting /mnt/disk2 Jun 14 08:44:57 Tower emhttpd: shcmd (417): mkdir -m 0666 -p /mnt/disk2 Jun 14 08:44:57 Tower emhttpd: /sbin/blkid /dev/md2p1 2>&1 Jun 14 08:44:57 Tower emhttpd: /dev/md2p1: UUID="bfbc647c-28b8-4ea0-bf42-2da9a2153d85" TYPE="crypto_LUKS" Jun 14 08:44:57 Tower emhttpd: shcmd (418): rmdir /mnt/disk2 Jun 14 08:44:57 Tower emhttpd: disk2: mount error: wrong file system Jun 14 08:44:57 Tower emhttpd: mounting /mnt/disk3It is tempting to assume this is hardware related. If so it can be a bad motherboard, CPU or RAM. Not sure where to begin. I might try a process of elimination by taking out RAM sticks. I did a memtest which came out good. I used the Intel Processor Diagnostic Tool to check the CPU for errors. No issues were found. I have tried a brand new flash drive with Unraid 7.1.3 with no configs or any changes. I don't want to sound counterproductive but, to be honest I have had nothing but problems since I started using Unraid a year ago. Since I am new to Linux and Unraid I don't know what to expect. Some hours ago I had some problems where a docker app (plex) would not shutdown and the server would not shut down. Feel free to have a look at the logs:

I have updated to 7.1.3 and I still have problems with disks not mounting. Why are you so certain this is related to encryption?

Description: I have a problem where disks fail to mount and instead displays as Unmountable: wrong file system. This happens whenever I try to start the array. I can start and stop the array several times and each time a different drive/s will fail to mount until eventually, if I am lucky, all drives will mount successfully, but it might take me 10 times. Restarting the server has no significant effect. This can happen to any of the disk; array disks no matter the file system or type of disk; pool SSDs, cache SSD... I was advised to update to Unraid 7.1.3 because of the "Encryption passphrase sometimes incorrect"-fix. So I did update. In fact I just made a completely new flash drive with 7.1.3- This did not make a difference unfortunately. This problem has been bugging me since I started using Unraid and I have tried to remove HBA card, remove a disk that gave me errors, switched out cables, I have made several new flash keys, switch position on the drives and so on. Steps to Reproduce: Make a new USB flash key with Unraid 7.1.3 Connect the flash drive and start the server. Enter the passphrase for the encrypted drives. Start the array. Random disks will fail to mount. Stop the array. Start the array. Some other random disks will fail to mount. Continue this process until eventually all drives will mount. Environment: OS: Unraid 7.1.3 Unraid USB Creator: 1.0.1 Hardware: CPU: Intel Core i9-14900K RAM: 128 GB (Kingston 4x32GB ECC Registered DDR5 4800 PC5 38400) Motherboard: ASUS PRO WS W680-ACE IPMI GPU: Intel Arc B580 Unraid USB flash disk: Samsung FIT Plus 3.1 PSU: Corsair RM850i Additional Context: All my drives are encrypted. I do have some SMART errors (UDMA CRC errors on some drives) but I am fairly certain this problem occurred before I had any such errors.

Now that Unraid supports wireless NICs it would be nice to know which one is tested successfully. At least which vendor or driver should work.

I really need to find out why mounting fails so frequently. The disks are completely random, almost never the same. I even tried to remove some of them but the problem remained. By the way, my encryption password is approximately 40 characters long. Is this a problem?