feins

Hi @JorgeB I recycle the Server and the NVME are back online and i got this error what should i do now? pool: cache state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: resilvered 127G in 00:03:09 with 0 errors on Thu Aug 20 15:31:05 2026 config: NAME STATE READ WRITE CKSUM cache ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 /dev/nvme0n1p1 ONLINE 0 0 0 /dev/nvme2n1p1 ONLINE 0 0 4 /dev/nvme1n1p1 ONLINE 0 0 0 errors: No known data errors rnmy-diagnostics-20260820-2343.zip

Dear all gurus, Today i just notice one of my NVME drive on my ZFS Cache Pool indicated Removed. What should i do as i don't dare to touch it or did any thing to it until i got the instruction to do so. pool: cache state: DEGRADED status: One or more devices have been removed. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using zpool online' or replace the device with 'zpool replace'. scan: scrub repaired 0B in 00:02:23 with 0 errors on Sat Aug 15 03:02:24 2026 config: NAME STATE READ WRITE CKSUM cache DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 /dev/nvme0n1p1 ONLINE 0 0 0 /dev/nvme2n1p1 REMOVED 0 0 0 /dev/nvme1n1p1 ONLINE 0 0 0 errors: No known data errors rnmy-diagnostics-20260819-2341.zip

I never had any problem since day 1 till now.

Oooh. any idea which plugin? because i dont have much plugin install.

The diags are during the problem occur i ssh into the system and run the diagnostics command.

Hi this two day i encounter this issue suddenly i cant login after the server are up for a while and i need to SSH into the server and do a reboot and everything are working again. Attach are the diagnostic file and hope that the issue is been log. ts-p500-diagnostics-20260115-1531.zip

Oooh i dont have VMs just Docker and Plugins. and this attach screen shot is the only Docker that being used. Did it indicate what's been utilized more RAM than normal? Ive reboot the server in normal mode since yesterday and seems stable till now. The logs now looks pretty quiets aswell. On the Disk3 filesystem corruption can it be recover? syslog-10.10.2.10.log

Hi @JorgeB its been a while, yesterday my server crashed and as usual a hard reboot and the server back online. I got some error on the logs. and i attach here together with the diagnostic. My last message received from my Telegram at 4.40am and i found the server crashed and cannot login or SSH at 12:40pm where i initial a hard reboot. Hope the logs find something on your site. ts-p500-diagnostics-20251229-1822.zip syslog-10.10.2.10.log

It's been up for over a day on Plugin only. there no additional errors till now. Seems everything are ok by now. ts-p500-diagnostics-20250808-1803.zip

Seems Plugins wont be available in safe mode.

Hi @JorgeB Thanks for your info. So by now how should i proceed? The server already on safe mode and docker only for over 4days without any issue and assume its stable till now. Should i reboot it back to normal mode and enable all plugin?

@JorgeB The server are running on Save Mode and Docker only for 4 days and everything stable. But i notice that's and Error on the log. Aug 4 20:04:16 TS-P500 kernel: ata1.00: exception Emask 0x10 SAct 0x8 SErr 0x400100 action 0x6 frozen Aug 4 20:04:16 TS-P500 kernel: ata1.00: irq_stat 0x08000000, interface fatal error Aug 4 20:04:16 TS-P500 kernel: ata1: SError: { UnrecovData Handshk } Aug 4 20:04:16 TS-P500 kernel: ata1.00: failed command: WRITE FPDMA QUEUED Aug 4 20:04:16 TS-P500 kernel: ata1.00: cmd 61/10:18:c0:b1:68/03:00:2d:01:00/40 tag 3 ncq dma 401408 out Aug 4 20:04:16 TS-P500 kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error) Aug 4 20:04:16 TS-P500 kernel: ata1.00: status: { DRDY } Aug 4 20:04:16 TS-P500 kernel: ata1: hard resetting link Aug 4 20:04:17 TS-P500 kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Aug 4 20:04:17 TS-P500 kernel: ata1.00: configured for UDMA/100 Aug 4 20:04:17 TS-P500 kernel: ata1: EH complete Aug 4 20:05:28 TS-P500 monitor_nchan: Stop running nchan processes

Ok Array started. Lets monitor for sometime. If the Array stable later on I'll enable Docker and keep monitor if it Crash again. @JorgeB I've notice some of this error and any of them are critical? So far the Array and Docker are enable as request, the server are still very responsive and seems no issue for nearly a day. Aug 1 19:05:17 TS-P500 kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks Aug 1 19:05:17 TS-P500 kernel: ACPI: Early table checksum verification disabled Aug 1 19:05:17 TS-P500 kernel: floppy0: no floppy controllers found Aug 1 19:05:17 TS-P500 kernel: i915 0000:00:02.0: [drm] [ENCODER:240:DDI A/PHY A] failed to retrieve link info, disabling eDP Aug 1 19:05:34 TS-P500 mcelog: failed to prefill DIMM database from DMI data Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 8: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 8: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 8: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 8: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd/udp: socket 1: sendto() error: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ] Aug 1 19:05:17 TS-P500 rsyslogd: omfwd: socket 1: error 101 sending via udp: Network is unreachable [v8.2102.0 try https://www.rsyslog.com/e/2354 ]

Hi @JorgeB Ive force shutdown and now boot into save mode with HDD auto start Disable, Docker auto Start Disable, I dont use VM so its Disable. What should i do next? ts-p500-diagnostics-20250801-1814.zip

Hi @JorgeB Sorry for the late update because im having issue with the Server and seems the server keep stop responding. Lastly ive manage to force reboot and just completed finish the Parity check with 0 Error, and i've complete Preclear the 8TB Drive without any issue. But when i try to shutdown the server manually and it dont wana shutdown. Ive manage to download the log file but when i try to generate a Diagnostic file it just down wana complete the diagnostic. So i only attach the system log file and hope you could find something that keep the server not shutting down. ts-p500-syslog-20250801-0645.zip