Hi all, hoping to get some help diagnosing an unexpected reboot on my server.

System:

- Unraid 7.2.3, kernel 6.12.54

- ASUS ROG MAXIMUS XI HERO (Z390)

- Intel i9-9900K

- Three ZFS pools: fleabottom (8x14TB RAIDZ1), greensight (10x12TB RAIDZ2), maester (2x2TB NVMe mirror)

What I was doing:

Running a large data transfer (~58TB total) from a NAS Pro 8 over 10GbE using parallel rsync workers in screen sessions. The transfer was writing directly to the fleabottom ZFS pool, bypassing shfs. Transfer was running well at sustained 600-700 MB/s for several hours.

What happened:

Approximately 5-6 hours into the transfer, the server rebooted unexpectedly. No manual intervention. After reboot all three ZFS pools came back ONLINE with no errors. The rsync workers were lost and the transfer had to be restarted, but rsync resumed from where it left off.

What I've checked:

- All pools show ONLINE with zero READ/WRITE/CKSUM errors after reboot

- No kernel panic message visible in syslog (log cuts off before the crash as expected)

- System is stable after reboot

Questions:

1. What's the best way to capture pre-crash data on Unraid 7 if it happens again?

2. Has anyone seen unexpected reboots during sustained heavy ZFS writes on 7.2.3?

3. Any recommended diagnostic steps for tracking down the root cause?

Diagnostics attached.

Thanks in advance.

kingslanding-diagnostics-20260223-2148.zip

Server rebooting by itself is almost always a hardware issue; RAM or PSU would be good first candidates.

Questions:

3 hours ago, wesman said:

1. What's the best way to capture pre-crash data on Unraid 7 if it happens again?

The syslog in the diagnostics is the RAM version that starts afresh every time the system is booted. You should enable the syslog server (probably with the option to Mirror to Flash set) to get a syslog that survives a reboot so we can see what leads up to the problem. The mirror to flash option is the easiest to set up (and if used the file is then automatically included in any diagnostics), but if you are worried about excessive wear on the flash drive you can put your server's address into the remote server field and log to a share instead.

It appears your server experienced a "hard" reboot during heavy ZFS write operations.

The potential causes:

• RAM Instability (XMP/DOCP): Sustained 10GbE transfers at 600-700 MB/s put extreme pressure on the memory subsystem. Your syslog shows 4/4 memory slots populated. On Z390 motherboards with an i9-9900K, running four sticks of RAM with XMP enabled can be unstable. Try disabling XMP or dropping the frequency slightly.

• ZFS ARC and Memory Pressure: ZFS uses a significant portion of RAM for its Adaptive Replacement Cache (ARC). Under heavy parallel rsync load, if the ARC doesn't release memory fast enough, the Linux kernel's Out-Of-Memory (OOM) killer might trigger, occasionally leading to a system hang or reboot if a critical process is affected.

• Thermal Protection: While you suspected HDD thermal throttling, that usually leads to slow performance rather than a reboot. However, a VRM or CPU overheat on the ASUS ROG MAXIMUS XI HERO during long-duration high-throughput tasks can trigger a hardware-level thermal shutdown.

Potential fixes:

1. Check for "C-States" Issues: Some Intel/ASUS combinations are sensitive to deep sleep states. In your BIOS, try setting Typical Current Idle or disabling C-States to see if stability improves during high-load transitions.

2. Limit ZFS ARC: To prevent potential OOM issues, you can limit the maximum size of the ZFS ARC by adding zfs_arc_max to your sysctl settings or boot parameters.

2 hours ago, Quinta said:

It appears your server experienced a "hard" reboot during heavy ZFS write operations.

The potential causes:

• RAM Instability (XMP/DOCP): Sustained 10GbE transfers at 600-700 MB/s put extreme pressure on the memory subsystem. Your syslog shows 4/4 memory slots populated. On Z390 motherboards with an i9-9900K, running four sticks of RAM with XMP enabled can be unstable. Try disabling XMP or dropping the frequency slightly.

• ZFS ARC and Memory Pressure: ZFS uses a significant portion of RAM for its Adaptive Replacement Cache (ARC). Under heavy parallel rsync load, if the ARC doesn't release memory fast enough, the Linux kernel's Out-Of-Memory (OOM) killer might trigger, occasionally leading to a system hang or reboot if a critical process is affected.

• Thermal Protection: While you suspected HDD thermal throttling, that usually leads to slow performance rather than a reboot. However, a VRM or CPU overheat on the ASUS ROG MAXIMUS XI HERO during long-duration high-throughput tasks can trigger a hardware-level thermal shutdown.

Potential fixes:

1. Check for "C-States" Issues: Some Intel/ASUS combinations are sensitive to deep sleep states. In your BIOS, try setting Typical Current Idle or disabling C-States to see if stability improves during high-load transitions.

2. Limit ZFS ARC: To prevent potential OOM issues, you can limit the maximum size of the ZFS ARC by adding zfs_arc_max to your sysctl settings or boot parameters.

5 hours ago, itimpi said:

Questions:

The syslog in the diagnostics is the RAM version that starts afresh every time the system is booted. You should enable the syslog server (probably with the option to Mirror to Flash set) to get a syslog that survives a reboot so we can see what leads up to the problem. The mirror to flash option is the easiest to set up (and if used the file is then automatically included in any diagnostics), but if you are worried about excessive wear on the flash drive you can put your server's address into the remote server field and log to a share instead.

Captured the crash again, and this time had "Mirror to Flash" setm diagnostics attachedkingslanding-diagnostics-20260224-0723.zip

There's nothing relevant logged, and this is to be expected in a case like this, since the server just reboots by itself.

31 minutes ago, JorgeB said:

There's nothing relevant logged, and this is to be expected in a case like this, since the server just reboots by itself.

Why would it "reboot by itself" ? I mean, why would it ever do such a thing?

8 hours ago, JorgeB said:

Server rebooting by itself is almost always a hardware issue; RAM or PSU would be good first candidates.

38 minutes ago, JorgeB said:

  9 hours ago, JorgeB said:

Server rebooting by itself is almost always a hardware issue; RAM or PSU would be good first candidates.

Right, thanks..
Update for the thread — we've had two more crashes since the original post and have done a lot more digging.

New diagnostics are attached from the third crash this morning.

Mirror to Flash was enabled so we have the pre-crash syslog. We also have mcelog output now which gives us something more specific to work with:

CPU 5, Bank 0, Internal parity error, Uncorrected error, Processor context corrupt. STATUS b200000000030005.
Interwebs say Bank 0 is the CPU's internal logic rather than a DIMM directly.

Ruled out a few things. Swapped the PSU to a Dark Power Pro 12 1200W and it crashed again.

Temps are completely clean — CPU cores were sitting at 31-34°C under full transfer load with no thermal warnings anywhere in the logs. XMP is disabled and we confirmed via dmidecode that all four DIMMs are running at stock 2133 MT/s.

What's interesting is the crashes are escalating. First crash was 5-6 hours in. Second was about 2 hours in. Third was only 28 minutes in. The system is completely stable at idle — this only happens under sustained heavy load.

Running memtest86+ now and will report back with results.

Main question is: whether a Bank 0 internal parity error on the CPU points toward a failing RAM stick stressing the memory controller, or whether it's more likely the CPU itself degrading. Any guidance there would be appreciated.

kingslanding-diagnostics-20260224-1042.zip

Edited February 24Feb 24 by wesman

Start by running Memtest, or because Memtest is only definitive if it finds errors, since you have multiple RAM sticks, try using the server with just one pair, if the same try with the other one, that will basically rule out bad RAM.

seems solved, been running for a few hours now.

MEMTEST86 Identified a bad ram stick. removed and we are now moving along wit zero errors