Hello folks,

 

First off: it's a long post, bear with me - I appreciate it. 🧡

I'd like help on the best course of action to get to the root of crashes I've recently started having.

 

Diagnostics attached to post #3

Unraid versions: v6.12.10 > v6.12.11

This is the mcelog event:

 

mcelog: failed to prefill DIMM database from DMI data
Kernel does not support page offline interface
Hardware event. This is not a software error.
MCE 0
CPU 3 BANK 0 
TIME 1721651313 Mon Jul 22 14:28:33 2024
MCG status:
MCi status:
Uncorrected error
Error enabled
Processor context corrupt
MCA: Internal parity error
STATUS b200000000030005 MCGSTATUS 0
MCGCAP c0e APICID 6 SOCKETID 0 
MICROCODE f4
CPUID Vendor Intel Family 6 Model 158 Step 12

 

Right now the system is doing a MemTest that seems successful so far (58%).

 

My system ran stable for several months before this happened, however, I did some hardware changes and upgrades all at once (my mistake).

1. changed RTX2080 to RTX3070Ti (because it's thinner and doesn't block PCIe #2)

2. moved HBA (12G SAS3008) from PCIe #3 (x4) to PCIe #2 (x8)

3. disabled memory XMP (it was on the entire time. Never noticed)

--- started at this midway point ---

4. changed 5G network card (1x #3) to 10G network card (PCIe #3)

5. changed 2x500GB SSDs (mirror) to 6x500GB SSDs (two-pair raidz1)

 

Essentially I wanted 10G network and more (+faster) cache to keep up. The GPU change was required to physically free the PCI slot to allow adding the 10G card that requires PCIe x4; the 5G ran on 1x

 

Things I noticed prior to the the issue

(Pre Midway Point)

In PhotoPrism I did a Library Sync/Import to get the pics. The system 'froze' during the process and I attempted `powerdown` via console. Upon executing the command the attached screen came alive but it couldn't shutdown, as it couldn't unmount the `/mnt/user` directory.

I couldn't unmount it manually with online guides so after it seemed there was no file in transfer, I forced the shutdown.

Upon restarting I let it run a parity check which it finished completely.

 

(Post Midway Point)

Added 4 SSDs and replaced NIC.

in preparation to replace the cache drives, I ensured Docker+VMs aren't enabled; changed cache-using shares to 'move to array' and invoked the mover... However, this left me with 140GB of data on the cache drive anyway. Folders I set up to 'move to array' were partially still present on it.

As nothing was running, I moved the files manually off cache to a subfolder on the NVMe (`mv cache\{,.[^.]}* hypercache\cachebak\`).

 

Once it completed, I erased the cache drives, changed size 2 > 6, set it to `ZFS Encrypted` fs `raidz1 - 2 pairs` and moved the cache files back from cachebak.

 

-----

 

Now, all in all that seems fine, despite the hiccup I had with the mover. But I've had plenty crashes since, within seconds after booting to unraid to  a few minutes if I'm lucky.

It doesn't matter if parity check is canceled or not. It also doesn't matter if Docker is running or not.

For that reason I assume the GPU isn't an issue, as it's effectively not being used by anything at all.

The HBA I just moved to another slot so that should also be fine.

 

My hunch is that maybe something got corrupted,

my system LIKES xmp and wants it turned back on,

something's not right with the cache/mover

Edited July 23, 20241 yr by BentoFox

Things tried:

- updated outdated plugins (unraid did an update recently)

- skipped parity check (would crash in the middle anyway) to see if it's related to disk activity

- started with dockers disabled

- checked cables

- fresh install of Unraid (v6.12.11)

- ran memtest (success, 2 passes)

 

- start in safe mode

- pull diagnostics (safe mode)

- run parity in safe mode, if it makes it through I'll wipe plugins and...

- start in normal mode (and reinstall plugins one by one with a few hours inbetween)

- pull diagnostics

Sadly it crashes as soon as unraid boots, even in safe mode.

 

Next up:

- enable XMP again (it doesn't make sense to enable it, but it was enabled for months without issue...)

- dreaded 1-by-1 rollback for which I need a stable system to make the changes

Edited July 22, 20241 yr by BentoFox

I've enabled XMP again and can now get into Safe Mode (maybe regular, too, I just haven't tried) and am running a parity check.

Cya in 2 days 💀 If it makes it through that, I'll try booting into normal mode and leave the array online for a day (no docker / vm). If that works I'll give dockers a go.

 

I'd be very happy if, albeit somewhat unintuitively and not recommended, enabling XMP would fix the crashes.

bentobox-diagnostics-20240722-2340.zip

The parity check has run through completely now and it hasn't crashed since then (in safe-mode with XMP on).

I now started all regular docker containers I run and it's been stable for half an hour at least.

 

As previously it'd crash within minutes of booting even with no dockers running and the array has now been running for 2 days,

I'd conclude that the issue was, indeed, that XMP was disabled as this is often recommended to do.

More precisely I assume the issue is that the 'AUTO' settings which appear to reduce the RAM clock speed may have attempted to boost RAM timings, whereas XMP would use an actually working preset the RAM is rated for.

 

So - if you recently disabled XMP and you now crash, try turning it on again if you ran stable with it before.

Must admit that this is the first time I have seen a suggestion that enabling XMP increases stability.

I totally get where you come from. Though it does seem to be completely down to that. The system's been stable since then.

Also, mind I do not think it's 'enabling XMP' itself that fixes the issue, but rather that the RAM is more stable at its rated XMP speed than effectively underclocked on auto.

In general I wouldn't recommend enabling XMP in a server environment - but if you had it on before and crash after disabling it ... consider enabling it 🦊