My Unraid system has started to randomly crash multiple times a week, both when untouched (with things running of course) but also when performing one specific action -- loading the web UI.

 

Diagnostics attached.

 

Graph showing frequency of crashes. Variable recovery time is due to me needing to manually start the array, as these health checks require a specific container to be running (which requires the array to be started).
 

 

System specs: https://uk.pcpartpicker.com/user/oatie/saved/#view=RfbCD3

...and because they're not available to select on PC Part Picker:

    Motherboard: Asus Pro WS W680-ACE IPMI

    HBA card: LSI 9201-8i

 

What I've tried:

• Rolling back to older Unraid versions: 6.12.13 and 6.12.11 still yield the same behaviour. Has the 6.12.4 release been pulled/withdrawn?
• Switching my Docker network from macvlan to ipvlan
• Running the Plex container without passing through the iGPU device, so that the iGPU remains idle (not used by any other workload on the server)
• 4 successful passes of Memtest86 that is bundled with Unraid. This is after running 2x 4 passes of the then-latest version of Memtest, over 2 days, when I first build the system and before I moved Unraid into it. I'm pretty sure the ECC DDR5 RAM is okay.
• Sending syslog to my secondary Unraid server (attached) -- I don't see anything in there relating to the crashes. The system crashes so hard and fast that I'm guessing there's no time to log anything
• I've tinkered with a bunch of BIOS settings that I thought might be relevant, but with this system once being stable, I don't think that's the issue

 

Some history/context, in case it helps

1. I built this system when I re-platformed my 2012 Unraid server, so I've been through and seen a lot. I've never experienced such severe issues in >12 years of running Unraid, since v5!
2. It came to storage expansion time, so I ordered the LSI 9201-8i to replace two SATA cards I had (which were faultless in both the original 2012 and new 2023 system, but the community persuaded me SATA cards are evil and I followed)
3. I had some challenges and quirks to get through to have the HBA card fully working, ultimately resulting in needing to enable Compatibility Support Module (CSM) on my very modern motherboard for it to detect the ancient HBA card.
4. Everything seemed to be working fine and was stable, except it took me weeks to realise my media transcoding was no longer using the iGPU and my CPU was regularly hitting 100% and 90°C
5. After a lot of experimentation, I realised the iGPU would be disabled or not detected during boot when CSM was enabled. However, now that my array was spread across the HBA, I couldn't just disable CSM to get my iGPU back because I won't be able to start the array
6. I left it in this state for many months, giving up my iGPU in favour of a running Unraid system with CPU transcoding  Meanwhile, the whole Intel 13th/14th gen fiasco was happening and I was keeping an eye on motherboard BIOS updates for the microcode fix
7. I recently (Christmas 2024) stumbled upon a random forum post where someone suggested removing the HBA card's BIOS to improve boot up speed and not require CSM. I was super excited, so worked quickly to get the HBA's BIOS removed and disable CSM
8. I was finally booting into Unraid with all my drives, able to start the array and take advantage of the CPU's iGPU - success!
9. I took the opportunity to upgrade Unraid from 6.12.4 to 6.12.14
10. I walked away and it was a day later when I woke up to see my Unraid server had rebooted and was waiting for me to manually start the array
11. Since then, I've been scratching my head, reading many Unraid forum and Reddit posts, whilst trying the things I mentioned above. 

 

The only thing I've not tried yet and can't see an easy approach for is to roll back to 6.12.4 which I know for sure the system was stable with prior to removing the HBA's BIOS. I had not previously run 6.12.11, 6.12.13 or 6.12.14 until I starting booting with HBA and iGPU (CSM disabled). Sometimes opening the web UI only to hear the faint "tick" from the server and see the page not load completely (because of a crash) makes me think this is an Unraid/Linux kernel issue -- but that doesn't explain why I once had stability with months of uptime.

 

Could removing the HBA's BIOS really have something to do with it?

 

A 14700K should arrive today so I can confirm my 13700K isn't a victim of the Intel microcode issue/burnout. I have an open RMA for the 13700K, on the assumption that it is already toast, but I've been told they will run tests on it to first confirm an issue before sending me a replacement. However, if the issue is Unraid, they won't find an issue with the CPU and will send it back to me. I am hesitant to send it off if I experience the same issue with the new 14700K I have separately bought.

 

I'm running out of options and getting desperate

 

Thanks in advance. Sorry for the mega post.

unraid-diagnostics-20250111-1701.zip syslog-unraid.log

Edited January 11, 20251 yr by Omid

14700K installed...

...and it crashed again, so my CPU isn't the problem.

So the issue is either with my motherboard, PSU or Unraid?

I'm quite suspicious of the fact that opening the Unraid web GUI sometimes triggers the crash.

 

I should try rolling back 6.12.4 (last known stable version) before replacing any other hardware.

Edited January 12, 20251 yr by Omid

Okay, I'm back on 6.12.4...

 

Still on the 14700k but will switch back to my 13700k in near future.

I don't want to speak too soon, but I think the crashes started when I upgraded Unraid 😥

It's so far been stable since I downgraded to 6.12.4

 

Lesson learned: don't remove HBA BIOS, disable CSM in motherboard and upgrade Unraid on the same day, especially when you're model of CPU is also known to have problems.

 

Edited January 13, 20251 yr by Omid

I believe to have enough data to conclude that upgrading to Unraid 6.12.10+ is the cause of my crashes.

 

I know that 6.12.11, 6.12.13 and 6.12.14 all cause crashes.

I know that 6.12.4 and 6.12.5 are stable.

I don't know about the versions between 6.12.5 and 6.12.11

 

 

What to do from here? I can't afford to upgrade to a bad version to troubleshoot further.

Do I just never upgrade?

 

 

Edited January 16, 20251 yr by Omid

Try 7.0.0

Unraid is for me also unreliable since 6.10.

 

Different RAM, CPU and MB, new stick, different locations - its just bad - a nas which cant even stay online. Most importantly for me it freezes or crashes without any usefull log. (like just ending, without anything)

 

Do you use fritzbox?

Edited January 17, 20251 yr by NewDisplayName

Thanks @JorgeB Just blindly upgrading to 7.0.0 has crossed my mind, but without a RCA I feel like I'm shooting in the dark and just being hopeful 😅.

To be honest, I'm already quite exhausted from what I just went through the last 3 weeks and can't really afford to upgrade and then [inevitably] downgrade all the way back down to 6.12.5; reading through all the rollback notes of each version, juggling plugin versions and tweaking settings, etc. I'm going to hold on to 6.12 for now and see how things evolve in the coming months.

 

I'm extremely surprised this issue isn't discussed more in the community, as I suspect the cause to be something pretty standard/common.

 

@NewDisplayName Nope, not using fritzbox (and hadn't heard of it until now). I have 23 apps (Docker) and just 1 Ubuntu VM I use as a dev box for (VS Code server).

Most intensive regular task performed by the server is probably Plex transcoding my home/family videos (when requested by one of the ~10 family members it's shared with).

 

And yeah, the really sudden crash means syslog and diagnostics don't really help. Even my remote syslog just cuts off abruptly   sad times

 

Please could you share your system specs? Or review mine in the OP and let me know what hardware we share?

Edited January 17, 20251 yr by Omid

i had these bugs starting with 6.10 and i6700k

 

current 1

M/B:ASRock Z390 Taichi Ultimate

BIOS:American Megatrends Inc. Version P4.30 Dated 11/26/2019

CPU:Intel® Core™ i9-9900K CPU @ 3.60GHz

HVM:Enabled

IOMMU:Enabled

Cache:L1 Cache: 512 KiB, L2 Cache: 2 MiB, L3 Cache: 16 MiB

Memory:64 GiB DDR4 (max. installable capacity 64 GiB)

 

 

 

So i dont think its special to a specific hardware. 

if you make a backup of your stick youre very fast back to the old version... btw

 

and all of my hardware is 1. fresh or 2. fresh from a working system.

Edited January 19, 20251 yr by NewDisplayName