I've been experiencing Unraid system crashes for the past 2 weeks and cannot seem to find the cause. 

I've eliminated docker and any of the docker images by stopping docker and its services and still the inevitable crash occur within hours (sometimes even minutes) for rebooting the system. 

I'm running Unraid  6.12.13 on Aoostar R7 with 2 x 16GB HDD in the array and no parity or cache drives. 

I've bough 2 x 20TB Ironwolf HDDs and a Terramaster D8 Hybrid to use in JBOD mode with the idea to use 1 x 20TB HDD as parity drive and I've enabled a 2TB nmve ssd as cache drive. The crashes started before adding the Terramaster D8 or additional drives.

I did perform an OS update prior to the crashes.

I'm attaching 2 diagnostic files (most recent and one for more than a week ago (if it helps). 

I suspect that it might not be very helpful as both diagnostic files we created post crash i.e. forced reboot...

I'm desperate for any advice on moving forward towards solving the problem. 

PS: I initially booted Unraid from a Proxmox VM. I booted Proxmox from a the 2GB NVME drive that I'm now using as my cache drive since I didn't use Proxmox for anything other than booting the system and then booted into Unraid. Both network and drive access were configured as "native pass-through" from Proxmox to Unraid. I simply bypassed Proxmox by booting directly from USB and when all seemed to be working well for a few days and no apparent issues I allocated the NVME drive to be used as cache in Unraid.     

I'll appreciate if you can have a look at the dianostic files and let me know if you see anything "suspect" but ANY advice on how to resolve this will be much appreciated. 

Regards

Andre 

unraid-diagnostics-20241006-1417.zip unraid-diagnostics-20240928-2055.zip

Enable the syslog server and post that after a crash, also make sure this has been taken care off:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

Thank you.
1. I've checked for Ryzen issues but I've got Ryzen 5700u which doesn't seem to be affected but I've diasbled c-states all the same.
2. I've also enabled Syslog Server

I will monitor and revert if problem persists.

 

I've had 2 more crashes since - find attached the syslogs.

I've also noticed parity disk heat warning at 51 degrees Celsius - hot but not critical I think (on a 20TB Ironwolf)?  

syslog syslog-previous

Unfortunately there's nothing relevant logged, this can also be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the docker containers.

Thank you - I'll try that next.
Any chance of network issues resulting in a crash?  

 

It's possible

I can confirm it's a hardware fault. 
It crashed several times in safe mode with NOTHING running.

I then installed the 1TB NVME drive that came with the Aoostar R7 and booted Windows 11. Within minutes the mini-PC crashed

This happened now 3 times within an hour - so I 100% convinced it's hardware - even crashed without login into Windows. 

I'll get in-touch with Aoostar directly wrt options.

Thank you for your support and prompt response @JorgeB  

 

Edited October 21, 20241 yr by ZAT1TAN
NOT Solved....

Nope - Windows crashing doesn't seem hardware failure related.... 
I've run numerous diagnostic tests for days a few days and all passed without issues.

I then went back to Unraid and decided to try the new V7 Beta 3

Did the upgrade yesterday afternoon and everything seemed fine at first but then I realised issues with samba mount and then noticed that when I use the terminal to access the /mnt directory it would freeze the terminal... same happened when using Midnight Commander ... I left the system running to see if I can get the parity drive going (which was around 30% late last night)

This morning I looked at the syslog file and saw that there seemed to be a kernel panic around 5:47 this morning and all I could do was to restart the system....

I managed to save a copy of the syslog file - see attached.

 

At this time I'm starting to suspect that my problem (system freeze) might be 1 of 3 things:
1. Network related (I read something about Macvlan issues)

2. Samba share related (saw some mount issues/errors)

3. ZFS related (as I also see ZFS errors) 

 

I'll appreciate ANY advice / feedback looking at the attached syslog or syslog-previus files.     

Regards

Andre  

syslog-previous syslog

macvlan should no longer be an issue with current releases, there are zfs crashes, which could be a bad filesystem, or a hardware issue.

What is causing this "blocking / disabled" and is it normal? 

Oct 21 13:41:01 UNRAID kernel: eth0: renamed from veth9c1de7f
Oct 21 13:41:01 UNRAID kernel: br-74e9dea89883: port 1(veth913a8c3) entered blocking state
Oct 21 13:41:01 UNRAID kernel: br-74e9dea89883: port 1(veth913a8c3) entered forwarding state
Oct 21 13:41:02 UNRAID rc.docker: container_add_route audiobookshelf
Oct 21 13:41:02 UNRAID rc.docker: audiobookshelf: started successfully!
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 2(veth6348eb6) entered blocking state
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 2(veth6348eb6) entered disabled state
Oct 21 13:41:02 UNRAID kernel: veth6348eb6: entered allmulticast mode
Oct 21 13:41:02 UNRAID kernel: veth6348eb6: entered promiscuous mode
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 2(veth6348eb6) entered blocking state
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 2(veth6348eb6) entered forwarding state
Oct 21 13:41:02 UNRAID dnsmasq[18820]: started, version 2.90 cachesize 150
Oct 21 13:41:02 UNRAID dnsmasq[18820]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile
Oct 21 13:41:02 UNRAID dnsmasq-dhcp[18820]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Oct 21 13:41:02 UNRAID dnsmasq-dhcp[18820]: DHCP, sockets bound exclusively to interface virbr0
Oct 21 13:41:02 UNRAID dnsmasq[18820]: reading /etc/resolv.conf
Oct 21 13:41:02 UNRAID dnsmasq[18820]: using nameserver 192.168.1.1#53
Oct 21 13:41:02 UNRAID dnsmasq[18820]: using nameserver 1.1.1.1#53
Oct 21 13:41:02 UNRAID dnsmasq[18820]: using nameserver 8.8.8.8#53
Oct 21 13:41:02 UNRAID dnsmasq[18820]: read /etc/hosts - 3 names
Oct 21 13:41:02 UNRAID dnsmasq[18820]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names
Oct 21 13:41:02 UNRAID dnsmasq-dhcp[18820]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Oct 21 13:41:02 UNRAID kernel: eth0: renamed from veth9aa1e70
Oct 21 13:41:02 UNRAID rc.docker: container_add_route binhex-lidarr
Oct 21 13:41:02 UNRAID rc.docker: binhex-lidarr: started successfully!
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered blocking state
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered disabled state
Oct 21 13:41:02 UNRAID kernel: veth6c7c404: entered allmulticast mode
Oct 21 13:41:02 UNRAID kernel: veth6c7c404: entered promiscuous mode
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered blocking state
Oct 21 13:41:02 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered forwarding state
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered disabled state
Oct 21 13:41:03 UNRAID kernel: eth0: renamed from veth01e1fb2
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered blocking state
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 3(veth6c7c404) entered forwarding state
Oct 21 13:41:03 UNRAID rc.docker: container_add_route binhex-readarr
Oct 21 13:41:03 UNRAID rc.docker: binhex-readarr: started successfully!
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 4(veth7c09f57) entered blocking state
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 4(veth7c09f57) entered disabled state
Oct 21 13:41:03 UNRAID kernel: veth7c09f57: entered allmulticast mode
Oct 21 13:41:03 UNRAID kernel: veth7c09f57: entered promiscuous mode
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 4(veth7c09f57) entered blocking state
Oct 21 13:41:03 UNRAID kernel: br-74e9dea89883: port 4(veth7c09f57) entered forwarding state
Oct 21 13:41:03 UNRAID unassigned.devices: Mounting 'Auto Mount' Remote Shares...
Oct 21 13:41:03 UNRAID unassigned.devices: Using Gateway '192.168.1.1' for Remote Shares.
Oct 21 13:41:03 UNRAID unassigned.devices: Waiting 5 secs before mounting Remote Shares...
Oct 21 13:41:04 UNRAID kernel: eth0: renamed from veth4d7c29b
 

It ns quite normal when docker containers start up.   If it keeps occurring then you probably have a container continually crashing and being restarted.  In such a case looking at the uptime can help determine which one(s).