Good morning,

Since switching to version 7 of unRaid, the server no longer responds (GUI/Docker/ect) every 8 to 12 hours.

I updated the bios and Intel ME firmware but it doesn't change anything.

I performed a memtest86 on 5 passes but the RAM is ok.

The solution was a rollback to 6.12. No more problems since.

Here is my configuration :

  - CPU : Intel I5 13500

  - RAM : One stick 32Gb DDR5 ECC

  - MB : Asus W680 Pro

  - PCIe : LSI9300-i8

  - PSU : 550W BiQuiet Gold

  - HDD : 4*1To Sata + 2*1To Sata partition

  - SSD : 2*800Go SAS Cache Docker + 2*800Go SAS Cache VM

 

Nothing appears in the logs. We see that nothing is published between the moment the server freezes and the moment I reboot it.

 

Just an exemple below (RAM-Disk run every 30min) :

Quote

Jan 30 04:00:01 Poufsouffle docker: Success: Backup of RAM-Disk created.
Jan 30 04:10:15 Poufsouffle emhttpd: read SMART /dev/sdj
Jan 30 04:10:15 Poufsouffle emhttpd: read SMART /dev/sdk
Jan 30 04:10:15 Poufsouffle emhttpd: read SMART /dev/sdd
Jan 30 07:59:04 Poufsouffle rc.rsyslogd: Syslog server daemon...  Started.
Jan 30 07:59:04 Poufsouffle cache_dirs: Arguments=-p 10 -i backup -i domains -i isos -i media -i nextcloud -i serveur_games -i syslog_server -l off
Jan 30 07:59:04 Poufsouffle cache_dirs: Max Scan Secs=10, Min Scan Secs=1

 

Edited February 2, 20251 yr by Dreizarf

Enable the syslog server and post that after a crash together with the complete diagnostics.

The syslog is already enabled and i show you in my previous comment that nothing appear.

Same issue here, literally just restarted because it was unresponsive.

This has been going on ever since I upgraded to v7 😞

unraid-diagnostics-20250203-1523.zip unraid-syslog-20250203-1523.zip

3 minutes ago, Chrison said:

Same issue here

The syslog you posted is the same syslog already in Diagnostics, so it doesn't show anything before last boot.

 

Setup syslog server

pretty sure that its already enabled, but cant check right now because I got another freeze ... 

It should also mirror to my usb stick.

Just as a sidenote, unraid runs as a VM in proxmox and since everything becomes unresponsive, I just stop end restart the VM.
I also use NFS connected drives on a physical NAS as the hard drives in Unraid, which at least up until v7 worked relatively well.

 

I tried to grab the syslog and diagnostic before the reboot, but it was so slow and laggy it wouldnt finish within a reasonable time (30 minutes or so)

And just in case:

unraid-diagnostics-20250203-1958.zip unraid-syslog-20250203-1959.zip

and another one shortly after ...

unraid-diagnostics-20250203-2027.zip unraid-syslog-20250203-0727.zip

No, those are the same too.

6 hours ago, trurl said:

syslog you posted is the same syslog already in Diagnostics

You also have to set Remote syslog server to the IP of whichever syslog server you want it sent to, such as your Unraid IP. Then you should be able to access it at the path you specified (your system share).

The actual setup of syslog.

 

Attached, you have the server syslog. To make it easier to find, I put a #CRASH tag. I also replaced the email addresses displayed in the file.

Same i put a #ROLLBACK tag when im back to 6.12

I don't have the diagnostic zip because I am currently in 6.12

I saw the lines below which seemed strange to me regardless of the version.

Quote

Jan 30 07:59:13 Poufsouffle emhttpd: shcmd (107): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1
Jan 30 07:59:13 Poufsouffle root: Specified filename /mnt//system/libvirt/libvirt.img does not exist.

 

syslog-127.0.0.1.log

Unfortunately, there's nothing relevant logged, this can also be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers. 

 

Additionally, look in the BIOS for a "Global C-States" or similar setting and disable that to retest, it's been known to be a problem with some boards, with both Intel and AMD CPUs.
 

 

Thank you for your quick response. By rolling back to version 6.12 this solved my problem. The server no longer crashed.

Can C-state pose a problem on version 7.0?

It can sometimes be kernel related, hardware that worked fine with a kernel may have an issue with a different one.

basically done that, reduced my containers down to Plex and qbittorrent and it runs a bit more stable. (still not good enough though but went from having to reboot 3-4 times a day down to once every day or two, which the missus appreciates)

Anyway, moving away from Unraid after this. v7 doesnt seem to be very stable especially in my scenario where my drives are provided via NFS as HDDs through Proxmox.

I'm also not too happy with the docker implementation in Unraid so already started pulling out all containers, just moved them back temporarily due to the instability that came upon us with v7

plain debian with mergefs will just do as fine as unraid until you guys got those instabilities sorted out  Good thing I got my NAS virtualised and not on bare metal, as this wouldve been a desaster  

Just chiming in: I have the same issue. Unraid 7 stops responding after some time. Right now it seems to be about 4 days for me.

Only thing I can do is physically keep the shutdown button pushed in and then restart it. I have enabled syslog and will post here the next time it locks up.

 

I do think this should have more attention from the developers. There are already to many cases to shove this of as anything else than version 7 related. I really do not like that being implied. Hardware does not "respond" to kernels. Some HW might not be supported, the kernel might send different instructions causing the issue: fine. But that is not a HW problem but an Unraid problem that needs to be fixed. My HW is fine, been running fine for years on 6 and in the first 2 days after upgrading to 7 my whole system froze.

I'm having the same problem. I was running on 6.12.x since I moved to Unraid in November without issues. Waited for the general release of 7 and upgraded. It started freezing and becoming unresponsive the same day. I had to manually reboot the server to get back online. I went through this about 3 times and decided to try downgrading back to 6.12.14 and the problem went away. That solved the problem and was using 6.12.14 until this past Thursday when I decided to try and upgrade again. . I did a fresh install of 7.0 and moved my config over. Roughly 24 hours later it become unresponsive. I had moved to overlay2 for the storage driver as well.

 

I booted to safe mode and re-enabled docker. I was going to downgrade so I moved back to the native storage driver with a btrfs vdisk. I decided to hold off on the downgrade right now.  It's been online for about 14 hours now. I just installed a container (plex) to see if anything happens. I have uploaded my diagnostics and syslog. I don't see anything in the syslog. It happened around 14:00 yesterday (2/14). I'll also note that when I got a chance to look at the issue, there was nothing on the monitor connected to the server and it wouldn't respond to any keys. The monitor acted like the PC was a sleep and wouldn't wake up.

 

I'm going to look at the c-states and see.

citadel-diagnostics-20250215-1327.zip syslog-citadel.log

I migrated away 90% of my docker containers and that seems to have helped. It didnt freeze ever since (mustve been about a week now?)

 

Only thing still running on unraid is qbittorrent and plex. the rest runs via SMB and 100% more stable than on unraid.

Honestly? Sounds to me like the already crappy docker implementation on unraid has finally been broken for good.

 

Lesson learned, not gonna rely on unraid any longer for hosting my containers 🙂

Edited February 15, 20251 yr by Chrison

14 hours ago, sefato2038 said:

I don't see anything in the syslog. It happened around 14:00 yesterday (2/14).

Nothing at that time, but there appears to be a container constantly restarting, check their uptimes to see if you can find the culprit.

Alright, now my CPU went to almost 100% and stayed there. It did not completely freeze yet. And the weird part: I just turned off all my docker containers.

I could not see what process was using the CPU that much, see the screenshot. I did see this error in the logs:

 

Feb 17 21:36:15 Tower kernel: cgroup: fork rejected by pids controller in /docker/3e90a24dbde517b34431e7d3a42ff057e1583e34280f14cec81c2371ca3792c9
Feb 17 21:36:15 Tower kernel: traps: .NET TP Gate[838131] general protection fault ip:148510f6650f sp:1443dccd9e60 error:0 in libc.so.6[148510f66000+155000]

9 hours ago, Mark12 said:

fork rejected

This suggests an issue with one of the containers, and could be causing the high CPU.

Some news.

 

I make some adujstement in the bios and now my uptime its 1 day and 1 hour without crash. Let see if i can go to 1 week.

 

For anyone who read this, in the bios, i remove the "Max power Saving" option and i set "APSM Control" to "Os"

Ok, so after more than a week i can tell the problem was done. No more crash.

I am just going to continue here. Since the upgrade to 7 (and now 7.1) Unraid has been in effect useless to me. Uptime varies between one and 5 days max. It is always docker that stops responding first. Sometimes I can still use the GUI, but as soon as I hit the docker tab it's end of story. The only thing I am running now is Nextcloud AIO docker, this is what I 100% need.

 

I have already turned off UASP for my external enclosure for the kernel, otherwise it would not even remain up for one day. I turned on diagnostic logging, hopefully you can help solve this persistent problem.

 

When I try to collect logs it seems to stop at the appdata part en simply does not continue. Tried the gui and via ssl: no log is made/downloaded.

 

I do see this in the Syslog:

Mar 22 05:02:31 Tower kernel: br-5e11601cd858: port 2(vethf588ddb) entered disabled state
Mar 22 05:02:31 Tower kernel: vethf588ddb (unregistering): left allmulticast mode
Mar 22 05:02:31 Tower kernel: vethf588ddb (unregistering): left promiscuous mode
Mar 22 05:02:31 Tower kernel: br-5e11601cd858: port 2(vethf588ddb) entered disabled state
Mar 22 05:02:41 Tower kernel: br-5e11601cd858: port 10(vethb5af300) entered disabled state
Mar 22 05:02:41 Tower kernel: veth74fb0b9: renamed from eth0
Mar 22 05:02:41 Tower kernel: br-5e11601cd858: port 10(vethb5af300) entered disabled state
Mar 22 05:02:41 Tower kernel: vethb5af300 (unregistering): left allmulticast mode
Mar 22 05:02:41 Tower kernel: vethb5af300 (unregistering): left promiscuous mode
Mar 22 05:02:41 Tower kernel: br-5e11601cd858: port 10(vethb5af300) entered disabled state
Mar 22 05:02:43 Tower kernel: br-5e11601cd858: port 2(veth3bc8150) entered blocking state
Mar 22 05:02:43 Tower kernel: br-5e11601cd858: port 2(veth3bc8150) entered disabled state
Mar 22 05:02:43 Tower kernel: veth3bc8150: entered allmulticast mode
Mar 22 05:02:43 Tower kernel: veth3bc8150: entered promiscuous mode
Mar 22 06:12:04 Tower webgui: Jellyfin: Could not download icon https://raw.githubusercontent.com/SmartPhoneLover/unraid-docker-templates/main/templates/icons/jellyfin_200x200.png
Mar 22 07:33:02 Tower kernel: cgroup: fork rejected by pids controller in /docker/60eb014042accab0fdf16a93b68d78c08275f4d671f4dcd87a46a0fb37db13f3
Mar 22 07:33:02 Tower kernel: traps: .NET TP Gate[837222] general protection fault ip:14e708d4650f sp:14a5d4aace60 error:0 in libc.so.6[14e708d46000+155000]
Mar 22 09:25:40 Tower webgui: Successful login user root from 192.168.1.217
Mar 22 09:28:19 Tower kernel: block nvme0n1: the capability attribute has been deprecated.
Mar 22 10:54:33 Tower sshd-session[1881719]: Connection from 192.168.1.217 port 54593 on 192.168.1.92 port 22 rdomain ""
Mar 22 10:54:36 Tower sshd-session[1881719]: Postponed keyboard-interactive for root from 192.168.1.217 port 54593 ssh2 [preauth]
Mar 22 10:54:46 Tower sshd-session[1881719]: Postponed keyboard-interactive/pam for root from 192.168.1.217 port 54593 ssh2 [preauth]
Mar 22 10:54:46 Tower sshd-session[1881719]: Accepted keyboard-interactive/pam for root from 192.168.1.217 port 54593 ssh2
Mar 22 10:54:46 Tower sshd-session[1881719]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Mar 22 10:54:46 Tower sshd-session[1881719]: User child is on pid 1881991
Mar 22 10:54:47 Tower sshd-session[1881991]: Starting session: shell on pts/0 for root from 192.168.1.217 port 54593 id 0

 

Edited March 22, 20251 yr by Mark12
Added syslog part.