I'm having trouble with my server ever since moving to Unraid. Machine was perfectly stable on Proxmox, but as I wanted to use this server as a file server and not just an application server I moved to Unraid, starting with 6.12rc3. Issues appear anywhere between a few hours to a few days after boot, with a bit over 4 days being the longest uptime I've achieved on Unraid so far.

 

Server is an 11th gen Intel NUC (NUC11ATKPE) with a Pentium N6005 processor (Jasper Lake processor, Tremont cores), 16GB of RAM. I've run memory tests (latest version of Memtest86) with no errors, have updated the system's BIOS, with no effect. Pool SSD with appdata, domains, system shares is internal NVMe PCIe, Array is in external USB storage (2x16TB Exos drives in an Icy Box enclosure). Have an additional "spool" pool SSD in the external enclosure doing nothing.

 

Symptoms include:

* Complete loss of network connectivity (ping, SSH, HTTP, etc.).

* Display connected directly to server (in text-only boot) either freezes, glitches, or in the most recent boot the display no longer thought it was plugged in until rebooting.

* Keyboard connected to computer no longer responds (eg pressing caps lock doesn't affect caps lock light).

 

Computer requires a hard reset to begin working again.

 

Previously, I had also had both BTRFS and ZFS pool volumes corrupt, which caused further system instability particularly when the ZFS pool stopped working. Since before moving to 6.12rc4 I have all volumes running as XFS as a result.

 

I've tried a few boot time options too. None have had an effect on this issue. To go through them:

* pci=nommconf - I was having errors in the log which I traced back to the on-board Ethernet adapter dropping packets. Easier to parse the syslog without said errors.

* i915.enable_guc=2 - Required for hardware accelerated transcoding in Plex when reading from H265 files. I'm inclined to believe that, since transcoding doesn't work without this option and judging from the Arch Linux wiki documentation on this boot option, that enable_guc=0 is the default. Which is to say, I experienced crashes/system instability both before and after adding this option.

* intel_idle.max_cstate=1 - The above Arch Linux wiki page suggested a few potential fixes for crashing on Intel low power processors, but said wiki article they're all for older generations of processors. I tried this one, but still experienced a crash today.

 

I've had two crashes since enabling syslog mirroring to flash. The first one happened after I went to bed, the system was running no containers/VMs and was only performing a File Integrity plugin build (the lack of load on the machine at the time of that crash is what led to me trying the idle.max_cstate option). That appears to have been around 00:33 on 6 May, but a lot of error entries appear in the log afterwards. Either way the "crashed/unresponsive" symptoms above were there when I woke up. I also tried removing all of the htop/Intel GPU stats plugin stuff in case that was having an effect. Today's crash was around 18:03, 7 May, and there is no indication in the mirrored syslog file as to what caused the crash. Speaking of, I've added it to the diagnostics zip file in the logs folder as "syslog". There are a bunch of Docker network changes leading up to the crash, this is because I have a new and exciting problem with the LinuxServer SWAG tool where it's deciding to just not work half the time that I'm trying to troubleshoot. Can't see how network changes would cause the system to be unresponsive to direct keyboard/display connections though!

 

Any help gratefully received - really want to like Unraid but I've had nothing but problems so far! Rapidly approaching throw-toys-out-of-pram point.

 

Thanks in advance!

 

Edit: I've tried changing intel_idle.max_cstate=1 for i915.enable_dc=0 which was suggested in an 11th gen Intel Core/Unraid issue thread. Maybe it'll help? 🤷

cyllene-diagnostics-20230507-1817.zip

Edited May 7, 20233 yr by fneb

Enable the mirror to flash option in the syslog server and post that after a crash.

1 hour ago, JorgeB said:

Enable the mirror to flash option in the syslog server and post that after a crash.

I did - as mentioned in the original post, it's in the diagnostics zip added to the logs folder. I mistakenly said it was called "syslog"; I actually called it "syslog2" but hey ho. Anyway, I've attached it again for your convenience.

syslog2

May  6 23:57:34 cyllene kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
May  6 23:57:34 cyllene kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

1 minute ago, JorgeB said:

May  6 23:57:34 cyllene kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
May  6 23:57:34 cyllene kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

Thanks for your response JorgeB. I've already switched over to ipvlan but sadly I can't remember if I did that before/after the second crash to know if that made a difference. I think I was having crashes before I originally moved to macvlan but well, by this point it all kinda blurs together!

 

Speaking of, can you see anything that could have caused the second crash in the syslog? The log itself seems to show absolutely nothing, but the diagnostics dump was taken straight after rebooting.

 

Thanks!

Syslog does show other issues, including filesystem corruption and other call traces, but if there were changes delete current syslog and post a new clean one after a crash.

Semi-update - computer crashed again but I’m on holiday so can’t do anything about it. Will post a response/proper update early next week when I’m home.

 

For what it’s worth, I had seen the file system corruption and done xfs_repairs previously (think corruption was caused by all the crashes) but did it again after your last message.

On 5/8/2023 at 10:32 AM, JorgeB said:

Syslog does show other issues, including filesystem corruption and other call traces, but if there were changes delete current syslog and post a new clean one after a crash.

Syslog attached (I tried generating a diagnostics zip but after nearly an hour of doing so the browser tab crashed) - I've trimmed the syslog to between the start of the boot of the boot which crashed, and soon after I rebooted it. Backblaze shows the computer (more accurately, the Docker container running on the computer) stopped responding at 6:10am on 10/05/2023 which is a bit before the last syslog entry, but it could be either log or Backblaze is showing UTC and the other is showing my current local timezone (BST, GMT+1).

 

The log has a bunch of errors relating to the Parity Check Tuning plugin, I'm assuming as it was outside of scheduled runtime? But otherwise the log abruptly stops. Same symptom as before, the machine was completely unresponsive including the USB keyboard having no response (eg caps lock key didn't change caps lock LED), required forced power down.

 

I had to go back to macvlan in Docker due to other issues with accessibility to containers. I've not seen anything saying there are issues with macvlan causing full system crashes outside of UnRaid, but I'm pretty sure I've had crashes even when on ipvlan. Currently I'm running the server without Docker running at all (disabled in Settings/Docker) and I am backing up my data to my old server so I can change to Proxmox or something other than UnRaid. I'd rather just use UnRaid but at some point I'll just need to accept I'm better off not continuing and write the time and money spent on UnRaid off. At least running the system with Docker disabled will let me see if the issue lies somewhere in there I guess. Any help resolving this so I can stick with UnRaid gratefully received. Thanks!

syslog3.txt

There are macvlan call traces on the 9th, and that will usually make the server crash within 1 or 2 days.

3 minutes ago, JorgeB said:

There are macvlan call traces on the 9th, and that will usually make the server crash within 1 or 2 days.

Why does this seem to disproportionately affect UnRaid? I haven't been able to find any issues online with macvlan Docker networking outside of posts about UnRaid. Asides from "don't use macvlan", what are the possible fixes? This thread seems to indicate it's a new issue with 6.12 and that turning off bridging may be a bypass solution (though I imagine that could cause container networking troubles) but it would be good to know if there is a better solution - hopefully if it is down to UnRaid it's being investigated prior to the full 6.12 release!

 

Thanks.

Current solutions are using ipvlan, disabling bridge if no VM needs it or use two NICs.

12 hours ago, JorgeB said:

Current solutions are using ipvlan, disabling bridge if no VM needs it or use two NICs.

I've turned off bridging for now, though I'll look forward to a fix. Thanks for your help!