Unraid randomly crashing after a few days or weeks of uptime

[Bertoo1337]

Hey there.

For the past 3 months i encounter random crashes of Unraid. I've searched for possible causes but since i dont know whats the trigger, its hard to boil down. System log is giving me no clue atm. The System is a little less than a year old, i swapped the PSU already because of paranoia, checked the RAM by only leaving one dimm at a time in there (usually 4x8GB), Processor never exceeds 35°C except for running an occasional game server, other than that - no unusal system load. No specific time of day for the crashes - sometimes in the night, sometimes in the afternoon...

 

I got 8 Dockers running:

- FileBrowser

- MeTube

- Nginx

- Nginx Proxy Manager

- paperless

- pihole

- Plex Media Server

- Redis

 

System Specs:
Gigabyte H410M H V2
i5 10400, cooled by a BeQuiet PureRock 2
32GB GSkill DDR4-2400 CL17 RAM
560 Watt LC-Power LC6560GP3

4x 4TB WD Red Plus (dont worry, they're not SMR drives) 1 Parity
1X 512 GB Samsung 970 EVO as  Application drive
1x 512 GB Crucial MX300 as Cache drive to eliminate any bottlenecks while transfering Data

 

I'd be happy if any1 can help me identifying the issue, or at least how i can get a usable log.

 

[JorgeB]

You can still post the persistent syslog to see if there's something there, another thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

[Bertoo1337]

On 5/16/2024 at 12:11 PM, JorgeB said:

You can still post the persistent syslog to see if there's something there, another thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

that would take ages to hopefully find the issue... isnt there a crashdump somewhere?

[JorgeB]

If there's nothing in the persistent syslog there won't be anywhere else, if you don't have the persistent log enabled, enable the syslog server and post that after a crash.

 

 

[Bertoo1337]

On 5/19/2024 at 1:18 PM, JorgeB said:

If there's nothing in the persistent syslog there won't be anywhere else, if you don't have the persistent log enabled, enable the syslog server and post that after a crash.

 

 

i enabled it now, lets see if i can find something on the next crash

[Bertoo1337]

On 5/16/2024 at 12:11 PM, JorgeB said:

You can still post the persistent syslog

Now i got a log. Tried to find something, but my coding skills are almost non existent... Can any1 more skilled find something here that tells us why its crashing?
https://pastebin.com/kufAG0Rp

[JorgeB]

May 22 03:28:00 NAS kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
May 22 03:28:00 NAS kernel: ? _raw_spin_unlock+0x14/0x29
May 22 03:28:00 NAS kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

[Bertoo1337]

On 5/23/2024 at 12:24 PM, JorgeB said:

May 22 03:28:00 NAS kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
May 22 03:28:00 NAS kernel: ? _raw_spin_unlock+0x14/0x29
May 22 03:28:00 NAS kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

 

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

 

i mean, its "only" been 10 days so far, but it looks like this is it... thanks a lot man.

EDIT: now i can't reach my file browser from outside tho... i have a forwarding in nginx proxy manager, i can reach it locally, but not from the outside anymore. is there something else i need to change in order to regain that acess?

Edited June 2 by Bertoo1337

[JorgeB]

8 hours ago, Bertoo1337 said:

i can reach it locally, but not from the outside anymore. is there something else i need to change in order to regain that acess?

Not sure I can help with that, but what are you using to access the server from outside?

[Bertoo1337]

19 minutes ago, JorgeB said:

Not sure I can help with that, but what are you using to access the server from outside?

I use cloudflare to resolve my adress via the remote interface of my router and linked my DNS records for my domain accordingly. With nginx Proxy Manager i forward the adress to my local adress of the file browser docker. Worked fine before switching to ipvlan, and i cant see a reason why it shouldnt work now, since no adress changed since then 😕

[Kilrah]

If you have that container on ipvlan try moving it to bridge and not have a dedicated ip, it should not be needed. Or give more details about your network setup/what router you have.

[Bertoo1337]

6 hours ago, Kilrah said:

If you have that container on ipvlan try moving it to bridge and not have a dedicated ip, it should not be needed. Or give more details about your network setup/what router you have.

For a quick test i reverted wo macvlan without even rebooting and my domain was reachable again and was able to log into my file browser. Same goes for nginx Prox Managager. I wasn't able to create an SSL Certificate for another domain, which worked perfectly fine after reverting to macvlan. All containers are working fine with their local network adresses in both vlan types.

For the Network setup:
my PC and the Unraid server are connected to a HUAWEI WiFi AX3 Router via LAN, that one is connected to my FritzBox 6591. Each router has its own Subnet. (Fritzbox -- Main Router which is connected to the internet got 192.168.178.xxx, the Huawei Router has 192.168.3.xxx. All my dockers which need connection to the outside have a dedicated 192.168.3.xxx adress. The router also runs in UPnP mode and within the FritzBox the Huawei router is able to manage port-forwarding as needed on its own.

For the containers in working macvlan configuration:
Nginx Proxy Manager: br0 LAN1 got the 192.168.3.2 with TCP Ports 80 and 443
File Browser: br0 LAN1 got the 192.168.3.101 with TCP Port 80

The Physical adress of the Machine is 192.168.3.10
 

Changing the FileBrowser docker to Bridge configuration doesnt change anything, still not reachable. With macvlan i get a 504 Bad gateway when trying to reach it from the outside in Bridge mode.

Edited June 3 by Bertoo1337

[Bertoo1337]

Solved it!
I've put the Nginx Proxy in host mode, edited my portforwarding in my router, switched the FileBrowser to bridge and set my proxy accordingly. now its running like i want it to.