System hung, IPMI not accepting input

[sonofdbn]

This morning I found my server (6.12.6) had hung: couldn't access the GUI or the shares. IPMI showed an unRAID screen with a field for user name, but I couldn't type anything into the field, even using the virtual keyboard.

 

In the end I had to reboot, so there was the dreaded unclean shutdown, and now parity-check is running.

 

Don't think I've done anything drastic, although over the last few weeks fiddling with docker container access I switched from ipvlan back to macvlan and vice versa a couple of times, but the server has been running fine (on macvlan without any further changes) for at least a week.

 

Hoping someone can make sense of the diagnostics and tell me what caused the problem.

tower-diagnostics-20240224-0945.zip

[JorgeB]

If it keeps happening enable the syslog server and post that after a crash.

[sonofdbn]

Well it so happens that I forgot to disable syslog server from some past investigation, so I actually have a syslog. I've removed earlier things and lines after rebooting, but can add those back if needed. Looks like the crash was around 3.00 am on Feb 24.

syslog_1.log

[JorgeB]

Many call traces but unclear to me what's causing them, it doesn't look macvlan related, it could be a specific container causing issues, if it keeps happening you can try stopping all and then re-enabling one by one.

[sonofdbn]

The GUI has become unresponsive at least twice more in the last month. Not sure if it's the same problem, but would like to know if there's something obvious that I should be doing. Still on 6.12.6. This resulted in unclean shutdowns each time.

 

Diagnostics and syslog are attached. Syslog is lightly redacted; hope I haven't removed anything that's pertinent. (\\IP address\Tower_repo is a Tailscale connection.)

 

This time the GUI got stuck while updating the DuckDNS container - I saw some error message (unfortunately can't remember what now) and then the GUI gradually froze - various tabs became unresponsive.

tower-diagnostics-20240329-1010.zip syslog-192.168.1.14.log

[JorgeB]

Mar 29 10:06:31 Tower kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Mar 29 10:06:31 Tower kernel: ? _raw_spin_unlock+0x14/0x29
Mar 29 10:06:31 Tower kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]

Macvlan call traces will usually end up crashing the server, switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)), then reboot.

 

Also, the docker image is corrupt, it should be recreated:

https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file

Also see below if you have any custom docker networks:

https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks

 

[sonofdbn]

Thanks so much. Will carry out the fixes once the parity check has completed.

[sonofdbn]

I've switched to ipvlan, recreated the custom docker network (same name "proxynet") and recreated the docker image. But now my docker containers (Swag and Nextcloud) aren't connecting to the custom network. Host access to custom networks is enabled, as is Preserve user defined networks. I'm sure I've done this before, so I think I'm missing some obvious step.

 

Here's the result of docker network ls:

 

root@Tower:~# docker network ls
NETWORK ID     NAME       DRIVER    SCOPE
16453c5dced9   br0        ipvlan    local
7c5d56aee35f   bridge     bridge    local
83cf0e1b1ef5   host       host      local
9e74c89874cb   none       null      local
69ed37939ece   proxynet   bridge    local

 

tower-diagnostics-20240331-1848.zip

[JorgeB]

Sorry but I cannot help with that, suggest creating a new post since it's unrelated to the thread title.