Hey folks, my server is inconsistently freezing a few minutes after boot/enabling docker. I've posted previously about an error in the log (here) with no response, but this seems to be related to docker/network settings. As per recommendation of upgrading to 6.12.4, I have docker configured to use ipvlan which has worked fine - when the sever does not crash within the first 10mins of booting, it will run perfectly until the next time I manually shutdown/reboot for maintenance.

 

I have however noticed that after a reboot, the docker network setting is reverting to maclan - possibly I'm not noticing this everytime it reverts and could be the times it freezes up? not sure. I'm also suspicious of my USB flash drive which is a new replacement about 2 weeks ago after the prior seemed to become faulty. Where in the USB drive would this 'ipvlan' setting be, so I can explore closer? 

 

Any support/suggestions appreciated, diagnostics attached.

nostromo-diagnostics-20230916-1220.zip

Enable the syslog server and post that after a crash.

 

15 hours ago, Rusty6285 said:

Where in the USB drive would this 'ipvlan' setting be, so I can explore closer? 

If the line below is present on /config/docker.cfg it's using ipvlan

DOCKER_NETWORK_TYPE="1"

 

 

 

Thanks @JorgeB, I do see 'DOCKER_NETWORK_TYPE="1"' in the current docker.cfg file, so I'll keep an eye on it in future. I'm fairly sure my docker.cfg file keeps getting reset, I'v included docker.cfg from an export a week prior and it has significantly less values in the file for some reason.

 

My syslog is sent to my 2nd unraid server, linked below - I never see meaningful syslog entries when the crash occurs, the last crash was at 'Sep 16 13:16:15'. I've also attached the syslog directly from the crashing server (Nostromo), after the last crash hope these help.

 

 

volta-syslog-10.168.1.90-20230917-1330.zip nostromo-syslog-20230916-1728.zip docker.cfg

17 hours ago, Rusty6285 said:

I never see meaningful syslog entries when the crash occurs,

Yep, unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Hey @JorgeB, I replaced my PSU and USB stick again this week but the problem persists - my first boot with the new hardware, I verified that Docker was set to ipvlan and started the array. Within a few moments it crashed. On the next boot, I went back to view docker settings, and as expected, the network type has reverted to macvlan - see image below.

 

Any possible explanation for why this in particular seems to be resetting? if my docker.cfg or network settings are  somehow corrupted, is there a way to restore? not making a lot of sense to me right now. This server has been running for many years, with lots of er, tinkering throughout that time  it's quite possible I've done something odd with my network configs too, per edit 2 below. Any best practice to eliminate possible issues here?

 

Edit - added diagnostics and the docker (after).cfg file from the flash, obtained on the 2nd boot. I also included docker (before) which is a copy of what I moved to my new flash drive last night. - you can see that a bunch of the docker settings are absent in the 'after' file.

 

Edit 2 - looking at my docker (before), is DOCKER_CUSTOM_NETWORKS=" " a valid value to have in there? seems to be where the files first differ

 

 

nostromo-diagnostics-20230923-0928.zip

docker (before).cfg docker (after).cfg

Edited September 23, 20232 yr by Rusty6285

As an experiment, I replaced both 'docker.cfg' and 'network.cfg' from the clean Unraid install template, and my new docker.cfg looks quite different upon starting the array:

 

DOCKER_ENABLED="yes"
DOCKER_IMAGE_FILE="/mnt/user/system/docker/docker.img"
DOCKER_IMAGE_SIZE="20"
DOCKER_APP_CONFIG_PATH="/mnt/user/appdata/"
DOCKER_APP_UNRAID_PATH=""
DOCKER_READMORE="yes"
DOCKER_NETWORK_TYPE="1"
 

I don't see 'DOCKER_CUSTOM_NETWORKS=" "' any longer either... were the other values redundant/retired?

 

Edit - Gah, after about 10 minutes the system froze up again. Booting into safemode now as per your suggestion.

 

Edit 2 - After booting in safe mode, guess what's back in my docker.cfg:

 

DOCKER_ENABLED="no"
DOCKER_IMAGE_FILE="/mnt/user/system/docker/docker.img"
DOCKER_IMAGE_SIZE="20"
DOCKER_APP_CONFIG_PATH="/mnt/user/appdata/"
DOCKER_APP_UNRAID_PATH=""
DOCKER_READMORE="yes"
DOCKER_NETWORK_TYPE="1"
DOCKER_CUSTOM_NETWORKS=" "
DOCKER_TIMEOUT="10"
 

Maybe I'm barking up the tree with that value, but why does it so randomly appear at different stages of this debugging journey?

 

Edited September 23, 20232 yr by Rusty6285

17 hours ago, Rusty6285 said:

DOCKER_NETWORK_TYPE="1"

It's still using ipvlan.

Seems the null value beneath it was a red herring - I remembered that I had a user script starting powertop on array start and disabling that has so far returned stability, need to monitor for a few days! thank you for your patience and the troubleshooting steps!