Hi,

 

I've upgraded to 6.12, then 6.12.2, and having issues with Docker crashing the machine. I ran with Docker disabled for 24hrs with no issue, then enabled to try and force the error before hanging.

 

Symptoms are:

• df command hangs
• diagnostics command also hangs, I've attached what I can before it hangs up. I also left one running last night with nohup, and it appears to have never finished
• Bars showing usage in WebUI go grey and stop working
• Eventually, system becomes unstable, hangs, and requires a forced reboot.

 

I've looked for bad plugins, stopped almost all docker containers (when they were running), disabled SMB and NFS in case that was the issue, and swapped from macvlan to ipvlan. Still no avail. Any ideas?

 

Diagnostics attached. Any help appreciated before manually backing out to 6.11.x

 

asdfghjkai

 

tower-diagnostics-20230705-2156.zip

Edited July 5, 20232 yr by asdfghjkai
Removing unanonymised diag logs

Hangs at this point

 

/usr/local/emhttp/webGui/scripts/error_interfaces|sed 's///' >>'/tower-diagnostics-20230705-2156/config/listen.txt' sed -ri 's/^(share(Comment|ReadList|WriteList)=")[^"]+/\1.../' '/tower-diagnostics-20230705-2156/shares/A--------e.cfg' 2>/dev/null

Now it really is being curious

When the issue starts post the output of:

df -h

 

Would do, but like I say it hangs. As does doing an ls on /mnt

 

Doing a 1 level deep du against the log directory shows it isn’t full though, so I know that

See if you can check if /run is full

Sure. Seems unlikely looking at this though 
 

root@Tower:/run# du -h --max-depth=1
0       ./containerd
140K    ./docker
4.0K    ./avahi-daemon
0       ./samba
0       ./cryptsetup
0       ./cron
12K     ./elogind
0       ./user
4.0K    ./dbus
0       ./faillock
0       ./lock
0       ./lvm
512K    ./udev
0       ./mount
8.0K    ./blkid
744K    .
root@Tower:/run#

 

And at this moment your are having the issue? If yes it's not the /run filling up issue some users are seeing.

Yes currently having the issue. I stopped the docker service, and it has remained stable, in that it has not crashed, but still has these obvious issues unfortunately.

 

If I start the Docker service again, the machine will likely halt within 12-18hrs.

If you are using IPv6 anywhere try using IPv4 only, it's been reported to help some users with docker issues.

IPv6 Already Disabled

root@Tower:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq master bond0 state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,PROMISC,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
5: bond0.20@bond0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.20 state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
6: bond0.100@bond0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.100 state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
7: bond0.120@bond0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0.120 state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 scope global br0
       valid_lft forever preferred_lft forever
9: br0.20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.4/24 scope global br0.20
       valid_lft forever preferred_lft forever
10: br0.100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
11: br0.120: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f4:6d:04:ac:5f:fc brd ff:ff:ff:ff:ff:ff
    inet 10.0.120.10/24 scope global br0.120
       valid_lft forever preferred_lft forever
13: br-7e1157f31345: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:d3:ff:42:0d brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-7e1157f31345
       valid_lft forever preferred_lft forever

 

Looks like I can still SSH, but the Web UI has stopped working now. Any other queries worth running before it stalls?

Edited July 6, 20232 yr by asdfghjkai
Correction

3 minutes ago, asdfghjkai said:

IPv4 Already Disabled

IPv6 is the one that should be disabled, use IPv4 only.

My bad. Should be obvious from the above output. IPv6 is already disabled. 

Anyone who finds this, the system has remained stable having disabled the bridging interface, a config carried over from a machine a years ago, given I now use a single NIC.
 

Catch here is that upon doing this, any containers mapped over to the previous interface, for example a vlan on the bridge (br0.100) will fail to start due to the missing interface, and you’ll need to both select a new equivalent (likely eth0.100) and re-enter the IP you chose previously.
 

System up for about 20 hours with this config and no issues yet, even with the run and log dies.