System (and containers) have no Internet access after CA Appdata backup

[CorneliousJD]

The issue is that after doing a CA apdpata backup, my system loses Internet access for quite some time (6-12 hours usually) before coming back, it's very weird. the CA appdata backup only takes about an hour to backup.

 

I noticed that I got errors connecting to github to check for updates, and errors connecting to dockerhub to check for docker updates, etc. It has been a weird issue to track down.

 

I am on 6.10.3, and this issue only started after the 6.10.x branch.

I changed from having a docker VLAN to combat the macvlan docker issue, and just used the new ipvlan issue to prevent crashing.

No crashes so far, but now I'm running into this really bizarre issue. no idea if they are related.

 

Everything is totally fine until I run the CA appdata backup, at which point things go offline and I'm unable to connect to the Internet from the server or some containers as well. Very strange indeed. Any help is appreciated.

 

Note, I turned on checking for app updates after an appdata backup, and it hangs there too now.

Here's relevant system log info below that happens when invoking a CA appdata backup. 

 

I am unable to ping known-good addreses, such as 8.8.8.8 or 9.9.9.9 I just get

From 10.0.0.10 icmp_seq=1 Destination Host Unreachable

 

Jun 27 11:45:41 Server CA Backup/Restore: Stopping Vikunja-API
Jun 27 11:46:41 Server kernel: docker0: port 11(veth56fb8cf) entered disabled state
Jun 27 11:46:41 Server kernel: vethd5c5710: renamed from eth0
Jun 27 11:46:41 Server avahi-daemon[9164]: Interface veth56fb8cf.IPv6 no longer relevant for mDNS.
Jun 27 11:46:41 Server avahi-daemon[9164]: Leaving mDNS multicast group on interface veth56fb8cf.IPv6 with address fe80::c41a:f2ff:fea7:160c.
Jun 27 11:46:41 Server kernel: docker0: port 11(veth56fb8cf) entered disabled state
Jun 27 11:46:41 Server kernel: device veth56fb8cf left promiscuous mode
Jun 27 11:46:41 Server kernel: docker0: port 11(veth56fb8cf) entered disabled state
Jun 27 11:46:41 Server avahi-daemon[9164]: Withdrawing address record for fe80::c41a:f2ff:fea7:160c on veth56fb8cf.
Jun 27 11:46:41 Server CA Backup/Restore: docker stop -t 60 Vikunja-API
Jun 27 11:46:41 Server CA Backup/Restore: Stopping WizNote
Jun 27 11:47:42 Server kernel: docker0: port 12(vethce1e5eb) entered disabled state
Jun 27 11:47:42 Server kernel: veth2a8fc33: renamed from eth0
Jun 27 11:47:42 Server avahi-daemon[9164]: Interface vethce1e5eb.IPv6 no longer relevant for mDNS.
Jun 27 11:47:42 Server avahi-daemon[9164]: Leaving mDNS multicast group on interface vethce1e5eb.IPv6 with address fe80::4002:8ff:fec8:c984.
Jun 27 11:47:42 Server kernel: docker0: port 12(vethce1e5eb) entered disabled state
Jun 27 11:47:42 Server kernel: device vethce1e5eb left promiscuous mode
Jun 27 11:47:42 Server kernel: docker0: port 12(vethce1e5eb) entered disabled state
Jun 27 11:47:42 Server avahi-daemon[9164]: Withdrawing address record for fe80::4002:8ff:fec8:c984 on vethce1e5eb.
Jun 27 11:47:42 Server CA Backup/Restore: docker stop -t 60 WizNote
Jun 27 11:47:42 Server CA Backup/Restore: Backing up USB Flash drive config folder to 
Jun 27 11:47:42 Server CA Backup/Restore: Using command: /usr/bin/rsync  -avXHq --delete  --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" /boot/ "/mnt/user/backups/unRAID/flash/" > /dev/null 2>&1
Jun 27 11:47:51 Server CA Backup/Restore: Changing permissions on backup
Jun 27 11:47:53 Server CA Backup/Restore: Backing up libvirt.img to /mnt/user/backups/unRAID/libvirt/
Jun 27 11:47:53 Server CA Backup/Restore: Using Command: /usr/bin/rsync  -avXHq --delete  --log-file="/var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log" "/mnt/user/system/libvirt/libvirt.img" "/mnt/user/backups/unRAID/libvirt/" > /dev/null 2>&1
Jun 27 11:47:53 Server CA Backup/Restore: Changing permissions on backup
Jun 27 11:47:53 Server CA Backup/Restore: Backing Up appData from /mnt/user/appdata/ to /mnt/user/backups/unRAID/appdata/[email protected]
Jun 27 11:47:53 Server CA Backup/Restore: Using command: cd '/mnt/user/appdata/' && /usr/bin/tar -cvaf '/mnt/user/backups/unRAID/appdata/[email protected]/CA_backup.tar'  --exclude "plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder" --exclude "plex/Library/Application Support/Plex Media Server/Cache/Transcode" --exclude "sonarr/MediaCover" --exclude "sonarr-uhd/MediaCover" --exclude "radarr/MediaCover" --exclude "radarr-uhd/MediaCover" --exclude "lidarr/MediaCover" --exclude "tautulli/cache" --exclude "joplinapp"  * >> /var/lib/docker/unraid/ca.backup2.datastore/appdata_backup.log 2>&1 & echo $! > /tmp/ca.backup2/tempFiles/backupInProgress
Jun 27 12:07:48 Server emhttpd: spinning down /dev/sdk
Jun 27 12:09:15 Server emhttpd: spinning down /dev/sdn
Jun 27 12:10:14 Server emhttpd: spinning down /dev/sdh
Jun 27 12:18:23 Server nginx: 2022/06/27 12:18:23 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:18:23 Server nginx: 2022/06/27 12:18:23 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:18:26 Server nginx: 2022/06/27 12:18:26 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:18:26 Server nginx: 2022/06/27 12:18:26 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:24:22 Server nginx: 2022/06/27 12:24:22 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:24:22 Server nginx: 2022/06/27 12:24:22 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:24:25 Server nginx: 2022/06/27 12:24:25 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:24:25 Server nginx: 2022/06/27 12:24:25 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:30:23 Server nginx: 2022/06/27 12:30:23 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:30:23 Server nginx: 2022/06/27 12:30:23 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:30:26 Server nginx: 2022/06/27 12:30:26 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:30:26 Server nginx: 2022/06/27 12:30:26 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:36:23 Server nginx: 2022/06/27 12:36:23 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:36:23 Server nginx: 2022/06/27 12:36:23 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:36:26 Server nginx: 2022/06/27 12:36:26 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:36:26 Server nginx: 2022/06/27 12:36:26 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:42:22 Server nginx: 2022/06/27 12:42:22 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:42:22 Server nginx: 2022/06/27 12:42:22 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:42:25 Server nginx: 2022/06/27 12:42:25 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:42:25 Server nginx: 2022/06/27 12:42:25 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:48:23 Server nginx: 2022/06/27 12:48:23 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:48:23 Server nginx: 2022/06/27 12:48:23 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:48:26 Server nginx: 2022/06/27 12:48:26 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:48:26 Server nginx: 2022/06/27 12:48:26 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:52:39 Server CA Backup/Restore: Backup Complete
Jun 27 12:52:39 Server Docker Auto Update: Community Applications Docker Autoupdate running
Jun 27 12:52:39 Server Docker Auto Update: Checking for available updates
Jun 27 12:54:21 Server nginx: 2022/06/27 12:54:21 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:54:21 Server nginx: 2022/06/27 12:54:21 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.168:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:54:24 Server nginx: 2022/06/27 12:54:24 [error] 4045#4045: recv() failed (113: No route to host) while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"
Jun 27 12:54:24 Server nginx: 2022/06/27 12:54:24 [error] 4045#4045: OCSP responder prematurely closed connection while requesting certificate status, responder: r3.o.lencr.org, peer: 76.73.236.177:80, certificate: "/boot/config/ssl/certs/certificate_bundle.pem"

 

[CorneliousJD]

During a ping test I stopped docker so I could check on some network configs to see if making any changes would resolve the issue, but it looks like just stopping the docker service itself corrected the issue.

 

 

EDIT: Here are my docker settings. Nothing unusual here that would cause this, and again it only seems to happen when I run a CA appdata backup where it stops all containers, backs them up, and then starts them again.

 

 

Edited June 27, 2022 by CorneliousJD

[CorneliousJD]

bumping this - hoping someone has an option for me to at least try for troubleshooting this, as it's an odd one to say the least.

[CorneliousJD]

Swapped back to macvlan mode w/ a dedicated docker vlan for few containers that need static IPs.

 

so far issue has not resurfaced.

bug report created here.

 

 

[greenflash24]

This could be potentially related to the issue i have described in the following post:

(maybe this thread is related to your issue.)

[CorneliousJD]

On 7/4/2022 at 4:06 PM, greenflash24 said:

This could be potentially related to the issue i have described in the following post:

(maybe this thread is related to your issue.)

 

This certainly looks like it could be related, for me my network is fine up until I run a CA Appdata backup and then it stops working, very weird. It certainly seems to be ipvlan related for me too!