Slow pulldown

[Fiservedpi]

When I pull down/update a container it takes at least 5-8 min to complete, 90% of the time it works 10% it stalls out and doesn't finish.

1. Anyway I can see what's happening (/var/lo/docker.log)

2. This problem is recent it never used to be like this maybe 2-3 weeks ago it started.

3. Could this be related to worldwide bandwidth cap?

here is /var/logdocker/log

 

2020-04-04 22:35:19.916441 W | wal: sync duration of 1.230827326s, expected less than 1s
time="2020-04-04T22:35:19.916520110-04:00" level=error msg="error creating cluster object" error="name conflicts with an existing object" module=node node.id=pkh71xjdu81eie6xiprf7xjnc
2020-04-04 22:35:22.038938 W | wal: sync duration of 1.356385242s, expected less than 1s
time="2020-04-04T22:35:22.551929134-04:00" level=error msg="agent: session failed" backoff=100ms error="session initiation timed out" module=node/agent node.id=pkh71xjdu81eie6xiprf7xjnc
time="2020-04-04T22:35:23.897859341-04:00" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"
2020-04-05 22:35:23.408243 W | wal: sync duration of 3.460660432s, expected less than 1s
time="2020-04-06T00:04:34.848541359-04:00" level=error msg="stream copy error: reading from a closed fifo"
time="2020-04-06T00:04:34.848555923-04:00" level=error msg="stream copy error: reading from a closed fifo"
time="2020-04-06T00:25:51.051354158-04:00" level=error msg="stream copy error: reading from a closed fifo"
time="2020-04-06T00:25:51.051379921-04:00" level=error msg="stream copy error: reading from a closed fifo"
time="2020-04-06T10:14:24.247694905-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-06T21:16:28.333742527-04:00" level=error msg="Not continuing with pull after error: context canceled"
2020-04-06 22:35:22.012352 W | wal: sync duration of 2.061016111s, expected less than 1s
time="2020-04-08T02:12:56.452449600-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:12:56.452623038-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:12:56.452650813-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:12:56.452628502-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:15:45.686901979-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:15:45.687041547-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:15:45.687100009-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:15:45.687506259-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:18:10.337973256-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:18:10.337974329-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:21:47.026458987-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:21:47.026558447-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T02:33:19.360768836-04:00" level=error msg="Handler for POST /v1.37/containers/48b66735f418/restart returned error: Cannot restart container 48b66735f418: container is marked for removal and cannot be started"
time="2020-04-08T11:38:59.223867708-04:00" level=error msg="Not continuing with pull after error: context canceled"
time="2020-04-08T11:43:26.171070927-04:00" level=error msg="4996f60ab5ffdb18983a61f61369b4e3d8287348ed513053407dcd43addb5914 cleanup: failed to delete container from containerd: no such container"
time="2020-04-08T11:43:26.171101471-04:00" level=error msg="Handler for POST /v1.40/containers/4996f60ab5ffdb18983a61f61369b4e3d8287348ed513053407dcd43addb5914/start returned error: driver failed programming external connectivity on endpoint Virt-Manager (d7e04025416c94683db0208675e3e22efe9e1c6fdd04e8ec38b9cb11b9d528ac): Bind for 0.0.0.0:8080 failed: port is already allocated"
time="2020-04-08T12:26:57.292769913-04:00" level=error msg="54dae4a9f418e55668fd95eeaeb45748f12a9202b335fe1f6eec0a2724de8687 cleanup: failed to delete container from containerd: no such container"
time="2020-04-08T12:26:57.292801322-04:00" level=error msg="Handler for POST /v1.40/containers/54dae4a9f418e55668fd95eeaeb45748f12a9202b335fe1f6eec0a2724de8687/start returned error: driver failed programming external connectivity on endpoint Virt-Manager (6fa2a8fa99163a288330f96916519098d921dca7bce21a06f71baf56a115ef18): Bind for 0.0.0.0:8080 failed: port is already allocated"
time="2020-04-08T12:31:10.565678728-04:00" level=error msg="6bd0474f3e836f9b344a85d362751cd57acdd7bef1c61aedcea90b328bdfc788 cleanup: failed to delete container from containerd: no such container"
time="2020-04-08T12:31:10.565707694-04:00" level=error msg="Handler for POST /v1.37/containers/6bd0474f3e83/start returned error: driver failed programming external connectivity on endpoint Virt-Manager (4b08b4a37e6477379145667eb6fa118656f88855e462f26f7e1bf8f831b9119e): Bind for 0.0.0.0:8080 failed: port is already allocated"
time="2020-04-08T12:32:04.733791889-04:00" level=error msg="6bd0474f3e836f9b344a85d362751cd57acdd7bef1c61aedcea90b328bdfc788 cleanup: failed to delete container from containerd: no such container"
time="2020-04-08T12:32:04.733819158-04:00" level=error msg="Handler for POST /v1.37/containers/6bd0474f3e83/start returned error: driver failed programming external connectivity on endpoint Virt-Manager (0af2bce04b30593516dee481e41537496f5037eaa04b2a651df9caab3cba02bd): Bind for 0.0.0.0:8080 failed: port is already allocated"
2020-04-08 22:35:24.519657 W | wal: sync duration of 4.555785529s, expected less than 1s

 

tower-diagnostics-20200408-1955.zip

Edited April 9, 2020 by Fiservedpi
added docker log/tower log