Not wanting to hijack this thread... but are your dockers hosted on a btrfs disk pool???

 

I had something like what your describing happen to me while I had my dockers hosted on this disk pool... It wasnt till I moved all of my dockers back to the array did things start working properly...

 

little known bug??? Dont know... but something to consider

 

to note... I had two SSD's in the disk pool... 

 

I had issues within some of the dockers, odd issues, but mostly when trying to update them, even had a docker become 'orphaned' due to this... After I moved everything over to the array, things started working normally... 

 

Hope that helps

Yes indeed, I will try it , if I change the format to xfs or zfs on the SSD if it helps.


 

I don’t think for you @xrqpthe Docker image is causing your current problem. If Docker were unable to write to the image, you would typically see space/write related errors like

No space left on device

Input/output error

 

Instead, having read @Squid asking @LanceG0dif Jumbo frames are enabled i think maybe he suspects an mtu mismatch, which I also think could be your issue possibly if you also are using jumbo frames?
Consumer routers usually don’t support jumbo frames. If you are using Jumbo frames on your server, to test this, try running a ping command to the router that forces the do not fragment flag with a large packet size. That way you can see if your router does support large mtu set on the server

 

ping -M do -s 8000 your routers ip

so for me, my router ip is 10.10.20.1 and running the command looks like the below

 

root@Nebuchadnezzar:~# ping -M do -s 8000 10.10.20.1
PING 10.10.20.1 (10.10.20.1) 8000(8028) bytes of data.
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long

 

So for me i get the errors as my router doesnt support mtu 8192 so i get message too long & mtu=1500 because my router doesnt support jumbo frames

 

So if you get message too long or mtu=1500 errors your router doesnt support jumbo frames either. So then in your case the server would be sending large packets that must be broken into smaller fragments. All these fragments need to arrive successfully and be reassembled. Losing a single fragment means retransmitting the entire packet and so this can cause delays or failures, and may lead to the timeout errors you’re currently experiencing when updating the containers. I would try setting the server back to 1500 and see if the issue still happens.
@LanceG0d I would also try setting back the mtu to 1500 your router doesnt support a high mtu if you find the docker image isnt your issue.

 

I hope this helps.

8 hours ago, LanceG0d said:

Yes indeed, I will try it , if I change the format to xfs or zfs on the SSD if it helps.


 

changing the format shouldnt make a difference... just move your docker files to the array and see how it goes... IMO should 'fix' your problems...

@SpaceInvaderOne im using jumbo frames, an my router is supporting jumbo frames.
I did the test:
8008 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.745 ms
8008 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.770 ms
8008 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.798 ms
8008 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=0.498 ms
and so on..
 

Currently im testing with XFS cache pool and i hope that this is the fix,
Because i have 2 pools
1 for docker images and 1 only for appdata for the dockers.
for the images it was btrs format , for the appdata xfs. Now i formated the image pool to xfs and i will test if still bug´s.

@mathomas3 my array is only HDD´s so it doesnt make sense for me because its to slow
Im using SSD pools for a longer time and had never problems until one day. Its very weird, beacuse i changed nothing, other then adding one more HDD for the Array

7 minutes ago, LanceG0d said:

@SpaceInvaderOne im using jumbo frames, an my router is supporting jumbo frames.
I did the test:
8008 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.745 ms
8008 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.770 ms
8008 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.798 ms
8008 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=0.498 ms
and so on..
 

Currently im testing with XFS cache pool and i hope that this is the fix,
Because i have 2 pools
1 for docker images and 1 only for appdata for the dockers.
for the images it was btrs format , for the appdata xfs. Now i formated the image pool to xfs and i will test if still bug´s.

@mathomas3 my array is only HDD´s so it doesnt make sense for me because its to slow
Im using SSD pools for a longer time and had never problems until one day. Its very weird, beacuse i changed nothing, other then adding one more HDD for the Array

As you wish... Im just saying what happened to me and what my fix was. It's worth trying out IMO... to help narrow down what's going on... Best of luck

When it working then its good to know. if it doesnt work at my way i will give it a shot an try it your way.
 

4 hours ago, LanceG0d said:

When it working then its good to know. if it doesnt work at my way i will give it a shot an try it your way.
 

It pains me reading your English.... Yikes...