SeRiusMe Posted May 23 Author Share Posted May 23 (edited) 12 hours ago, Vr2Io said: set that switch port ( connect to eth2 ) with a new vlan id then vlan3 won't go there. You mean that set the Id to an unused/fake vlan so traffic doesn't go anywhere? I think I could do the same by changing vlan 3 on that port from untagged to tagged. (It's already tagged) But I would want to take that IP configuration out there. I think something messed as I started the server in just one network, as every other appliance and evolved to vlans. EDIT: I was missing permissions for the proxmox server in 10.1.5.xx for mounting backups share. As soon as I added them, I accessed properly. So the only server that fails at mounting and accessing shares is at 10.1.3.xx segment. That is the configuration that I want to fix in the main section of the interface. I bet that the server is trying to serve all the DNS requests through the 10.1.3.xx gateway/route, and it's failing. (It must be the default route) Edited May 23 by SeRiusMe Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 (edited) 22 hours ago, SeRiusMe said: And I can't change the empty gateway in eth0.2. And you can see that it's not creating a custom network for eth0. This indicate problem on docker system, I have reach a case, the OP have similar problem, always can't found the custom network, finally OP mark the case resolve by remove other previous docker network. So pls check does other create network left and remove it by "docker network rm xxxxx". Also perform some check on all existing docker network. That's also / may be why we change all network setting and problem still occur. Sometimes, docker network like a black box, when you confirm your physical network haven't issue, but always got trouble. Then you may need further verify by VM network first. For DNS problem, do you confirm not problem on router routing ? For me, all docker could reach private DNS and internet, also other subnet. docker network ls NETWORK ID NAME DRIVER SCOPE 669a79230488 bridge bridge local 61f4b403738e eth0 macvlan local a5af90fcb721 eth0.2 macvlan local e4473591c365 eth0.666 macvlan local a642478c8a6d host host local 33db889adf1b none null local docker network inspect a5af90fcb721 [ { "Name": "eth0.2", "Id": "a5af90fcb721b1133c84995351c516a77c6d3464c66a60b0a505047507bd8ef4", "Created": "2024-05-18T10:51:40.796286014+08:00", "Scope": "local", "Driver": "macvlan", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "192.168.2.0/24", "Gateway": "192.168.2.9" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "1e404021ac2b09835323d92888db216c69b46dfe26a213c33a4599fce449e1a9": { "Name": "Pihole", "EndpointID": "7489add3c1bc4b7eacc43bb65541a914b39a4a63ae40ef1acfdeb08265bf0564", "MacAddress": "02:42:c0:a8:02:05", "IPv4Address": "192.168.2.5/24", "IPv6Address": "" }, "cfaa6db42d9c4023309fa94ef10458b77d3cb184d7b5eb03b4c064f8996fed66": { "Name": "NTP", "EndpointID": "7009d52484bc8ca96171d088b7f1fbe592a6c435753448952d51b560b1369c2e", "MacAddress": "02:42:c0:a8:02:06", "IPv4Address": "192.168.2.6/24", "IPv6Address": "" }, "d7ffa44f7c40585d323f540599adf7880750480881f2d1163bfb43541dbba245": { "Name": "Syslogserver2", "EndpointID": "8d800260429264b6278caf086f5a77be0cade18ab7bfd9028155ac6f9037fee0", "MacAddress": "02:42:c0:a8:02:07", "IPv4Address": "192.168.2.7/24", "IPv6Address": "" } }, "Options": { "parent": "vhost0.2" }, "Labels": {} } ] 19 hours ago, SeRiusMe said: You mean that set the Id to an unused/fake vlan so traffic doesn't go anywhere? Yes. Each interface have a PVID, so if you don't want untag traffic going to a port, then you can assign an unuse PVID to it. Edited May 24 by Vr2Io Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 (edited) 16 hours ago, SeRiusMe said: I bet that the server is trying to serve all the DNS requests through the 10.1.3.xx gateway/route, and it's failing. (It must be the default route) No, for 10.1.2.xx or 10.1.5.xx, if DNS was 10.1.3.xx it should direct route through router. For example, traceroute in Unraid console, 192.168.9.x subnet route through 192.168.9.9 gateway traceroute 192.168.2.5 traceroute to 192.168.2.5 (192.168.2.5), 30 hops max, 60 byte packets 1 192.168.9.9 (192.168.9.9) 0.115 ms 0.107 ms 0.229 ms 2 * * * 3 * * * traceroute in docker console, 192.168.68.x subnet route through 192.168.68.9 gateway traceroute 192.168.2.5 traceroute to 192.168.2.5 (192.168.2.5), 30 hops max, 46 byte packets 1 192.168.68.9 (192.168.68.9) 0.088 ms 0.099 ms 0.089 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * Edited May 24 by Vr2Io Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 (edited) 53 minutes ago, Vr2Io said: No, for 10.1.2.xx or 10.1.5.xx, if DNS was 10.1.3.xx it should direct route through router. My router listens for DNS queries on all interfaces. LAN, IOT and SYS. I don't have any blocked DNS entries on my router, but anyways I've created a firewall rule allowing access for DNS address in 10.1.3.xx segment and the problem still persists. This is a traceroute inside a container on vlan 5 to a computer in vlan 2: # traceroute 10.1.2.100 traceroute to 10.1.2.100 (10.1.2.100), 30 hops max, 46 byte packets 1 10.1.5.251 (10.1.5.251) 0.091 ms 0.121 ms 0.072 ms 2 10.1.2.100 (10.1.2.100) 1.211 ms 1.347 ms 1.230 ms I can't find any way of inspecting a DNS request inside any of may containers, but all have in resolv.conf 127.0.0.11 as DNS resolver and fails. Yes, it seems a docker configuration problem with the DNS resolver. If I could "see" what requests are really coming from the containers... Now I'm trying with bridging active. I found in the release notes of unraid (https://docs.unraid.net/unraid-os/release-notes/6.12.4/#fix-for-macvlan-call-traces) a writeup talking about a problem and trying it, but doesn't change anything. Perhaps you are right when you say that perhaps the docker networking configuration is not reconfiguring and it's frozen at some point. I don't know, but I'm not advancing anything. EDIT: There isn't any problem that I know it the routing at the router. My computer, for example has total access to the devices in other vlans, and devices in all vlans can resolve DNS queries. LXC containers on a Proxmox server in ITO segment can resolve DNS. Even the Unraid Host can perfectly resolve. Something is wrong between the host and the dockers. And if the containers use the default bridge (dropouts). If I put a container manually in the 10.1.3.xx net it runs without dropouts (with no DNS): I also checks the DHCP in case there where conflicts, but no. Edited May 24 by SeRiusMe Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 (edited) I found a container for network diagnosing: # dig google.com ; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47311 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;google.com. IN A ;; Query time: 4001 msec ;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP) ;; WHEN: Fri May 24 13:56:00 CEST 2024 ;; MSG SIZE rcvd: 28 # ping www.google.com ping: www.google.com: Temporary failure in name resolution # host www.google.com Host www.google.com not found: 2(SERVFAIL) From one container to other works, but not to host: # nslookup d10ef2ae9349 Server: 127.0.0.11 Address: 127.0.0.11#53 Non-authoritative answer: Name: d10ef2ae9349 Address: 10.1.5.28 # nslookup deathshadow Server: 127.0.0.11 Address: 127.0.0.11#53 ** server can't find deathshadow: SERVFAIL Seems like dockers DNS gateway is not working?? Edited May 24 by SeRiusMe Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 (edited) Your dig result haven't answer session, anyway depends on docker itself, my HA docker have dig and nslookup. The DNS translate also 127.0.0.11 I am not sure how to diagnostic if DNS resolve not work in docker. Edited May 24 by Vr2Io Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 2 hours ago, Vr2Io said: This indicate problem on docker system, I have reach a case, the OP have similar problem, always can't found the custom network, finally OP mark the case resolve by remove other previous docker network. So pls check does other create network left by "docker network rm xxxxx". Also perform some check on all existing docker network. Restarting docker brings this two errors, that I found online tht are related to trying to create routes and failing because they exist. # /etc/rc.d/rc.docker restart stopping dockerd ... ... Waiting to die. starting dockerd ... RTNETLINK answers: File exists RTNETLINK answers: File exists Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 3 hours ago, Vr2Io said: I have reach a case, the OP have similar problem, always can't found the custom network, finally OP mark the case resolve by remove other previous docker network. I've seen that post, but I can't understand what is being said there. BTW, I've seen the route table you posted there and I'm missing several routes that are there, for example docker0. Can you please explain me how did the other guy solved it? Now, I can't make changes on network. Because docker is stopped, but it's still saying that has to be stopped for changes: Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 2 minutes ago, SeRiusMe said: Restarting docker brings this two errors, that I found online tht are related to trying to create routes and failing because they exist. # /etc/rc.d/rc.docker restart stopping dockerd ... ... Waiting to die. starting dockerd ... RTNETLINK answers: File exists RTNETLINK answers: File exists Yes, problem pointing to docker system instead network or network setting. You may try start from scratch by remove docker image / folder to fix it. Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 (edited) 19 minutes ago, SeRiusMe said: Can you please explain me how did the other guy solved it? In fact I am not sure OP really fix or not, but no matter how I change the network setting, I couldn't reproduce OP's docker problem. I store docker in /tmp ( ram file system ), so each reboot will redownload all docker ( only need restore appdata ) and I never do extra docker setting, that may be help always keep docker system clean up and so no trouble at all. Edited May 24 by Vr2Io Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 (edited) Unfortunately deleting the docker image didn't work. 😢 But docker did create the missing routes: # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.1.3.251 0.0.0.0 UG 1 0 0 eth0 0.0.0.0 10.1.2.251 0.0.0.0 UG 2 0 0 eth0.2 0.0.0.0 10.1.5.251 0.0.0.0 UG 3 0 0 eth0.5 10.1.2.0 0.0.0.0 255.255.255.0 U 0 0 0 vhost0.2 10.1.2.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0.2 10.1.3.0 0.0.0.0 255.255.255.0 U 0 0 0 vhost0 10.1.3.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 10.1.5.0 0.0.0.0 255.255.255.0 U 0 0 0 vhost0.5 10.1.5.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0.5 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 Edited May 24 by SeRiusMe Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 (edited) After start docker service Still no gateway at eth0 ? Unraid routing table look nornal ? Edited May 24 by Vr2Io Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 (edited) 48 minutes ago, Vr2Io said: After start docker service Still no gateway at eth0 ? Unraid routing table look nornal ? Yes, even removing the docker configuration file from /boot and rebooting. # cat /boot/config/docker.cfg DOCKER_ENABLED="yes" DOCKER_IMAGE_FILE="/mnt/services/system/docker/docker.img" DOCKER_IMAGE_SIZE="20" DOCKER_APP_CONFIG_PATH="/mnt/services/appdata/" DOCKER_APP_UNRAID_PATH="" DOCKER_READMORE="yes" DOCKER_CUSTOM_NETWORKS="eth1 eth2 " <=== IF THOSE REFER TO HOST INTERFACES, THEY ARE DISABLED. ONLY ETH0 IS UP BUT I EDIT THE FILE AND AFTER DOCKER STARTS REVERTS TO THOSE VALUES. DOCKER_TIMEOUT="10" DOCKER_LOG_ROTATION="yes" DOCKER_LOG_SIZE="50m" DOCKER_LOG_FILES="1" DOCKER_AUTHORING_MODE="no" DOCKER_USER_NETWORKS="remove" Host routing table: # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.1.3.251 0.0.0.0 UG 1 0 0 eth0 0.0.0.0 10.1.2.251 0.0.0.0 UG 2 0 0 eth0.2 0.0.0.0 10.1.5.251 0.0.0.0 UG 3 0 0 eth0.5 10.1.2.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0.2 10.1.3.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0 10.1.5.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0.5 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 Resolution in container in Bridge works: # dig google.com ; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53262 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;google.com. IN A ;; ANSWER SECTION: google.com. 300 IN A 142.250.201.78 ;; Query time: 23 msec ;; SERVER: 10.1.3.251#53(10.1.3.251) (UDP) ;; WHEN: Fri May 24 19:51:18 CEST 2024 ;; MSG SIZE rcvd: 55 Edited May 24 by SeRiusMe Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 (edited) RESOLUTION IN CONTAINER IN VLAN5 WORKS IF DNS OF VLAN5 IS FORCED INSTEAD OF THE DEFAULT (10.1.3.251) # nslookup google.com 10.1.5.251 Server: 10.1.5.251 Address: 10.1.5.251#53 Non-authoritative answer: Name: google.com Address: 142.250.201.78 Name: google.com Address: 2a00:1450:4003:803::200e The host DNS server doesn't work, and I guess the internal docker DNS is trying the same: # nslookup google.com 10.1.3.251 ;; communications error to 10.1.3.251#53: timed out ;; communications error to 10.1.3.251#53: timed out ;; communications error to 10.1.3.251#53: timed out ;; no servers could be reached # nslookup google.com 127.0.0.11 Server: 127.0.0.11 Address: 127.0.0.11#53 ** server can't find google.com: SERVFAIL Even if I add the other DNS servers to unraids network, it doesn't work: # UNRAID HOST # cat /etc/resolv.conf # Generated by rc.inet1 nameserver 10.1.3.251 nameserver 10.1.2.251 nameserver 10.1.5.251 Edited May 24 by SeRiusMe Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 3 minutes ago, SeRiusMe said: RESOLUTION IN CONTAINER IN VLAN5 WORKS IF DNS OF VLAN5 IS FORCED INSTEAD OF THE DEFAULT (10.1.3.251) Interesting, but don't know why 10.1.3.251 not work, because eth0 no gateway ? This also the difference on my setup, my private DNS ( Pihole ) was on eth0.2, not the router or public DNS. But my private DNS endup also lookup on a public DNS. Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 If I change the order for the DNS at the host and put the 10.1.5.251 at the top, The container at VLAN5 resolves DNS. BUT ALSO OTHER ONE IN VLAN2. (But it takes a while, I suppose because is trying 10.1.5.251 first? # ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 45: eth0@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:0a:01:02:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.1.2.25/24 brd 10.1.2.255 scope global eth0 valid_lft forever preferred_lft forever # nslookup google.com Server: 127.0.0.11 Address: 127.0.0.11#53 Non-authoritative answer: Name: google.com Address: 142.250.201.78 Name: google.com Address: 2a00:1450:4003:803::200e WTF?? Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 6 minutes ago, Vr2Io said: Interesting, but don't know why 10.1.3.251 not work, because eth0 no gateway ? No, by now there's no interface with an empty gateway. DOCKER: IPv4 custom network on interface eth0: Subnet: 10.1.3.0/24 Gateway: 10.1.3.251 DHCP pool: not set IPv4 custom network on interface eth0.2: Subnet: 10.1.2.0/24 Gateway: 10.1.2.251 DHCP pool: not set IPv4 custom network on interface eth0.5: Subnet: 10.1.5.0/24 Gateway: 10.1.5.251 DHCP pool: not set UNRAID: # ifconfig docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 ether REDACTED txqueuelen 0 (Ethernet) RX packets 4 bytes 168 (168.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4 bytes 268 (268.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.1.3.30 netmask 255.255.255.0 broadcast 0.0.0.0 ether REDACTED txqueuelen 1000 (Ethernet) RX packets 8061 bytes 1604692 (1.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6701 bytes 5185374 (4.9 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0.2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet REDACTED netmask 255.255.255.0 broadcast 0.0.0.0 ether f4:52:14:c6:05:c2 txqueuelen 1000 (Ethernet) RX packets 2879 bytes 863444 (843.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2278 bytes 4808849 (4.5 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0.5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.1.5.30 netmask 255.255.255.0 broadcast 0.0.0.0 ether REDACTED txqueuelen 1000 (Ethernet) RX packets 3589 bytes 265242 (259.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 690 bytes 74546 (72.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 12 minutes ago, Vr2Io said: This also the difference on my setup, my private DNS ( Pihole ) was on eth0.2, not the router or public DNS. But my private DNS endup also lookup on a public DNS. Same on me. But I have Adguard installed on my router machine. Listening on port 53 and I changed the routers DNS server to 53530. Adguard resolves to router and router to upstreams. Quote Link to comment
Vr2Io Posted May 24 Share Posted May 24 (edited) 8 minutes ago, SeRiusMe said: No, by now there's no interface with an empty gateway. Then it look like no problem on docker system now. 14 minutes ago, SeRiusMe said: I change the order for the DNS at the host and put the 10.1.5.251 at the top, The container at VLAN5 resolves DNS. BUT ALSO OTHER ONE IN VLAN2. (But it takes a while, I suppose because is trying 10.1.5.251 first? All DNS server must accessible by all subnet, client will randomly resolve on all DNS server according the setting, not 1st one then 2nd then 3rd. Edited May 24 by Vr2Io Quote Link to comment
SeRiusMe Posted May 24 Author Share Posted May 24 Still container network is hanging after a while if attached to Bridge or Host. 🤷♀️ Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.