I'm having issues setting up an Unraid to Unraid ZT network for off-site backups. I've got the community applications version installed on both servers, and both servers are currently on the same physical network until I get this working and can move one elsewhere.
On server A - ZT installs and tries to connect to the ZT network. It succeeds, but has a totally unpredictable uptime. Sometimes it'll stay connected a day, sometimes for 10 minutes. my.zerotier.com shows that it had been connected, I authorize it, and shows that it's just not online. I've tried reboot, fresh install, new ZT network, etc. Logs haven't been helpful at all.
Output from server A (Removed ZT Network ID) -
/ # zerotier-cli listnetworks
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks dbXXXXXXXXXX95 96:1d:00:34:ab:9e REQUESTING_CONFIGURATION PRIVATE ztkseuc3dy -
200 listnetworks 9fXXXXXXXXXXf2 Fireflight f2:d5:a3:e7:1d:e7 OK PRIVATE ztuzezwewm 10.1.100.169/24
/ # zerotier-cli status
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
200 info fec4d9241b 1.8.2 OFFLINE
/ # zerotier-cli peers
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
200 peers
<ztaddr> <ver> <role> <lat> <link> <lastTX> <lastRX> <path>
62f865ae71 - PLANET -1 RELAY
778cde7190 - PLANET -1 RELAY
9f77fc393e 1.8.5 LEAF -1 RELAY
cafe04eba9 - PLANET -1 RELAY
cafe9efeb9 - PLANET -1 RELAY
/ #
Server B - This is the one I'll be moving off-site. I haven't had any issues with this one, ZT control shows it's online constantly.
Output from Server B (removed ZT Network ID) -
/ # zerotier-cli status
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
200 info b65b5961a1 1.8.2 ONLINE
/ # zerotier-cli listnetworks
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks 9f7XXXXXXXXXXX2 Fireflight f2:9d:3c:67:58:5d OK PRIVATE ztuzezwewm 10.1.100.102/24
/ # zerotier-cli peers
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
zerotier-cli: /usr/lib/libstdc++.so.6: no version information available (required by zerotier-cli)
200 peers
<ztaddr> <ver> <role> <lat> <link> <lastTX> <lastRX> <path>
0cd5db9b0b 1.8.2 LEAF -1 RELAY
33b9213523 1.8.7 LEAF 2 DIRECT 6342 7673 192.168.1.132/9993
345c25e0e6 1.8.2 LEAF -1 RELAY
562b69b5d4 1.8.2 LEAF -1 RELAY
62f865ae71 - PLANET 265 DIRECT 6342 1067 50.7.252.138/9993
6ab80a8212 1.8.7 LEAF 1 DIRECT 1865 1865 192.168.1.172/29994
778cde7190 - PLANET 51 DIRECT 11347 1282 103.195.103.66/9993
9f77fc393e 1.8.5 LEAF 29 DIRECT 7921 7921 35.222.184.52/21015
b726b407e9 1.8.6 LEAF -1 RELAY
cafe04eba9 - PLANET 129 DIRECT 11347 1205 84.17.53.155/9993
cafe9efeb9 - PLANET 64 DIRECT 7904 1272 104.194.8.134/9993
fec4d9241b 1.8.2 LEAF -1 RELAY
/ #
I can't think of what could possibly be different about the two. Server A has been around a while and is used for a lot of different apps, and Server B is built new. The only real issue I've caught over the last few weeks is that Server A's Routing Table (in unraid network settings) doesn't always show the proper routing for the ZT network, where as Server B does. However, even with proper routing rules in place, Server A doesn't have a reliable connection.
It's also worth pointing out that when Server A does get a connection to ZT, it's not able to ping Server B.
Finally, the Unraid logs for Server A showed this when it last connected to the ZT network -
Apr 14 11:52:20 UNICRON root: /usr/sbin/winbindd -D
Apr 14 11:52:20 UNICRON wsdd[19807]: set_multicast: Failed to set IPv4 multicast
Apr 14 11:52:20 UNICRON wsdd[19807]: Failed to add multicast for WSDD: Address already in use
Apr 14 11:52:20 UNICRON wsdd[19807]: set_multicast: Failed to set IPv4 multicast
Apr 14 11:52:20 UNICRON winbindd[19816]: [2022/04/14 11:52:20.378088, 0] ../../source3/winbindd/winbindd_cache.c:3203(initialize_winbindd_cache)
Apr 14 11:52:20 UNICRON winbindd[19816]: initialize_winbindd_cache: clearing cache and re-creating with version number 2
Apr 14 11:52:20 UNICRON winbindd[19816]: [2022/04/14 11:52:20.378466, 0] ../../lib/util/become_daemon.c:135(daemon_ready)
Apr 14 11:52:20 UNICRON winbindd[19816]: daemon_ready: daemon 'winbindd' finished starting up and ready to serve connections
Apr 14 11:52:42 UNICRON nmbd[19800]: [2022/04/14 11:52:42.371489, 0] ../../source3/nmbd/nmbd_become_lmb.c:397(become_local_master_stage2)
Apr 14 11:52:42 UNICRON nmbd[19800]: *****
Apr 14 11:52:42 UNICRON nmbd[19800]:
Apr 14 11:52:42 UNICRON nmbd[19800]: Samba name server UNICRON is now a local master browser for workgroup WORKGROUP on subnet 172.17.0.1
Apr 14 11:52:42 UNICRON nmbd[19800]:
Apr 14 11:52:42 UNICRON nmbd[19800]: *****
Apr 14 11:52:42 UNICRON nmbd[19800]: [2022/04/14 11:52:42.371560, 0] ../../source3/nmbd/nmbd_become_lmb.c:397(become_local_master_stage2)
Apr 14 11:52:42 UNICRON nmbd[19800]: *****
Apr 14 11:52:42 UNICRON nmbd[19800]:
Apr 14 11:52:42 UNICRON nmbd[19800]: Samba name server UNICRON is now a local master browser for workgroup WORKGROUP on subnet 10.1.100.169
Apr 14 11:52:42 UNICRON nmbd[19800]:
Apr 14 11:52:42 UNICRON nmbd[19800]: *****
I'm not really sure where to go for fixing this, the logs have been pretty unhelpful, I've read through this entire forum topic, and any help would be appreciated!