[Plugin] Tailscale

[ailliano]

Noticed some of my dockers with the infamous "not available" problem, I looked at the logs and seems that tailscale plug in is rate limiting DNS queries ?
I use tailscale up --advertise-exit-node --accept-routes --advertise-routes=mysubnet/24

 

May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:03 [RATELIMIT] format("dns udp query: %v")
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 [RATELIMIT] format("dns udp query: %v") (1 dropped)
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 [RATELIMIT] format("dns udp query: %v")
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:32 [RATELIMIT] format("dns udp query: %v") (5 dropped)
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:32 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:32 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:40 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:40 dns udp query: context deadline exceeded

After I do tailscale down and check for updates all my dockers are green again. 
Any suggestions ? 

[EDACerton]

9 hours ago, reverend remiel said:

Sure thing.

aram-diagnostics-20230510-1429.zip 169 kB · 1 download

It seems like something messed with the version of glibc on your system, and in doing so broke the system "at" command, which the plugin uses to handle starting the service:

Quote

May 10 12:38:30 aram atd[11741]: PAM unable to dlopen(/lib64/security/pam_unix.so): /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /lib64/libpthread.so.0)
May 10 12:38:30 aram atd[11741]: PAM adding faulty module: /lib64/security/pam_unix.so
May 10 12:38:30 aram kernel: atd[11741]: segfault at 0 ip 000014a50a6da886 sp 00007ffe1b9ecc10 error 4 in libpthread-2.33.so[14a50a6da000+f000]

 

I can't see far enough back in syslog to see what might have done that, though, and it's possible that it's a plugin that has since been uninstalled (if the server hasn't been restarted since then).

 

This has been a known issue for other plugins when running on older builds of Unraid. I can think of a few things that you could try to see if they help:

• Restart the server, see if that clears up the changed library version.
• Update Unraid; the issues with glibc should go away then.
• Install the "Unassigned Devices Preclear" plugin, that plugin updates glibc and might help (or might not, that's hard to tell for certain).

[EDACerton]

21 minutes ago, ailliano said:

Noticed some of my dockers with the infamous "not available" problem, I looked at the logs and seems that tailscale plug in is rate limiting DNS queries ?
I use tailscale up --advertise-exit-node --accept-routes --advertise-routes=mysubnet/24

 

May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:03 [RATELIMIT] format("dns udp query: %v")
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 [RATELIMIT] format("dns udp query: %v") (1 dropped)
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:12 [RATELIMIT] format("dns udp query: %v")
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:32 [RATELIMIT] format("dns udp query: %v") (5 dropped)
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:32 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:32 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:40 dns udp query: context deadline exceeded
May 10 17:11:13 Loki tailscaled: 2023/05/10 17:10:40 dns udp query: context deadline exceeded

After I do tailscale down and check for updates all my dockers are green again. 
Any suggestions ? 

The DNS queries aren't being rate limited, the error logging is what's being rate limited.

 

Please post diagnostics. That gives me more info that I can use to try and help.

[ailliano]

9 minutes ago, EDACerton said:

The DNS queries aren't being rate limited, the error logging is what's being rate limited.

 

Please post diagnostics. That gives me more info that I can use to try and help.

sure !
 

loki-diagnostics-20230510-1754.zip

[EDACerton]

8 minutes ago, ailliano said:

sure !
 

loki-diagnostics-20230510-1754.zip 338.42 kB · 0 downloads

This seems like a weird issue with Tailscale, my best guess being that it has something to do with all of the following being enabled:

• Accept Routes
• Accept DNS
• Advertised Route
• DNS server is inside the advertised route

Do you need for your server to be able to resolve MagicDNS addresses? If not, you could probably just run

tailscale set --accept-dns=false

and make the problem go away.

[ailliano]

3 minutes ago, EDACerton said:

This seems like a weird issue with Tailscale, my best guess being that it has something to do with all of the following being enabled:

• Accept Routes
• Accept DNS
• Advertised Route
• DNS server is inside the advertised route

Do you need for your server to be able to resolve MagicDNS addresses? If not, you could probably just run

tailscale set --accept-dns=false

and make the problem go away.

okay just tried, now having another issue when I do 
tailscale up --accept-routes --advertise-exit-node --advertise-routes=172.18.108.0/24 --accept-dns=false
I lose connection to unraid locally, ssh dies right after the command, all my dockers are unreachable as well.

[EDACerton]

3 minutes ago, ailliano said:

okay just tried, now having another issue when I do 
tailscale up --accept-routes --advertise-exit-node --advertise-routes=172.18.108.0/24 --accept-dns=false
I lose connection to unraid locally, ssh dies right after the command, all my dockers are unreachable as well.

 

A couple questions…

 

Do you have any other subnet routers that are advertising the same route?

 

Can you access Unraid via its Tailscale address?

 

If you set accept routes to false, can you get back in locally?

[ailliano]

1 minute ago, EDACerton said:

 

A couple questions…

 

Do you have any other subnet routers that are advertising the same route?

 

Can you access Unraid via its Tailscale address?

 

If you set accept routes to false, can you get back in locally?

Do you have any other subnet routers that are advertising the same route?
I have a synology running tailscale too 
sudo tailscale up --advertise-exit-node --advertise-routes=172.18.108.0/24 --reset
Can you access Unraid via its Tailscale address?
Yes
If you set accept routes to false, can you get back in locally?
yes I can just tried
root@Loki:~# tailscale up --accept-routes=false --advertise-exit-node --advertise-routes=172.18.108.0/24 --accept-dns=false
Some peers are advertising routes but --accept-routes is false

[ailliano]

3 minutes ago, ailliano said:

Do you have any other subnet routers that are advertising the same route?
I have a synology running tailscale too 
sudo tailscale up --advertise-exit-node --advertise-routes=172.18.108.0/24 --reset
Can you access Unraid via its Tailscale address?
Yes
If you set accept routes to false, can you get back in locally?
yes I can just tried
root@Loki:~# tailscale up --accept-routes=false --advertise-exit-node --advertise-routes=172.18.108.0/24 --accept-dns=false
Some peers are advertising routes but --accept-routes is false

okay seems like everything is good, even Dockers can find updates now, only issue is my log getting spammed with 

 

May 10 18:24:31 Loki tailscaled: 2023/05/10 18:22:57 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:22:57 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:22:57 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:07 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:07 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:07 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:07 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:16 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (10 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:16 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:16 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:16 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:26 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:26 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:26 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:26 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:36 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:36 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:36 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:36 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:46 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:46 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:46 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:46 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:56 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:56 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:56 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:23:56 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:06 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:06 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:06 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:06 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:16 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (22 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:16 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:16 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:16 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL")
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:31 [RATELIMIT] format("dns: resolver: forward: no upstream resolvers set, returning SERVFAIL") (16 dropped)
May 10 18:24:31 Loki tailscaled: 2023/05/10 18:24:31 dns: resolver: forward: no upstream resolvers set, returning SERVFAIL

 

[EDACerton]

The network config might be a little weird after making a bunch of changes, you can try restarting the daemon with

 

/etc/rc.d/rc.tailscale restart

 

If that doesn’t work, I’d just reboot. 

[ailliano]

1 minute ago, EDACerton said:

The network config might be a little weird after making a bunch of changes, you can try restarting the daemon with

 

/etc/rc.d/rc.tailscale restart

 

If that doesn’t work, I’d just reboot. 

just tried that, NICE no dns issues, I can get in too. 
You're the best !

last q, why did tailscale up --accept-routes=false --advertise-exit-node --advertise-routes=172.18.108.0/24 --accept-dns=false this work instead of my regular command
what do --accept-routes=false and --accept-dns=false fix ? Trying to understand what they do in the first place, I have another unraid in different site I need to set up

[EDACerton]

3 hours ago, ailliano said:

just tried that, NICE no dns issues, I can get in too. 
You're the best !

last q, why did tailscale up --accept-routes=false --advertise-exit-node --advertise-routes=172.18.108.0/24 --accept-dns=false this work instead of my regular command
what do --accept-routes=false and --accept-dns=false fix ? Trying to understand what they do in the first place, I have another unraid in different site I need to set up

• --accept-routes: causes Tailscale to use the routes that are being advertised by subnet routers (most frequently -- used by clients to access servers/devices that can't run Tailscale).
• --accept-dns: causes Tailscale to handle DNS resolution... this allows clients to enter (for example) "unraid" to access the "unraid" device on Tailscale.

Both features are generally more useful for clients than servers. (Unless you need to initiate connections from Unraid to something else on Tailscale, but most use cases rely on a client initiating the connection to Unraid).

 

As for why disabling those fixed your issue -- some combinations of the "advanced" network settings for Tailscale work on Unraid, and some don't work so well. In this case, I think it's a combination of the Synology subnet router + accept-routes that causes things to go awry.

 

This is an effect of the way the network configuration works in Unraid. Unraid doesn't use tools like NetworkManager or systemd-resolvd to manage network configurations. This means that applications like Tailscale have to manually update the network configuration to work -- which can run into problems if multiple things are all trying to do that.

Edited May 11, 2023 by EDACerton

[EDACerton]

Release: 2023.05.10

• Added help page to settings
• Added info page to settings
• Moved Tailscale to Network Services section of Settings
• Updated Tailscale to 1.40.1

[rainformpurple]

11 hours ago, EDACerton said:

It seems like something messed with the version of glibc on your system, and in doing so broke the system "at" command, which the plugin uses to handle starting the service:

 

I can't see far enough back in syslog to see what might have done that, though, and it's possible that it's a plugin that has since been uninstalled (if the server hasn't been restarted since then).

 

This has been a known issue for other plugins when running on older builds of Unraid. I can think of a few things that you could try to see if they help:

• Restart the server, see if that clears up the changed library version.
• Update Unraid; the issues with glibc should go away then.
• Install the "Unassigned Devices Preclear" plugin, that plugin updates glibc and might help (or might not, that's hard to tell for certain).

Yeah, I keep putting off updating, because I never seem to find the time to deal with eventual problems that may arise by doing so. I should probably just do it. Some day.

 

I'll install the Unassigned Devices Preclear plugin and see if that fixes it temporarily.

 

Thanks!

[YaneonY]

I've faced the same problem as some people here: I am unable to access any Docker container running on br0 with a static address set. I've tested different commands, such as:

 

tailscale up --advertise-routes=192.168.1.0/24 --accept-routes --accept-dns --advertise-exit-node
tailscale up --advertise-routes=192.168.1.0/24 --accept-routes=false --accept-dns=false --advertise-exit-node

 

Basically, I don't want my home server to be an exit node, so I would remove it later.

However, regardless of whether it's included or not, I can't access anything from Unraid Docker. I can access Unraid itself and other IPs running on my network, like Unifi, Printer GUI, etc.
 

Is it possible that some routes need to be added to the routing tables?

 

[YaneonY]

Okay, for everyone facing same problem:

 

Go to Settings -> Docker and change "Host access to custom networks" to "Enabled".

[ailliano]

12 hours ago, EDACerton said:

• --accept-routes: causes Tailscale to use the routes that are being advertised by subnet routers (most frequently -- used by clients to access servers/devices that can't run Tailscale).
• --accept-dns: causes Tailscale to handle DNS resolution... this allows clients to enter (for example) "unraid" to access the "unraid" device on Tailscale.

Both features are generally more useful for clients than servers. (Unless you need to initiate connections from Unraid to something else on Tailscale, but most use cases rely on a client initiating the connection to Unraid).

 

As for why disabling those fixed your issue -- some combinations of the "advanced" network settings for Tailscale work on Unraid, and some don't work so well. In this case, I think it's a combination of the Synology subnet router + accept-routes that causes things to go awry.

 

This is an effect of the way the network configuration works in Unraid. Unraid doesn't use tools like NetworkManager or systemd-resolvd to manage network configurations. This means that applications like Tailscale have to manually update the network configuration to work -- which can run into problems if multiple things are all trying to do that.

@EDACertonThank you for great explanation, makes a lot of sense  

I have another host in a different location, I want to be able to vpn in and navigate throughout the network so i advertise it as an exit node, however in this case tailscale is installed on WSL and has a separate network (172.27.16.0/20) than the host Windows 192.168.0.0/24, I tried to advertise both routes, but I still have no access to the internet or the 192 network, I'm sure since the wsl adds another layer I'm not reaching something network wise.
Setup is this WindowsHost(192.168.0.0/24)<>WSL Tailscale(172.27.16.0/20)<>Internet
command:
sudo tailscale up --advertise-exit-node --advertise-routes=192.168.0.0/24, 172.27.16.0/20 --reset

 

Thank you

[rainformpurple]

17 hours ago, EDACerton said:

It seems like something messed with the version of glibc on your system, and in doing so broke the system "at" command, which the plugin uses to handle starting the service:

 

I can't see far enough back in syslog to see what might have done that, though, and it's possible that it's a plugin that has since been uninstalled (if the server hasn't been restarted since then).

 

This has been a known issue for other plugins when running on older builds of Unraid. I can think of a few things that you could try to see if they help:

• Restart the server, see if that clears up the changed library version.
• Update Unraid; the issues with glibc should go away then.
• Install the "Unassigned Devices Preclear" plugin, that plugin updates glibc and might help (or might not, that's hard to tell for certain).

 

The plugin didn't make any difference, so I'm currently upgrading my server to 6.11.5, which I should've done many moons ago anyway.

 

I'll report back once the upgrade is complete.

[rainformpurple]

Yeah, that went straight to hell in a handbasket. Upon rebooting into the supposedly upgraded system, the console starts spewing error messages stating GLIBC_2.34 cannot be found (required by rm and sleep). My USB key has also been blacklisted for some  reason, and no dockers nor VMs are started. The WebUI is only half-way functioning and doesn't display any meaningful data (no disks show up, no server stats, no plugins show up, no controls work, nothing.

 

Exactly what I was afraid of what would happen, in other words.

 

I'm not saying it's related to the tailscale plugin in any shape way or form, just to make that clear.

 

Currently trying GUI safe mode to see what's going on and to see if I can get it working enough to contact Limetech support to have un-blacklist my USB drive.

 

Edit 1:

Downgrading to 6.9.2 got rid of the error spewing, and it appears my USB drive isn't blacklisted anymore... I'm not entirely sure what's going on, but something isn't right.

 

As a curious aside, the tailscale plugin now works and I've re-registered my server with the Tailscale mothership and everything works as expected.

 

Edit 2:

I couldn't let this lie and had  to tinker with it. I nuked all my plugins and now it works with 6.11.5, so I'm adding back the plugins one by one to see which (if any) breaks my server.

 

Edit 3:

The culprit is a user script I have written, which tries to install some packages and that breaks everything, requiring a hard reboot to get out of. The script is now disabled and things look fine.

 

Edit 4:

The problem was that I for some ducked up reason installed a package named aaa_glibc_somethingsomething.txz that apparently worked enough on 6.9.2 to allow the audio tools I installed via the same script to work.

 

That caused various problems when installed on 6.11.5, as the installed version of glibc is newer.

 

After booting into safe mode a number of times I have verified that this was indeed the cause and that package is no longer installed on boot.

 

I'm currently working my way through the list of packages to find compatible versions, and I feel that this is something that maybe could be installed via Nerd Tools.

 

The mismatched glibc also caused all the other problems with the unRAID WebUI, registration retrieval status, network issues, etc. My USB key is not blacklisted, the mismatch caused the verification to fail miserably and probably receiving garbage data back, leading it to think that the USB key was blacklisted. That is sorted now and everything works.

 

In any case, my server is now finally upgraded to 6.11.5 and the tailscale plugin works perfectly

Edited May 11, 2023 by reverend remiel
Cleaning up and summarizing my posts in the thread

[EDACerton]

It sounds like you had quite the adventure! I’m glad it’s working now. 

[rainformpurple]

14 hours ago, EDACerton said:

It sounds like you had quite the adventure! I’m glad it’s working now. 

 

Yeah, it was a rollercoaster. I thought I'd get away with a quick 30 minute upgrade, but it turned into a 4 hour forensics job.

 

Today I've been slowly fixing docker issues that popped up because I forgot that newer versions of unRAID weren't happy about how things were on 6.9. As I don't trust automated scripts I don't know what actually do, I'm changing the permissions by hand for the shares I know I have to change, and then I'll see about the rest later, if at all necessary.

[SC8198]

Quick question does this plugin run tailscale on startup of unraid? or will I need to make a script?

[EDACerton]

1 minute ago, SC8198 said:

Quick question does this plugin run tailscale on startup of unraid? or will I need to make a script?

It runs on startup. 

[SC8198]

9 minutes ago, EDACerton said:

It runs on startup. 

Thats awesome! thanks for making the plugin!

[Angeloc]

After upgrading Unraid to 6.12.0-rc6, ssh and samba through tailscale interface are broken, maybe it was caused by the following update:

Quote

• rc.samba - let smb, nmb service listen on regular interfaces only which have an IP address, this includes the primary interface + set ipv4 / ipv6 support (also for wsdd2)
• rc.ssh - listen on regular interfaces only which have an IP address, this includes the primary interface + set ipv4 / ipv6 support

 

Are there any workaround for this？