Every few months I come back to this repo to check if they finally got Tailnet lock running or if someone security audited them in the meanwhile. Unfortunately neither of these things seem to make any progress and thus, I’ve grown uncertain in how much I can trust this as a core part of my infrastructure.
The entire premise of Tailscale SaaS builds on creating tunnels around your firewalls, then enabling the user to police what is allowed to be routed through these tunnels in a intuitive and unified way.
Headscale seems to have nailed down the part of bypassing the firewall and doing fancy NAT-traversal, but can they also fulfill the second part by providing enough of their own security to make up for anything they just bypassed, or will they descend to just being a tool for exposing anything to the internet to fuck around with your local network admin?
To me, not giving your Tailscale implementation any way for the user to understand or veto what the control server is instructing the clients to do while also not auditing your servers code at all sure seems daring…
nativeit 18 hours ago [-]
> Headscale seems to have nailed down the part of bypassing the firewall and doing fancy NAT-traversal
Did they really roll-their-own for those functions? I thought this was just a control layer on top of Tailscale’s stock services on the backend, are they facilitating connections with novel methods? Apologies if I’m asking obvious questions, I use ZeroTier pretty regularly, but I am not too familiar with Tailscale.
i think they mean headscale's implementation specifics
xrd 10 hours ago [-]
Can you share why you use ZeroTier over Tailscale? I run several headscale control planes and it really is nice to self-host. But, I'm curious about other options.
password4321 7 hours ago [-]
Not OP but I'm on ZeroTier because it was one of the best free tiers available before Tailscale could run as a Windows service.
Also I believe it implements a lower layer of the network stack so more options are supported, though I haven't needed to investigate in detail.
tailnet lock seems way way less important for headscale than tailscale, given you personally control the headscale infra.
codethief 9 hours ago [-]
Depends on your threat model. Mine definitely includes one of my servers getting compromised. (Which, tbh, is probably more likely than Tailscale getting hacked.)
SuperShibe 9 hours ago [-]
only until someone finds a zeroday in headscale (remember, it never got audited) or until the server running headscale itself gets compromised. Especially in countries where getting a dedicated public IPv4+IPv6 from your ISP is hard-impossible and you‘d have to rely on a server hosted externally (unless you’re large enough to make deals with the ISP) some company hosting your server still retains at minimum physical control over your headscale infra. For why this is a problem, see the recent Oracle cloud breach.
botto 11 hours ago [-]
This is my thought as well, if you are in control then you also control which nodes go on your tailnet
gpi 19 hours ago [-]
One of the maintainers work for tailscale now.
wutwutwat 19 hours ago [-]
maintainer's employment != security audit
gpi 19 hours ago [-]
My thinking is their time is divided now and could lead to less efforts spent on headscale.
palotasb 16 hours ago [-]
Not compared to the previous state where he worked for an unrelated company and only had his free time to contribute to Headscale.
If you're interested in self-hosting your orchestration server, you can look into Netbird. It's a very similar tool, but has the server open sourced as well. So you have a self-hosted control server with a nice GUI and all the features the paid version does.
I've been slowly moving everything over from Tailscale to Netbird and aside from some shenanigans with Tailscale taking over the entire CGNAT route, it works wonderfully!
Tailscale is still running for now, but I'm getting closer and closer to decommissioning it and switching entirely to Netbird.
davidcollantes 9 hours ago [-]
Compared to Headscale, Netbird has so many moving pieces! It looks robust, and powerful, and featureful... yet, self-hosting Headscale is super simple, and less demanding.
I think it would be neat if headscale allowed peering / federating between instances. (Maybe after the ACL rework.) One of the main problems is address collisions.
So here's my proposal: commit to ipv6-only overlay network in the unique local address (ULA) range, then split up the remaining 121 bits into 20 low bits for device addresses (~1M) and 101 high bits that are the hash of the server's public key. Federate by adding the public key of the other instance and use policy and ACLs to manage comms between nodes.
Should add the project name, Headscale, to the title
Headscale has been on HN many times.
voxadam 18 hours ago [-]
Does it run on Plan 9?
bradfitz 9 hours ago [-]
Ha! But seriously, it's written in Go, so probably.
mountainriver 21 hours ago [-]
Love headscale, we just took it to production and it’s been great
linsomniac 9 hours ago [-]
I've been running headscale for 2.5 years and it's been pretty good. We use our gmail domain for logging in, which gives a big benefit that users can self-serve their devices. Unlike with OpenVPN in the past where ops had to hand off the certs and configs. Really the only downside has been when they accidentally connect to the tailscale login server instead of our own and then can't figure out why they can't reach any services. We use user groups to set up what services users can access.
We are still running the old headscale, because we have some integrations that will need to be ported to the new control plane. According to "headscale node list | wc" we have ~250 nodes, most of them are servers.
One thing I really don't love about tailscale some of the magic it does with the routing tables and adding firewall rules, but it has mostly not been an issue. Tailscale has worked really quite well.
syntaxing 21 hours ago [-]
As in you rolled out an internal service for the whole company?!
cassianoleal 15 hours ago [-]
As opposed to what? This seems pretty normal.
We considered it as well but there was a feature missing that meant we couldn’t use it for one of our main requirements. Had that not been the case, we’d have rolled it out.
mrklol 10 hours ago [-]
Mind sharing which feature?
cassianoleal 9 hours ago [-]
Honestly I'm hazy on the details but we're running a fairly complex environment in GCP with PSC everywhere, connections to on-prem and other external environments, and something wouldn't quite work due to all that.
Sorry I can't provide any more details but I really don't remember the specifics. We were in touch with Tailscale engineers and they offered some workarounds that we had already worked out but that wouldn't help us achieve what we were after.
sshine 18 hours ago [-]
I’d love to see a write-up on that.
Especially in the unlikely event that you used Nix for the deployment.
benley 17 hours ago [-]
I've done exactly that: headscale in production at work, a few hundred client devices, infrastructure mostly powered by nix. What would you want to hear about it?
sshine 3 hours ago [-]
> headscale in production at work
- How much effort do you put into key management compared to plain WireGuard?
- How automated is the onboarding process; do you generate and hand over keys?
- How do you cope without the commercial Tailscale dashboard?
- Do you run some kind of dashboard or metrics system?
- How long did it take to set up?
- Were there any gotchas?
avtar 29 minutes ago [-]
> How do you cope without the commercial Tailscale dashboard?
* Does it work well?
* Do you recommend it?
* Do your users care?
* Is it difficult? Do you have to maintain it or is it basically set it and forget it?
* What was memorable about setting it up?
* Why did you go for Headscale vs Tailscale or Netbird or some other solution?
aborsy 6 hours ago [-]
How much is the risk of my devices being compromised if Tailscale coordination server is compromised, and tailnet lock is enabled?
pilif 20 hours ago [-]
Keep in mind that for many use cases (mobile access, GUI on macOS), this relies on the official Tailscale clients keeping the ability to set the control server.
The moment the inevitable enshitification will start at Tailscale, this feature will go away.
I’m saying this as a currently super happy Tailscale customer who was burned multiple times in the past by other companies being sold or running out of VC money
miki123211 3 hours ago [-]
I may be misremembering, but I think they have said somewhere that Headscale is actually revenue positive for them.
That feels right to me. Headscale is mostly used by home labbers and small hobby users, it competes with self-hosted OpenVPN and WireGuard, not Pulsesecure, Cisco Anyconnect or GlobalProtect. It's a way to introduce Tailscale to people who love to try new shiny tech in their spare time, but don't want to give up control over their infrastructure.
Those people will then bring their Tailscale expertise and enthusiasm to work. Work really doesn't like managing IT infrastructure unless it's one of their core competencies.
Sure, some companies will actually choose Headscale over Tailscale proper, but I suspect that's a small minority (especially if you take company size and the money involved into account). That's just cost of revenue, not unlike Facebook advertising or billboards on the side of a road in Silicon Valley.
comex 2 hours ago [-]
> I think they have said somewhere that Headscale is actually revenue positive for them.
I have the same memory. But they may not feel that way forever. Many a company started by attracting developers with a generous free tier or open-source offering, then started to clamp down once the going got tough.
Heck, it happened to one of Tailscale's competitors, ZeroTier, which used to release their client software under GPLv3 but eventually switched to BSL.
risho 19 hours ago [-]
arent most of the the tailscale clients open source aside from the gui portion of the non open source os's?
pilif 16 hours ago [-]
Yes they are, unless you're using a mainstream OS and/or want to use a GUI, which is probably the most common use case.
__float 13 hours ago [-]
While the GUI is somewhat helpful, at the end of the day it's not the key piece, and it could easily be rebuilt.
notpushkin 10 hours ago [-]
I think the whole Windows client is closed. On macOS though you can use it from the command line just fine (apart from a couple quirks due to a completely different VPN implementation [1]).
"This repository contains the majority of Tailscale's open source code. Notably, it includes the tailscaled daemon and the tailscale CLI tool. The tailscaled daemon runs on Linux, Windows, macOS, and to varying degrees on FreeBSD and OpenBSD. The Tailscale iOS and Android apps use this repo's code, but this repo doesn't contain the mobile GUI code."
and
"The macOS, iOS, and Windows clients use the code in this repository but additionally include small GUI wrappers. The GUI wrappers on non-open source platforms are themselves not open source."
On the other hand, while I suppose the Windows app is probably reasonably straightforward to replicate, I guess it would be much harder to produce an iOS or Android app because of the vagaries of mobile programming.
pilif 9 hours ago [-]
> I guess it would be much harder to produce an iOS or Android app because of the vagaries of mobile programming.
on iOS you also need a special entitlement that's only available on specific request and only to known developers, so practically impossible for any open source project to acquire.
dcow 8 hours ago [-]
This was true in 2015. It is not true anymore.
notpushkin 10 hours ago [-]
Thanks, I stand corrected then!
Android client is open source (and you can get in from F-Droid, even), so that only leaves iOS I guess.
freedomben 7 hours ago [-]
Yep, Tailscale takes a pretty reasonable approach to that IMHO. Open source on platforms that are open source. I think that works out pretty well because it meets people where they are. For example the people who care about open source (and thus are running linux or android) get their open source needs met, and people who don't care about open source strongly or at all (as evidenced in part by them running closed/proprietary OSes) such as mac or windows users are also met where they are. Of course this also helps protect their business model because then competitors can't just take the open source versions and run off with them, and the number of linux users is quite small compared to mac and windows so it keeps the majority of the client closed while still providing the openness to those who truly care about it.
*In my perfect world everybody would care about open source, but the evidence is pretty clear that only a tiny minority of people actually do, even among engineers
sixothree 9 hours ago [-]
Tailscale clients are the thing I am least happy about with Tailscale. Specifically mobile clients and battery usage.
The reason I can't use Tailscale at work is because it routes traffic through servers we can't control.
I would _love_ to use tailscale at work. It would solve so many problems. I am okay with being forced to open ports. But tunneling traffic through them is extremely worrysome.
techsupporter 9 hours ago [-]
> The reason I can't use Tailscale at work is because it routes traffic through servers we can't control.
yes. Battery usage is super bad, mainly because of their DNS features which forces every DNS resolution to go through their network extension. At least recent updates have stopped the background power usage when you disconnect from the network in the app.
>But tunneling traffic through them is extremely worrysome.
it only does that in case of super bad NATs that make the usual NAT traversal techniques impossible. And presumably, the traffic is end-to-end-encrypted, so it doesn't matter if they have to be in the loop.
If you don't trust them to properly end-to-end encrypt, then it really doesn't matter whether they are in the loop for forwarding a packet or not because if you don't trust them to encrypt properly, all bets are off to begin with.
If you trust them however, it doesn't matter where the traffic is flowing through because only the intended machine is able to decrypt it.
dcow 8 hours ago [-]
On the battery topic I’m curious if you have anything more than anecdotal evidence. A basic full tunnel wg network extension doesn’t affect battery in a noticeable or unacceptable way, in my experience. Is tailscale’s implementation doing more in a way you can isolate and attribute to poor battery?
sixothree 8 hours ago [-]
I can see it (tailscale) in my battery usage on multiple devices. 20 hours of background usage per day is a bit much if you ask me.
CharlesW 7 hours ago [-]
FWIW: On iOS 18.4 my Battery report for the last 10 days is ~128h of background activity, using ~2% of my battery life.
3abiton 21 hours ago [-]
This looks interesting! What's the added value over wireguard + openwrt setup?
watusername 20 hours ago [-]
Your devices will connect to each other peer-to-peer (even behind complex NATs) with no manual configuration, subject to ACLs you centrally manage. It just works.
People sometimes dismiss Tailscale as "just" a WireGuard orchestrator, but it's actually much more than that - From a product perspective, WireGuard is just an implementation detail.
compootr 20 hours ago [-]
it's wireguard that doesn't make me hate myself :)
usagisushi 20 hours ago [-]
It's a mesh VPN, so peers communicate directly without additional delay.
I opted for Netbird myself because Headscale's UI felt too basic for me back then. Has that improved over the years probably?
udev4096 17 hours ago [-]
How is netbird? Is it more stable than tailscale/headscale? How is your performance while streaming a video?
avtar 20 minutes ago [-]
Netbird seems (or perhaps is?) newer. It didn't have some basic features baked in when I last looked into it, e.g. you couldn't switch accounts on the client https://github.com/netbirdio/netbird/issues/3273 and if I had an account associated with a single team, then that account couldn't be invited to or be associated with additional teams.
usagisushi 12 hours ago [-]
They are both based on WireGuard (kernel-space and user-space `wireguard-go`), so I guess there's no significant difference in performance for typical usage.
In terms of stability, Netbird has been pretty good for me. I've been using Netbird as the backhaul network for my laptop, phone and inter-site k3s cluster for several years without major issues.
One major downside of Netbird is that its Android client can be quite a battery drainer [1]. (It keeps your fingers warm during winter, though!)
As for Tailscale, it offers some neat features like Funnel, which is missing in Netbird, but in my case, covered by DNS and k8s Ingress.
Tailscale's value prop is "Wireguard that the merely somewhat-technically-inclined can set up and manage unassisted". Across tons and tons of clients (my AppleTVs connect to my Tailscale network, this took maybe a minute to configure—and they can act as gateways)
sunshine-o 16 hours ago [-]
Some do not want/have a fixed IP address or anything listening on their home network.
Tailscale or having Headscale hosted somewhere else allows you to do that.
20 hours ago [-]
1vuio0pswjnm7 5 hours ago [-]
"To me, not giving your Tailscale implementation any way for the user to understand or veto what the control server is instructing the clients to do while also not auditing your servers code at all sure seems daring..."
This statement sugggests that publishing the Headscale control server source code is not enough to allow the user to "understand or veto what the control server is instructing the clients to do".
If using the Headscale control server, the user can "understand or veto" anything "the control server is instructing the clients to do". This may be accomplished by reading, editing and compiling the source code.
If using the Tailscale control server, the user can only "understand or veto what the control server is instruction the clients to do" to the extent that the Tailscale company permits. The user is prohibited from editing or compiling the source code.
Not all users want the option to read, edit and compile third party software that they use. Some users may be comfortable relying on the ongoing assurances of companies funded by Silicon Valley VC. For those users that want the option of 100% open source projects, not dependent on venture capital, Headscale can be useful.
The author of Headscale calls the Tailscale coordination server "essentially a shared dropbox for public keys".
snvzz 20 hours ago [-]
Headscale has been serving me well for half a year now. It is great, to the point I have no idea how I lived without a tailscale network before.
It is packaged in openbsd, and that package is the server I am using.
udev4096 17 hours ago [-]
How does headscale hold up when you're streaming video over jellyfin/plex?
scottyeager 17 hours ago [-]
Do you mean when using it as a relay because p2p connectivity isn't possible? The preferred operating mode of Tailscale networks is for the bulk of traffic to go p2p, using various tricks for NAT and firewall traversal.
cassianoleal 15 hours ago [-]
I’ve used it extensively to stream video across continents. No issues as long as you can get a P2P connection going. If it needs to go through a DERP server, then it may suffer but in my experience that’s pretty rare.
watusername 9 hours ago [-]
> If it needs to go through a DERP server, then it may suffer but in my experience that’s pretty rare.
It's semi-frequent in my case, and it's painful every time it does that since Tailscale's official DERP servers are very slow (they seem to have some aggressive QoS). It would be nice if Tailscale supported using regular TURN servers so I could just use one of the hosted solutions.
cassianoleal 9 hours ago [-]
You can self-host DERP if you're up for it.
LilBytes 9 hours ago [-]
Yep and most of us are already using Subnet routers it's not technically much harder.
Finding a cloud or VPS provider with free or cheap bandwidth (egress and ingress) is likely the biggest issue.
pluto_modadic 20 hours ago [-]
wonder if some of the bugs with self-managing it have been worked out :)
Rendered at 22:51:06 GMT+0000 (UTC) with Wasmer Edge.
The entire premise of Tailscale SaaS builds on creating tunnels around your firewalls, then enabling the user to police what is allowed to be routed through these tunnels in a intuitive and unified way.
Headscale seems to have nailed down the part of bypassing the firewall and doing fancy NAT-traversal, but can they also fulfill the second part by providing enough of their own security to make up for anything they just bypassed, or will they descend to just being a tool for exposing anything to the internet to fuck around with your local network admin? To me, not giving your Tailscale implementation any way for the user to understand or veto what the control server is instructing the clients to do while also not auditing your servers code at all sure seems daring…
Did they really roll-their-own for those functions? I thought this was just a control layer on top of Tailscale’s stock services on the backend, are they facilitating connections with novel methods? Apologies if I’m asking obvious questions, I use ZeroTier pretty regularly, but I am not too familiar with Tailscale.
Also I believe it implements a lower layer of the network stack so more options are supported, though I haven't needed to investigate in detail.
https://netbird.io/knowledge-hub/tailscale-vs-netbird
Tailscale is still running for now, but I'm getting closer and closer to decommissioning it and switching entirely to Netbird.
So here's my proposal: commit to ipv6-only overlay network in the unique local address (ULA) range, then split up the remaining 121 bits into 20 low bits for device addresses (~1M) and 101 high bits that are the hash of the server's public key. Federate by adding the public key of the other instance and use policy and ACLs to manage comms between nodes.
I think it's a nice idea, but the maintainer kradalby said it's out of scope when I brought it up in 2023: https://github.com/juanfont/headscale/issues/1370
Headscale has been on HN many times.
We are still running the old headscale, because we have some integrations that will need to be ported to the new control plane. According to "headscale node list | wc" we have ~250 nodes, most of them are servers.
One thing I really don't love about tailscale some of the magic it does with the routing tables and adding firewall rules, but it has mostly not been an issue. Tailscale has worked really quite well.
We considered it as well but there was a feature missing that meant we couldn’t use it for one of our main requirements. Had that not been the case, we’d have rolled it out.
Sorry I can't provide any more details but I really don't remember the specifics. We were in touch with Tailscale engineers and they offered some workarounds that we had already worked out but that wouldn't help us achieve what we were after.
Especially in the unlikely event that you used Nix for the deployment.
There are a couple open source dashboard options but right now only this one comes to mind: https://github.com/tale/headplane
The moment the inevitable enshitification will start at Tailscale, this feature will go away.
I’m saying this as a currently super happy Tailscale customer who was burned multiple times in the past by other companies being sold or running out of VC money
That feels right to me. Headscale is mostly used by home labbers and small hobby users, it competes with self-hosted OpenVPN and WireGuard, not Pulsesecure, Cisco Anyconnect or GlobalProtect. It's a way to introduce Tailscale to people who love to try new shiny tech in their spare time, but don't want to give up control over their infrastructure.
Those people will then bring their Tailscale expertise and enthusiasm to work. Work really doesn't like managing IT infrastructure unless it's one of their core competencies.
Sure, some companies will actually choose Headscale over Tailscale proper, but I suspect that's a small minority (especially if you take company size and the money involved into account). That's just cost of revenue, not unlike Facebook advertising or billboards on the side of a road in Silicon Valley.
I have the same memory. But they may not feel that way forever. Many a company started by attracting developers with a generous free tier or open-source offering, then started to clamp down once the going got tough.
Heck, it happened to one of Tailscale's competitors, ZeroTier, which used to release their client software under GPLv3 but eventually switched to BSL.
[1]: they have three: https://tailscale.com/kb/1065/macos-variants
"This repository contains the majority of Tailscale's open source code. Notably, it includes the tailscaled daemon and the tailscale CLI tool. The tailscaled daemon runs on Linux, Windows, macOS, and to varying degrees on FreeBSD and OpenBSD. The Tailscale iOS and Android apps use this repo's code, but this repo doesn't contain the mobile GUI code."
and
"The macOS, iOS, and Windows clients use the code in this repository but additionally include small GUI wrappers. The GUI wrappers on non-open source platforms are themselves not open source."
Moreover, there's https://github.com/tailscale/tailscale-chocolatey to aid the build process. I haven't built it or run it.
On the other hand, while I suppose the Windows app is probably reasonably straightforward to replicate, I guess it would be much harder to produce an iOS or Android app because of the vagaries of mobile programming.
on iOS you also need a special entitlement that's only available on specific request and only to known developers, so practically impossible for any open source project to acquire.
Android client is open source (and you can get in from F-Droid, even), so that only leaves iOS I guess.
*In my perfect world everybody would care about open source, but the evidence is pretty clear that only a tiny minority of people actually do, even among engineers
The reason I can't use Tailscale at work is because it routes traffic through servers we can't control.
I would _love_ to use tailscale at work. It would solve so many problems. I am okay with being forced to open ports. But tunneling traffic through them is extremely worrysome.
You can run your own DERP servers and exclude the Tailscale ones even if you don’t run your own Headscale server: https://tailscale.com/kb/1118/custom-derp-servers
yes. Battery usage is super bad, mainly because of their DNS features which forces every DNS resolution to go through their network extension. At least recent updates have stopped the background power usage when you disconnect from the network in the app.
>But tunneling traffic through them is extremely worrysome.
it only does that in case of super bad NATs that make the usual NAT traversal techniques impossible. And presumably, the traffic is end-to-end-encrypted, so it doesn't matter if they have to be in the loop.
If you don't trust them to properly end-to-end encrypt, then it really doesn't matter whether they are in the loop for forwarding a packet or not because if you don't trust them to encrypt properly, all bets are off to begin with.
If you trust them however, it doesn't matter where the traffic is flowing through because only the intended machine is able to decrypt it.
People sometimes dismiss Tailscale as "just" a WireGuard orchestrator, but it's actually much more than that - From a product perspective, WireGuard is just an implementation detail.
I opted for Netbird myself because Headscale's UI felt too basic for me back then. Has that improved over the years probably?
In terms of stability, Netbird has been pretty good for me. I've been using Netbird as the backhaul network for my laptop, phone and inter-site k3s cluster for several years without major issues.
One major downside of Netbird is that its Android client can be quite a battery drainer [1]. (It keeps your fingers warm during winter, though!) As for Tailscale, it offers some neat features like Funnel, which is missing in Netbird, but in my case, covered by DNS and k8s Ingress.
[1]: https://github.com/netbirdio/netbird/pull/3379
Tailscale or having Headscale hosted somewhere else allows you to do that.
This statement sugggests that publishing the Headscale control server source code is not enough to allow the user to "understand or veto what the control server is instructing the clients to do".
If using the Headscale control server, the user can "understand or veto" anything "the control server is instructing the clients to do". This may be accomplished by reading, editing and compiling the source code.
If using the Tailscale control server, the user can only "understand or veto what the control server is instruction the clients to do" to the extent that the Tailscale company permits. The user is prohibited from editing or compiling the source code.
Not all users want the option to read, edit and compile third party software that they use. Some users may be comfortable relying on the ongoing assurances of companies funded by Silicon Valley VC. For those users that want the option of 100% open source projects, not dependent on venture capital, Headscale can be useful.
The author of Headscale calls the Tailscale coordination server "essentially a shared dropbox for public keys".
It is packaged in openbsd, and that package is the server I am using.
It's semi-frequent in my case, and it's painful every time it does that since Tailscale's official DERP servers are very slow (they seem to have some aggressive QoS). It would be nice if Tailscale supported using regular TURN servers so I could just use one of the hosted solutions.
Finding a cloud or VPS provider with free or cheap bandwidth (egress and ingress) is likely the biggest issue.