The next step for VPNs w/ Carl Dong from Obscura

Like, how did the idea of Obscura itself come about? How'd you end up here?

I was actually at Blockstream before that partly overlapped when I was in college at Berkeley. And when I was a Bitcoin core developer, I mainly worked on the Reproducible build system.

Now, for people who are unfamiliar with reproducible builds, what that basically is is when you download, let's say, Bitcoin Core's app, or Bitcoin D, or you Download the disk image, you download something that's a binary and then you have the source code that's on the Bitcoin core repository. So how do you know that the source code is what you're downloading?

How do you know that, you know, somebody didn't tamper with the source code before they, you know, gave you the download and things like that. And so that's what reproducible builds solves. And it was solved previously with the Gitian system, that dev random setup. I improved it in the sense of we were trying to solve for the trusting trust attack. So the trusting trust attack is this attack that was posed by Ken Thompson, who made the UNIX operating system.

And it was an attack whereby, if you can poison, if you think about compilers, right, compilers, okay, compilers are compiling something. But where do compilers come from? They come from a previous generation of compilers and it's sort of like, you know, turtles all the way down, right?

So what Ken Thompson posed was that if you poison one generation of compilers in a very smart way, you can actually have compilers that compile poisoned compilers, even if the compiler code is correct and sort of go all the way down. So that was sort of a supply chain attack. And nowadays software supply chain attacks are everywhere because people have found out it's the easiest way to get your bug in instead of trying to figure out, oh, how do I buffer overflow this thing?

You can just do a supply chain attack and, and you don't even need to worry about that. And so I always sort of bring a solution for that trusting trust attack to Bitcoin's reproducible build system, making it what we call bootstrappable with geeks. And now that is the release process for all Bitcoin core releases going forward. And so I, you know, throughout my career it's, it's, it's always been a focus on, you know, obviously user privacy and also about trust, right?

How, how do we make it such that users can trust our, what we build and, and the things that we distribute? I think that's especially important in a digital world where, you know, nobody's going to like disassemble gigabytes of, you know, gigabytes of binary just to know what's going on. You really need to have a good process and good story around that.

And so, you know, after working on Bitcoin core, I was looking around for things to do and things in the industry. And thankfully enough, I think it was Matt Carollo who was one of the other Bitcoin core developers. He told me he knew that I had sort of an interest in VPNs for a long time and he told me to look into Apple iCloud Relay. And I looked into Apple and I, you know, perhaps naively thought that, oh, you know, it's, it's an Apple thing.

When I first heard about it, I was like, oh, it's an Apple thing. It's probably just, you know, some ordinary vpn, oh, it's only for Safari, you know, what is it? But he was like, okay, actually look into the tech behind it. And I, I was really enthralled really by how they designed it. It was sort of a hidden gem almost. And there are two parts to this. One part is, you know, having two party. It's not only two hops, right? It's more than two hops, it's more than multi hop.

Because if you have multi hop, if it's the same provider that, you know, what, what are you gaining really, right? It has to be two separate parties, two separate entities, two party hop, whereby the first entity sort of knows who you are but doesn't know anything about your traffic and the second exit entity doesn't know who you are and serves you the traffic. And I think in Apple's case it was Apple as the first hop and cloudflare Akamai or Fastly as the second hop.

And that was how they designed their scheme. So I thought that was pretty ingenious. And I looked into the underlying sort of protocol that they were building, which was Masque M A S Q U E which is amazing name, amazing acronym. They backronymed it, I'm sure, and it was fantastic. And I looked into it and I actually implemented my own version of the Mask UDP protocol in Rust and go, just to see how things worked and everything.

And I went to the IETF to talk to the working group about this and sort of just to get to know people and you know, they were, I had a great time. They were networking nerds like I was. Right? But that's sort of not the only thing that was interesting about Apple iCloud Relay. The other big thing that was interesting about Apple iCloud Relay was the way that it did the transport layer. It used quite quick, it used quick and http3.

And I thought that was really ingenious in a very subtle way that people perhaps can't appreciate at first. So I grew up for a part of my childhood in Shanghai in China. And well, actually I was born there. I grew up There. And then in primary school, I went to London for a few years and then I came back to Shanghai for three years. And that was sort of when, you know, in London I was used to Facebook and YouTube and, you know, all these newfangled Google and all these things.

And Wikipedia, I mean, Wikipedia, think about that. And then coming back to Shanghai, the great firewall had just came up and I was like, okay, I have access to none of these things, right? And so I had to, you know, do my own, you know, build my own VPNs for friends and family, try out some of the commercial ones. I think back then the protocols were very nascent. It was like L2TP and PPTP. I think those were very ancient algorithms.

And so I followed the journey, the evolution of all of these VPN protocols all the way up to OpenVPN and Wireguard and things like that, right? And certainly, you know, I've been in the States since I was in high school, but every time, you know, I go back, visit family, I would test, let's say, my new new ways against the great firewall and see what, what's going on.

And I think one of the things that I realized was that for these network filters, let's say for, for all of these network filters, what they want to do at the end of the day is still, they still want to let benign traffic through. They want to let good traffic through, right? Like, you know, they want to let apple.com through, let's say, at the very least, right?

And so the best way, and this is sort of out of the community of there's a huge community on GitHub of basically either expats like program software engineer expats in China, or, you know, just networking nerds and engineer software engineers in China who are building these tunneling protocols. And the thing that they have found is that obfuscation is key. Obfuscating your traffic and making your traffic look like benign traffic is really the best way to get around it.

There's sort of a mismatch, let's say, between people who build VPN protocols in the first world and people who build VPN protocols behind a firewall, let's say in that people who build not behind a firewall are always thinking about how can we make the cryptography even better and more indistinguishable and the packets and whatever. And to a firewall they're like, okay, this stream looks too random. You know, like looks, you know, too well protected.

And so I'm just going to like, Cut off your connection and we'll see what happens. You know, and so it's really funny the different approaches to things. And so bringing it back to Apple iCloud Relay. By using Quic and HTTP 3 as transport, they now look more or less like the rest of Internet traffic, which means they'll get around more firewalls. Not to mention the fact that, you know, we can get into it later. I've already gone too long on this question.

Now that it's over udp, which means that there are massive improvements in terms of performance and latency and things like that.

Like you have a very different set of assumptions and things that you need to work around. Which means that, that there's a lot more, really more fascinating stuff there. I mean honestly, VPNs for the most part in like the western world, a lot of people are just using them to get around geo restrictions. So it's really just speed and changing the location. There's not nearly as much thought about the more complex blocking of VPNs as there is when you're facing something like a firewall.

Nowadays like if you go into a cafe or going to an air like airport, public WI fi or any kind of public WI fi or your coworking space or your hotel, they probably had some guy that was in the admin portal and saw it, Cisco Advanced protection or something like that and just take that box without knowing about it. I'm not saying all of these, you know, admins are malicious or trying to control you or whatever, just ticking a box and they tick a box and now your VPNs don't work. Right.

And these are like the networks where you want to use a vpn. I'm not trying to, you know, I want to use my VPN on a public network. Right. And most of these Public networks have these kind of advanced protection stuff. Um, and so, yeah, it's not only just relevant for people who are behind a firewall.

There's a lot of, obviously there are a lot of legitimate reasons to want to block VPNs or to block custom DNS from a system admin perspective, but obviously for the end user, especially one who's relying on VPN for personal privacy, it's a, it's a big detractor from the usefulness of vpn. So that's definitely a key aspect there.

Obviously it has the downside it is closed source, but it's pretty amazing to see the actual technology that they built. And so it's cool also to see that that kind of inspired some of obscura as well.

And I think they were one of the few where they had articles that were like, where are all the multi party relay VPNs? And talking about icloud relay and being like, this is great, like why aren't more people building this? I was like, yes, that's. Yeah, that's what we're doing.

They have a lot more incentives to sell that data. Mobile doesn't. So that's been like the normal paradigm that we've gotten used to with the really the only alternative up to this point being the Tor network, which is a very different set of assumptions and very different set of protections and threat models that you would want to use it for. So I think that something like Obscura kind of fits in pretty well in between that model. So what was it like?

What, what made it so important to you to make Obscura and to kind of shift this paradigm? Like maybe tell us a little bit more about Obscura here. Cause I feel like we've been beating around the bushes a little bit about what it is, but tell us a little bit about what it is and why you thought that was a necessary like missing piece in the ecosystem.

Now we've partnered with Mullvad because we think they're one of the most trustworthy VPN providers out there. And so with this two party scheme, as I talked about before us, as Obscura, we know who you are because we have to know that you paid, but we literally can't see anything about your Internet traffic, sort of, not even the metadata or anything like that. Right.

You know, even with tls, you know, there's the SNI indication, which means that's the, the unencrypted part of TLS packets that tells you the host name. And so we don't know anything about your traffic. And on Mobat side they don't know anything about you. They just know that you're one of, I don't know, a few hundred Obscura users and they serve you the Internet traffic as they go to google.com and YouTube and things like that. And so that's what Obscura is and that is what we've built.

And right now we are on Mac OS and I'm sure we'll go into sort of different platforms and all these things. But what we do, let's say as a service is we've adopted Mulvad's account number scheme so that you know, you don't need to provide an email address or anything to sign up. And we also accept Bitcoin over Lightning for payments because we don't like, you know, we see personal information as toxic waste and we want to collect the least amount possible about you.

We just want to offer you a good service.

And that I think the, the payment in Bitcoin or something like Monero is such a vital part of it as well because like you said like you especially if you weren't in the multi vendor model that you're doing now or two party system, it's even more problematic if you're giving over credit card info or shipping address or not shipping address but billing address to, to a VP provider because then they have basically the same info on you that an ISP does and you haven't really removed any trust

at all. You've just.

Like how do you see it comparing in terms of like who would use this versus a traditional VPN or something like a mixnet, like kind of the broad spectrum.

Right now I'm connected to Obscura talking to you, right? And I think the latency. Okay, well there we go, perfect. And I think the latency is pretty good. And I use it for, you know, all my calls. I use it to, you know, watch 4K HDR video. I want to make sure that the thing that we want to change is the default for the Internet, right? And to change the default and for people to adopt a different default, you can't be worse or you can't be substantially worse than the existing experience.

And so when we think about Obscura, that's sort of the niche that we fill in, that people who want to use it for everyday use, right. And so when it comes to the everyday use and the speed and the reliability in terms of mixnets or Tor, their advantages in terms of privacy and sort of freedom of participation is a double edged sword almost.

And that, you know, it becomes where, okay, you're not sure if any particular node is going to be that reliable, is going to be sort of that well maintained and things like that. And perhaps that makes it infeasible for daily use at times. Although I will say, like, I'm a big fan of these projects, right? I'm, I am all for new projects that advance Internet privacy and everything like that. And so.

Which I think Obscura has been filling that role really well for me. It is a cool test case that we're recording this both over Obscura, so should be a cool example of how, how low latency it is. We have video between the two of us as well, even though that's not recorded. So it's a cool test for that. But it is definitely one of those situations where it, it's, I mean, it's just kind of the crux of how useful will A privacy tool be is how good is the user experience?

And I think that's where something like Signal has just destroyed a lot of competition because they're just really easy to use. They also provide very strong privacy, and the default is strong privacy, and that's where a lot of these privacy tools really have to get. So I definitely, definitely agree on that.

And then do you have plans to expand to allow other exit nodes or exit partners in the

future?

I thought it was so ingenious. I was like, yeah, why did. Why do they need my email for this? Why? There's no reason they need my email for this. You know, and if I, you know, if I need support, I'll just email them. But like, you know, otherwise there doesn't need to be anything done with that. And it's just a random number, so you're not like fiddling around with some password generator to generate a number. Amazing.

And, you know, and also they make hard decisions, almost counterintuitive business decisions for better OPSEC and infrastructure security. Like, I think they're retiring OpenVPN pretty soon if they've already retired OpenVPN. I mean, I remember when I was looking into OpenVPN and the implementation and things like that, I was like, this, this is, like, this is great that it has all these features, but it's too configurable. And I'm sure there's a bunch of CVEs in here.

Like, this just doesn't look like it's going to be so, you know, retiring, things like that. They've been a real pioneer and that's why we chose them. I've also had a good, good, really great relationship with them. I remember the first time I reached out to Jan, their CEO. I think this was back in 2020 when I was first starting to resell Mulvad vouchers for Lightning. And I reached out to him and we communicated over PGP encrypted email. I was like, this is pretty, like this is pretty cool.

There are dozens of us going, yeah, exactly. So they really, they get it, they're really about it. And so we wouldn't have chosen anybody else to be our first. Now, in terms of giving users a choice and, you know, other providers for the exit and things like that, we really hope that we will be able to do this. I think, you know, one of my, one of my dreams, and perhaps, perhaps this is too optimistic a dream, is that I hope that we can elevate the VPN industry.

I hope that we can elevate the VPN industry to a place where perhaps all of the VPN providers could be second hops for each other and be interoperable with each other. Now that requires making, as I said before, hard decisions, let's say, about, you know, your business and things like that. But I think if we can get to that place, no matter how we get there, that would make the entire industry way more credible.

Right, Way more credible in terms of, you know, the claims that they make in terms of the base level of privacy. And we're actually doing something great for the health of the open Internet and I think that would be great. So I hope we get there and certainly, you know, I'll be reaching out to folks.

But I think it's been, it's been fantastic so far, kind of on that vein. So obviously it's going to be a little bit of a different bandwidth and latency system, I guess you could say, than if you're just using something like mulbot. Have you done any like hard numbers on what the expectation is in like latency increase or bandwidth loss for doing this two party system? It doesn't seem like much in my testing, but I'm just curious if you have any kind of real comparison there.

I think the real test is like twitch stream, low latency mode. Right. Because like YouTube you can buffer and things like that. Right. And with, you know, even 4K, do Twitch have 4K? At least the 1080p 60fps streams I'm able to stream just fine.

But obviously it is still possible for Obscura and Mobot to collude. Is there any technical prevention there to prevent that collusion or is it just kind of a system where you have to trust that these two entities won't both collude knowingly to collect this data that could have been collected by just a traditional one party vpn?

And this session is established between your wireguard key pair that's local and Mullvad's key pair that's on the servers and these keys are on their server page. You can literally verify these keys are the keys that Mulvad is using. And on a technical standpoint, I don't know anybody, any admin anywhere that will give their wireguard private key to a new partner. That's a small, like, it just doesn't make any sense, let's say from that standpoint.

And so, you know, this is, this is a reason why we designed our protocol. We could have gone full mask, which is, you know, over HTTP 3, you know, doing their whole custom encryption and things like that, and authentication and things like that. But we chose to go with sort of tunneling wireguard over quic, our custom protocol, so that it is easier for the user to verify, hey, I'm directly connected to Mulvat. This is not some like software tricking me or anything like that.

That's how it works now. You know, from, I think from a business standpoint, I think it's, it's a good point. We, this is sort of why we were so careful in choosing our first exit partner. Right. We wanted someone who was well known for being trustworthy. Right. And I think we, they would not. A company like Mulvad would not stake the reputation that they built over the years for a small new partner like us. You know, not to mention that they'd have to share their wireguard private keys.

It just wouldn't make any sense. Is, is the real answer.

And I don't blame them. It like took a while so we'll want to have reproducible client. I think client code should always be open source. Especially for something as, as, as, as critical as you know, a VPN that like you know, has. Handles all of your packets. Right. God knows what, what, what it's doing with them. It should always be open source.

I think one interesting thing to note here is that on macOS and on Apple platforms, they have this API called the Network Extensions API, which is what we use. And this is how normally VPNs get set up. And with the Network Extensions API, our network extension that handles the packets run separately from the app as what they call a system extension. And it's very interesting on a technical level. It runs as root, but it is sandboxed, which is a crazy thing that only macOS can do. It just.

I think the UID is zero, but it can't access any of your other files, which is nice, I think. Right. Like, if you have a piece of code that's handling all your traffic and is somewhat privileged, you want to lock that down as much as possible. So I'm happy to see that macOS took steps on that front, I think. I'm not sure that there's a great story for Linux there.

I know that there's some progress being made with Landlock and things like that, although I think the Landlock API is still very nascent. I just saw an interesting thing called Land Run the other day that helps with that. But yes, we want to be as transparent with our users as possible. And I think it's definitely important that our client is open source.

For example, like, you know, for a lot of our wait lists and things like that, we allow you to use PGP keys and submit your BGP key so we. That we can, you know, encrypt that message. I sort of wrote up the first version of that without implementing how I would send it later. And when I, when I ended up having to send things, there was a lot of Python scripts and trying to mangle things and trying to get the encryption correct.

And a bunch of people, you know, who you are, submitted keys that were either expired or not on any key server. So I was like, I don't even know what to do with this. So we'll see. But yes, I mean, like, you know, at every turn we try to think about how can we implement this feature in a way that is private for the user. And it's been an exciting journey for sure.

And of course, I think with what's exciting about some of the features that we're going to be working on after we get more platform support is that we'll be able to work on interesting UX problems in the, in the VPN space. For example, split tunneling. I think split tunneling is one of those things that is really crucial for us to get to a place where we offer our users something that they could use without turning it on and off. And the normal finagling that you have to do with VPNs.

And so we're thinking about, okay, at what abstraction level could we do this? Is this sort of on the addressing layer, on sort of the socket layer in the browser with a different profile. So we're experimenting with that. One thing that I forgot to talk about actually about things that are interesting that we're working on is with our obfuscation, we're currently already doing really well in terms of it's working in most network scenarios.

There is one scenario in which there is still some that we're still lacking in and that is when the network admin is really not the smartest and has said, well, maybe they know what they're doing. They're like only TCP over 480 and 443. There are networks like that. I have gone back and forth with a few users where it's just like that. And it frustrates me to no end because then you're not loading your websites quickly over quick.

And they, but anyway, we, you know, the reason why we went with QUIC was to avoid the TCP over TCP meltdown, which is why with a lot of TCP based VPNs you actually see a bunch of stuttering. What we're, what we've actually found is this is sort of, this is again one of those performance performance things that, you know, our engineers have found a good excuse to work on and they're like, yes, let's do this.

We found a way to tunnel packets over TCP in a cooperative way between the client and our servers, such that we avoid TCP over TCP meltdown by sort of dynamically resizing buffers and dropping packets and things like that. So I think that's really exciting, let's say. And we'll make sure that, you know, we even work in situations where they only allow TCP over 443.

But I've heard a lot of requests for Monero from the users and I was just looking into it a few weeks back, sort of the logistical complexities of supporting it doesn't seem to be that much. So I'm going to continue looking into it. I think it is inevitable that we add Monero payment support to Obscura and I look forward to announcing it the day that we do.

Any idea on when that's coming?

And I think that's quite an important feature to have, let's say, for people. And certainly one of our engineers has been complaining about it as well. So you can be sure that, that.

You know, we've architected our infrastructure such sort of the different modules of our code such that it is more or less reusable across the platforms. But there are still some integration problems to do what I'm really happy about with. But we'll definitely do Android because all of our engineers are on Android and have been complaining forever. And I will say that right now we're working on a wireguard configuration generator.

Now, of course, with that wireguard configuration generator, you do get the privacy of you're not able to correlate your IP with your Internet traffic, but the obfuscation side is sort of lost because, hey, it's vanilla wireguard, what can you do? But it should be able to serve most people who are on Android.

But one of the other things I want to say is one of the things I was really happy to see was a bunch of users actually reached out to me and was like, hey, I know you're going to be on X platform Android, let's say. And here are all the pitfalls with integrating with Android's VPN subsystem and things like that. I'm like, thank you. You just saved me, like, you know, a few days of engineering time trying to figure this out. So I'm happy.

Please reach out if you have insights about VPN integration on different platforms. For sure.

The Mac experience has been fantastic from day one, and I think that shows that y'all understood if we tried to do everything at once, we're going to overreach and there are going to be things that suffer. And I think that that kind of. That focus showed for me in the initial launch and gives me, again, just good hope that y'all are going to do a really good job on other platforms as well as you have time.

And I think, you know, this was more, you know, last or last, last generation of privacy products in that consumers sort of have this thing of like, oh, if it's going to be the more private alternative, it probably is much harder to use than the other way around. And I think, you know, the privacy tech industry needs to come together, and I think, you know, Signal started these things and things like that.

It needs to come together and make sure that that is not the case, because really, we can solve a lot of these problems if we just invested in the tech, the cryptography, the different ways that we do things. And that's one of the things that we hope to drive forward as well. And that privacy products can also be the most usable product and that put together.

we do have some listener questions we can dive through a little bit. Um, I think some of these we already answered, so I'll skip some and I'll just, I'll drop people who did ask these questions. Links to the right timestamp in the episode. Um, but one of the, I think good questions that was asked by Mark Molly on X was he was curious about knowing what the biggest technical challenge you had was building it, and what challenges you anticipate facing next in this kind of privacy arms race.

And you inevitably run into corner cases where you have to go to the developer forums and then somewhere some guy from two years ago said something that, that helps you along. And, and I think it speaks to the fact that, you know, these closed platforms, sometimes they have huge innovations, right? Like icloud private relay, like the different ways of sandboxing on macOS that's good for security, good for privacy and things like that.

But these, these APIs are really hard to work with, really opaque. When it fails, it's really hard to discern what's going on. In fact, we had a few bugs where. I mean, I was desperate enough that, you know, I was attending the IETF at the time. The. The IETF conference. The IETF is the body that makes Internet standards like HTTP 3 and Qwik and all those. And funnily enough, it was just a bug that we really couldn't figure out.

And I knew that someone from Apple was the guy that would know the answer. And so I just waited in front of the only escalator in the conference for him. I think I bothered a couple people that, like, kind of looked like him, but, like, wasn't him. But I finally, like, saw him and I just like ran up to him and be like, hey, like, we have this problem. And he finally was like, okay, look for these logs, look for this. And, you know, showed us how to do it.

And so these were the, these were the challenges. In terms of the other stuff, I think, you know, we have a really superb engineering team who's able to truck forward, right? Like, if there is a problem with our dependencies, we're not up beyond you know, just patching our dependencies and looking at things like that. It is when there is the opaque. I think I was talking to one of the guys at Google who worked with the APIs, he said it this way.

I like this analogy of the Apple platform is you have the XNU kernel which is open source. Like I actually didn't fully grasp that at first, but the XNU kernel is fully open source, right? And then you have the APIs that the apple opens you, that gives you. And then you have this fog of war in the middle. Like, you know, if you play a strategy game, there's fog of war, right? Fog of war in the middle of how they, you know, interact with each other that you have no idea about.

And all you can do is twiddle the APIs on the API end and see what happens on the XNU end. And so I like that analogy. And you know, it's sometimes the problems with closed platforms for sure.

we, we're always on the lookout for different ways to do stuff and, and of course port forwarding is one of those things that people request and rightfully request for peer to peer file sharing. We, as of right now, we can't do it because Mulvad doesn't have support for port forwarding. And at the end of the day what is exiting out to the Internet is Mullvad's servers. And I think that it is correct in that when you allow port forwarding there is the problem of abuse, right?

It is very, you know, and we don't want to know anything about the user. So we're not trying to like go visit everybody's open port and try to snoop on what they do and all those things. So it's a fine balance to make.

I think there's something to be done, let's say of a, of a hybrid, you know, consumer VPN versus tail scale where you know, if you connect it on your home device or your home server or whatever, you know, you can connect to Obscura using your phone and be able to go, go visit Your home server and things like that. Yeah, I think that's something that we'll definitely look into in the future.

And so good people suffer because sometimes that has to be cut out so that you can still offer a product. So I know that that's a really hard thing to offer in the space, but good insight.

Whereas if you go up a level and use things like UBlock origin, even though I know that on Chrome now they. The manifest v3, all of those things, if you use something like UBlock origin, I think that's a better way to go about doing things and may confuse the JavaScript on the page a little less. But yes, we plan to support this for sure.

How are you prioritizing those two in terms of like what you're trying to bring out first?

And, and I think for Linux what we'll likely see is first a CLI version so that we can get something out really quick and then a GUI. Certainly GUI's on Linux there's a whole, I don't know if you're familiar, there's a whole fight about GTK versus QT and libidwaita and client side declarations and Wayland and exit. It's quite a fragmented ecosystem.

And wrote like a whole essay about like all the different ways to avoid problems with the Windows VPN stuff. So that's going to speed us along. That's going to speed us along, yeah.