Welcome to Cybersecurity Today on the Weekend. This is another of those shows where I have a fascinating discussion with someone doing some interesting stuff in the cybersecurity arena. Advanced warning. We wander around on a lot of topics, but I think you'll find it interesting. My guest is Anton Levi from a firm called Distrust. I'm happy to meet you. You've had a pretty incredible career.
First question I wanted to ask you was, how does a guy with a BA in English literature from York become a security guy? Except that I have, my undergraduate is in literature from York. No way. Okay. There you go. Just 20 years before you. Gotcha. See, you would understand better than most, yeah. How you get from a place like that to being in tech. But yeah, it's I wasn't really sure what to study.
I always liked reading and writing, so that's how I ended up doing my Bachelor of Arts or English literature. And then when I finished my degree, it was time to get a job. I. And it was, what am I gonna do with this degree? Or either become a teacher or an author, maybe pivot into journalism. But I got lucky and my friend got me a job at a startup that had a SaaS product that was if you imagine Netflix, but for Android apps that you would usually pay for.
So you pay like a few dollars a month and then you get access to a huge catalog of paid apps that have a overall value of, I dunno, $10,000 or something. So I got my first job there and my first job out of university and I had to do business sales.
And my job was to go on the Google Play Store and look for developers that hadn't been contracted yet to the platform, and then check our internal CRM And then if we didn't have those developers in there, I had to go and reach out to them and try to sell them on joining the platform. I had already had a little bit of. Programming experience, but a lot of my work seemed very like something you could automate.
So I started practicing my scripting chops with Ruby and I actually wrote a web crawler that would get me all of the developer names from the Google Play Store. And then I had a script that would check in our CRM who's not in there, and it would spit out a list of people to contact and then even further automated that. So that would generate my emails based on a template.
And I basically went from, spending all day clicking through Google Play Store to running my bot and actually very quickly won first place for most developers contracted in a month. And that was my foray into real programming where I really fell in love with the idea of automation. And it just showed me the power of programming and that kind of set me on that path.
Now, somewhere between there and here, you got really interested in Bitcoin, or at least blockchain and I'm, I want to, I wanna separate the two 'cause I, the first question I have is blockchain, is it still around? Yeah, actually it's definitely around and there's been a little bit of a struggle with finding good use cases, but a subset of these projects have been pretty successful in finding real life use cases that are quite useful.
There's a lot of opportunity seeking individuals who are just creating vaporware and not really contributing anything that, is worthwhile. But there are projects that are actually doing useful things. So one example I always give is stable coins, which for those who aren't familiar with is a digital asset that is pegged to the value of some currency.
this is useful because not necessarily in places where we have reliable financial systems, like in Canada or the United States, but for example, in Argentina where there have been events where the banks did predatory conversion of people saving. So people had a lot of US dollars saved up and the bank overnight without permission converted that at a very. Bad rate and not in favor of the user to their local currency and wiped out a huge amount of people's savings overnight.
And so for those people having a means to bank themselves and have their stable coins sitting in a digital asset wallet that they control is a really nice thing because they can no longer trust their banks. And you can extrapolate that to other jurisdictions. If you look at Venezuela very unstable financially. And so this is actually a very cool use case that's come out. And stable coins are much easier to send around as well. It can take a few days up to a week and it is expensive.
There's a lot of clearances. I could not believe the hoops I had to go through. because of anti-money laundering. You sent anything over a thousand dollars. you are just, tied up for days. Precisely. I always talk about Bitcoin as well, being a great alternative financial system that's been growing and getting a lot more adoption in all sorts of industries, of course, in the financial sector, but that's its own kind of thing. We could spend a whole podcast talking about.
We could, but the essence of this, because this is a security podcast. Yes. The reason I was tracking back this, I'm always surprised at how much hacking goes on in Bitcoin and crypto. Why? Because I thought blockchain was inherently secure. Remember, we were sold that bill of goods? Yes. Why is it so pop? Obviously it's popular for the reason that people say you rob Banks 'cause that's where the money is. You rob crypto because that's where the money is. But why is it so prevalent in attacks?
Yeah. So it's inherently secure. The cryptography of blockchains is solid. It uses the same cryptography used for TLS and elsewhere and has not been broken yet. So it's designed well, but the problem arises from how people manage their. Private keys, they're cryptographic keys that are actually the thing that enable us to move funds around and issue commands to blockchains.
This is where things fail because there is this impression even for people that are in blockchains, that it's, this isolated island that is protected from the rest of the world, but in fact, all of the security flaws of traditional systems that came before that their security footprint is inherited by blockchains. Because at the end of the day, if you're interacting with a blockchain, you're using some sort of a computer with an operating system that you might be using for a number of things.
Maybe you're opening PDFs from an email, which is actually a very common vector that's resulted in huge hacks in this space. One time. This is a crazy story where one of the biggest hacks in recent history was $600 million with this project called the Ronin occurred because threat actors were able to convince.
somebody, a developer that was part of this project that they were trying to hire them for a very lucrative position, and it was a multi-stage process where they had a screening and then they would pass them off to another person. And eventually they sent them a job offered to sign, which was A PDF, and they received this PDF on their work computer and opened it and basically installed malware on their machine.
And they were able to use this malware to exfiltrate keys, which were used for signing off on certain things happening in the protocol. And they were able to actually, by, by. That way steal $600 million. This is so funny 'cause we, I mean we always make these fancy terms for this, but people are talking about, link less phishing now and what you really described as link less phishing. Oh, then I ship you the, the PDF or whatever it is.
And yeah I wouldn't open a PDF that I got on the first meeting with anybody, even if I said it to myself, exactly. But most people are very naive about this back in the day, washing hands before doing things even for doctors was like a crazy idea. And it took us about a hundred years to normalize that. And I think it's gonna take us a while to normalize the idea of isolation.
So when I work with clients in my work and personally, I actually use an operating system called Cubes os which you may have come across. If not it, it's. Just what was, what? Cube's os cube Os with QEBE. Yeah. Yeah. I know the name. I don't know anything about it. Okay. So it's a really amazing tool and very simple. Basically they took a hypervisor, the Xian hypervisor, and wrapped it up into an operating system.
And so what it does is it just makes it really easy to spin up virtual machines that are for, different purposes. And so when you're opening A PDF, you don't trust, you could just spin up a new virtual machine really quickly that's disposable, open your PDF in it and then just trash it. And so it creates this really simple way based on templates. So you have templates. Let's say maybe you have a Fedora template or a Debian template.
You can create a cube and they call them cubes, but they're basically just virtual machines. And so you can create purpose specific vm. So all of a sudden it's not such a huge burden to create a separate VM for accessing your AWS. Account or managing an API token that's for production. So you can create these little VMs and it just makes managing them a lot easier. You could do it by other means.
Of course, there are other tools for virtualization, but this just wraps everything up into a nice seamless workstation. Let's go back to our blockchain example, because I did want to, I did want to go through that and talk about that. So you spent a lot of time working in this area over the past few years.
Yeah. I've been in the industry for about 10 years now, and the way I got into it was really funny because one of my early jobs, after I described to you how I discovered programming, but then I spent a lot of time studying and going to meetups. I met some mentors in Toronto that really helped me early on. I had this friend Josh, who's from a funny name place in, I think it's Manitoba Flynn Ffl. And I would go and Flynn Flo's.
Not funny, to me it's, I'm sure it's a lovely place, but I find the name yeah, just funny. You got bigger Saskatchewan and Flynn Flo's, just flin font's dull. That's, yeah. You've got Tisdale, the land of rape and honey in Saskatchewan. You've got, oh my goodness. There's all kinds of funny names on the Prairie. Yeah. So I just, watch him program over his shoulder and that's how I learned. But I landed my first, proper programmer job at a company called Wild Apricot.
And it just so happened that the owner of that company and the CEO was Vitalic but's father. So the inventor of Ethereum, I worked for his dad. And so very early on, when Ethereum was becoming a thing. Vitalik would come around and everyone would be talking about Ethereum, and I was like, oh, cool. That's interesting. I knew about Bitcoin already, but I never, I didn't dive into Bitcoin properly. So my actual like full entry into blockchains was through Ethereum.
And eventually I started learning how to write smart contracts on Ethereum, joined a hackathon. And actually what participated in that hackathon with somebody who now runs a big layer two, which is basically just a throughput a way to increase the throughput of a main net blockchain. I fell into it, and then I really fell in love with what blockchains were about, which was this really interesting cross-section of game theory and economics and of course cryptography and distributed systems.
And as a, young programmer, it really grabbed my attention. And that was something that also really pushed me into security because all of a sudden, your private key is actually money, right? It controls money or something that's of value. And that kind of raised the stakes and made me go interesting. Yeah, this changes the game in a lot of ways. And eventually North Korea also realized this, and now a major portion of their GDP is crypto theft.
And then unfortunately they use that to fund among studies things, their nuclear program. Yeah. That and working for American companies Yes. The, another area that I find just. Spectacularly amazing again, where you have all of these things that make hiring someone secure, but you can still hire somebody from North Korea. Yes. And that, I, I find that astonishing. I have, and I've been doing this for a long time.
I, back in 2003, had somebody working for me and they phoned me and said, I've been working for you for six months. Don't you think we should actually meet? And I went yeah, we could do that. But I ran a worldwide consulting practice right after nine 11. And we weren't allowed to travel, so I got very used to doing phones primitive sort of conferencing stuff at that time.
Sure. It didn't make a lot of sense to fight traffic in Toronto, so even then, I still talked to my staff enough that I think I would've spotted whether they were North Korean or not. Just amazing. I'd love to know how they pulled this off.
Whoa. I think now more than ever, it's becoming a lot more dangerous because of deepfake technologies, and that's something that's been haunting me for the last few years because, even three, four or five years ago I realized that this would eventually become a thing, once you can in real time do video and audio deep fakes a lot of what we use currently or used to use to verify who is, who just is out the window. And North Korea seems to have started doing that a little bit.
They're still mostly just, doing old school social engineering, but I'm sure soon enough there's gonna be a lot more deep fake stuff going on, and that will make it much harder. But yeah, within the kind of blockchain and digital asset space I've definitely seen an increase in this type of thing where, north Koreans are trying to become an insider and.
A lot of my research and interest and what I work on with clients is, will be around how do we protect from insider threats, and that's a very tricky game. Coinbase just got nailed by People just simply doing the old, we spent all this time talking about the technology of hacking and all of that, when just do the old fashioned bribe some guy, exactly.
And basically that whole hack seems to have been, they just bribed a few people and they Were in positions where they had trust, so they managed to gain all of that access to the blockchain. and by the way, I think they were being paid in stablecoin, That even hackers don't wanna be speculators, yeah. It's interesting.
I find that there's a lot of regulatory requirements around collecting KYC, but not nearly as much about protecting that KYC or it's not as stringent, if it was up to me you would have to have technical controls that dependent on the sensitivity of the personal information or data in general, requires increasingly more individuals and maybe across different teams.
So you could imagine if you need to access someone's passport, it's not just you're a support person, you can just go in and extract it. You would need to have, maybe you request it, and then two other people have to with the correct role approve that. And on top of that, of course, layering things like how much data and how quickly you can access. So if you're trying to access more than one record a minute, you're just not gonna be able to do it and just layer on those types of controls.
And I think that would really drastically reduce the blast radius. Of what would happen in a case that someone gets bribed. Of course it's not gonna stop every attack, but it'll definitely reduce the surface area drastically. But companies just don't do that because it's not commonplace. No one's doing it. So why would we do it? We'll see.
Maybe Coinbase does something like this now, and yet if you had a file cabinet in your office and somebody had to go and look up the personnel files, they'd have to come to you for the key. And the first question you might ask is, why do you need that file? Sure. Yeah. And good luck hauling, a hundred thousand files out of the cabinet. I get that part but I'm just saying that we don't seem to transport the things that we would find natural to an automated security environment.
We would like this whole idea of, need to know. Is something we always had with physical files always, and. I appreciate that. You're never gonna be perfect. My wife always says, when we leave the house, lock the door. We live in the country. Somebody could just smash the window and get in. It's not that hard a deal. They're gonna meet a dog they don't want to meet. but then I look at it and say, she's right. You make it a little more difficult.
Sure. Making, making your attacker's life more difficult. It's, and it's never gonna be impenetrable, but you can always make it hard enough that they give up. Yeah. Yeah. So you stayed with it, with this, with blockchain. What are some of the other things about blockchain that people you'd stand back and go, security people should really be thinking about this. What are some of the other ideas that you've seen over the time you spent with this?
I've spent quite a bit of time working on vaulting systems specifically, or custody systems that are meant to protect private keys that manage cryptocurrency or digital assets. And this is where you really need to go as deep as you can down the security rabbit hole because you are defending from state funded actors and all attacks are basically on the table when someone's trying to steal a few billion dollars worth of crypto.
Which are cases that I've dealt with and I've, with my colleagues worked on some of the world's leading custodial platforms, like Bit Go and Unit four 10 and turnkey. And so in these systems, you really need to go and explore every possible vector. And so one, for example that came up is a very interesting vector is compiler level attacks. Or more on the supply chain side of trying to, we saw the XE backdoor attempt.
I'm sure there are other attempts that are in progress or maybe some that already successfully were deployed that we're not even aware of. And so how do we close off? Some of those doors. we have a show rule. Before you go on, you can't just say the XE thing and assume everybody knows it. So Fair enough. You have to explain what that is. 'cause we have people who may have lives.
And do other things, Sure. So XE Backdoor was this really scary initiative that was a super long campaign spanning about three years, where some, somebody were unclear who it is but likely a state funded actor or very sophisticated attacker at the least, won the trust of a maintainer, a very widely used library. That's a compression library. And they use this as a means to try to compromise the SSH demon that's called, that's used to connect to servers remotely in a secure manner.
And they were super close to actually successfully getting their backdoor in. And the only reason it was caught was because I believe somebody on Microsoft was doing benchmarking, using a tool called Val Grind to figure out the speed of SSH connections and notice that there's, maybe a 10 millisecond difference.
It was a really small difference, but they noticed, and that's the reason we weren't, the fallout would've been crazy, a backdoor where you can just go around the, all the defense mechanism and connect directly to most servers around the world if they updated to that version would've been horrendous. And so the, I raised this as an example of a type of attack and there seems to be a trend towards more supply chain attacks because. We're not really doing enough on it.
but this example, this is such a great example and that's, I'm glad you went through the whole story because it's one of those things we had I had a whistleblower on the program about two weeks ago, and the amount of pressure that he got to stop investigating things. Because he'd find an anomaly and he'd raise it and he'd have to go I better really figure this out because I can't go talking about this with my boss, right?
And I went, oh my God, this guy at Microsoft sitting there, God knows when, but let's say it's 10 o'clock at night, he probably should be home. He sees this little blip and he goes, I gotta find out what that is. And God bless them, whoever. Absolutely. His management was. That he felt that was a good thing to do. I don't know how much damage he saved the world from. Oh my goodness. The curiosity of security people is not something we talk a lot about. And it's so essential. Absolutely.
Caring, caring about security always stays, half the game, people are like, so what do we need to do to be secure? I'm like, first you job number one, care about security care, care enough about security and be curious, like you said otherwise you're not gonna make it. . So we were talking about blockchain and just some of the things that you discovered or seen there that really got right.
So I started talking about some methods that we saw were underutilized and that you need to think about because of the sophisticated supply chain attacks. So the two things that I've been spending a lot of time and energy on is full source bootstrapping which I can talk a little bit about and at length, but essentially just building from pure source code and not hiding anything inside of binary.
So you don't want binaries when you're building things because they're hard to inspect, they're opaque, and so we don't really know what's inside of them. So when we build, for example, even a compiler, ideally. That's a really good thing to full source bootstrap because we want to make sure that our compiler can't go and introduce vulnerabilities. I'll talk about another really exciting attack now that happened but I only discovered it recently.
I was surprised that I hadn't known about it, but it's called Xcode Ghost, and it was this fascinating attack where in China, developers were having a hard time connecting to Apple servers.
The problem is this version of Xcode was fully functioning except the compiler was modified so that when you build an application with it for iOS, it actually injects a backdoor and they successfully compromised a number of different applications and downstream it impacted close to a million devices and it was basically exfiltrating data from these devices. And so this is a cool example. It's a scary example of how a compiler can be the source of compromise if you don't know.
Was it intentional or accidental? It was intentional. It was definitely intentional and relied on the desperation of the developers that were trying to get their tooling and they couldn't because of the, great firewall of China. And so they basically used this to their advantage and, got people to download from a source they shouldn't have. So exactly.
And so this is something that Ken Thompson, who was super prescient back at, it was in the eighties at MIT, realized even if your source code is fully reviewed and trusted, the compiler itself could actually inject malicious behavior into your binary. And so this is something that we thought about a lot and realized that, when you're building systems that manage billions, you need to actually make sure that all of the software ideally is full source bootstrapped.
The other method that we came upon is deterministic or reproducible builds, right? So the idea that your software should always compile through the exact bit for bit binary. And when you do this, it gives you this really easy way to check the integrity of your software. I can talk about another attack right now that. Most viewers probably know about, or listeners but maybe not the SolarWinds incident that happened. Yep. Some years back we're all familiar with SolarWinds.
That attack could have been avoided with the help of reproducible builds. And SolarWinds actually put out a paper after some time saying as much. But the idea basically, or the failure of SolarWinds was that once the binary was built, because it wasn't deterministic, and so like you build it now, and then you build it the same version five minutes later, the binaries would be slightly different.
Not because of being functionally different, but because there might be a timestamp or some artifact from the chip set that the compiling is being done on, or an environment variable that's slightly different. But essentially, once the binary is out of the build pipeline, there's no way to check if that binary is what it's supposed to be. With reproducible builds because we're forcing the binary to be identical.
Now, what that enables us to do is even if the binaries are already out of the build pipeline and be being, maybe it's uploaded to a page where you can download it, you, for example, as a developer could go and build it on your machine and then compare that binary to whatever came out of the build pipeline. And now even if the build pipeline was compromised, the likelihood that your developer computer and the build pipeline was compromised is much lower.
And of course, you can extend this and say, okay, we're a really big tech IT company and our tools are used by Fortune 500 and government organizations. So what we're gonna do when we build our software is we're gonna have three different build servers on three different cloud platforms. We with separate access permissions for this infrastructure. And we're gonna build the same software and then compare it and make sure that all the binaries are exactly the same.
Now, the likelihood that all those three systems are compromised in the same way is very low. And so this is a really cool method that isn't really being used as widely. And yeah, we spent some time working on this stuff. And actually me and me, my colleagues and friends built our own Linux distribution that makes this a fundamental part of how we build it.
We start by full source bootstrapping and compiler, and then we do that deterministically of course, as well, and then build the whole tree of all the packages that are available. Also with that compiler that we now know is trusted fully deterministically. So anyone can go and reproduce the whole thing and see if their hashes match. Of course, that doesn't solve the problem of trusting the source code.
You still need to, know what's in the source code, but it does eliminate the risk of what if the compiler injects something or what if the build. Environment has something malicious and modifies the code. And most software companies actually are exposed to that risk today. But how do you, these differences are introduced by factors like, as you said, timestamps the peculiarities of the chip.
How do you filter that stuff out and still know that you've got two identical source or two identical binary files? How do you basically get a rid of these little variations that make your binaries non-deterministic? A lot of the time it's as simple as. Holding a gun up to the timestamp and saying, you're not changing so you can just fix the timestamp and, set UTC to one and that will actually force all of the timestamps to always be identical.
And for some software, that's enough for other software. You have to play with the compiler flags because, for example, when you're doing certain parts of the build process it might be non-deterministic and it might do things in parallel. And so because it's doing things in parallel, it might do them in different order depending on when you're running it.
Sometimes you have to go and actually patch the software to change the way it's actually built to, maybe it just intentionally goes and grabs some details about the chip set it's being built on and injects that. And so we've had to actually go in for no js, this was the case, and we had to work with the developers of no JS two. Change this about how they build their software so that, so we can make it deterministic.
But yeah, it can be time consuming for some software, for other software it's easier. Of course, you also want to do full source bootstrapping for languages. So self compiled languages like rust. If you download a rust binary from somewhere that you don't know how it was built and then use that to build the next version of rust, the trust level on that is much lower than if you go and first bootstrap the GCC compiler and then or a tiny c compiler.
And then you go to the first version of rust and then the next one and iteratively build up until you have a clear path from before. Rest was self compiling all the way to the latest version of rest. So this is another thing that we discovered as a good defense mechanism on making sure that cer basically it allows you to say certain. Categories or classes of attacks are no longer viable, you basically cut those off. And so you still have things to worry about, but at least less things it, yeah.
And again, it's, obviously when we're talking about it in this matter and you talk about doing it the first time it's probably labor intensive. Probably takes a lot to figure out. Yeah. But I'm just amazed that with all of the problems we've had with open source code, that people have not put out a protocol or something that says, Hey, this is how you established that what you've got is a binary compatible version of the original and has not been interfered with.
Yeah. I find this astonishing that no one's really pursued that. There are a few distributions that are working on this. So the one that we built, it's called Stage X. And there are two other ones that have similar approaches where they do boots, full source, bootstrapping and determinism. They're called nicks and geeks. And geeks is actually a fork of nicks with slightly different approaches.
But none of the existing distributions were as strict as we want it to be on this idea of full source bootstrapping determinism. So the one we built, we basically made a rule and said only if something is full source, bootstrapped and deterministic, and any changes that are made during packaging, the packaging steps of actually describing how the package is created in our distribution has to be reviewed and signed by two individuals.
And so we basically took these things and tried to do something akin to what you were just matching. That was the idea. How do we make something that's fully open source and free and helps people get determinism and full source bootstrapping into their systems as a kind of a building foundation, of course, you need to force your application to also be deterministic. To fully close off the loop on that, but that's exactly the idea. How do we get this into more people's hands?
And now it's actually fairly easy to use. It's it's almost an alpine replacement, like a drop in replacement. So if you're using Alpine, Python or rust, you could just pull in an image. It's on Docker hub of Stage X and it's the same software, same version, except it's built differently. So it's built using a full source Bootstrap compiler. All the languages are full source, bootstrap. And it's fully deterministic, so it's pretty cool. And yeah. And this is available now.
This is, you've made this open source freely available? Yeah, it's open source. And we did this because a lot of our clients cared about this kind of attack factor. So it's a cool feedback loop where blockchain companies spent money to do a lot of this work and then we open source the work and made it available to everyone. And it's really nice because it's already being used. Actually, if you're familiar with Talos Linux they're very widely used.
The Linux distribution that specializes in Kubernetes. They decided to use our distribution to build theirs. Ours is essentially a secure tool chain for building software. And they thought, Hey, this would be a good way to build our software because we close off a bunch of attack vectors on this compiler level and environment kind of risk level. I will be able to say, I knew you when, boy, this is me.
You can you send me a link and I'll make sure I post that in the show notes, at least for the YouTube version. That'd be very helpful because the link, yeah, we spend a lot of, and continue to spend a lot of time and energy essentially all of our free time. We, when we're not working on projects that make us money, we go and pivot to this, and sometimes we're able lucky enough to actually get some of this stuff funded too.
But it's a labor of passion and love and we love to see people use it because we believe it can help basically anyone. That's the beautiful thing, it's oh, you're using Python in your stack. Great. Just drop in to our Python image. Or if you're using rust or. No JS or whatever. Yeah, no, this would be handy.
The other thing that just drives me crazy when we talk about blockchain was one of the first use cases for blockchain that I ever heard that made business sense other than crypto, was the idea of traceability. And in the food system, like a really messy system full of people who are not necessarily PhDs through this. And I'm not making fun of them, I'm just saying this is not an organized system.
yet I can trace a piece of lettuce that is here in Minden, Ontario, which is in the middle of nowhere, back to the original field where it was. Because if they get an outbreak of some sort of e coli or something like that, they need to be able to pull it off the shelves. And yet. With all of the brain power we have in blockchain, we've never been able to establish the providence of software and software elements or modules in the same way.
Yeah. that's a good question and a very difficult problem that people have been trying to solve for a long time, and especially because blockchains fit the shape. It's yeah, this should be the solution. I think a part of the problem is that. Blockchains are very good at preserving data once it's on the chain, but it's very hard to get high quality data into it.
And if you think about the blockchain trying to fit into the operational complexity of a supply chain that spans, maybe multiple countries and different points where the data has to be entered, I think that's where things start to break down. If the how do you make sure that reliable stuff gets in a blockchain? Because if you get poo on a blockchain, it'll just be poo on a blockchain and it's not gonna be great.
So I think that's the operational and logistical parts of how these supply chains work is the hard part. Once you have good data, you can throw it basically on anything. In a sense, the blockchain's just the. Distributed data layer. It's not really anything crazy. It's not anything crazy. It's actually pretty crazy. Yeah. It's, yeah. It's not something you do in an afternoon, but it is.
But this, have you given any thought to this and I'm, I just asked this while we're just talking as to, you've obviously conquered one piece of this and say, I can establish the providence of a binary. I can make sure that it's the same. Could we apply some logic to that in the open source world so that we are just making sure we are actually getting the correct modules and that we're not getting somebody's Yes. Yeah. The answer is yes.
And we've been thinking about this a lot this idea of how do we eliminate single points of trust in any individual or computer. I think that's like a core idea and and yeah. Concept to filter ideas through and trying to. Think about holistic security that, that, thinks about the full life cycle of software that we consume. And so we talked about, yeah, like you said, this part of, compilers and and determinism and what that, that can address.
But still a lot of holes remain even on the level of, if you look at compression algorithms and tarballs. So what with the XXI attack, what was actually happening was that there were additional artifacts in the tarball that were not in the source code. And so it was hiding in the tarball. And this is not something you would think about. You think, oh, source code goes into the tar ball, which is just the source code, and then you un tar it and you do stuff with it.
But how do you make sure that's actually the case? So that's the one step before what we were talking about where it's how do we verify that we're actually building from the source code we're supposed to be building from without any extra stuff in there. So here we are trying to build some tooling as well. There's a project that I found that is, I forgot what it's called. It's like predictable binaries, or it's by a Fosse. I would've to look it up.
But it's essentially a tool that detects binaries and source code. And so that's one idea. You can try to find binaries, eliminate binaries, 'cause those could sneak things in. But then also can we create a way to standardize the source code and make sure that anyone can easily check that the source code is exactly the source code you're meant to be building from.
But we can go even earlier than that where we need to review the source code to make sure that there isn't, if someone writes some malicious code into the source code and no one's looking at it, that there's no way to really address it. So we need reviews. So one way that, that we've been thinking about that in my group of friends and colleagues, is how about we create a crowdsourced kind of protocol? We are calling it SIG Rev or yeah, SIG Rev, which is like sign reviews.
So the idea is let's make it easy to create reviews for specific versions of software, and then cryptographically sign them using PGP. Or you could use, you could sign in whatever you like, but PGP is probably a good case.
And then start building a database of software that's been reviewed, and you can go and let's say you're an independent researcher, if you reviewed some piece of software, you could say exactly which commit or which state you reviewed, have a hash of the tree, and then say, I reviewed for this and that, and then sign it off. And this is another thing that I feel is a huge hole right now. If you look at the NPM and Python package ecosystems, they're full of literal malware.
And for whatever reason, we got to a place where we just feel okay with downloading tons of third party libraries without actually reading the code. And we just say we use static analysis, so it'll be fine. But it's not fine because static analysis only catches things that we've previously detected. And so novel stuff we don't have signatures for, similar to how malware is, right?
Like polymorphic malware, it just slightly changes the shape of the malware and all of a sudden we're blind to it, right? We're trying to develop tools to fight this, but why aren't we reviewing all the source code? And there are a few answers for this. Often 80% of an application is third party code. That's just open source libraries, right? So how are we gonna review all this? It's hundreds of thousands or millions of lines.
But the responsible thing to do, especially if you're running a business that is maybe financial institution or manages a ton of personal formation, would be to literally review every line of code, right? And you actually say, okay, what if we all collectively, as we reviewed software, posted our results and said, yeah, we reviewed this stuff and we actually found it to be secure for this specific version.
Now you might use a different version and say this hasn't been reviewed, so I have to review the difference between those two. But over time, if we were all doing this sort of thing, we would start building up a repository of knowledge of which source code. Has at least a slightly higher level of trust. 'cause right now it's just a lot of people use it. Probably somebody looked at it, but we don't really know.
I have to they come and take my podcast or license away if I don't ask about AI at some point in the conversation. It's just the way it's forgive me on this one, but why hasn't anybody started to look at using AI to do some of these tasks? Especially when you talk about, multi-lines of code. One of the things you, we can have a great debate about whether AI can write code or not and we can, that's a religion now is to what you believe rather than anything else.
But the one thing I think we all agree on is it's really good at documenting code. Definitely, and I've been definitely very surprised to not be reading a lot about people who are saying, we've got this problem. I'm gonna apply AI to it. It, have you heard of anybody doing this? I haven't looked into it too much, but I am very much in favor of it. I think it still probably is in its infancy. I wouldn't trust a tool like that as like a, oh, we have this now we can hands off.
But the way I think about it is why not have an additional input or, a layer that you can put into your automation, the whole shift left to use that term. you have your linter, you have your static analysis tools, and now you have your AI analysis tools. And as people figure it out and they become more sophisticated and advance, I think yeah, they could be very effective. I don't know when we get to a point where we can fully trust. These tools to make judgements.
Why would you have to fully trust it? The, it's the old thing of, that's why we build in layers. 'cause we don't fully trust anything. Exactly. Exactly. Fact, my friend David Shipley would tell me that organizations that believe they're totally protected are the most likely to get hacked. You need Andy Grove said only the paranoid survive. I think that's true in security. but having another layer We're already using AI to look for anomalous.
Things in, in software that says, Hey, we don't have a signature for this, but this looks weird. Totally. Yeah. It doesn't hurt it. When I first started thinking about it, I was kinda like, wait, no, but we don't need to rely on ai. It's literally just another set of inputs. So Great. I'll take more inputs. Anything? We rely on people. Sure. And people make mistakes. I always find this just totally amazing, is that everybody says yeah, but AI made a mistake. And you go yeah.
And so did a person, yeah. It learned from us and what's your point? Yeah. It's, nothing's perfect. That's why we have layers. That was at least my understanding of it. Yeah, absolutely. I have to ask you just in a, just going onto this thing, I was looking through your LinkedIn and I was reading about this Milk Casad disclosure. Oh yeah. And I glad you brought that up. I know I'm drifting into a new topic, but this was so fascinating.
We've talked about some of the human things we can do, but this really was. A mechanical or a technical error. Yeah. That just went by everybody. Yeah. It was very, a very interesting bug that actually impacted one of my friends. And they lost a bunch of Bitcoin. And the really scary part is that they went and followed a very widely used and popular book called Mastering Bitcoin. And this book recommended using a specific software to generate your bitcoin wallet.
And so it recommended that you do it in an offline environment, on an air gap system that's not internet connected. And so they followed these instructions and did all this, and then years later, their wallet got drained. we assembled a research group of friends who started looking into how this happened. eventually we discovered it was. A mistake in how they implemented the cryptography around randomness or entropy.
And they used what's called the mesan twister, which is a random number generator algorithm, but it's not a cryptographically secure random number generator. So it's actually usually used for montecarlo simulations. It's not meant to be used for generating private keys. By doing this, they reduce the key size to only 32 bits. With a well optimized cracking algorithm you can map out fully in about a day.
So this is exactly what we did, and we were able to reproduce, basically build the whole, all of the keys that, were part of that key space. And so on milk Sad Info, there are a series of blogs that one of our teammates, Christian. Wrote about. And so it goes into the details, really low level of how it all worked and how we actually reproduced the vulnerability. And we had a responsible disclosure and back and forth with the developers.
But it was a very interesting, we actually presented it at K Communication Congress last year in, in Germany and Hamburg. So there's also a talk recorded about it. So if you prefer video format, you can watch that. 'cause I found this just fascinating again. Yeah, it was, it's for all the technology we have, it's really nobody being curious enough to go, that's only 32 bits, man. The gaming machine will get through that day. It's a very tricky.
Bug, because you need to know cryptography relatively well to figure this one out. And also, I've seen a lot of, books that teach computer science recommend using the current time as the seed for randomness and time isn't random. And so we also have this layer of just bad information floating around there. And that's partially what leads to these mistakes. And developers can overestimate their ability a little bit when it comes to security and cryptography.
And cryptography is something that even I as somebody who spent a lot of time working on cryptography take very seriously. And I'm basically scared of, like, whenever I work with cryptographic algorithms it's like dealing with nuclear codes, exactly. But how do you make sure that you're, how do you satisfy yourself that you've done the due diligence? I actually go and work with like multiple different cryptographers Exactly.
That's the only way around we all go on about how security has to be built in, not bolted on, and sorry if I sound preachy on this thing, but it's just, what I would. Have done. 'cause I was the world's lousy. Maybe you've gotta be stupider as a programmer and 'cause every time I did something, I went, I don't really know how to handle this. I'm gonna go to somebody who really knows this. Exactly.
And we would do what we call chalk talk at the time, which became even weird when we were actually in a basement of someone where, like Waterloo, you could, where they actually have chalkboards. They probably still 'em to this day, but what would chalkboard, we'd be in somebody's on somebody's whiteboard with markers and still call it chalk talk. But it was that whole thing of go to somebody smarter than you have them walk you through what you're doing.
They may not even be a total expert, but they'll ask dumb questions. Absolutely. Which are exactly the ones that, that's it. That always get you into trouble. Yeah. I'll go talk to my friends who have expertise in areas that I haven't been or delved into as deeply. Or I'll go on IRC and find those, really, arcane knowledge people on something that, I need to find out about.
And so yeah, there are people you can talk to, but yeah, you just need to be a little bit humble Exactly, because it's not the times when you don't know it. Even, like I said, I, because I was never the smartest programmer in the world, I always had to ask people, make sure I was doing things right. But it's the guys who really think they know stuff that scare me. Yeah. 'cause they don't ever have to ask.
And I used to work with one architect and he was like, he just wanted to get the program to be the smallest thing in the world. Even after, like in the early days when I started, if you wrote a program that was more than 5K, you had to do an overlay. Because we did, that's, we didn't, we only had about three and a half K memory that was available. I wish we still had such constraints. That's one of the things that really irks me.
Like we have so much processing power now, and instead of making our software super fast and efficient, it's actually some of the least efficient software ever. Yeah. I think Word has 30 million lines of code. Oh my God. I'm going to, to a word processor, that's I've given up on Microsoft a long time ago, but, yeah. I finally gave up. Even if it's seven bucks a month, I gave up paying the tax to Microsoft to use. Yeah, good. A word processor. Yeah. That, I'm just like living in Vim now.
Markdown files are my, go-to and it's been a while now that I've. I've been left all that behind. This has been a most amazing chat. I want to talk, just close it off a little bit about you. You've and I always do these things. I tell people they can't do a commercial on here, but you're working for a firm called Distrust. I always figure if people get to the end of the show that you've done a good job then, and they're still listening.
'cause the audience that I've got in, particularly in cybersecurity, if you actually try to sell them anything they'll, it evil on me. They'll say but let's talk about your company distrust. First of all, who came up with the name that Yeah. So it was my co-founder.
And it's basically this idea of you shouldn't trust, you should be able to verify and so therefore distrust It's a company that my co-founder started a while back, probably five years back, and then I joined and started working with him maybe three years ago. But it's basically a firm that specializes in working with high risk clients. So a lot of it came from working with. Custodians and vaulting solutions like I mentioned earlier.
And we basically learned a lot of really great methods to protect sensitive systems. So we work with electrical grid operators and we work with hedge funds. And anyone who has really sensitive data or cryptographic keys, they want to protect, we help them. And we think very holistically. And we think about this idea that I also mentioned earlier of how do we eliminate single points of failure in a system because it's like nuclear, launch.
Like you have two keys but what if we say we need four keys, or, we play with this idea of a quorum of people that are required to do something. So maybe you have seven people that have the right permissions, and you need four of those seven people for something to happen. So we methodically think of ways to, at every layer, including, the Linux distribution or the operating system that's built. How do we make sure there's no one maintainer, for example, they can go.
And insert some code somewhere into one of the dependencies and undermine the integrity of everything that comes after it. And that's basically the game we play. We have a threat model on our website. It's basically talks about these different levels of how we think about it, but our approach to threat modeling is basically how do we defend from state funded actors at the most extreme end.
instead of trying to think about specific vulnerabilities like CBEs, we think about how can we eliminate entire categories of attacks and make it so that you don't rely on one person or computer for the integrity of your system a lot of what we do is consulting and we'll be on retainers with companies, like a fractional csar, fractional security engineering team. all of the rest of our time goes to building these open source tools and we have a bunch of them.
We have some custom operating systems. Work a lot with remote attestation technology. So getting a server to cryptographically prove what software it's running and a prerequisite for this are deterministic builds. You need the software to be deterministic because essentially the idea is, let's say you use a VPN you don't want to just trust the VPN provider that they don't log anything like Pinky Promise. What if the server could actually cryptographically attest to the software it's running?
So imagine that portion of their stack is open source software. And it's basically the thing that you connect to. Now, if it's open source, you can read and review the code and say, okay, great. This actually doesn't log anything amazing. You build the software locally. Because it's deterministic. You expect the hash that you get locally to be the same as what the server tells you is running on that server.
And so there is a special type of hardware called a TPM, a trusted platform module, which is basically a secure chip that can go and it has its own keys that only it has access to. There are these things called PCRs. So between those two, you can basically get the measurement of the server, figure out what the binary is that's running on the server, and then cryptographically sign it with the secure chip.
And now you get a. Cryptographic at the station from the server proving that it's running the code that you reviewed locally. So this amazing building block and a level of trust that I think in the future we're going to see a lot more, but it's just a fundamental building block that is underutilized right now. And so that's another example of, where we spend time building, we built an operating system called Enclave Os that's specifically to make this sort of thing easier.
So basically wherever we find there are opportunities to make it easier to access these defense mechanisms, we create open source software and say, here we go, please use it. Help us build it. And everything we do basically is open source. So yeah that's a remote, that's a station is a really interesting topic as well. And then the Caymans, I actually.
Ca came here and opened a company called Caution because Benjamin Franklin once said that the parents of security are distrust and caution will actually be creating some products that are like managed products to help. For example, one of our products will be a service where you can ask us to build your software and give you a hash of what we built. So when you're doing deterministic or reproducible builds as a part of your CICD, you could say, Hey, what's the hash of this?
And so you have another separate system reproducing your software that you don't have to set up for yourself. So that's just one example of what we'll be doing. But I'm gonna tell you guys, if you want to get outta the security game, you could go to most of the AI companies and help them name their products. So they didn't have stupid names. All of our stuff is named very simply. Like we have an operating system that's for offline operations. It's called Air Gap os.
We have an operating system for enclaves. It's called Enclave Os. We have, the Linux distribution, that full source bootstraps everything. It's called stage X because use stages to bootstrap things. So this is a new stage. And so yeah, we try to use just very simple names. This has been fantastic, Anton. it was great to meet you.
And for anybody who's listening to this, if they're still here with us we met because a friend recommended you and I urge people to do this if somebody's got, knows somebody's doing interesting stuff. I love to, every once in a while just do an interview that is just talking about interesting stuff that people are doing. So that's, this has been great. Thank you.
Before we go though, I always say that people, when I was heading up things, everybody thinks that the CEO or the head of an organization is running it. Now. We're just selling trust me and I always took away one thing was that was when you left a room, what you really wanted to do when you were in the room was make people remember you when you left. In other words, the question was, what are they thinking about when you leave, when you're not there.
So at the end of this, what do you want people to remember from this conversation we had, we'll have to remember. I mean it, I don't wanna sound too generic, but that it really matters what impact your work has. And, for me it is thinking about what my skills can best be used for to improve the security, privacy, and freedom of as many people as I can. And the way I found to do this is through, primarily open source software that creates this, like building blocks that improve security.
And so I would just encourage everyone to try to find. There a sweet spot for that, where their skills meet, what the world needs, what they can also get paid for and what they love. This is like this, I ripped the Japanese concept of Ikigai, but I'll like, say five or Moe Gaudet who says, stop working for money, start working for purpose. That's it. And I've, but my passion of, freeing these tools and open sourcing everything is definitely the trade I take any day over just making money.
This has been fascinating. My guest has been Anton Levi, and he is with a firm called Distrust. Love to hear your comments, and if you know someone who's doing some fascinating work or if you're one of those people, or if you've just got an idea for a show, drop me a note. Maybe we'll get a chance to chat and some of the chats actually make it onto the show. You can reach me at [email protected], or you can find me on LinkedIn, or if you're watching on YouTube, you know what to do.
Just leave a comment under the video. David Shipley will be sitting in on Monday, and I'll be back in the news chair on Wednesday. I'm your host, Jim Love. Thanks for listening.