EPISODE 1948


[INTRODUCTION]


[0:00:01] ANNOUNCER: Open-source software underpins virtually every modern application. That ubiquity is a superpower for developers, but it is also an expanding attack surface.
Software supply chain attacks were once rare, but are now happening daily with malicious actors exploiting the trust developers' place in public registries, package managers, and CI/CD pipelines.

Chainguard is a secure software supply chain platform. The company started with hardened container images and has expanded to cover domains including VMs, language libraries, GitHub actions, and agent skills. Matt Moore is a co-founder and CTO of Chainguard and a veteran of Google's open-source container and security infrastructure work. In this episode, Matt joins Gregor Vand to discuss lessons from recent supply chain attacks, why CI/CD pipelines are now a primary attack surface, the challenge of meaningful software inventories, the EU Cyber Resilience Act, and what the arrival of Anthropic's Mythos model means for the pace of vulnerability discovery and the urgency of patching at machine speed.

Gregor Vand is a security-focused technologist having previously been a CTO across cyber security, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at vand.hk HK or on LinkedIn.

[INTERVIEW]


[0:01:41] GV: Hello, and welcome to Software Engineering Daily. Today is a fun one where we get to talk to a person in a company again. At the end of many episodes we say we'll be following along, and that's exactly what we've been doing here. Very happy to have Matt Moore of Chainguard back with us. Yeah, welcome, Matt.

[0:01:59] MM: Thanks for having me.

[0:02:01] GV: Yeah, some of you may remember the episode we did probably around 2 years ago, I think, at this point, something like that, with Chainguard. Everything, talking about hardened containers, about security. Spoiler alert, a lot has happened over the last two years from that perspective. I think it's going to be super interesting to dive into how Chainguard has evolved with that and how it's protecting things and evolved. But as usual, just in case anyone didn't catch that episode, just a very TLDR of who is Matt Moore? How did you get to Chainguard?

[0:02:34] MM: Yeah, I'm Matt. I'm one of the founders of Chainguard and the CTO. My co- founders and I, we all met over many years of collaboration at Google. And we collaborated on things across open source, the container space, the security space, the developer experience space. And we have been working on parts of these problems for over 10 years together.

But one of my co-founders and I had left Google briefly, and I was on a break barbecuing and playing video games trying to figure out what I wanted to do next. And our CEO, Dan, was trying to recruit me back to Google, and I was like, "Ah, I don't really want to go back to big tech just yet." And so I Uno reverse-carded him, and I was like, "Have you thought about starting up?"

And we'd been working on aspects of this problem space for a while, and really nobody cared. But at that time, SolarWinds had just happened, and overnight everyone cared. And so the time was right. And we had spent a lot of time studying this space, and so it felt like right team, right time. And we decided to go for it. And it's been a pretty wild ride ever since.

[0:03:43] GV: Yeah. Amazing. And yeah, it's interesting how you mentioned SolarWinds there, like right time. And then we're going to get into almost equivalents that might have happened over the last 2 years. And then, again, for anyone not familiar, just what is Chainguard? What are hardened containers.

[0:04:00] MM: Yeah. We started with containers. Chainguard on the whole, we sort of want to give folks a safe way of consuming whatever open source they're writing. These days, you can't really write modern software without open source influencing it in some way. Basically, every language runtime, or compiler, whatever, is open source. Java, Python, Node, right? These are open source language runtimes.

And then on top of a lot of them, those sit on a lot of open-source projects, like the system packages that go into every Linux distribution, the Linux kernel. If you're running containers, the container runtimes are open source. Kubernetes is open source. Open source just permeates everything that we do. And it's basically unavoidable these days if you want to write a modern application. But that's also sort of a superpower, right?

If you think about before the proliferation of open source, how much you had to write in order to ship an HTTP server? Back 25 years ago, Amazon was racking servers, right? Google was racking servers to build a startup. You were racking your own servers and going in and dealing with the hardware aspects of this. But cloud has made that so that you can just take that and use it. Open source has made it so that the amount of software you need to write, to write an API, is so much smaller, right? You write a little bit of glue on top of frameworks that sit on top of HTTP servers, right?

And so it's this amazing accelerant that allows folks to write much, much, much less software to build a functional app. But really, you need a way of safely getting all of that software. And so we started with containers because what we found was the way folks were consuming containers today, you sort of start with a baseline of hundreds of unpatched vulnerabilities. And it's not that patches for these don't exist in the upstream. It's that the way modern distributions work, they're just very slow to get those into folks' hands.

And so we wanted to give folks a better way of doing it where we would fully patch everything that goes into the containers that they're getting. And really, for many years early on, one of our big struggles was convincing folks that we weren't somehow lying. That this was actually a tangible result. They had this learned helplessness that what the result that we were showing them wasn't actually possible. But it is. It's just a lot of work that nobody prior to us had really done to the level that we did it.

And so we've sort of evolved as a company and gone beyond just containers. Really, the way I look at what we do now is being a secure software supply chain as a service, right? We give folks a safe way of consuming that open source. One of the analogies I really like, and I realize that it dates me a little bit, but in the early days of digital music, right? Folks got their music off Napster, right? And ignoring the licensing aspects of it, it was a way you could get a lot of music.

And sometimes you'd get music, but sometimes you'd get malware, right? That's the state of open source today.

If you're just consuming stuff from the public registries, it's a lot like the result you'd get from getting stuff off Napster. And so the role we fill is a lot like when iTunes came onto the scene and started allowing folks to buy MP3s, right? You could buy a song and then you got access to that song. It was coming from Apple, a reputable company. And you had a safe way of getting that music. And you weren't going to get malware from Apple, right?

And our initial product, we sort of oriented around unit cost. So you buy an image from us, you get that image just like you'd get that MP3. And then over the years, our catalog grew a lot. And we had customers who wanted to standardize on us and wanted access to that whole catalog. And so we shifted towards we still have that model if folks want that, but we sort of evolved into more of a Spotify model, right? Where folks could get access to the whole catalog, listen to what they want, and consume whatever container images they want from us. And it's a developer seat pricing model. Depending on which model folks are interested in, they have a safe way of getting their open source from us. We started with containers and then we've expanded to a variety of other different ways folks consume open source.

[0:08:07] GV: Yeah, I love that analogy. I also use Napster back in the day, moved on to, I think, Kazaa and all sorts of other things. And then, yes, now I'm Spotify because it's very quick and easy, and it's definitely the actual song and all that kind of stuff. I think that's a great analogy.

When we talked a couple of years ago, as you've called out, it was originally hardened containers and it is now more of a platform play here. And then almost back then, it was like a CVE reduction story, if you want to call it that. We've got smaller images. We've got fewer packages actually. Fewer but hardened. Fewer known vulnerabilities. But how's that conversation, I guess, changed with your customers and with just people interested in the product over the last two years?

[0:08:51] MM: Yeah, it's shifted a lot. I mean, hardened containers was always just the beginning. When you're trying to solve for a space like supply chain security, it's an enormous

place. We had to pick somewhere to start. And containers was a space we knew exceptionally well. We decided to tackle that first.

But the real question is how do you secure your entire software stack? All of that open source that your developers aren't writing but it's running in your environment, right? It's subject to all the compliance of your environment, right? And every unpatched piece in there or every piece that may carry in malware potentially presents risk to the business. We have rapidly expanded beyond containers. We started to target VMs and language libraries a bit over a year ago. And then at our most recent user conference, we announced that we were expanding into hardened actions and hardened agent skills.

And so one of the ways I think about the framing beyond CVE reduction is I think about CISA's Secure by Design initiative that launched a couple years ago. And one of the tenets in there is seeking to eliminate entire classes of vulnerabilities. Not just patching vulnerabilities, but this is things like writing things in memory safe languages or leveraging an ORM to avoid SQL injection, right?

Part of the way I look at it is by building things from source, we're actually able to eliminate entire classes of vulnerabilities that come from trusting those public artifact registries. I mentioned the Napster analogy, right? Just by building stuff from source, we're able to eliminate 98% to 99% of the source of recent malware attacks, right? It's just a huge class of vulnerabilities. Basically, the vast majority of how most of these things happen, just building from source, gets us that. But it also gives us a level of control that allows us to eliminate other classes of vulnerabilities, like folks running unpatched software downstream of us, right? We can make sure that all the patches are applied from upstream.

And so I think two things have also materially changed in the past couple years since we last spoke. The number of CVEs being reported has just exploded. And this is only going to get an order of magnitude worse post Mythos. Right? We saw data coming from Firefox and some other high-profile projects where the number of CVEs patched in a recent release, I think it was 150, was 10 to 20x what most prior releases had found, right?

And so you've got the CVE graph just growing exponentially. But then you've also got malware attacks. These used to be relatively rare. And they've gone to monthly occurrences, to weekly occurrences, to daily occurrences. And I posted something on LinkedIn talking about how these things were now happening daily. And someone left a comment being like, "Soon it's going to happen a couple times a day." And literally the next day, there were three attacks that happened on the same day.

And so while no layer of defense is going to be able to promise 100% protection, one of the things that also hasn't changed in the last two years is that none of our customers have been impacted by any of these attacks where they were using our products, right? Our libraries product has been able to defend against all of the malware attacks that have happened over the last several months, and the rate at which they're happening is just frankly mind-blowing.

[0:12:06] GV: Yeah, absolutely. Certainly, open source supply chain attacks have had a moment through the last two years especially. I mean, it was kind of, for many of us, we could almost see it coming. I think it was just you have actually taken action on it and started a company to defend against it. Most of us were like, "Ah, it will happen probably. But hey, it's like nothing's going too wrong yet." But here we are and it really has become very clear to a lot of unfortunately malicious people that that is a great way to actually that's how you should distribute your malware.

Let's go on to like a case here. There was the XZ Utils case. We did actually cover this and one other that we're going to jump onto in another episode, but from a different angle. Some listeners may be familiar, but this was XZ utils was basically where a maintainer got more and more trust with the community that maintained that set of packages. And eventually, they were able to then push in a malicious - I think it was a malicious macro, basically, into the build system. This went into a package liblzma. I'll stop there. That's the super factual. Someone gains trust, manages to push a malicious macro into liblzma. How did Chainguard, I guess, respond to that?

[0:13:24] MM: I think one thing worth calling out is like, still in my mind, that attack is still in a class of its own with respect to both the level of sophistication as well as the patience, right? A lot of what we've seen recently is much more opportunistic large blast radius attacks that try and

just smash as much stuff and then get as many credentials as possible, and then figure out what other fun they can have with those. This was an incredibly long period of time that -

[0:13:56] GV: It's like two and a half years, I think, they spent.


[0:13:58] MM: "Person." I say that with air quotes right, like? Jia Tan, right? They spent time gaining the trust of the community doing legitimate contributions. And basically, it resembles Spycraft, right? It is a level of sophistication that I don't think I have seen since. That doesn't mean it's not happening. I'm almost certain it is. And I think people are worried about it happening all sorts of different places. But that said, the attacker had maintainer level access, but the attack still required the use of an intermediate artifact.

Jia Tan basically published an altered release distribution, which a lot of distributions build from, right? And in this case, they were like, "Ha-ha. The source is this." But then they altered the release distribution that they attached the release that they knew downstream distributions were building from and it carried modifications that activated the malware. Building from source defended against it in this case. But it's also possible that if Jia Tan had taken another path, it might not have, right? Because they had done all of that leg work to get that sophisticated and that deep level of control over the project before they were doing their attack, and they were doing it effectively in plain sight.

But I think this touches on why defense in depth is so critical. And my point about no one layer of defense is ever going to be 100%. And if someone's telling you that, then they're lying to you.

[0:15:22] GV: The Swiss cheese model, right?


[0:15:23] MM: Yeah.


[0:15:24] GV: Yeah, the Swiss cheese model. Yeah.


[0:15:24] MM: I love the Swiss cheese model. Building from source, minimizing what you're running, scanning for anomalous behaviors. I think one of the things that's interesting about scanning for anomalous behaviors, not just necessarily scanning for like known malware

signatures, but scanning for things that programs you want running in your environment aren't doing. It's not always malware.

One of the things that we've been starting to talk about scanning for and blocking is what we're calling gray wear, right? Classes of software that maybe aren't compromised, but do similar things to malware that you don't want running in your environment. They may establish remote access terminals. They may read or transmit sensitive credentials to places they probably shouldn't.

There are projects out there that do these things that the projects that haven't been compromised and people use them. Like remote access terminal, that sounds really great for debugging. Maybe I'll stick it in there, but maybe you shouldn't, right? Maybe your product security team doesn't want that running in your production environment.

Yeah, I mean, this notion of grayware where you flag behaviors that you don't want running in your production environment, whether it's matching a malware signature or not, is I think one of the things that I think we need to start getting real about what you'll let in and what you won't let into your environments.

[0:16:37] GV: Yeah. What's interesting here as well is the packaging of liblzma that Chainguard actually uses, it runs on Wolfi as opposed to Debian or x86. This wasn't actually specifically targeted. It was sitting, I guess, within a build that you had put together. But because it wasn't actually targeting Wolfi, technically it wasn't a threat at least in that moment compared to anyone running on Debian. But can you walk me through the mechanics of that as well? Do you think that was accidental that they didn't also target Wolfi? And I guess, also, maybe walk us through if they had been targeting Wolfi, what would that have changed for you guys, for example?

[0:17:17] MM: Yeah. I mean, I think I mentioned there were a few things. One, it went through the release distribution. I think another one was I believe there was something in the modifications that targeted specifically systems that were producing I think devs and RPMs, which we don't build devs and RPMs. And so, yeah. I mean, we built the release that was tainted but it didn't impact us because the way we built it did not activate the attack code.

I think that this is my point about source level access gives folks a way of potentially doing malicious things. And Jia Tan had the level of control to potentially do some of those nasty things. But it's also why we complement the fact that we build from source with so many other techniques including running different behavioral scans around the type of syscalls that are being made in in one release versus the next.

And it's not to say that any of these things are foolproof. And I think one of the interesting things about dealing with open source is so much stuff tends to be out in the open. And how you scan for some of this stuff? Actually, you almost want it to be somewhat secret, which is counterintuitive to an environment that biases towards transparency and whatnot.

But say I have an open source malware scanner that we run to detect these kinds of things. If I were an attacker, what's the first thing I'd do? I'd scan it to make sure you weren't going to catch me, right? And so there's, I think, limits to just how much you can be completely transparent about everything you're doing and still be protected against folks who know how you're going to scan what comes out the other side and can check in advance basically whether you're going to detect them or not.

And so it's a balancing act that we have to deal with. But yeah, we are doing various behavioral scans of the package builds where we try and classify behaviors of what they're doing, but also check them for known signatures of malware and things like that.

[0:19:23] GV: And I think chain guard factory flags an update. If there's a code update that makes a new network connection, that's actually not mentioned in chain logs, for example. That's interesting.

[0:19:34] MM: Yeah, we try and look at a wide variety of signals. I think one of the behavioral classification things that one of the tools we do - and we do sort of differential scanning. If the previous release, the .1 release, didn't have it and the .2 release did, patch releases are typically places where you don't expect significant behavioral changes either at the build time or in the resulting binary. Maybe you enabled some new piece of functionality. In which case, flagging is probably the right thing to do. You can review that. And then from there on, that becomes a baseline.

But there was actually a breach on the library side that was able to compromise the source code, and our scanning was able to block that. I forget the name of the recent attack, but it basically embedded a PTH file into a Python library that forced the interpreter to basically load that. Even without importing the library, it loads that at runtime and executes it. That got flagged by our malware scans and blocked. So, we didn't rebuild the malicious release even though the source for it was available because we detected that PTH. We actually built the release right before, which was totally fine.

And after they detected this, they actually cut a new release and we immediately built that, but we didn't build the bad release because the malware scans that we run detected that malicious behavior and blocked it in our system. There's a wide variety of stuff that we are doing to both scan the broader ecosystem, scan the stuff that we are building and running through our systems and then taking an approach to how we build these things from source that tries to eliminate classes of vulnerabilities.

We can't eliminate all classes of vulnerabilities, but we try and do what we can to eliminate classes of vulnerabilities where we can. There's building from source. There's things like compiler hardening flags that enable different types of memory safety things. There's potentially exploring replacing components with memory safe alternatives. Things like PseudoRS. And I think we were one of the first folks to ever package PsuedoRS, which is a Rust reimplementation of Pseudo. But there's a whole wide variety of these.

We've had the uutils project, which is a Rust port of the Coreutils project in our distribution forever. And so there's a lot of these different projects that are coming onto the scene that where possible we try and make those available. Rust CLS is another really good one where you can swap out the TLS implementation used by curl, libcurl, etc., with a version that's been written in Rust for handling cryptography.

[0:22:07] GV: Yeah, just moving on. I just want to briefly touch on one other case which was tj- actions. Again, we did cover a bit of this from a different angle in a different episode. Some may be familiar. This was related to GitHub Actions. Basically, this GitHub Action, it was compromised. And what that did then was extract CI/CD secrets.

And we have seen this through other attacks as well, like CI/CD pipelines are now a primary attack service. And at the end of the day, Chainguard images, they are built in CI. How do you guys think about security of the build pipeline itself as well?

[0:22:47] MM: All day every day. I think one of the things we've been saying from the very beginning is treat your build systems like production systems. CI/CD workflows operate with some of the highest levels of privilege in modern software delivery, right? But they often remain one of the least protected components in the development stack.

You worry about least privilege in production. I don't want this one microservice doing more than it should be doing. But then you've got your CI system that has permissions to push everything to production, right? And so it's got these incredible credentials, this incredible power to publish things into your production environments. And so we need to protect those.

I think the principles laid out in the Salsa framework are a good start. But I think one of the things that in my opinion I think it's the biggest miss in Salsa was not mandating the use of short-lived credentials. Because if you look at virtually every supply chain attack that has happened - and you mentioned some of the fun ones, tj-action, Shai-Hulud, they exfiltrate long- lived credentials. Some of them were made possible because someone got a hold of a GitHub pad or some other long-lived credential. Right?

I think that Salsa fundamentally missed by not mandating the use of short-lived credentials everywhere. And I think at Chainguard, we feel a little bit like Cassandra, whatever the story of Troy, right? They bring Cassandra back, and she's a prophet, right? She can see the future, right?

Several years ago, we built Octo STS to solve this problem for us, right? It is a credential federation service for GitHub because GitHub does not operate one. I've heard one's coming, but I think it's on their public road map. But this would have prevented a great many of these attacks. And we operated as a public good. Anyone can use Octo STS. And the one long-lived credential that sits underneath it lives in a KMS system that nobody can access without it basically paging our security team. But it gives you short-lived credentials. And I even flagged

that to the tj-actions maintainer when this happened because, like I said, we operate this as a public good.

I think another good one that's happened recently is the Trivy breach. This built on a technique that Billy Lynch, who is one of our engineers, disclosed to GitHub three and a half years ago. We call it impostor commits, which is this technique where folks are pinning a GitHub action to a commit. That commit actually doesn't have to live in the repo that it looks like it comes from the rest of the action line. It can actually come from a fork. If you Google Chainguard impostor commits, you'll find the blog post. But this is part of the technique that was used to compromise Trivy.

And so treat your build systems like production systems. Harden them. Use short-lived credentials. Practice least privilege. And make damn sure that every link in your supply chain is auditable, right? Just like you would in your production environments, so you know what came from where, what happened where, etc.

[0:25:49] GV: Yeah, that's great, great advice. Yeah, if anyone takes away anything today, take that away. Looking at like Chainguard as a product that, as we've been talking about, has evolved over the last of couple years. I'm kind of interested in, I guess, a few things actually.
The first thing, I guess, is actually just the number of images that you now actually maintain, which I think early '23 it was about 400. Then it was, I think, over a thousand by the end of '24. And now we're, I think, pushing over 2,000. And my guess it's like how do you guys think about just maintaining that consistency and quality? I mean, that's just a crazy amount of stuff to maintain.

[0:26:28] MM: Yeah, there's a lot. I think we're adding 200 to 300 images every quarter. And we've got an amazing team that has been building a lot of those, in some cases, by hand increasingly leveraging automation. But on the upkeep side right, 200 to 300 isn't necessarily growing super rapidly. But maintaining those images keeps growing, right? It grows by two to 300 every quarter, right? We're looking at automation really across the board.

And a lot of people talk about human-in-the-loop, right? If you think about the purpose of automation, it's generally getting humans out of the loop, right? We're slow, we sleep, and we

too can make mistakes. And we've always emphasized automation long before things like agents arrived on the scene. And we would try and automate as much as we could. And then humans would get involved when things broke down. And when we did get involved, we would look at ways of avoiding repeating that breakage as you might look at a post-mortem, right? You always reflect, and you're like, "Oh, how can we do better next time?"

And over time, those systems have improved. But agents have come onto the scene, and we aren't ignoring them, right? We are looking at ways we can safely use them. But the way we look at them is they're sort of yet another tool for pushing our automation further handling more of that without needing human intervention. But, ultimately, just because our automation is becoming agentic, just because you stick that word in front, it doesn't change any of our commitments or our SLAs. We're trying to find ways to adopt it responsibly and figure out those best practices. Because the way I see a lot of what we're doing is operationalizing best practices at extreme scale.

One of the very first places we actually employed agents was to bolster our testing across our inventory, right? Can it generate more tests that exercise different behaviors of libraries or packages and do things that could help us catch regressions coming in through new updates. And so we still very much have humans-in-the-loop, but the mindset shift is increasingly going from how do we solve a problem to how do we solve entire classes of problem?

[0:28:35] GV: Yeah. And the average patch time, I believe, is still incredibly impressive. A critical is like less than 20 hours. And then high is 2 days, something like that.

[0:28:44] MM: And we recently announced an SLA for the Kev list. If a vulnerability is known to be exploited, we have a 24-hour SLA for getting that fixed and in folks' hands.

[0:28:55] GV: That's amazing. I guess looking from a product perspective, but I guess more on the competitor landscape as well. I think maybe a few people might be thinking, "Hey, doesn't Docker now do this?" Yes, they do. And I guess that must have been an interesting moment for you guys. It does happen unfortunately to very great ideas of companies. The bigger guy comes along and says, "That looks interesting. We can do that too." GitHub, I think, I've done that. And GitHub actions was basically a clone of other startups into GitHub. Made sense. Yeah, how has

that affected things? How you approached this problem? Docker has a free tier that's, I think, interesting to talk about as well.

[0:29:33] MM: Yeah. And speaking of things that have changed in the last two years since we last spoke, competition has really come out. And I mentioned we had to convince people that what we do is possible. Thankfully, that's actually a conversation I have to have a lot less now that other competitors have come out with products that at least claim to do the same thing that we do.

I think one of the confusing things with DHI is that they claim two things only one of which can be true. One is that they are building on top of existing distributions like Alpine and Debian. They also claim that they're building everything from source, right? But if they're building everything from source, then they're not Alpine or Debian, right? They can't carry patches. They can't build versions that Alpine or Debian aren't carrying and build those things from source and then claim it's Alpine or Debian, right? Those two things are sort of in conflict.

I think the other thing is we were very familiar with the Alpine ecosystem and the Debian ecosystem, and we use them for many years. Our CEO Dan and I really built the first real hardened images back in 2017, 2018, which were called the Google Distroless Images. And we actually did build that on top of an existing distribution. We built it on top of Debian packages.
And it was because of this that we knew that to get to the results that Chainguard delivers that we couldn't build it on top of another distribution.

If you take any of those images which are running on the latest Debian, they're minimized and have the latest versions of all those packages. You take any one of those images and scan them with the exception of one, they all have unpatched CVEs. Because Debian is deciding what to backport, what not to backport, etc. If you scan our equivalents, none of those have any CVEs.

And so I think I'm glad that more folks are starting to take this problem seriously and not just because I don't have to explain that what we do is actually possible, but I don't think all of the approaches are necessarily created equal. And I think our approach is informed by an extremely

long history in this space doing things like this to harden these images and figure out the path that we actually have to take in order to achieve the result that we are getting.

[0:31:43] GV: Yeah, I think usually in this case it is so that rising tide raises all boat, or just like a bigger pie, or whatever analogy you want to give it. But it should overall be positive. But yeah, it's obviously also interesting when such a large company does come along claiming to do sort of a very similar thing. And I think that distinction that you've already made is very helpful. One other thing that was interesting. I think end of last year, you guys announced Emirates OSS. This is like maintaining deprecated open source projects.

[0:32:14] MM: EmeritOSS.


[0:32:16] GV: Oh, EmeritOSS. Yeah, I was like Emirate? Is that like to do with -


[0:32:19] MM: With the OSS at the end. Yeah.


[0:32:20] GV: Yeah. EmeritOSS. There we go. Nice. I was going to ask what the name meant, but you just - yeah, there we go.

[0:32:26] MM: Yeah. I think that the origin of this is that certain projects that are in some cases critically loadbearing for folks' workloads stop getting maintained. And so there was one that was near and dear to us. So one of the many, many, many, many projects that Dan and I and others collaborated on Google was actually the starter project for one of our engineering managers when she joined reporting to Dan at Google, Priya, was a project called Kaniko. She has since left Google and joined us, and leads our containers team actually.

This project, Google never really had - they never productized it. They never support it. But this project literally closed a 3, 4-year-old issue for how do I build a container with a Docker file on Kubernetes without mounting the host demon socket without running with privilege so that I can start my own Docker demon, Docker and Docker, stuff like that? The kinds of things you don't want folks doing in your cluster if you're an SRE or you're product security. Kaniko closed that issue. It showed folks this is how you can do it.

And I think Gitlab documented as one of the ways to build container images. I joke that I can't meet with a bank without them talking about how they use Kaniko for image builds. And then one day, Google stopped really merging any pull requests. And I think it took about 6 months before they openly acknowledged they were basically archiving the repo. And so we had a lot of history with this project obviously. And we knew how important it was to a lot of folks. And so we decided we would take it on and we would make sure that it was receiving patches for various security problems in the libraries that are in there, but we weren't going to do a lot of feature work.

The nature of sort of some point a lot of open source projects reached this "done status". And it was pretty mature. It had a lot of the features that it needed from supporting the Docker file spec. And we wanted to give folks a stable and safe way of consuming that piece of software where we would keep patching the security stuff but not do a lot of feature work that would potentially destabilize their workflows.

And so that was really the seed from which this idea for EmeritOSS. And there have been a few other projects that we've taken on with a similar sort of philosophy, right? We will build images from them. We will patch any vulnerabilities that come in through dependencies. If things are disclosed in the package itself, we will look into that and try and remediate those things. But we're not going to do feature work, right?

I think one of the great things about open source is if someone really wants to take on making substantial changes to a project, they can fork it and they can add their features to that. And if that fork becomes a successful alternative, then people can just start consuming that, and that's great. That's open source working as intended. And that can become the source of truth that folks use if they find the features that are being added valuable.

And so I think there's some great examples of really successful forks that become bigger than the original project. I mean, I think a lot of folks don't appreciate these days that Jenkins started as a fork of a project called Hudson, because Hudson was owned by Sun, and then Sun got bought by Oracle. And the person who started Hudson, Kohsuke, left Oracle and wanted to continue his open source project. He forked it as Jenkins. And now, who talks about Hudson? I mean, I think it's probably a vanishingly small set of folks who even know what it is.

[0:36:07] GV: I didn't know that story. I had not heard of Hudson. That's, yeah, super interesting. I guess as we kind of look forwards, or I guess in the present and forwards, you have mentioned that Chainguard is not just hardened containers anymore. Kind of using that framing. I think looking at things like SBOMs. This is Software Build of Materials. We've covered these in past episodes as well. This is more like runtime behavior. What is being fetched? What is the provenance?

This feeds into the fact that we do have regulatory frameworks coming in towards the end of 2027. One is the EU Cyber Resilience Act, which, okay, it's EU. But it will no doubt frame still a lot of what happens on the US side. And no doubt you have EU customers. So they have to abide by that. And that is saying mandatory SBOMs, things like 24-hour vulnerability reporting. Are you kind of taking in that kind of thing into where the product's going as well, or you're already thinking about? Yeah, what are you guys thinking about all of this?

[0:37:08] MM: I think it's a nice tailwind for us because I think we do a lot of the things that folks need in order to satisfy these things. All of our images carry SBOMs that we generate at build time. But we also go out of our way to make sure that SCA tooling that generates SBOMs works exceptionally well with our software and the binaries that we include in that software. And so we definitely do a ton of stuff to make it so that folks can either get the artifacts that they need from us or generate them from the images that we are producing.

And I think one of the interesting things just to sort of circle back to the Trivy breach, I think the European Commission itself was actually compromised by that Trivy breach, right? And so they disclosed that they were compromised by it because of that 24-hour reporting deadline. I think it was beginning of May. And something like 340 gigs of EU citizens' data was leaked as a result of that, right? And that's something that wouldn't have impacted them if they were using Chainguard's images, right?

And so I think that even if folks are seeking this really just to be compliant, there's a really nice security properties that you get out of adopting hardened images, adopting software that knows exactly what's going into it. I worry that when folks talk about things like SBOMs, that they aren't talking about the complete picture a lot of the time though. And so everything that goes into our images is captured in our SBOMs. And I can assert that because everything that goes into our

images comes from one of our packages. And all of our packages capture everything. Every file that ends up in the final images file system comes from one of our packages. And so we have this sort of chain of provenance for every single file, we know exactly where it came from. And you can trace the builds all the way back. But that's because that we own that full end-to-end.

And so I think that folks who aren't doing this, right? You look at the way folks traditionally have built container images with Docker files. Docker files, most of the like official images are bootstrapped from like a random tarball somewhere that has a rootfs. Hopefully, that's the set of packages that that rootfs was built from. And then they start to add stuff up on top of that. Some of those come from the package manager. Some of those curlpipe to bash in the Docker file build.

One of my favorite things to call out around SBOMs is what our CEO Dan calls the WordPress test. Right? If I have an SCA tool, can the SCA tool tell me that WordPress is in the official WordPress image on Docker Hub? And I think we found one that can actually tell you that WordPress is in there. I think it's the Black Duck scanner which uses a sort of signature, file signature-based scanning. But because the official image pulls down the source code and builds it in the container, and that's what you get, there's no metadata telling you that WordPress is actually in that container, right?

If you scan our WordPress images, because it comes in through the system package manager, it's represented in our package database. Every SCA tool on the planet can tell you that WordPress is in our images, but it's also in the SBOM that you're getting from us. I think one of the things that I worry about, it's not just mandating SBOMs that I think is important is this notion of dark matter, right? How much coverage does the SBOM actually have of the files that are in your system?

I mean, can I give you an empty JSON document and be like, "Here's your SBOM," satisfies the schema but there's nothing in it. It has 0% coverage. Does it pass the mandatory SBOMB check, right? And so, okay, that's an extreme example. But say I give you an SBOM that covers like 10% of what's in the image in terms of packages that are installed, does that count for mandatory SBOMs? This is one of the tensions that I think that I've always struggled with with people being like, "Yeah, mandatory SBOMs." It's like this isn't a yes or no thing.

This isn't a binary thing. We should look at it sort of like code coverage, right? What is the coverage your SBOM has of your image, right? Because an empty JSON document would obviously be zero. But I think the intent of it is to have 100% coverage. But they're not saying that, right? And I think that's one of the things that I just take issue with in general when folks talk about SBOMs, they're not talking about the bar you should be clearing when you are generating that kind of metadata about images.

[0:41:33] GV: That's super interesting. I guess these frameworks, and especially if they're trying to like implement them as mandatory at all, then well, I guess it's this question. Yeah. Is it mandatory? Should it be mandatory? Or should it be there's a bar, but then you say mandatory, and I guess they had to pick one? But yeah, I totally agree. I think this will initially lead to probably a lot of subpar SBOMs. I think at least getting the terminology and getting a lot more. I mean, I can think of so many developers if I was just to say SBOM, they have no idea what I'm talking about. That's going to be helpful.

[0:42:02] MM: Yeah. When the original executive order happened in the US, there was also this tension where like they didn't want to be prescriptive about what format you use. And there's two major formats right there. CycloneDX and there's SPDX, right? Which both have community backing, but they didn't want to pick a winner, right? And so there's no prescription around like what format you use.

And I think because of that, they also didn't really do too much around prescribing the structure because you'd have to do it twice and do all this other stuff. I would go so far as to argue that the package database that distributions include in their images is effectively an SBOM, right? And so I think that you could arguably check the box. Every SCA tool on the planet can generate you a fantastic SBOM from a distributions package database. It's the first thing any of them look at.

It's all the other stuff that doesn't come in through the package manager where things start to struggle, like the WordPress test I was talking about. And so that I think is one of the things that there's some tension around and yet another gap in the, "Okay. Well, what mandatory SBOM?" You're not talking about coverage. What format? What are the minimum elements? What is the depth of resolution? I could tell at a package install level. I could go down into the file level

within like a go binary file. I could tell you the transitive dependencies with respect to every version of every Go package that got pulled into that binary.

Then there's other fun things too. Go is really good about including that dependency information, but Rust doesn't do it by default, right? And so you have to enable auditable mode in Rust compiler builds in order to get that same metadata. And we do that by default in our tool chains because like I said, we go out of our way to make it so that SCA tools work really well with Chainguard-produced images and binaries.

But you can get images today from AWS. They build a lot of stuff in Rust, but they're not turning on auditable mode. And so there may be vulnerabilities in some of the libraries they're using, but you can't tell because that metadata is not there, right? And if that's not something an SCA tool can recover because the metadata is not there and it's not captured in the SBOM, who knows, right? You don't know whether or not you were vulnerable to that.

And so that's part of why we go out of our way to work really well with SCA tools in addition to capturing as much metadata as we can. But there's file coverage, package coverage, there's depth of resolution, there's how you actually structure all of that information as well. There's so many things that are not well scoped. And so saying mandatory SBOMs without any of that other stuff means it's really hard to satisfy that in a way that is particularly meaningful.

And so as a result of that, we do produce SBOMs, but like they don't necessarily have the same shape everyone expects. Because everyone think - you ask 10 different tools what the right shape to produce an SBOM is, you'll get 10 different answers. But if we work really well with SCA tools that can generate their own SBOMs and you can verify the image came from us via our image signatures, you can generate your own SBOMs from our images that we know will work with your tool.

And so that's why we do it sort of both ways, right? And we don't think that the SBOM space has really matured enough that just generating an SBOM is enough. And it's because of all of these unanswered questions about shape, coverage, depth, etc., that I think make it hard thing to satisfy at least in a useful way.

[0:45:32] GV: Yeah, absolutely. Kind of related like as we're sort of cruising to the end of the episode, but there's a couple things. First is related to what you've just been saying, enterprise adoption, what friction points could you, I guess, speak to that's not a friction point? It's just a perceived friction point. And actually, I guess, using Chainguard is you should just talk to us.

[0:45:53] MM: There's definitely some FUD out there about stuff like migration and things like that. Fear of the unknown. It's like I don't know how hard it is to migrate, so it must be impossible, right? And so this is some of what we deal with coming from other distributions, like Debian, like a RHEL-based distribution. And so, I think that we've gone out of our way to accommodate in various ways as broad a base of compatibility as we can, right?

So, we use the same package manager that Alpine does. But one of the big sources of incompatibility in Alpine-based images is the musl libc. They use a different libc than virtually every commercial distribution on the planet as well as many of the non-commercial but widely used distribution. Debian, Ubuntu, all the RPM based - Amazon, Linux, RHEL, etc., they all use glibc.

And so when we bootstrapped our distribution, even though we decided to use the APK package manager, we bootstrapped our distribution around glibc to get that sort of maximally broad compatibility. And so I think there's definitely a lot of FUD around migration, but it's definitely a spectrum.

And I think the main thing is, okay, depending on the package manager you're coming from, if you're coming from Alpine, it'll look very familiar; apk add package name. If you're coming from Debian, yeah, you'll have to change apt-get into apk add. Or if you're coming from the Red Hat ecosystem, you'll have to change yum install into apk add.

But we have tools that help streamline this, make this easy. We have a new tool that we call the gardener that helps drive a local migration for you, leveraging our agents to help you do some of that package mapping and stuff like that. If you're coming from Debian, we can be like, "Oh, this package is called this in the Debian ecosystem, but it's this within the Chainguard ecosystem." Or if you're coming from the RHEL ecosystem, it may be called something else, right?

And so mapping package names, doing those kinds of things and really helping you drive that further and further to completion. And we have a bunch of other stuff that we're working on to sort of smooth migration. But in some cases, it's really just a matter of dropping one of our images in. The vast majority of our images are application images which most folks just take and run, right? Something like an envoy, you typically plug it in and run it as a proxy with some configuration. And maybe you're running something like Istio. How many folks are modifying their Istio images? They're mostly just taking it and running it through a Helm chart or something like that.

I think the one caveat to that is folks who want to inject their own root certificates into it. Maybe they are using something like Zscaler, something like that. But other than that, it's actually relatively uncommon for folks to be modifying the application images. And so those will just drop in and run as the application. You don't have to worry about it having a different package manager, you're not installing packages. And then when it comes to migrating stuff like Docker file-based builds, we have tools that will help drive that migration and help you get as close to that finish line as possible, if not all the way there.

And then with things like our library products, a lot of times our customers goal is to effectively just turn a switch. A lot of our customers will proxy our language packages through something like Artifactory, or Cloudsmith, or Nexus, right? And they've already done that to sort of insulate their internal networks from developers just being able to pull whatever.

And so when you have that kind of mature setup where you can regulate what packages your developers consume and it's coming in through Artifactory, maybe it's just passed through from the outside world, but you have that cut point. You can cut that over from pulling from, say, Maven Central to pulling those from Chainguard. And your developers may not even know that you switched it to Chainguard.

On the library side, it's much more transparent in terms of dropping in and replacing things. That's our goal, to be as drop-in as possible. But I think some of that can deviate a little bit, it's the same migration problem you'd have. If you want to move to one of our images that doesn't have a package manager, it's the same as moving to a Google Distroless image, which half of this cloud-native ecosystem is using these days. It's clearly possible.

But if I were to move from a Debian-based image to a RHEL-image, I would have to rewrite my package install lines anyways and figure out the package mappings and get it working again anyways. And so any sort of migration like that will have a little bit of work. But folks are already going through those kinds of migrations too, one LTS release to the next, and where a lot of system packages are changing.

And so we try and give folks something where we aren't an LTS-based distro. We are a rolling distro. So if they migrate to us once, which we can help them do as smoothly as possible, they shouldn't have to go through that every four years when there's a new LTS as well, right? We can hopefully be the last migration that you ever do. And we, like I said, are going out of our way to make that as easy and painless as possible.

[0:50:54] GV: Yeah. Amazing. For anyone, I guess, at that sort of enterprise level, but medium company, whatever you want to call it, just that on-ramp isn't maybe as scary as it sounds. Very briefly, we were talking about it before we started recording, Anthropic Mythos, Fable model, or Fable, which is derived from Mythos. Super curious. This is very bleeding edge because it's only been announced in the last 24, 48 hours. But how does that affect your day-to-day? Or you've been, I think, using it a little bit. What are your first thoughts?

[0:51:23] MM: I think when it first came out, everyone freaked out. And then there was a whole group of folks who were like, "This is just marketing." I've seen some of the findings. It is not just marketing. These are real vulnerabilities. It has very good capabilities. There are several different aspects of it.

I think I mentioned the Firefox numbers, right? They found a whole slew of vulnerabilities that other agentic tools that they had been running weren't finding, right? And I think one of the things that makes it exceptionally interesting is its ability to chain together multiple vulnerabilities, which in real time we're watching it close the skills gap that's needed. This notion of sort of scarcity, right? Right now, there's the skills needed to find and weaponize vulnerabilities are scarce. It's hard. It takes time, right? And that time is dropping precipitously.

And so I think the Fable model that they released today has all kinds of security controls. The first thing I threw at it mostly for grins was to audit security isolation tool that I've been having

Opus just audit in a loop and find and fix vulnerabilities in this abstraction. I've been very impressed with how good Opus has been. I've been trying 4.7, 4.8. So, I threw this at Fable, and I got this prompt back being like, "It looks like you're trying to do security stuff. I'm downgrading to Opus 4.8." It wouldn't let me do it.

They said in the announcement that they had put in security controls around keeping folks from doing this for vulnerability research, and it caught me. So, I think there are controls that will keep folks from using it in that way. Folks always find a way around those controls though. And so I think the thing to bear in mind is this is not marketing. This is very real. This will find tons of vulnerabilities that other models probably haven't found yet. And folks need to be prepared to patch things at machine speed because you could already weaponize vulnerabilities at machine speed.

And now with Mythos coming into the picture, being able to find even more and more effectively weaponize vulnerabilities at machine speed. If you aren't prepared to patch things at machine speed, which is really what we've been building towards at Chainguard since before we even realized this was a problem folks were going to have, you're going to be pretty unprepared for the world of hurt that I think is going to play out over the next 6 to 12 months as at least the initial tranche of Mythos vulnerabilities become known to the world.

And then it's a muscle you need to build even if you get that initial tranche patched. Because I would be very surprised if we didn't continue to see step function improvements in model capability, the quality of the harnesses for leveraging the models to find these vulnerabilities. And it's going to be an interesting next year.

[0:54:11] GV: For sure. It really is. On that note, thank you so much again for coming on again. Learned a lot this time as well as I, and I think the audience did last time. Yeah, who knows?
We'll maybe get you back on in another two years. We'll hear. Because this landscape is just changing so much and so fast. Thanks again. And I'm sure we'll catch up again.

[0:54:29] MM: Thanks for having me.


[END]