EPISODE 1865

[INTRODUCTION]

[0:00:00] ANNOUNCER: Homebrew is a widely used package manager that simplifies the installation of open-source software on Mac OS. It was created in response to the growing demand for a lightweight developer-friendly tool suited to an increasingly Mac-centric development ecosystem. Today, Homebrew is a near-essential part of the Mac OS software development toolkit. Mike McQuaid joined the project early on and collaborated closely with its creator, Max Howell. He joins the podcast with Kevin Ball to discuss Homebrew's origins, architecture, its emphasis on automation and CI/CD, long-term sustainability, controversial trade-offs, and much more.

Kevin Ball, or KBall, is the Vice President of engineering at Mento, and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meet-Up, and organizes the AI in Action Discussion Group through latent space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

[INTERVIEW]

[0:01:16] KB: Mike, welcome to the show.

[0:01:18] MM: Hey, thanks for having me, Kevin.

[0:01:19] KB: Yeah, excited to get to talk to you. Let's maybe start with a little bit about you. Can you talk about your background and then how you got into Homebrew and where we're going to go today?

[0:01:29] MM: Yeah, sure. I feel like in tech, I've had two lives, right? There's my maybe a little bit being a really rubbish superhero, right? Where, I guess, my commercial job-related life, I'm a guy from Scotland, I interest in computers, did the computer science degree thing, got the tech job thing. I've been doing that since 2007, just getting jobs, changing jobs, paying the bills, having fun, all that stuff. But then there's my open-source life, which is generally, what's of more interest to people with Homebrew and all that type of background.

My love of open source, I guess, probably started while I was at university. I came to university, I heard about when I was in high school, people talking about installing Linux on the machine the same way that you might talk about taking illicit drugs, or whatever. It was this risky, dangerous, somewhat admirable thing. Yeah, so I got to university, a bit of a Windows power user. Had not done meaningful programming, really, beyond just trying to get games work in my computer and peer pressure, happened pretty quickly. I was like, okay, I need to get on the installing Linux on my desktop bandwagon.

Got Linux on my desktop machine, got a little home server of an old computer running at my parents' house, so I could send data back and forth and all this type of stuff. Yeah, and obviously, desktop Linux, one of the nice things about it, I mean, probably even now, it's certainly back in 2003 for when I was first downloading with it is like, open source is front and center. You very quickly realized like, oh, this is a community of people building a thing.

As much as I feel like I'm a consumer and I'm using this thing, if I file a bug, there's a very real chance that the guy, or girl who wrote that is the person who responds to my tickets. Yeah, so I guess, that growing awareness. I started helping out with things and helping out on IRC channels and bug trackers and forums and all this type of stuff. Then forking stuff myself, modifying it a little bit, publishing up the changes in case anyone cared. Yeah, but I guess, for me, the main thing was probably like, I did Google Summer of Code back in 2000 and, what it'd be? 2007, for the KD desktop environment. Still around. Yeah, and I basically fell in love with the open-source community through that, really, from spending a three-month project, just running lots of code, seeing very high degrees of trust and mentorship and help and whatever.

Then I just went on from that, being a huge proponent of it in my work life. Then, I guess, the two streams probably crossed again in 2009. I was in London working for a startup called Mandalay, across the other side of London, friend of a friend, was a guy called Max Howell, who is also working in London, we start called Last FM. That probably brings back memories for a certain number of people.

Yeah. He had been tasked with doing various bits and pieces and building desktop applications. I believe, the official story is he was in the pub one night complaining about all the package and it's just being terrible. Someone said to him, "Well, if you hate them all so much, why don't you make your own one, right?" He left the pub, went down, wrote a outline of what Homebrew should be, started building it. Then I got involved, I can't remember what it was, like four or five months later, heard about this thing. I'd tried to build something vaguely similar myself. Yeah, and that was, I guess, 16 years ago, like in September that I've been working on this thing. Yeah, and that's my open-source story, I guess.

[0:05:19] KB: That's awesome. Yeah, I feel like a lot of us have that experience with open source of like, this stuff is all crap. Wait, with this stuff though, I can actually do it myself and make it my version.

[0:05:28] MM: Yeah, totally.

[0:05:30] KB: I feel like, in today's age, especially with Max having become, at least in the US and much of Europe, the de facto development environment for many, many developers, a huge number of our listeners are probably Homebrew users. But I don't know how many of them have actually looked below the hood and how it works, or why it became the de facto norm. Why is this one not terrible the way that all the package managers that Max was complaining about are? How would you describe like, what are the core choices, or approaches that have made Homebrew so successful?

[0:06:04] MM: Yeah. Before we get to that, I just noticed this is, again, I've been working on for 16 years. It's the first time I've ever thought of this, I think, because you were saying in that sentence and you're talking about Max, the Apple computers and Max, the creator of Homebrew. It's only just twig. I'm like, "Oh, yeah. That's a fun little coincidence that -"

[0:06:20] KB: That is a fun one. Yeah.

[0:06:23] MM: Anyway, yeah, so how Homebrew works. I think one of the things I've found in my career in general is I'm a maintainer. I'm not a creator. I'm someone who is very good at taking something that someone else has made and riffing off and evolving it and growing it and blah, blah, blah. I am rubbish at coming up with new ideas myself, right? A lot of the brilliance of Homebrew, I think, goes back to Max's original design.

I guess, the less interesting parts of the design first, but people might not know this, so let's tell them anyway. Homebrew is and was from day one written in Ruby primarily. At the time, Ruby and Rails was starting to take off. Ruby was becoming a popular language in 2009. Nowadays, it's the same. There's still a bunch of codes that probably goes right back to the first few commits. If you squint a lot older, you can see that the DSLs and stuff that are being used in Homebrew today are still much the same as back then.

I think that brings you to the first part of our move that I think was initially brilliant, which is something Ruby is very good at is building these DSLs, Domain Specific Languages, I guess, we would call them, where because Ruby coded can be at least made to look very much like just normal English, right? And because Ruby is a very low syntax heavy language, you can build these things that look a little bit more like you're just declaring - it almost looks a little bit like YAML, without as much indentation stuff, sometimes, where you can just look you're declaring data, but you're actually calling functions inside a class and things like that.

As a result of that, Homebrew start off with these things called Formula. The Formula is basically a definition of how a package is built. Originally, Homebrew started off being from source package managers, so everything is building your machine. Nowadays, basically everything is built by someone else, usually Homebrew, somewhere else, elsewhere. That initial formula description was very, very easy to read and write and contribute to. I think that was a key thing to begin with. If you've ever worked with other package managers, particularly back in that era and had to package something in AppGet, or RPM, or whatever, it was a nightmare, frankly. It was horrendous, how many steps it would take and how many things you'd have to do to pass all the lints and test all this stuff when you look at machine. Whereas, the Ruby formula made it very easy to read and understand what's going on and design and contribute.

That's the next thing I think that was the really smart choice. I was speaking with a CEO recently, who said something on the lines of, all the best engineers are lazy, right? Often, engineers that he might have a problem with are the ones who are being insufficiently lazy, not working hard enough. But there was a certain amount of that from the outside with Max, where he said like, "Hey, if this package manager is going to be successful, we're probably going to have 5, 6, 10,000 packages." I don't want to maintain 10,000 packages. I also don't want to go down the route of something like Debian or whatever, where I have to find a thousand people who are willing to maintain 10,000 packages.

This was around the era that GitHub was starting to take off. He basically was like, "Right, we're going to design the package manager, so it's a GitHub based workflow." Homebrew actually predated pull requests even on GitHub. The original flow, I mean, the first few commits I had would be IOB on IRC. I would DM Max with a commit in my fork. He would be like, "Yeah, that looks good." He would then cherry pick that from my fork and then push that to the Homebrew repo. Then over time, we ended up pull requests and reviews and all this type of good stuff.

Yeah, I think that model from the outset of designing it to be both very, very easy to contribute to, but also designing community contribution and community maintenance as being a core part of the overall flow of Homebrew is, I think, what resulted in it being as successful as it's been.

[0:10:40] KB: That's fascinating. I think for open source in general, the packages that sustain themselves seem to be those, who, that managed to build that community early. So, thinking about that from the beginning is great. Another thing that I think makes Homebrew different from at least the package managers that went before in Mac land and also a lot of the Linux package managers is it's pretty much all user space, right?

[0:11:06] MM: Yeah. Yeah. That's one of the nice things about Mac OS, and one of the reasons I'm still a fairly diehard Mac fan today is back in the Linux package manager days when I was working with that, I mean, the Linux package managers, I would still say to this day that I don't think AppGet is better than Homebrew in every way, but I think overall, it is a more powerful package manager. It has a lot more stuff, a lot more sorted than Homebrew does.

Always, there was a slight weirdness to me always, where essentially, some random utility that I'm pulling off someone's GitHub, that essentially, if it breaks, it has no consequence to me. My libc, or kernel are essentially managed in exactly the same way, as far as the package manager is concerned, right? One of the nice things about Mac OS, or I mean, even nowadays, again, if you look at it from a funny perspective, things like WSL, or there's Linux issuers that are doing this now as well, where you have a immutable file system like a modern Mac OS does and Homebrew in Linux is taking off on those as well, is this idea that as you say, you have this user space package manager, which is living in a non-protected part of the file system. You don't need to run pseudo to run it and stuff like that once it's installed.

Then basically, you limit the damage that that can do to your system. Nowadays, I think partly because of maybe Homebrew becoming more popular, the Mac OS-based system is super locked down. You can get access and modify things if you want to, but you're going to have to boot your Mac in a special mode and blah, blah, blah. That basically just means that developers on Macs now don't have to fiddle with things in user bin. User bin is up to Apple, like October bin is up to Homebrew and you can just remove auto bin and every at least desktop application on your system should just continue to work fine as is, or your programming projects may explode, but it has less of a degree of concern that you're actually going to brick your system, or whatever, which was at least theoretically possible in the battle days.

[0:13:37] KB: Yeah. Well, and it makes, for example, staying up to date with Apple's upgrades a lot less painful as well. I feel like, it used to be I would, and I still have some of these instincts, but I would resist forever upgrading to the newer OS's, because I knew my whole dev environment would break and I'd have to go and rebuild everything. I feel like that problem has more or less gone away.

[0:13:59] MM: Yeah. I'm glad to hear your perception is that problem is mostly going away. I mean, to be fair, Apple are better on some of the stuff than they used to be, but it's definitely an area of pain that projects like Homebrew mainly feel. But in some ways, if Homebrew can do one thing well, it's providing a abstraction layer over all of this stuff, so that we have to care and worry about and fix this stuff and you don't. Yeah, that's very much a goal of our package manager is to isolate -

[0:14:33] KB: I wouldn't say it's completely painless at this point, but it's - I mean, it used to be days. Now, it's like, "Oh, I've got a couple hours of cleaning things up."

[0:14:42] MM: Yeah, for sure.

[0:14:43] KB: Let's maybe, actually, then peel back that abstraction layer a little bit, because I as a user, I just know the APIs you provide. What are actually the key pieces that go into making a package manager work?

[0:14:56] MM: Yeah. I think Homebrew is a bit of a special snowflake of a package manager in lots of ways. I guess, I've mentioned some of them already. Some of the package managers that have come after have really leaned into the same model of community contributions and stuff like that. Some haven't. I think one of the things we do that often surprises people is we have, I guess our, our stats and best estimates or whatever are like 5, 10 million relatively active users of homebrew, which is a scary amount of people.

Then in terms of maintainers, we have 30 people, right? Probably on a day-to-day basis, probably 10 to 15 of them are active, and you tend to have this parallel function of even within those maintainers there's some people who do a really disproportionate amount of the work. In the total history of all Homebrew maintainers ever, I think there's been probably less than the 60 people, maybe even less than 50 people. Obviously, that's good scaling, like how you can get a relatively small number of people to provide that amount of service. Some of that is the contributors I mentioned before, like getting third parties to go and help and submit changes. Probably got like, yeah, I mean, definitely over 10,000 contributors. I don't think we're anywhere near to a 100,000, but in the five figures there.

Still, there's scaling effects and you're like, how can you make that happen for 20,000 packages, which are getting probably 10, 20, 100 updates a day across that. Something we have done, certainly at least from the very early days of me being involved with Homebrew is going back to what we said before about laziness, like I'm an exceptional lazy person. I like the people I work with to be lazy and I really encourage that. One of my positions for a long time, which is funny, and now we've got AI, because it's encourage you to think about that in a different way still, was almost like, I guess, I wrote a blog post about this called Robot Pedantry Human Empathy, right? Because at the time, I felt like I was seeing people were almost being like, "Oh, I'm going to write a bot to thank first time contributors to a project, or whatever."

My observation was is that like, if people have a bot doing it, it doesn't mean anything. They don't really feel valued and considered by a bot. On the flip side, people are very tolerant of bots being - I don't know if you've ever had one of those pull requests reviews in a work environment, whatever, where -

[0:17:27] KB: That were brutal?

[0:17:28] MM: Yeah. Well, not even brutal. I would say, excessively pedantic, right? We're saying, you're using semicolons in JavaScript the way that they don't like and you did it 20 times, and they go through. On every single occurrence, they're like, use semicolons. You just do it. When a human does that, you hate that person, right? Whereas, when a, why would call a robot, or a CI job, or a linter, or whatever it does it, you're like, "That is actively helpful. If you didn't tell me all of the occurrences, you would not be a useful tool."

Something I try to encourage in myself and in others in Homebrew from the early days was being like, okay, any time you are regularly making the same comment on, say, three pull requests, figure out a way to turn that into an automated check that can run, that can make the CI read, so that the user has that indication that there is a problem here. I guess, to adopt horrible tech industry jargon, shifting things left as well. Also, that idea of like, okay, ideally, we move from a human comment to being a CI comment on a pull request or whatever. Then ideally, still, we move from a CI comment to a local development environment comment.

Again, we have a bunch of automated checks. The most common linter in Ruby land is a tool called Rubocop. We now have it configured, so that if you open Homebrew in VS code, you'll get prompted to install the Rubocop plugin. If you install that, then when you're writing Homebrew code, we will pop-up straight in your editor and tell you like, "This is against, essentially, the description of this formula is not in the format we like. You have started with a or an and we don't like you start with a or an, because it doesn't look nice in our output, or whatever."

Then you can have a user who's working on this for the first time who gets that input straight away. If you've enabled auto formatting in your editor, then some of those Rubocops have auto-correction. You could start your description with an a or an, I can't remember whether this one does auto-correction or not, but we'll assume it does for this anecdote. You could type that in and you press save, and if you've got format and save enabled, then you'll just see that text just disappear. Obviously, that contribution experience is delightful when all the stars align and all that stuff happens. Instead of a human being like, "Don't do this, don't do this, don't do this," your editor is automatically making your codes the way we need it to look, so it passes the I.

Even if it does make it to CI, again, we have a pretty time soon distributed team now, so we probably have decent follow the sun coverage. Before we had that, there was something really delightful about seeing someone open pull request, see our linter spits out 10 warnings, they read them all, they action them all, they improve everything, then you wake up in the morning and you're like, "Oh, there was 20 things I would have said in the first version of this pull request." Instead, our CI tooling and the robots, or whatever said all this. The person has got it ready and now I can just merge it straight away. Then everyone has a better experience, and that's how the projects can scale dramatically better in that way.

I also might, while I'm on my soapbox, my personal experience is, a lot of companies could do with moving a lot more in that direction they think as well, because generally, humans don't like doing that stuff and generally, pretty much every software company in the world, if you're like, I can give you a way to move faster without negatively impacting your quality, most people and most companies, most developers would say, "Yes, please." That's how you get there. You get there by very heavy, but reliable automation.

[0:21:02] KB: There's a mindset shift there that takes a while to sink in. I find, I'm climbing that whole mental climb again now with these AI coding tools, because there's things they're good at, there's things they're not good at, but a thing they are very good at is writing tools, so you can use them to automate your check that you are going to do. It's mind-boggling how much faster you can go as you accumulate those.

[0:21:25] MM: I'm in the same space right now, where I would say I like AI, I'm a user and mild optimist, but it's figuring out the places in which these tools are very well suited and very badly suited. Because it's like, all the expression used to be of, I can't remember where I first heard this, but your average not tech person would be like, "I hate computers, because they never do what I tell them to." Then the tech person responses, "No, you hate them, because they do exactly what you tell them to," when that's not what you mean.

I think it's interesting, because I feel like a lot of the time, some of the modern LM tools are getting better at inferring what you meant, rather than what you told them to do. Then on the flip side, the responses are now non-deterministic. Those examples before with the CI and the linting or whatever, I'm sure I could describe and prose and feed that into ChatGPT and have that somehow run in my CI pipelines. The problem is if you rerun that on the same code three times, you're going to get - you don't get the same response every time, without some deterministic code layer living underneath or on top that is making sure that that stuff works as it does. Yeah, I still feel like that's a big point of growth that we're still figuring out is figuring out how do we make these two tools play together in the nicest possible fashion.

[0:22:53] KB: The automated validations that you're doing are actually great for that, because if you can put this non-deterministic thing in a loop with deterministic validation for correctness, your quality gets so much higher.

[0:23:06] MM: Yeah. Yeah, I've definitely felt that when I've been doing stuff. I'm yet to really go hard on agentic stuff, particularly agentic, not in a window in the foreground approaches. It's on my to-do list to play around with that in the next few weeks. But even on that, I definitely find in tools like cursors, agent mode or whatever, if you can say, here's how you run the test, here's how you run the code, here's how you type check the code. Homebrew nowadays uses Sorbet, which is a type checker built by Stripe type checker, runtime type system. Runs exceptionally fast to check the entire code base.

Then, yeah, the more of those guardrails you have, then the better these tools do, because they're able to validate their own behavior without you having to say like, "Oh, no. That should be an integer being passed this method, not a number one in quotes, or whatever."

[0:24:04] KB: Let's come back a little bit to the pieces that make Homebrew work. We talked about the automations for contribution and the key piece of formula, which are building DSLs. What's the engine inside of Homebrew? Presumably, you have some dependency resolution, or a thing like that going on. I don't know the software architecture of a package manager.

[0:24:26] MM: Yeah. I mean, I don't know that I know the software architecture of a package manager either, but I can speak to, yeah, Homebrew basically, where you touch upon dependency resolution. That's the most common thing of value in some ways that a package manager is providing. If you don't use, or have never really thought about a package manager before, essentially, the two key things you might say that they are doing for you is one is just a way of essentially specifying, here's a bit of software. I want this installed. You figure out however that gets installed, and the way of indicating you want two pieces of software installed should be the same, even if the underlying build mechanism distribution mechanism, whatever is radically different between those two things. It's that abstraction layer.

Generally, as well as that, there's some version-based behavior as well. This is something where Homebrews go a little flack over the years and we're better than we used to. Again, to let's differentiate between different types of package managers for a moment, right? You have, I guess, what you might call, I guess, you were saying user space earlier, some might say system package managers, which would be Homebrew would be an example, but definitely something like AppGet on Debian, or Ubuntu, something like Yum or whatever on Fedora. I think that's been called DNF maybe now. I can't remember. It's been a while. Emerge on gentoo, Pacman on Arch, etc. Those are responsible, generally, for installing everything that might be on a system or not.

Again, you can debate whether Homebrew is one of those or not. What homebrew is definitely not, which people are often more familiar with, is a language package manager. Say, something like NPM, or cargo in Rust land, or pip in Python land, or gem or bundler in Ruby land, etc., etc. These packs managers where everything you install through that thing will be written at least primarily in that language. They may also have C extensions, or whatever, depending on the language. Essentially, you're installing libraries mainly for the writing software yourself in that language.

One of the big differences between the language package managers and the system package managers is the language package managers generally have this model of like, okay, well, we basically can let anyone publish anything and ultimately, the people who are in control of uploading a new version and deciding whether people get upgrades or not are the publishers of that package and generally, the authors and the creators and the repo owners on GitHub.

Whereas, with system package managers, generally there's a bit of a like, okay, because of the dependencies, which we mentioned earlier, you need to verify that it is okay for everyone to get a new version of a thing. If I decide I have some library that 1,000 packages in Homebrew depend on, I release my version two and I say, "Okay, Homebrew. We're ready for a version two, the package." If I'm in entire control of that happening, then again, depending on the package manager, that might be okay. It might be that they can just start with version two and then everything sticks on version one until we manually change it and it says that it goes to version two.

Homebrew was designed from the early days to be, again, more package manager terminology, I'm afraid, what you call a rolling release package manager. Things like Ubuntu or Debian, you might be familiar that they have every so often, they've got, I can't remember what, and I've been to one is, but it's Poetic Pelican. It's always like a two letters the same, or Debian, Buster or whatever. Generally, the way they do that is they say, okay, we're going to branch off a while before we essentially get all the packages, so they're vaguely stable together. Then we have a point where we say, okay, we've released this new thing. We basically are only going to do security updates and bug fixes, until there's ways of you working around that. But by default, that's how we're going to do things.

Then you have a rolling release package manager, which is what Homebrew is, which is essentially, you get the newest version of everything all the time. If you just type brew install some package name, say, MySQL, in Homebrew, at least that's simplified things and look at Homebrew 10 years ago to start with. If you type brew install MySQL, you will always get the newest version of MySQL. If yesterday, whoever it is that owns MySQL nowadays, Oracle dropped a new major version and we're on MySQL 10, and it has no backwards compatibility with MySQL 8, tough luck, right? All your shit's broken. But Homebrew is internally consistent, so it's fine.

We evolved from there to being like, okay, well, then we're going to at least have CI, where we're going to test and make sure that everything works within Homebrew's ecosystem at least, so you're not breaking any other packages in Homebrew. That's one of the fun artifacts of our CI is that when that happens with something like OpenSSL, which has thousands of dependencies, what that looks like is a CI job that will take two to three days to run, of continuously churning away for 40 hours. We've had various CI providers and hosting providers who assume that that's a typo. We say that like, "Hey, your timeouts are triggering too long. They're kicking in after -" I think once I was like, "We're kicking off to 40 hours." They're like, "Oh, do you mean 48 minutes?" "No, 48 hours." "Oh, yeah, that's surely, it's not taking longer than 40." "No, yeah. This job takes up to 72 hours to run." That's been a little bit of a surprise.

Again, that's part of the reason why people, when they might critique Homebrew and Homebrew's model, it's like, yeah, it's hard to do this stuff at this sort of scale. Then more recently, Homebrew has done a bit more around versioning. Now you can install MySQL at 5.7, right? That means that you will sit on that version forever. But because of Homebrew's original architecture, the way we do that is we maintain a separate MySQL at 5.7 package, and we have to maintain that indefinitely. If there's something that has to happen to all the MySQL versions for a new Mac OS version, we have to essentially port it between all of those, right? There's a little bit of overhead there, which is why we don't do this for every single package all the time.

[0:30:59] KB: Yeah. That is interesting. What CI/CD provider are you using today?

[0:31:05] MM: Nowadays, we use GitHub Actions, but we have our own self-hosted runners for doing a lot of the hard work, basically. Particularly on Mac OS, essentially, we need the newest version of Mac OS pretty much as a month, two months before Apple released the new version, because we want to be able to test things, or verify things, or whatever. We would ideally use an entirely hosted solution. We are yet to ever find an entirely hosted solution who can do things as fast as we need them to be done, but also provide 72-hour timeouts of things. 

Our self-hosted runners used to be, again, a little bit of fun open-source lore, it used to be physical Mac-minis that were originally installed in a data center by me taking them down in a suitcase from Scotland on a train, which ended up going in the car of someone whose podcast I listened to, who let me stay in his house and then put them in the data center of his ISP. Then later, I took another train and then moved them up back to Scotland and blah, blah, blah. Yeah, so there's a bunch of fun games that happens with this before this stuff was available for wide cloud providers.

Nowadays, there's a company called Mac Stadium, who we've worked with for a long time now, probably coming up to 10 years, who provide hosted Macs for us to use. They've got a nice Kubernetes-like abstraction layer that lets us spin up and spin down VMs of the various Mac OS versions we need to run. Yeah, so that's basically what powers our CI, the actual, where the code is being run and built side of things. Yeah, but then the main almost centralized server now is all on GitHub Actions.

[0:33:00] KB: Who funds that? Do they donate those resources?

[0:33:02] MM: Yeah. Originally, Mac Stadium donated all of the resources. Then Apple helped when we were in the Apple Silicon transition, that's the first time, I guess, Apple gave us something, which was nice. They got us access to basically, to a bunch of free Apple Silicon hardware. To be very clear, this is completely fair enough of Mac Stadium. Essentially, we kept on being like, we need more, bigger, faster, etc. They were like, yeah, I think you can pay for some stuff now.

Basically, we pay Mac Stadium, but we receive a very heavy discount. You can see if you're interested in Homebrew's finances and financial situation, I guess, that's another fun little thing where all our finances now are on, I think, called Open Collective, which is essentially, if you imagine your personal banking app, you can go and see all the transactions and oh, 2-23, on this day you spent this amount with this vendor or whatever. It's essentially that, but the ledger is public. Kevin, you can go and see how much money Homebrew received and spent and from whom in the last year. You can also see, we do a small maintainer stipend of whatever it is, like $300 a month for people who are still active on the project. You can see who got that stipend and when and how much and when it was paid out and all that type of stuff.

Obviously, this sounds like a privacy accident waiting to happen, but hoping to be very good about making sure everything that should be private remains private and whatever. Yeah, it's quite cool. It's a quite cool open-source way of doing funding, where rather than having very strict rules and how money can be spent, essentially, you have this open ledger approach. Yeah, that's essentially how that's funded. You can literally go and see how much did we spend with Max Stadium last month and all that type of stuff on our open collective, if you were so interested.

[0:35:07] KB: Yeah, that's actually wild. I'm looking at it, just glancing at it here. That's very cool to have that level of transparency and to have for a registry, you need a lot of resources. So, having that visible and having people able to see like, hey, this is what it costs to make your life just work like this.

[0:35:28] MM: Yeah. I mean, we're lucky as well. I guess, I would be remiss if I didn't mention, like we get a lot of free resources from a bunch of companies as well. DNS simple, one password, formerly Digital Ocean in the past. We basically have a lot of vendors who've given us a lot of free stuff, which is very nice and is obviously, I don't know how much of that reaches across the Atlantic to where you are, Kevin. Suddenly, Scotsman such as myself are stereotypically very cheap and very careful with our money. My preferred price for any vendor arrangement is always free. That is what I've always attempted to negotiate. Yeah, we very like to have that.

I guess, the biggest one, a former employer of mine, and it's been easier to be a bit more sycophantic about them before I work there and after I work there, because of Homebrew both ends is GitHub has contributed a huge amount to Homebrew in terms of why they both given us a bunch of money, but also, when you download any of Homebrew's binary packages nowadays, that's all on GitHub packages infrastructure, which is primarily a Docker registry, which is why it might be a little bit confusing as to how we are using that for our stuff.

Yeah, but basically, we had a situation, I can't remember how long it was, five years ago, maybe, where our previous hosting provider, Bintray, with about 90 days' notice were like, "Yeah, we can't post your stuff anymore. Sorry." We had to find the new provider and migrate everything over there and whatever. That was very good of GitHub to be willing to do that. I remember at the time seeing, essentially, when the switch ever happened, seeing all of the internal graphs of usage and being like, yeah, this is - GitHub actions has had a lot more usage over time. At the time, it was 50%, 75% of the usage of the entire system was people using Homebrew. We couldn't do that without a big, funder like GitHub. I feel like, we're also used to GitHub at this point that we expect -

[0:37:27] KB: It's just infrastructure. We expect it to just work, right?

[0:37:30] MM: Exactly. We expect it to all be, oh, just work all the time and be free to everyone, so that everyone do your open source. Yeah, like it. I think a lot of my former co-workers have and do work very hard to make all this stuff work. Yeah, particularly shout out to GitHub there.

[0:37:48] KB: Yeah, for sure. Let's maybe actually talk about, so you've mentioned a couple of big changes that have happened over the years. One was migrating to GitHub packages. I don't know if there's more to that story that's worth typing into. Some of the other ones, actually, one I'm curious around is you mentioned originally all of the building happened on developer machines, right? You install and it builds for you right there. Nowadays, very little is happening on developer machines. What drove that transition? Were there any interesting technical challenges to make it a reality?

[0:38:22] MM: Yeah. Let me get to that in one second. First, we'll set on the GitHub packages migration. I think it's quite interesting for people of a certain ilk. If that does interest you, then I did a talk, I think it was the Star Plus Conference in London, two or three years ago. You can go and find out on their web page. If you go on my website, mikemcquaid.com, under the Talk section, I think there's a link over there or whatever. That's basically the, if you're interested in a dedicated 30-minute discussion of why that was an interesting technical thing to work on, you might find that interesting.

In terms of the source migration, yeah, I spearhead that the most. It's pretty much the only concept in Homebrew that I have created myself, which is why if you hate the naming, then that's my fault. I'm sorry. We call keeping with the beer metaphor in Homebrew, we call our binary packages bottles, which are then poured onto the file system. I created bottles originally as just a way to speed up a select number of packages. I've been talking to Max and there was, I guess, QT, the cross-platform programming framework that both Max and I worked on with the past. We were both using it when I was at Mandalay, and he was at Last FM. We had quite a lot of experience with that. It was an early package in Homebrew that got reasonable levels of uptake.

The build times were just, well, were and still are bonkers. It would take multiple - it would take often close to over an hour on fairly standard Mac hardware at the time. There was a element of like, this is not a great user experience for someone to type brewing still QT and then have to wait an hour before they get what they want. There was also the aspect of just errors, right? You could sometimes get to a point where when you were compiling stuff, particularly back in the day when it was easier to play around with your Mac OS-based system, where QT would build for say, it was going to take an hour, 59 minutes and 59 seconds and then have an error and be like, oops, something went wrong. Essentially, you lose all of your progress and state and you file a bug report, right?

We would notice that these bug reports were sometimes, well, very often. When building things from source, the number of bugs that were just like, this user has done something weird with their machine. We were able to defensively improve some of that, but it became pretty apparent that essentially, when you build a bottle rather than say, something like QT, you run a compiler and linker to do various things, or various libraries and move things around and then install things and blah, blah, blah. When that was all happening in the user's machine, when it was building from source, somebody would say, there's just so many things that can go wrong. 

Essentially, you're running probably over the course of that build system, probably in excessive a million, maybe 10 million, maybe even a billion separate shell commands, any of which failing for a particular reason will take down the whole thing, basically, right? Whereas, this bottle architecture we moved to was essentially, again, I'm not a particularly smart guy, so I believe in the simplest solution to any problem. Essentially, bottles are just a toggle, right? It's basically, just, we build that, we run all those commands. Originally, the first few bottles were literally, I just build it on my personal machine, on my MacBook, and then we run brew bottle, QT, it spits out a toggle, we upload the toggle, we provide the check sums for the toggle, so that you know it hasn't been tampered with, and then people download that toggle and it saves them an hour.

That started off being just on, and I guess I say, the commands from before, essentially, you go from a million commands to being, essentially, download toggle, extract toggle. The ways in which that could go wrong were dramatically fewer. It dramatically reduced our support burden, and going back to, we said, way earlier, again, a big motivation in Homebrew has been some of the changes which I pushed through that have been very unpopular in the package manager, have basically happened, because I'm like, without this, this project will die, right? We do not have the resources to deal with the amount of incoming support requests we have for supporting this power user feature, which maybe lots of people love, but generates a really spectacular number of support requests.

We are going to do it this new way, which is a way that we are actually able to support. That was another motivation of this stuff, is it's just like, because fewer things can go wrong, fewer people submit issues, and we are able to maintain the package manager better at the cost, in some cases, of some flexibility for our users.

[0:43:13] KB: Yeah. Well, and it highlights, right? All engineering is about trade-offs. This is a trade-off that you absolutely had to make to, in order to support the number, or the scale that you're at. Do you do those binaries? Are they statically linked, or can they reference libraries on the system, or how do you navigate dealing with library code, or other system dependencies?

[0:43:36] MM: Yeah. Most of the time, they're dynamically linked. We will link to stuff provided by Apple, and we will link to stuff that's provided ourselves in other packages, bottles, libraries, etc., which is where things get a little bit more complex, because if you upgrade a library, and then you have to rebuild everything, and that's how you get your 72-hour CI jobs. Yeah, so that's essentially these cascading change of dependencies, where you have to ensure all the linking between all of them is consistent and stable, and all that good stuff.

[0:44:08] KB: You alluded to another change, which is this removing optional compile flags, or reducing the set of options available.

[0:44:15] MM: Yeah. That's probably my, I would say, most impactful in terms of making Homebrew long-term, scalable, and maintainable, but definitely the most overwhelming and negative feedback I've ever received for any bit of work I've ever done, and was probably the first thing that built me a much thicker skin in open source. In fact, speaking with thicker skin in open source, just as a funny anecdote, again, so right now I have a person who was banned from the Homebrew issue tracker last week, who is now in his third new email account that he signed up with just to send me a piece of emails. I'm having this fun back and forth of, he has now had two GitHub accounts banned, and I'm just going through not replying to any of his emails and one by one taking the time to make sure that every new email he signs up with gets banned by his email provider.

It's only with maybe 16 years open source could this become a fun pastime of recognizing every time he's going to try and log into some new email provider, or GitHub account, or whatever, seeing that it's banned and me taking a satisfaction in that, despite not seeing, him actually seeing any response from me. As an aside, so that example you mentioned with the options, again, this was a, I think, that became the natural end of the road with the - as I mentioned, we started off doing the binary packages just for a few select packages, and then we got to a point where we're like, okay, basically everything is going to be better with these binary packages.

The problem is if we provide options we don't really have, we have never built, and like all these things in open source, if someone had come along and built this, I would have been delighted, but no one did. We don't really have a way to have this optional behavior with our binary packages. What happens is when you provide these compile options, then those people are just falling back to building from source. We go back to this world in which everything is, again, very complex, lots of things could go wrong, we get lots of issues filed.

Also, when you have, again, this is obvious to anyone who's done a decent amount of even probably high school-level maths, but right, if you have one option in a formula, then you have one thing that can - then you essentially, have a - that can be on or off. That's two combinations, right?

[0:46:38] KB: You get the combinatoric explosion.

[0:46:40] MM: Exactly. You've spotted it, yeah. You have two, you have that. You have three, you have - Many of our popular formulas would have five, 10 different options. There was just a, from our perspective, felt obviously not literally, but a perceivably near infinite number of things that could go wrong, and just, we ended up constantly having this whack-a-mole. Some of the power users said, "Well, you could have just said that people shouldn't file issues, and you didn't need to take away our toys just because some people were misbehaving with them."

It's like, unfortunately, again, when the scale of something like Homebrew, you only need 0.1% of users to be regularly doing the wrong thing before you just have this absolute dilution of noise. What happens is, again, the most - It's funny, because there's a lot of talk for a while about problems in open source and funding and sustainability and scalability and all this type of stuff, right? I think a lot of that's overblown. But one of the things I do think is, if you want to talk about sustainability in open source, the scarcest resource available in open source is the time of a maintainer. Money is good when it can give you more time for existing maintainers, often. Often, it cannot. But in this case you can say, okay, well, just close these issues. Don't respond to them. But it's like, but every time a maintainer, and often with the way it helps notifications or whatever work, it's not just one maintainer reading that, it's one, or five, or 10, or 20.

Say, even one person reads this. If we're getting 75%, 90% of our issues are just this noise, then that is all time that that maintainer, I guess, in many cases, it was often me, cannot fix your bug. I cannot release your feature. I cannot release a critical security vulnerability update, because I'm spending all my time triaging just all of this noise, which essentially, the only way to make it go away, because we would tell people, again, don't file these issues, please. People keep filing. The only way to make it go away is to take your toys away and say, "Sorry. This option behavior, it does not work for us to be able to maintain a scalable package manager with these around."

Yeah, to this day, I still meet people who are very disappointed in this choice. But I think if they were aware that there was a time when it looked like, we either do this, or literally, the package manager would die and everyone will quit, then they would maybe realize that, well, that was preferable to the other outcome.

[0:49:13] KB: I mean, I spent a couple years, about a couple years as a primary maintainer for a big open-source package. I was paid to do it, right? It was a job, and it's still, it's so interesting to deal with the noise. Yeah, that is hard. This is one example of a way that you've made maintaining Homebrew a little bit more sustainable. How do you think about creating a sustainable environment for your team of maintainers?

[0:49:41] MM: Yeah. I guess, that somewhat alludes back to the thing I said earlier, well, two things, I guess. One is the most valuable primary resource is maintainer time. To me, what that looks like is - I mean, in a funny way, you could say, maybe even maintainer time as an output, and the input is maintainer motivation, right? What that looks like is most of our maintainers, as I mentioned, people will receive a small stipend of $300 a month. I mean, some of my maintainers are maintaining, they are merging 300 pull requests in a week. The dollar by time compensation -

[0:50:26] KB: You don't do this to make money.

[0:50:27] MM: Exactly. It's not a money-driven thing. Even, I guess, me working on it for 16 years, the vast majority of this time, I have never been paid anything to do any of my work in Homebrew. There's never been a time when I've had more than maybe a couple of months where my primary paid responsibility has been doing anything related to Homebrew directly.

As a result, you need to just be like, how can this stay interesting and fun and healthy and whatever? What that looks like, again, something which I've got a bit of flak for, but I very much stand behind, is Homebrew has to be a safe space for the people who are working on it. That doesn't mean we can't have challenging conversations, and it doesn't mean we can't disagree in whatever. What it does mean is if someone is being abusive, nowadays, it used to be able to give them two strikes, three strikes, whatever. Nowadays, my general policy with an open source, and somewhat in life is if you are being a very unpleasant, mean person, even if it's completely unintentional, I have found if you receive one notification saying, "Hey, this is not okay. Your behavior is not being appropriate." That you can judge how well that will go from the person's reaction to that.

Someone who even tries and gives a horrible apology about like, "I'm sorry, you felt that way, blah, blah, blah," you can still tell that they're trying. They maybe never heard a good apology, and they maybe never given a good apology. But you know what, they're trying and they're trying to adjust their behavior. Those people, I will give them a bunch of chances. When someone is called out on their behavior in that way, and their response is to double down, were getting incredibly defensive, we're saying, "Well, no. Actually, you are the problem. You, you go read your code of conduct. You're bullying me, or being mean, or whatever it may be," that never ends well.

If you read through people getting blocked on Homebrew, sometimes people get blocked on Homebrew for what seems like almost nothing. The reason why they get blocked is because A, we've had a conversation privately as maintainers about this and what's going on. Sometimes there's a bunch of stuff where someone only says one borderline thing in public, but they've sent a bunch of private emails to a bunch of people which are significantly worse. Often, it's just like, I've been doing this for 16 years, and frankly, I can just tell when this is not going to end well. The quicker we can just shut this down, the better, right? I think that's what those are.

Another, I guess, maybe more recent addition is for the last maybe six years, or a couple of years break with COVID, we try and have most of the maintainers meet in person once a year. If you looked at people's contribution off from GitHub, there is an explosion of activity when that happens because, again, people -

[0:53:24] KB: You re-energize for sure.

[0:53:26] MM: Yeah. We re-energize and you meet people. Also, when I promised I was going to review someone's PR and I've seen them in person and then over a drink. "Did you review that PR?" I'm like, "Oh, sure. I'm really sorry. I'm going to go review it right now," or whatever, right? That's helpful and it's energetic. It's something I felt like I learned back in the day from KDE of going to their conferences when it was hundreds of maintainers back in whatever it was, 2007-2008, and just seeing like, oh, yeah, this is, you need to build, something I'm not very good at personally, but you need to build a community. We've talked about a community of contributors. There's also a community of maintainers. That needs to be a small group of people who remain actually regularly using and contributing and maintaining Homebrew, who feel they're a bit of a team and they have some sense of collective identity together. I think that helps a lot.

It also helps with, as I mentioned before, with people being abusive. It's like, I just go into, even before I had kids, I go into protective dad mode and I'm like, "You know what? You can say a lot of shit to me, but the bar, what I'm going to tolerate for my team of people who are spending their evenings and weekends trying to build you shit that you get to use for free, my bar of the amount of abuse I'm going to let you give them is not very high." If that means that Homebrewers lost some value contributors over the years, who we just needed to tolerate that every few months, they were going to go and be very rude, or ruin someone's day, so be it. Fine. We can be less. That's the one area of almost productivity decrease I will happily accept is if we can have a twice as fast Homebrew where everyone has to deal with assholes all the time, or we can have a half as fast Homebrew, then we can have a half as fast Homebrew. It's a nicer place to deal with.

I'm aware of saying that if you Google Mike McQuaid asshole, there is a - It's a not minority position that I myself am an asshole, but I do this from a position of trying to make it a better, friendly place for people.

[0:55:21] KB: Yes. I think I have no stake in this and I'm not in the community, but I wish more parts of the tech ecosystem had that type of zero tolerance for assholes.

[0:55:30] MM: Yeah. I just, I don't know, again, not naming names, or employers, or whatever, but I definitely, I'm sure you're the same Kevin, where there's been people who've had problematic behaviors at previous companies I've worked for, where the signs were there in the first week. I have said that before when I was like, so long as I have any influence, if you push the boundaries and you're coming really close to the line in your first week where you should be in a good, happy place, you should be trying to impress your co-workers, we should be trying to impress you, if that's the first thing you do when you're into a company, not a good fit, right? As far as I'm concerned. Because at the end of the day -

Again, that's the other maybe difference with Homebrew compared to some other projects is that like, we have a lot of perks that you don't get at work, and that no one can really tell anyone what to do. I can tell people, I'm not going to let this PR be merged as is, but I can't say, you have to go off and write this code by tomorrow, or else. No one gets to do that. That's nice. You don't really have bosses bossing you around to the same extent. The flip side of that is like, I want us to have the level of interaction that would be considered standard in a workplace. That means that just being abusive and flaming people and whatever and losing your temper, these things happen, but essentially, how much slack you're going to get caught is proportional to how much time and effort and energy put into the project.

If the first interaction we've ever had with you is very negative and angry and whatever, then no. Not interested in continuing that. This has unfortunately happened with Homebrew in the past, if you're a prolific contributor who's been very involved, maybe a maintainer for a long period of time, and over time those rates of problematic behavior rise and rise and rise and rise and rise, eventually, we're going to have a conversation where it's like, either you need to fix this, or we need to part ways. Much as like in a job situation, you can sometimes have those people who they used to be great, but either they change, or in happier ways, the culture change, right? We all changed to be good. We were not going to tolerate that anymore.

I guess, if there's a last note on that, if you're a maintainer listening to this, I would encourage you to read one of my most cited posts, as far as I could tell, that I'm the proudest about is a post I wrote a few years ago called Open-Source Maintainers Owe You Nothing, which is basically, if you look through the licenses of open-source software, then that's what it says. Essentially, it says, we do this without warranties, or disclaimers, or promises of damage. How much of this is legally enforceable is debatable, but literally, most open-source licenses say, if a maintainer was to push out a change that deliberately destroyed all the files on your machine, in agreeing to the open-source license and using that, you are agreeing that you can't hold them responsible in any way for doing that.

I think most people would agree that's when it costs a line. Maybe you're legally okay, but that would, at the very least, make you a bit of a dick. The thing that I think so crucial about that way of thinking is it's like, if you're a maintainer, you do not have to do anything that people don't want you to do. If your project maintaining it is not fun, or interesting for you anymore and you're just being guilted into doing things, then you can stop and that's okay. Not only is it okay, you probably should do that. You need to find a way to have your responsibilities on that open-source project be something that actually you find interesting and engaging.

As you mentioned before, Kevin, sometimes the way you can do that is like, I take a job where I get paid to do this. Sometimes I don't like it, but then I look at my bank balance and I'm like, "Okay, this is fine because it's worth what that is."

[0:59:21] KB: Exactly. Yeah. If you're getting paid for doing a job, there will be parts of it you don't like that you still should do. If you're not getting paid, which most open-source maintainers are not, you darn well better enjoy it.

[0:59:33] MM: Yeah. Even on the you not getting paid, I guess, I would extend that to say, you're getting paid like a reasonable market rate.

[0:59:40] KB: Right. Right, right, right.

[0:59:41] MM: Someone who is getting $10 a month from GitHub sponsors also does not owe anyone anything, really.

[0:59:47] KB: Correct. Yeah, for sure.

[0:59:48] MM: I think this is the thing. To me, this is again, back to open-source sustainability, where I think without sounding too, I don't know, cliche or whatever. I think a lot of sustainability comes from within, and it's about figuring out what is sustainable for me as a human and what can I do and what is - since I become a husband and father and whatever, what can I do on Homebrew that is not going to negatively impact my family, right? That is, again, all of this stuff that goes into running an open-source project that people don't really think about, because you don't necessarily have anyone who's going to be saying to you, "Hey it's 11.00 on a Friday evening. You need to be up early tomorrow." There's a problem with Homebrew. You're staying up late dealing with this right now. Maybe you just need to let it slide. Maybe you just need to let people suffer until tomorrow morning. Because actually, you need to get a good night sleep, because you're going to be grumpy with your family, or whatever you've done, right?

In most good workplaces that I've been in, your boss, or your co-workers would be the people nudging you in that way. Whereas, an open source, again, historically, there's not been as much of that. Something, again, I hope for Homebrew's sake, and we do see a bit of that is when people are saying like, "Oh, I need to step till this is fixed." People are like, "I got this." Or, "This can wait, right?" You need to look after you. That's the stuff that keeps people around and happy and keeps humans maintainable, as well as the software.

[1:01:23] KB: 100%. All right. Well, we're coming close to the end of our time here. Is there anything we haven't talked about yet that you think would be important to touch on leave people with?

[1:01:32] MM: I don't think so, especially. I guess, just to maybe extend even more of what I was saying before, I guess, I found in working in the industry for whatever it's been, 18 years and open source for 16 with Homebrew and a few more, a lot of this stuff that I maybe dismissed when I was younger about the human interpersonal stuff and squishy feelings and therapists and boundaries and all of the cheesy grown-up words, or if you're Scottish, cheesy American words, it's all really important at the end of the day. This is what stuff is built on the back of. Something I really admire when I'm looking at the LinkedIn, or resume, or CV, or whatever you want to call it of a person who's applying for a job, or I'm going to work with, or whatever, is like, we've got this industry expectation of like, you don't want to be at too many jobs for a year or less

I really love when I see people who've done one thing for a decade or more. I guess, almost like a challenge, I would put to anyone listening to this is like, what would it take for you to be happy in your job, or your industry, or your open-source project, or maybe to be even deeper still, like your marriage, or your house, or your friendship, or whatever, what would it take for that to be still in a really good place in 10 years, right? Think about what you can do to get there, because I used to say to myself like, "Oh, I've got 10 years in Homebrew and then I'm going to quit." Then I'm like, "Well, maybe I've got 20, or 30 or whatever." I don't really know, because it remains sustainable for me to do. It remains, hopefully, at least beneficial for others to receive the work I'm doing.

I think the world is much better when people can focus on that. Often, to quote every flight attendant or whatever, putting on your own life mask first helps us all be able to do more better stuff together, than it does when we're monitoring ourselves and then we burn out in two years, or whatever.

[END]