EPISODE 1734

[INTRO]

[0:00:00] ANNOUNCER: Mozilla Firefox is an open-source web browser developed by the Mozilla Foundation. Since its first major release in 2004, and has stood out on the browser landscape for its emphasis on privacy, security, and customization.

Brian Grinstead is a senior principal engineer at Mozilla. He joins the podcast with Kevin Ball to talk about the Firefox architecture, the recent UI refactor, web performance, the role of Rust in the browser, and more.

Kevin Ball or KBall, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI inaction discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn or visit his website, kball.llc.

[EPISODE]

[0:01:02] KB: Brian, welcome to the show.

[0:01:02] BG: Thanks for having me.

[0:01:04] KB: Yes. I'm excited. Let's maybe start out a little bit. Can you introduce yourself to our listeners and share a little bit about what you do?

[0:01:12] BG: For sure. My name is Brian Grinstead. I'm a senior principal engineer at Mozilla working on Firefox and the web platform. I've worn a bunch of hats at Mozilla since I joined in 2013. I've spent a lot of time thinking about web developers, the Firefox desktop application and performance. These days I work with engineering teams across our organization to make sure we have excellent technical foundations for both Firefox, the browser, and Gecko, the engine that powers it.

Prior to joining Mozilla, I was a full stack web developer doing dot net and got really into front-end work at the time. But I constantly ran into problems with browsers and tools of the day. I remember having to support IE6 and 7.

[0:01:49] KB: Oh, my gosh, yes.

[0:01:51] BG: The headaches, compat issues, performance issues, hard to debug, and it just made it hard to ship good apps to users. So, I had done some open source work at the time to share some of the libraries and tools I had made and eventually kind of caught the browser in DevTools bug. I had made a small contribution to Chrome. I actually added the first color picker in inspector because I was tired of typing hex keys and hex values into the inspector. And once I saw that land and I was using it day in and day out, I caught the bug and got the opportunity to go join Firefox to help stand up built-in developer tools.

[0:02:21] KB: That's super cool. Yes, I remember, I used to estimate for people and be like, "Okay, so if you want to include IE7, double our estimates. If you want to go back to IE6, like triple it again. It's going to be that long."

[0:02:33] BG: Exactly.

[0:02:34] KB: So, I'd love to hear a little bit more about that sort of transition into working, not just working in front end, but working for the front end. How did you actually get connected into Mozilla? 

[0:02:45] BG: Yes. So, after I had done the Chrome contribution, I had written about that and I started talking about it. I was reached out. Joe Walker at Mozilla was they were - so at the time, they were starting to stand up a built-in developer tools. Previously, it was done in an add-on. So, when I joined, there wasn't even a toolbox yet. It was like little, I remember because I would use this like in the computer lab. I would have these separate pop-ups for the inspector and the console. I was like, "What is this?" So, they reached out and I looked at it. I was like, "I love to build these tools. I think it's great to add a little bit of competition in that space." And that's how I got out to it.

[0:03:19] KB: That's super cool. It's a great example of how doing something for yourself in open source can really lead to opportunities. I feel like when DevTools came out for the first time, that like, for those of us who weren't already floating in that space, it set the bar. It was like, "Whoa, wait, you mean we can actually do all this debugging in browser?" How did that actually come together? They reached out to you, but you were there for the first production shipment. So, what did it take to get it to happen? 

[0:03:45] BG: Good question. Big question. So, one thing that was notable, I actually didn't know this before I started at Mozilla, is that the Firefox desktop is basically a web app, so is DevTools. When you're building the inspector and you have the CSS tree and the rule view and stuff, that's all happening in HTML and JavaScript. There's a message passing system that is connecting to basically the DOM as if it was almost like a REST API and you're building that.

So, I think there was a lot of engineering work that had to happen to not only sort of make it work, but make it fast enough to not slow down the pages when it wasn't open, and to be acceptable on a huge, large pages. Really interesting technical work. A lot of UX work too. The Firefox DevTools started with really strong UX influence and team members. That was actually, after a couple of years, we had pivoted a lot of features in the design space to be sort of the best thing you could do for working with CSS Flexbox and grid and stuff like that.

[0:04:42] KB: Yes. That's interesting because that kind of in some ways came from this sort of self-connection, right? It's web technology. You want to make it better than you need to improve the web technology to be able to do it.

[0:04:53] BG: Exactly. I remember funny stories like the worst is if you make a bug in DevTools because in order DevTools, you need to use DevTools against itself. So, you would run into these loops where you're like, we ended up adding this feature where you could like change the port that connected to it, and then you could run a known good build of Firefox with the good DevTools that would point to the port of the local build so that you could actually effectively debug. So, I remember just all sorts of challenges like that.

[0:05:20] KB: That's awesome. Were there any pieces in that initial build that were not able to be directly in web technologies?

[0:05:27] BG: Yes. So, the Firefox front end is like mostly a normal web app, but not entirely. It has special powers. It can call directly back into the C++ back end, either like directly in Gecko or to do like application specific things like right through the file system or store something in a SQLite database. There's always that backdoor that you have when you're running in privilege code. I think there's that basic sort of application functionality, although we would try to store stuff in like IndexedDB or things like that when possible.

Then, yes, there's little things. Like a lot of web developers probably know that like getting layout bounds can be costly. If you add an element and then you need to immediately measure the layout, that forces the engine to do a whole bunch of work. We have like a special thing called dom-utils or something and there's a function called get bounds without flushing. It's like, if you want the bounds, but you're okay with them maybe being a bit wrong. It's not something that it's exposed to the web, but I remember using that heavily on the web console.

[0:06:23] KB: Got it. So, from your perspective then building it, were that all just exposed as a JavaScript API? Or did you have to dive into the C++?

[0:06:32] BG: Yes. So, if you needed to change the implementation, you would go work on the C++. But exactly, there's this sort of interface description language that has been around since the start of the Mozilla source code, where it's a syntax where you define the interfaces, and then you can actually implement that in either C++ or JavaScript or REST. So, it's a bit abstract, but when you're calling it from the front end, it just feels like a JS function call.

[0:06:56] KB: That's awesome. So, Firefox started this whole DevTools thing, and now there's a whole, like, developer addition of the browser, which feels like it's kind of taking that to a whole other level. Were you involved in that project also?

[0:07:07] BG: Yes. I think we had timed that with the 10-year anniversary of Firefox's first release. So, I had joined a year prior or so and had mostly worked on the kind of in the DevTools toolbox. If you don't know, if you're a web developer and you're working on a page, you have the main page content and then you press a keyboard shortcut and then it opens into a little toolbox below the tab. All of my workups at that point was like inside of that little container, which it turns out to be a very big and deep container, but in the UI, it's not taking up much space. I had the opportunity to go help stand up developer edition. That was like a bunch of component parts.

One was it was the first time that we made it really easy to run multiple profiles side by side because we wanted a user who's running like Firefox Nightly or release to also be able to download dev and have it be a different profile with different settings. So, there was some of that. There was new DevTools features that we timed with it, and then there was sort of a big push on theming and system integrations, which that was how I had started to get a bit more involved in the desktop front end. Then, also just different settings.

In Firefox release, we can't capture what we call async stacks. So, in the console, if you have a log and it's coming out of like asynchronous function calls, it's a performance cost, basically. We don't turn that on until you sort of open dev tools. But in developer edition, there's little knobs you can twist where the person's expressed some intent that maybe they're okay with a bit more developer activities.

[0:08:34] KB: That makes sense. So, there's defaults that you can change in that way because you know developer edition, it's clearly about developers. It's not just opt in. Are there other things that are different once you break out of just being in that little console space?

[0:08:47] BG: Yes. So, I think around that time we had launched the responsive design mode, which lets you sort of modify the whole page into different user agents and sizes. It was also a time when we were working, on as a company, Firefox OS, and so there was a lot of integrations to help debug remote devices, which actually a lot of that has carried over to the day where we have a page called about:debugging, about:preferences, et cetera. There's one about:debugging that helps you connect really easily to like an Android device or even a remote desktop instance. That launched around that time.

[0:09:19] KB: That makes sense. Where do you see this type of tooling going now? What is the new frontier when it comes to dev tooling in the web?

[0:09:29] BG: So, the short answer is I don't know. I think it's interesting the sort of balance between like authoring in the browser versus authoring outside of the browser. I think there was a vision. We have a feature called Style Editor where it basically loads the entire CSS as it was delivered to the browser, and it's a text editor. It's like a code mirror. You can go in and edit stuff, and then you can sync it back to your disk.

I think that in a world where there's like a good one-to-one mapping of the code that you've written and the code that goes to the browser, that's like a really compelling feature because you don't have this round trip where you're like, "Oh, okay, I'm going to tweak this. Now, it's red here. Now, I'm going to copy and paste that back to my editor. I'm sure people have done that a million times." But I don't know. I almost wonder if some more will move into the IDE because there's so much tooling and stuff that can happen sort of on the side on the way in. There's debugger protocols that the browsers export and sort of automation protocols. We just release support on WebDriver BiDi.

I actually don't know though. I'm curious if you have a take on it.

[0:10:27] KB: It's a good question. Well, and part of why I asked you is, like, it feels like in some ways, a lot of the innovation is happening in user space again now, rather than being something that's happening at the browser level where people are building plugins to these DevTool systems. So, you have like framework-specific DevTools and things like that. We've been in that kind of world for the last few years, partly because y'all did the work to build a plugin system, so did Chrome. It's like easy to plug into the DevTools. But it makes me wonder, is there a new round of consolidation at the browser level that needs to happen?

[0:11:02] BG: Yes. There's a lot of just like core continued work. I mean, it's interesting because a lot of the core functionality is so easy to just distribute to everybody. It's like, "Hey, if we make the console 30% faster," boom, like the next day, likely users have that. So, I think there's still a lot of healthy work to happen in that space on just, "Hey, what's a cool way to help you design your page better and stuff like that?" But yes, I do wonder if some kind of further extensibility could be useful.

[0:11:31] KB: So, let's move maybe into a project that you've done a little more recently. I saw you were also involved with the big refactor project of Firefox UI of moving away from XBL towards web components, moving more of the browser to be built with web technologies. That feels like a really big project. Anything with browsers feels like a huge project to me, because I'm, like, an application level guy a lot of times. But how did that all come together?

[0:11:56] BG: Yes, it was a really big fun project for me. I learned a lot from it. It's funny because in some ways, it's like a very boring project. It's one of these where, like, if you do to these were like, if you do everything right, nobody notices you did anything. But to set some context on this, I mentioned Firefox desktop is basically a web app. But I say basically because this architecture was set into place when the Mozilla open source code was released in 1998. There was this concept of a sort of very low-level, cross-platform desktop runtime that would sort of interface installing, and starting up, and interfacing with the file system and so on. The applications that were meant to be built. I don't think the team knew at the time that it was going to be a standalone browser. It was some set of web-connected applications.

Of course, there's a web engine there. But I think the really interesting choice that the team made at the time that was probably very weird at the time, but turned out to be, I think, somewhat predictive of where the world was going, is they built the whole thing on top of that engine. So, they built the actual - if you're on Firefox and you're dragging and dropping tabs, that's all JavaScript and web technology. But the problem was that at the time in like the early 2000s, none of the stuff that we used today existed.

So, they came up with this language, a sort of UI kit almost called XUL. That's X-U-L. It's an XML -based language. It looks like HTML, but it was tuned towards building applications. So, it had features like Flexbox and grid and little input sliders and all of this stuff that wouldn't appear until much later on the web platform. It also included a feature called XVL or XVL. So, it's like extensible binding language. 

It's exactly what you would think about if you're familiar with web components. It's like a way to attach custom functions and behavior onto existing elements. This was the sort of core architecture. Of course, it changed a lot over the years, but that was sort of set in pace a long time ago and had been built on and tuned and a million plus bugs had been sort of filed and fixed against this over the years. I had an indication that maybe we should do something about this XUL stuff. I had heard that since I had started. I think I had found discussions back in 2007 about it.

But it was actually never possible to do anything with this because of the add-on system. So, one other part of XUL was that it had these super powerful add-ons.

[0:14:17] KB: I remember building XUL-based add-ons back in the day.

[0:14:20] BG: Exactly. We love the add-ons and the add-on authors. However, one of the trade-offs of it was that it was impossible to change anything. Because the ways that the add-on worked is it ran in the parent process, like literally in the same global as the rest of Firefox. So, this is super powerful because you could just be like, "Oh, I just throw the tab over there. It's fine." Mostly, the code will just work. But it made it really hard. Even if you wanted to rename a function or change some function calls, it was like, "Oh, you just broke this add-on." The add-on authors are great. You would file a bug and ping them and they would go fix it.

But to imagine doing something that was like wide scale, like let's rewrite or refactor the front and onto a new technology set was just, there's no way. I think maybe a couple of years before I started, we had done the web extensions project and switched to a more sort of standardized format for extensions to where you could kind of refactor around them. I think around that point, I started to think about it. Nobody told me not to or that this was going to be like a ton of work. So, I said, "Oh, I think, web components are this thing. And XVL kind of looks like that. Maybe I'll go like remove XUL." Yes, I didn't really know what I was getting myself into at the time.

To give you a sense of scale, I guess, we started with 300 bindings and about 50,000 lines of code. If you think about a binding is basically like a web component. So, 300 separate elements around 50,000 lines of code. One funny story about that is I had done all this prep work. I researched it and I'd written these like code gen tools and trackers. I had done it all off of like static analysis. I'm just looking at the XML files for the XVL bindings. By the way, the XVL bindings are like XML files that have JavaScript inside of them.

So, like none of the tools worked. If you wanted to like link it, you couldn't even link it. I had looked at it. I said, I think I could taxonomize it. I had it whole planned. I said, "Okay, great, 200 bindings. We're good." And I had rallied support with leadership to kick off this project. I said, "I better run a CI job just like, I'm going to put a print dev every time one of these things gets constructed and just double check. I'm sure it's 200." The job comes back and there's 300. So, I had somehow missed 100, 50 %. I remember my coworker was like, "Really? Are you serious?" I said, "Well, sure, no problem." I kind of played it off. But I was like, "Okay, this is a big project."

[0:16:40] KB: Yes, absolutely. Well, it's fascinating the whole story there because it's almost as though you had exposed essentially an internal API to application developers. Your first layer you described was like, "Okay, give them a public way to do this that's using web technologies, that's not dependent on this." And that enables you to then consider ripping out or changing that kind of internal API a little bit.

I'm curious to dig into how you did the static analysis, because as you highlight, that type of file, like now it's more common because you have like Vue.js doing single file components and like JavaScript mixed with markup has become more of a thing. So, there's more tooling around it. But how did you approach that?

[0:17:20] BG: That's a great question, actually. I hadn't really connected the dots with the more modern stack and these languages that already have that sort of planning. It was super hacky. I'll tell you that script. I still have it on GitHub. It's scripts. It was like string comparing and parsing. Basically, I needed to make it work against like a known set of things. So, I would find the binding tag and I would kind of take till the next one and I would get the JS. It was always in like one of those C data comments. So, I would strip that out and put it. There was some tools at the time for like code transformations. I think I was using, there was an AST transformation tool maybe from Facebook at the time.

So, there was some tooling available, but it was very glue it together. I know this isn't shipping in the browser. This does not need to be good. I just need to get something that could help me sort of both understand, "Hey, what APIs, what functions are being used out of these bindings?" They give me like an 80% of the way their web component. It was never 100%. Even if we would build in improvements as we went through the project, but it was never 100% there.

[0:18:21] KB: Okay. So, that's kind of how you got it started, got it laid out. You have these 300. You have your CI job that's looking for it. What next? How did it play out? What were some of the biggest challenges you ran into?

[0:18:33] BG: Yes. So, one interesting thing is just sort of how do you structure a project like this? I think a lot of engineers will have seen projects like this where it's like, "Hey, it's this big horizontal thing where you're going to have to go kind of all across the code base and do something that's like sort of similar, but you can't automate it." Are you going to model this as I'm the tools person and I'm going to give you a tool and I'm going to work through the engineering management chain and make sure it's all prioritized and sort of delegated to the appropriate teams? They're going to use my tool, land the code, and sort of move on. That feels like very tidy.

But for this project, I will tell you, that would not have worked. I think probably for a lot of projects, there's pitfalls with that sort of engagement model. So, what we did is we basically went - I had a small engineering team, maybe four people or so, and we would kind of rotate over a couple of years. We just went in and started replacing the bindings. There's a bunch of like technical in terms of way that the binding would get constructed. It had this performance optimization where it wouldn't actually run the constructor until the page was laid out until the element had a layout frame. Whereas a web component gets attached the moment the element is created. This would cause all sorts of like race conditions and issues.

We would go in, replace the binding, and go get code review from like the appropriate person and sort of do testing and document and say, "Okay, oh, we learn this. Let's go add that to the tool." Or, "Oh, okay. Now, we know. Oh, I see. This one looks just like these other three." And a big spreadsheet and you're just kind of - we were just going. 

I think one downside to that is that you spend a lot of time yourself and you sort of have to structure a new organism around this project. But one pro is that you kind of buildup subject matter expertise and you know the edge cases. Because the first time you build a tool, it's for sure not going to work for everything. So, you would just say, "Oh, I get it." You just build up subject matter expertise on it, and then you just go. We went basically, honestly, at a pretty linear pace where I think we started with the easy ones, and then we started getting to harder ones, but we actually kept kind of a similar pace the whole time because we were getting better as we went.

We went for, yes, basically two years of working through every little corner of the front end and learning a whole bunch of how the browser works to do it.

[0:20:48] KB: That's really interesting. I feel like we could dive in two directions from here. We could talk about the sort of organizational piece of this and then the technical piece of this. I'm curious, starting with the organizational piece. This is you as this little kind of almost tiger team going in and interfacing with how many different teams did you have to kind of go into and say, "We're going to muck around with your code?"

[0:21:09] BG: It's interesting because, there were some components that didn't really have an owner. It's like a button component in the front end. We didn't have a like centralized design system team or something. We have very senior like solid front end people who kind of own the front end. So, there was some of those. There were some that were just like legit sort of in some corner of some page that nobody had looked at in 15 years or something. And then there was others that had very strong ownership.

The other thing is there was these like in-content bindings, which you might hear of now called user agent shadow DOM, where it's basically like the stuff that makes up like a date element or something. It's not exposed to the page, but it's like some web content that gets put in. That was a whole separate set of teams. I don't know what the answer. It's like all the teams across the front end at least.

[0:21:54] KB: Yes. That's like a tremendous social challenge. How did you go in and kind of interact with these teams be like, "Hey, let me go and muck with your code essentially and get them to buy in?" 

[0:22:06] BG: Yes. I think partly just having us be the team that was doing most of the legwork really smoothed a lot of that over. If you're coming in and you're saying, "Hey, I've got patches here. I do have a couple questions, but like, let's get this reviewed and landed." I think especially on sort of Firefox and a lot of open source projects, there's sort of some bandwidth to say, "Oh, a patch came in. I'm going to review and get this landed." The other thing is there was a big carrot on the other side of this for those teams, which was, "Hey, you can do ESLint now. You can go and actually refactor this stuff how you want to do it."

For the platform org, it's like someday, I will paint you a picture where we are going to remove the last line of XVL platform code that is like a completely parallel track to all of the standard web stuff that we rely on. We can focus all of our optimization energy on it. So, I felt great support like both on the ground and also, honestly, from the engineering leadership at Mozilla for the project. I didn't feel a lot of resistance actually socially on it.

[0:23:04] KB: Well, and I love that what you just pointed out there is you were able to go to these teams and say, "One, I'm going to do the work for you. Two, here's an immediate benefit you get out of this. You can now run more tooling against this. You have more validation. It's going to help you out. It gets through a lot of that sort of like, "Wait, this person who doesn't know my code is mucking with my code and I got to do the review piece because you can show them." Hey, here's what you get.

[0:23:29] BG: Yes. I also have regular sort of cadence of newsletter updates and, "Hey, we did 30 more bindings in the last couple whatever month and here's what we did." Hey, this contributor did it. It was actually wild because for this type of project, we would get like open source contributors. People were like, there's some sort of excitement about the idea I think of it. Sort of highlighting who had done what. I think it really helped sort of the on the ground feeling of, "Hey, this is happening."

[0:23:54] KB: Well, and that's, I think, as you get into these bigger and bigger projects, that type of work is a key part of getting this sort of engineering change to land. It's a two-year project. You have to build and maintain momentum through that.

[0:24:07] BG: Yes. It's like quite an investment. If you're looking at a short and midterm health of the project, it makes no sense to do this. I was quite grateful to be kind of a part of it and also to get the support I did internally on that project. 

[0:24:21] KB: Totally. I'd love to dive a little bit into the technical pieces because you talked about, "Oh, we would learn as we got to the harder ones. What were the things that were hard and easy? What did you learn about how you had to build your tooling and do these types of like adjustments as you went?

[0:24:37] BG: So, at the start, I had this whole taxonomy of, okay, we've got 300, like 150 of them are actually UI DOM elements. Some of them are expressed as a DOM element because that's how XVL works. But really, this should just be like a JS module. We had others that you would look at in and you'd be like, "You know what, I could just remove this entire piece of code." This UI, no way, either it's like literally not linked to anywhere in the browser or something like this, but we have some tests running that use it. So, I had sort of unused ones that should be JS modules. I had sort of the web component ones. There was the in-content ones, which were a whole sort of special ball of wax. I think that was sort of the basic set.

So, I think the biggest technical obstacle that I remember running into was this layout thing. If you think about the performance, the performance of engines at the time when all of this infrastructure landed was not that good. There was a JIT code in the JS. It would be terrible if you have the browser DOM. Think about it, you've got a file called like browser.xul or browser.xtml, and it has like tabs and it has a URL bar and it's got a little status bar and all of this stuff shows up. You had to then go run, I think, maybe there was 6,000 elements in the DOM and browser startup, something on that magnitude. You had to go run all of the constructors of all. This is slow. Firefox startup is like hyper-optimized. You cannot request that. You'll get yelled at by CI and by sheriffs and stuff if you do.

The optimization that they made was most of those elements are actually hidden at startup. It had a delicate interplay between the layout engine and the JavaScript engine, such that it would not actually run any of the constructors until it got a layout frame, or basically it's going to become visible. But like it sometimes would get constructed if you got an access to the object from like a DOM object. Super complex. The platform team like hated this sort of semantic and it was even hard for the front-end teams, because you would try to inspect the problem. The moment you expected the problem in DevTools, it would run the constructor.

[0:26:46] KB: Right.

[0:26:46] BG: So, you can imagine the sort of problems that we would run into doing these migrations. That was like the, I would say, fundamentally like one of the biggest technical problems, and we found ways around that, both in terms of tooling. There was a way that you could code gen it that would like emulate the semantics a little bit of the old way. But the preferred thing, of course, was to update it to just match web component semantics.

I think just in general, that was sort of the class of problem. Then, I do have a couple examples of like specifically hard things that we had worked on, but that's sort of the general class.

[0:27:20] KB: Yes. I mean, I'd like to dive in a little bit more because I think, and this feeds into a conversation, I think, we'll start having about performance and web performance. This is thing that is one of the big challenges in the web is how you deal with asynchronousity and how do you deal with like when are things loading and when are they fetching data and how are you cascading all these different pieces. So, how did you, when you're transforming them the web component semantics, how did you manage to maintain the performance when now suddenly they're being initialized at a different time?

[0:27:52] BG: Yes. So, part of it is that the JS engine is just much faster today. You would get a little bit less of a regression to say, "Yes, you know what? We're just going to run that at startup." It gets swallowed into the envelope of what you can get done in a few milliseconds. That was part of it. But also, I think we did have to add a couple special little hooks, special powers. We added this functionality called set custom element callback. This has, what it would do is basically one thing that would cost time is parsing and evaluating the definition. We had this - there was one file that was like 10,000 lines of code. But even if you step down to one that's a thousand lines of code, that actually just takes time for the engine to parse and kind of execute that, to even just call like custom elements of the file.

What we did is we added this little hook. So, what it would do is you would not import that at startup, but you would wait, and the DOM would notify you when that element was added into the DOM, and then you could run a function. Because all of this stuff is packaged into kind of, we call it a jar file, but it's packaged into the browser on the disk, you can actually just like import the - there's like a dynamic import that you could run. You say, "Oh, I found like a button element, go and import that JavaScript, which will then define the element before the DOM finishes parsing."

This is unfortunately not something that you can do with web components on the web platform, but I think probably for good reason, because the actual act of importing on the web would be very bad. That's basically like a sync XHR that you would have to do.

[0:29:25] KB: Well, it's interesting, though, because that's very similar to the way that tools like Astro and Quick, these front-end frameworks, do their JavaScript imports where they're essentially putting little event handlers, and they're not doing the whole component definition at that point, but they'll render something statically and then put an event handler on it that when you click it, now it loads all relevant JavaScript for you to load it. So, it's the same kind of wait until you actually need this to load and parse this JavaScript.

[0:29:56] BG: Yes. I think that's the sort of interesting balance of performance, right? You want that thing to be very responsive, but also you want to start responding as soon as possible. It's a really common thing we've seen in the Firefox front-end too, and I think it's been really cool, actually, in addition to like the benchmarking work that's happened, just the ecosystem. So, I do see, I'm not building production websites, but I say, I built some websites. It's really cool when you're kind of looking around at the tools and they're advertising performance.

[0:30:26] KB: Yes. It's really cool to see that. It's like a selling point now. Let's maybe move into that, because I think it is really interesting. I feel like we've had, as an industry, a big shift on performance in the last couple of years. Suddenly it's a focus. I have some theories about why that shift happened, but I'm curious, do you have any sense from your standpoint? What made that change where performance is suddenly top of everyone's mind?

[0:30:50] BG: My sense is that the accepted sort of common wisdom here is that people realize that this actually mattered for metrics that they sort of care about with their sites in terms of engagement, conversions, this kind of thing. I don't know if I have a different theory than that. There's like also this sort of interplay that's moving maybe more towards multi-page apps with server-rendered stuff and sort of more productive back to kind of progressive enhancement. I don't know if there's some relationship between those, but it's another one. I look your take on it. Maybe you have a perspective on this one.

[0:31:23] KB: I mean, I think you're right in some sense that people started to realize that it actually mattered. There's a chicken and egg thing, right? Like you all and Chrome and Google and those folks are putting more and more emphasis on performance and not just load time performance or first render performance, but like actual JavaScript performance and things like that. But my hypothesis, honestly, is that COVID did it because suddenly all of us were working from home networks instead of from fast dev networks. And we experienced what it's like when you have to load a megabyte of JavaScript before you can interact with an application on a home network.

[0:32:01] BG: There might be something to that. I think we've seen also like video conference,
like everything sort of is like, "Oh, now everybody's doing this." The people who are building it are dogfooding it a bit more. It could be.

[0:32:12] KB: I don't know. I mean, just the timeline of it was like, "Oh, suddenly we're talking about performance right when we're all not working from fast office networks anymore. Maybe." But whatever the reason, I think it's a good thing, honestly. Anytime you use a browser on a phone, you're using something, like you pay not only, as you highlight, not only is there the cost of fetching that JavaScript, but just parsing it, loading it, doing all those things is expensive.

So yes, let's maybe talk about it. You've been working a lot on web performance lately. What drew you into that domain and what's going on with it?

[0:32:47] BG: Yes. We had an opportunity to get involved with the Speedometer benchmark a few years back. Honestly, I hadn't been very involved in benchmarking, anything like that. I had never - I had written benchmarks for like my own applications or something here and there, but not much. I hadn't thought that much about it. I knew that we ran a bunch internally, but the more I started to look at it, we had sort of an opportunity to go partner with the other browser engines on this. The more I started to look at it. I was like, "Wow, this is such an interesting, challenging, and important space." I knew benchmarks were like kind of hard and that browser benchmarks in particular were kind of hard because you don't know what content you're going to run ahead of time. You're like in these remote applications and just executing them.

But when I started to look more into that, and we have people in Mozilla who like know this stuff and had sort of my coworker boss had given a presentation at the performance.now conference last year where they went like deep on the history of all the different benchmarks and their methodologies and stuff, and that's super interesting for people who are really into browser performance.

But Speedometer in particular is an interesting one. This was a WebKit project that had a design that was focused on both web, like running real web content, and the methodology by which it took measurements was based on responsiveness instead of throughput. A lot of benchmarks are based on throughput. How big of a semi-truck can you kind of drive through this crypto algorithm or codec or something like that? Mozilla's perspective is that responsiveness is actually much more important for the web in particular, because the way of webpage works is it's a single main thread and every time the user clicks or does anything, you need to run pages, JavaScript, you need to render and paint this thing. All of that kind of needs to happen before the input is fully processed.

If you've ever been on a site and you're typing and like the key doesn't go through and you're like, "Did that go through or not?" And then you press it again and then it goes through twice. It's super annoying. So, we in general, would rather aim towards things that are very high touch, very fast, but maybe not as much a full throughput-based test. That's what Speedometer was. Speedometer 2.0 was like 2018 maybe and we had done a release called Firefox Quantum in 2017 that actually was tuned towards the pre-release of that.

We rewrote the CSS engine in Rust and it was parallelized. It's still kind of state-of-the-art CSS performance. We had done tons of front-end improvements and profiling. We had improved the tools. We had a success years ago on this type of project. I think that the issues with Speedometer 2.0 were that, A, it was old. As you mentioned, like, the web had changed a lot since 2018. There's no doubt about that. That means both kind of the frameworks that were popular at the time are different. The types of stuff that happen on the web are different. That was sort of one class of issue. The other was more kind of like a governance and ecosystem question of, "Hey, can we get sort of cross-vendor buy-in and ownership over this thing?"

That's sort of important for each vendor to feel a stake in it and to feel ownership over it. It's also important on the substance where, "Hey, maybe we think something's important," and then you put it to scrutiny to the other people who were building engines. So, you know what? Actually, that's not. We should take it this way. That was sort of the structure that we kicked the project off. The big idea is that we wanted to align the content and the score in the test with something that would drive real user metric improvements on the web. That's really hard to do with the benchmark, and I think we did a pretty good job on Speedometer 3.0 with it.

[0:36:23] KB: Yes. So, lots of things I love about it. Can you maybe flesh out what exactly does Speedometer 3.0 end up testing and benchmarking?

[0:36:32] BG: Yes. So, if you can go to [inaudible 0:36:33] and sort of load it. It's a web page that anybody can load. It's a web page with like a parent runner in an eye frame that drives through test cases. The test cases that we focused on, something interesting. You could come up with infinite number of benchmark tests for the web. So, we decided to focus starting at the user experience, or we call it user journey as a group, and said, "Okay, some of the things that we would like to be fast, charts, in particular SVD charts, very popular now." You can look at things like HTTP archive and just see trends of stuff change over time. Kind of rich static pages like news sites, lots of layout, lots of state transitions. In this case, it's a single page app. There's sort of methodology issues that make it hard to test multi-page apps.

Speedometer 2.0 was based to do NBC, which is this sort of shared app shell that runs the same code in a bunch of different frameworks. Just kind of updating that, modernizing to the latest, most popular frameworks, and adding a more complex kind of DOM shell around it, and then also editing. So, text editing, rich text, WYSIWYG editing, and code editing. Those are the kind of big themes of stuff that we added.

[0:37:41] KB: That's awesome. Well, and as you highlight, one of the key things here is getting all of the different browser vendors aligned on this, which I can imagine was even more difficult than getting like all of your teams aligned inside of Mozilla and your UI refactor. So, how did you do that? How do you get everybody moving forward in these very different companies?

[0:38:03] BG: Yes. I'm really proud of the group. It's very hard to do. There's all sorts of different incentives and interests in terms of, "Hey, if you had a test and your browser is the slowest about it, are you are plusing that? Are you going to land it?" It's really tricky. And it's common with a lot of sort of cross-organization standards-type work. But I think we tried as a group to be really principal. We are aiming towards making the web faster. We are going to start with these user experiences. We're going to have little subgroups go and sort of, how would a reasonable web developer approach this problem? What are the most popular NPM packages for that? Let's go build a couple versions of this and just kind of see what it looks like.

So, I think there was a good sort of spirit of both scrutiny on like, "Hey, are you sure this is like the right thing to be testing? We've done the research on it. We just do our best of our knowledge. This is the right thing. I will eat the issue with our engine. And we think it's highlighting something we should fix. So, let's do it." We have actually an example from the Vue.js test in Firefox of that.

[0:39:04] KB: Awesome. And Speedometer 3.0, it's all, once again, built-in web technologies. You go to a website, renders this stuff and it goes, yes?

[0:39:11] BG: Yes. Anybody can go to the open-source repository and sort of build and run it locally and play around with it. So, if people have sort of ideas, it definitely welcome discussion and contribution to it.

[0:39:22] KB: That gets to a question I wanted to ask you, which is, if folks want to work at Mozilla, which, I mean, pretty cool place. Y'all are doing a lot of really interesting stuff. What should they be learning, doing, how should they get evolved? A lot of the stuff sounds like it's web technologies, but what else?

[0:39:39] BG: The great thing about the kind of browser and the surrounding ecosystem around it is there's a huge amount of breadth. So, there's like low level networking and crypto and operating system port work. There's layout and graphics. There's web GPU and there's front-end work. Actually, on the mobile clients use the native affordances on those clients. So, iOS, Android, developers, and then there's a whole space kind of connected services and ecosystem. Sync and new product ideas and we've just done this interesting, fully local in-page browser translation that translates languages in pages without sending anything to a server. So, there's sort of an ML training concept there.

I'd say quite broad in terms of sort of the types of stuff that Mozilla is working on. Certainly, the web, I think having kind of some familiarity interest in the web as a platform, as an open platform would be beneficial. But, yes, quite a lot of different stuff going on.

[0:40:38] KB: What's next for you? Are you still working on Speedometer? Or what's coming down the pipe for you at Mozilla?

[0:40:44] BG: Yes. Actually, we just had a really interesting conversation on Speedometer in the group this week and additional benchmarks. That's definitely something that I'm still focused on. I'm also working across our teams on different, just sort of working with engineering chains across both Gecko and Firefox and help with some of the design and review. I like to go kind of in bed with some of the experts internally and who know stuff that I don't and kind of help connect the dots. Right now, that's sort of what's on my plate.

[0:41:17] KB: Well, that's the fun thing about working at an org with all those different pieces, as you get to touch different parts and explore and learn about them. Random curiosity from me around Gecko and what's going on in Firefox. I know there was this big push of doing Rust and Servo for a while, and then pieces were getting pulled in. Is Servo still going? Are people mostly doing Rust now in the core Firefox grassroots? What happened to that whole constellation?

[0:41:42] BG: Yes, good question. We still have a lot of Rust code that we already have in the repo and we're landing. I think the sort of strategy around Rust is, primarily around sort of security and correctness. It's very nice properties for a systems program language compared to C++. Now, it is not a goal to go and rewrite all of Gecko and Rust. I think just in terms of the number of lines of code in the system and sort of some of the quirks that probably sites are relying on. That's not necessarily the explicit goal. What we've done is when we have opportunities to sort of replace a component and sort of encapsulate that piece and then you have sort of interfaces with the rest of the system, that's for sure still going.

In terms of Servo, we have a few component parts that have been pulled out of that, and I think are probably still shared upstream with that system. Mozilla isn't developing that anymore. I think Igalia is actually actively working on that, which is pretty cool. But like the style engine. Stylo came out of that. We have a rendering engine called Web Render that came out of that and we have other Rust projects. We're doing a web GPU back end in Rust that's actually being reused by other people in the ecosystem like the Dino runtime is using it as well. That's being developed sort of outside of Mozilla Central, but we pull that in for Firefox's WebGPU implementation. Yes, it's part of kind of a broader hardening strategy in terms of, "Hey, we're running like untrusted code and you're running in a systems programming language." So, Rust is a big part of that.

[0:43:09] KB: Another question I have for you, and this touches back to something we talked a little bit earlier. But you've kind of worked in a few different areas pushing the web forward, right? You worked on dev tooling in a few different ways. You're working now on performance. That's a big way to push the web forward. What do you see as outstanding, like, big, unsolved problems for the web?

[0:43:32] BG: Yes. I mean, I think for sure, like security and tracking. The fingerprinting story is so difficult from a client standpoint. We do so much client side to try and protect fingerprinting, which is so when sites use sort of ordinary web APIs, but combine them all in a way with enough entropy to say this is the same person.

We put all sorts of lockdowns to prevent that from happening. It's an extremely hard problem to solve because there's just so many web APIs that have so many legitimate uses and to actually be able to lock that down with the incentives that are in place. That's a big problem. I think another area is just sort of continued interoperability and sort of continuing the web as a platform that's defined by a set of standards rather than an implementation. I think that's super important just to maintain sort of opency, and openness, and reach of access. I think that's something that we constantly are, we'll have to work towards as a single vendor and also with the other browser.

[0:44:28] KB: Yes. Well, and I feel like that's one that's been - everybody's been devolving towards Chromium and the diversity of browser engines is really shrinking. In terms of usage, it's Chromium and you all, right?

[0:44:43] BG: Yes. I mean, WebKit as well is a big user base.

[0:44:47] KB: The linkages there are interesting.

[0:44:49] BG: In fact, the funny thing is when I had made the contribution to Chrome Dev tools, that was in WebKit. That was before the kind of fork. There's a lot of shared lineage with the implementation. But of course, over time, they've diverged a fair amount.

[0:45:02] KB: Yes, that's fair. Would you consider them fully enough separate to be protecting the diversity of the web?

[0:45:09] BG: Definitely very separate. I think in terms of just kind of the ongoing process of, "Hey, what APIs are we going to add? What kind of quirks and compatibility issues might there be?" They're definitely separate. One thing we invest a lot in is just web compatibility. So, this site doesn't work in Firefox and how are we like proactively going to discover those things sort of receive that information, how can we fix it. Sometimes you can just switch we have a system where you just switch the user agent and say, "Oh, Chrome and it's fine." And it actually works. 

That's pretty common. But if it's missing APIs and it's just part of a broader discussion around standards and sometimes, yes. It's really a difficult problem because you just don't know the set of web content that exists in the world.

[0:45:53] KB: Yes. Okay. One more web-related question, and then we can sort of start wrapping things a little bit. What do you see coming right now in the web world that has you the most jazzed up? What are you excited about?

[0:46:05] BG: In addition to sort of just this kind of continued march of performance. I didn't even - I forgot to even get into some of the numbers that we had in the last year. We've been just optimizing, we almost fixed 500 bugs against Speedometer. We've like doubled the score. I think we've seen page load increase by like a quarter. It's really been an amazing year just in terms of, "Hey, if you want like a fast-responsive web experience that's in the other engines as well." I think everybody's been optimizing there. So, that's something that's quite exciting.

I think in terms of sort of APIs, I've been really interested to watch the development of OPFS and like WASM SQLite is a sort of combination of two technologies that are like bringing WebSQL back in a way and in quite a really interesting and novel and I think pretty exciting way. I think that's going to be interesting to watch.

[0:46:54] KB: Awesome. Well, how can people, if they're interested in your work or they're interested in what's going on at Mozilla, what are the best ways for people to kind of follow along and stay up to date?

[0:47:05] BG: Yes, for sure. I mean, one thing is if you haven't used Firefox or haven't used it in a while, just download it, check it out. I think you'll be happy with the performance and other aspects of it. So, I think that's one thing. I think if you want to get more involved, we have an open-source project. It's an MPL license. Anybody's welcome to come and hack on it to get the code. You can build it and just run Firefox off of a local build. I do that myself because I like to make little tweaks to my daily driver. Just search building Mozilla code and there's a link there to good first bugs, our chat channels on Matrix, and everything you need if you want to kind of get started on the project.

[0:47:41] KB: Awesome. Do you have any closing thoughts or insights or things that we didn't get a chance to touch on that you'd like to leave folks with?

[0:47:50] BG: I think that pretty much covers it. This is really great conversation. Kevin, I appreciate it and that pretty much covers it.

[0:47:56] KB: Yes. This was super fun. It was fun to dig in. So, I guess then we will say farewell and thank you. Brian, this has been a great conversation. I'm KBall, and that's it for this Software Engineering Daily.

[0:48:07] BG: Thanks.

[END]