EPISODE 1854
 
[INTRODUCTION]
 
[0:00:01] ANNOUNCER: X-Plane is a popular flight simulator developed by Laminar Research. It features a first-principles physics engine, realistic aircraft systems, and a wide variety of aircraft. We wanted to understand the engineering that goes into creating a flight simulator, so we invited Ben Supnik on the show. Ben is a software engineer at Laminar, and he's been working on X-Plane for the past 20 years. He joins the show with Kevin Ball to talk about X-Plane and his career working on the simulator.
 
Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meet-Up, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter, or LinkedIn, or visit his website, kball.llc.
 
[INTERVIEW]
 
[0:01:06] KB: Ben, welcome to the show.
 
[0:01:08] BS: Thank you for having me.
 
[0:01:09] KB: Yeah. I am so excited to get to dig in. This is an area of software I haven't touched before, so I'm excited to go. But let's start with you. Can you introduce yourself a little bit and share about you and X-Plane, and what brought you here?
 
[0:01:21] BS: I am a software engineer. I've been working on X-Plane for over two decades, which is like 130 years in software engineering years. If you told me back in 2005, you'll be doing this in 2025, I would have been completely surprised. But I started working on X-Plane back in X-Plane 8. I was actually planning to get out of software engineering. My plan was to make a total career change and become an air traffic controller. I had been an X-Plane user and enthusiast, a modder. I called Austin up and said, "I'm doing this training to become an air traffic controller. I can work part-time while I'm doing this. How about one last thing before I go? I'm going to design a new Siri file format for you and get you some data. That'll be a capstone for all the modding I've done."
 
25 years later, here we are. I actually got the air traffic control training. While I was waiting for the FAA to call me back, there was just more to do on X-Plane and more to do on X-Plane. I stuck around. I started working on the rendering engine. By the time the FAA actually called me, we were knee deep in X-Plane 8 and making progress. I realized I was on this flight-sim track.
 
[0:02:39] KB: Yeah. I would love to dig in with you a little bit on just even what it looks like to be on a project for 20 years. Because that's a pretty unusual experience. Let's start with X-Plane. How would you describe, what is the core architecture of a flight simulator like this?
 
[0:02:54] BS: A flight simulator, probably from a software engineering standpoint, it looks closest to a AAA game. That's where our DNA comes from. Our business concerns are really, really different. We're outsiders. If there was a room with all the AAA game people, we'd be standing in the corner, maybe not one of the cool kids, because we have the same requirements for high performance as much graphics as we can possibly give them, realism, physically-based rendering has been huge for us, the whole games industry.
 
To a certain extent, games live and die based on being fun. If you don't do that, you're toast. Most games are going to be toast anyway. If you're going to do a sequel, you might just upgrade the whole engine. Whereas, we are a platform because people make content for the flight simulator. People make aircraft, people modify airports. The modding community is so big for us that it's almost like we're a platform, as much as we're an app. We need to have continuity, basically, as much as we possibly can, right?
 
That's where we differ from the games, where a game engine programmer might say, "Well, we've got this new technique. We'll have to modify all the art, and there's these constraints." I get my art director in and say, "Can we do this?" The art director says, "All right, it'll cost us much." The producer says, "Well, we can afford it." Then, they tell all the artists, "Go modify these things. Never use the alpha channel. Here's the restrictions. Here's the bugs you have to work around." That's how AAA games get made. We can't do that. There is aircraft that people have already spent many years on, that are already selling, and the stuff is what the stuff is. We have to ask ourselves, how do we upgrade the engine and still keep this content working? It's one foot in AAA games, and one foot and platforms, which is a unique niche that we sit in.
 
[0:04:42] KB: Yeah. That's really interesting. Before we dive into any of those, are there any things within the core software piece that would be not recognized by somebody coming from a gaming background?
 
[0:04:54] BS: I think from a gaming background, it's all going to look pretty familiar. Our physics engine is completely homegrown. Austin Meyer, our founder, benevolent dictator, the only Laminar employee for, gosh, at least 15 years, developed the physics engine for himself to simulate aircraft. His background is actually in aeronautical engineering. He has this deep understanding of why airplanes fly. The physics approaches used there are very different from what you might get in a havoc, or some general-purpose physics solver. Those systems tend to be very generalized, and they're trying to give you a bunch of very simple Lego bricks to do some physics in your game.
 
X-Plane really deeply understands airplane things. We model airfoils. We model wind flows. We break the wings down into elements. If you came from a game engine, you go, this is the physics engine. Then you probably go, this is not like any physics engine I've ever seen. What's going on here? It's because it's very domain-specific.
 
[0:05:56] KB: That makes a ton of sense. Well, and you have a realism requirement there that most games don't have. It doesn't have to just feel real. It has to accurately represent what somebody is going to experience in one of these planes.
 
[0:06:06] BS: In the case of the physics, it actually goes a step further, because X-Plane is not only used to simulate aircraft. That's the lion's share of our use case is people training for real aircraft with X-Plane and people playing, being pilots at home. We have gamers. We have a lot of older users who used to be real-world pilots, and they've lost their medical. If you've got cardiac problems or eyesight problems, you can't legally go fly a single-engine plane. These people discover that they can rebuild what they had in simulators and still get some joy out of flying.
 
We also have people who design new planes using X-Plane, and that's unusual even for flight simulators. Electric VTOLs is an area that's have a lot of growth recently, because the batteries have gotten good enough. Once your engine is a simple, inexpensive, small electric motor and not a complex, unreliable gasoline, or diesel engine, or turbine, you can put four engines on. You can do all sorts of novel designs. X-Plane can give you pretty good predictions of how that's going to fly just by inputting the design.
 
We didn't need to know the answer ahead of time and then spit it back to you, which is the conventional simulation approach. We can actually be predictive. That puts an even higher requirement on the physics, because we have to get it right, because no one's going to tell us what the answer is.
 
[0:07:31] KB: That's fascinating. I'm curious, then, what is the format for those airplanes look like? Is it similar to what someone is going to see, for example, as they're expecting things out for manufacturing, or what goes into that?
 
[0:07:43] BS: The format is proprietary to X-Plane, and it's pretty specific to X-Plane's needs. We have an aircraft format with just - it's really a giant key-value property table full of stuff. We need airfoils and their locations. We have a separate subfile to describe the airfoil. The airfoil is described as a table because we don't predict what the airfoil will do from first principles. That requires computational fluid dynamics, and that's a whole other ball of wax. Once your airfoil has been measured, maybe you have a wind tunnel, maybe you did CFD, maybe you just downloaded the answer, because there's a lot of airfoils that fortunately are published. You've said, "I'm going to use this airfoil." You can tell X-Plane, "Here's my wings. These are the airfoils. This is the shape." There's a lot of shaping you can do. They can twist, they can be tapered, they can move while you fly, they can be swept. Here's the control surfaces. This is how much of the space you have.
 
All of this detailed data really describes specifically how the aircraft operates in aircraft terms. You describe flaps, you describe ailerons. Once you have all this data, including here's your engines, here's your engine specification, X-Plane can take this. We can simulate the vehicle from the ground up. We're going to compute how much force did this jet engine put out and where on the airframe did it do it? What was the wind coming into the propeller? How was that wind modified by the propellers as it goes through? Was it redirected? Is it going to then hit the wings after? Is it going to hit the tail? Is the wind going to redirect the airflow away from a different part?
 
Essentially, we have this collection of parts, and we try to derive from the parts what is the net effect on the whole. There's a number of aerodynamic effects that are emergent and we don't code them in. There's this tendency of aircraft to pull to the side because of the twisting motion of the propeller. We didn't say this is how much it pulls. We said, what does the air do, and this comes out by itself.
 
[0:09:46] KB: That's fascinating. It's fascinating because, as you highlight, you don't want to do the CFD underneath it, because that's super expensive. But you are doing, it sounds like, a fairly low-level simulation and seeing those emergent effects arise out of it. How do you make that performant? I mean, you guys run all the way down to running on an iPhone or an Android. What do you do to make that all work?
 
[0:10:08] BS: We run on the mobile phones, but it's 2025. The phones are really fast. The phones are much faster than the computers that we're running X-Plane when I started, by a lot. Even an Android craplet, like a low-end Android device, is still much better than what we had with X-Plane, say, 8, which would have been a single-core, 2 gigahertz Pentium, or something. We don't live in the dark ages anymore.
 
The physics actually take a relatively small portion of the performance budget of the aircraft. I think the main reason for that is that the actual number of stations you need to simulate the aircraft isn't that bad. If you can do 40 different simulation points on the wings, that's more than enough to capture the individual effects. But 40 is not a big number for a computer, right? We think of it as like, "Oh, that would take me a long time to calculate." But when computer programs are slow, it's usually because the data input stream is really big. There is 10 million of something.
 
As X-Plane has matured, the physics has started taking a smaller and smaller amount of the CPU budget. We've much more spent the Moore's Law surplus on graphics, because our users have a unquenchable thirst for it to look more like the real world. There are some physics things we do now that we could not have done in the single-core Pentium 4 days. Modern X-Plane has fairly complicated interactions between the different surfaces. When the wind stream is deflected by one part of the aircraft, that carries through to the next. Those propagating effects are a higher, or bigger order than individual blade theory.
 
If you went back and looked at the original iPhone version of X-Plane back on the 3GS, I think it was, which seemed like a pretty nice phone back in the day, but now we would see this as a small computer, or even the original Pentiums, we were doing all the elements, but we were doing no propagation, and that is a computationally easier problem. Even with propagation, this is still a much more manageable problem than draw everything on the entire earth within a 100-mile view, which there's a lot of stuff there. Users have been always happy to have more, so that's been where we've sunk our dividends.
 
[0:12:27] KB: That makes sense. Well, and as you highlight, that is a similarity to gaming, right? You can never get good enough graphics. You can never do it. With your constraint of, oh, I have to be backwards compatible to all of these old files, how do you take advantage of those Moore's law dividends in the graphics space?
 
[0:12:43] BS: There's a couple things. First of all, it's a window of compatibility. The window might be bigger or smaller depending on how much stress we're under with that compatibility. I would say, having good separation of concerns has helped with the compatibility. I try to think of things in terms of what is this content modeling and how are we interpreting it, right? If you say this model is a drawable geometry with a lighting model attached, your compatibility options are zero, because it is exact in the spec of what this model is, how it will be drawn, and so, you're not allowed to change things.
 
If you say, this model is a physical material existing in the real world with some properties on the material, now the lighting model is not part of the model, right? You're allowed to make the lighting model better. Anytime we can use the real world as a reference for the content, we have the option of doing better. Anytime we've said the content's behavior is exactly specified, we're stuck. We can do a better job of simulating the aircraft if you tell us what the aircraft is. If you tell us how the aircraft works, we're stuck. It better work the same, because you told us the how.
 
In the graphics field, there's a pretty good separation between the what of what's in the world and how it looks. What we found is that in X-Plane 11, we moved to physically based rendering. We needed people to specify their materials in a physically based manner. We could look at the previous non-physically based manner and say, this is the least bad approximation we can do. We can say this color texture looks a lot like an albedo, so that's the best mapping. If all else fails, let's do the least bad compatibility path we can. Then going from X-Plane 11 to 12, where we went from it's physically-based to it's photometric, meaning not only is it a set of math on the lighting that is based on what real world optics do, but we're going to try to have all the calibrated numbers be in real units.
 
We're going to specify the intensity of light in candela. We're going to specify the luminance of a glowing thing in nits. When we did that, to the extent that the previous content was still giving us physical materials, giving us albedos, we could interpret those correctly, because we weren't changing those rules. A lot of that content just worked. We'd have to look and say, what don't we know that we wish we did? The answer is, well, it's got a self-illuminating texture, and they never told us the luminance. We're going to say, okay, for all aircraft surfaces, we're going to assume the luminance is 2,000 nits for everything in the cockpit. That's the least bad compatibility we can do, because we know it's an aircraft, we know the pilot can see these things, what's a realistic value they would need for the simulator to function? You need to see your displays during the day, 2,000 nits. It's not perfect, but it's pretty good. In that way, we can keep stuff working as best we can. Every time we make a change, we have to ask ourselves those.
 
There's one other aspect, which is a lot of AAA games use a lot of artist's time to make things work, right? They have a mission, they have a budget. If the art director says, "I want all the rainforest leaves to look like X," it might be the way they do that is they have code, they might write an algorithm that processes the art, or they might tell the artist, just go do it by hand. They can make that decision. In our case, I try to be conscious of how the dividends and computer graphics are driving up the price of making content. Because when I started as a modder, one person could make an airliner, the entire thing in about six months. It was half a man year to make an aircraft, because there just wasn't that much you could do.
 
The mesh came from the physics model. You couldn't attach custom meshes from an authoring program. You got about 1K of color texture, and that's it. There were no physical materials. The cockpit was a 2D drag and drop thing. It wasn't a 3D interactive cockpit you could click on. There was no code to write. You got the systems in X-Plane. There just wasn't that much to do, and that's all that the computers of the day could do.
 
As the computers get better and our users expect more, one of the problems has been that the authors making airplanes have borne the brunt of this, and that the labor to go into the airplane is going up. When we're going to add things to the feature, the other thing I think about besides, how am I going to map this content? The old stuff is how much labor are we going to bring in? Are we doing something that's going to be automatic and just works? Are we going to ask them to go and do a tremendous amount of work and erode the margin and the viability of making this content at all?
 
For example, we added proper 3D range of the windscreen in X-Plane 12. We take the airflow over the aircraft from that CPU physics model, we put it in the texture and send it to the GPU and now we can drive the particle system realistically, which is totally fun, because you're sitting there in a rainy day and the raindrops are falling down the windshield, and then you crank the engine and they go [engine noise 0:17:54], and they go right off the windshield. It's very satisfying. It just works because the GPU has the real shape of the windshield, it has the real flow of the engine, and we can just do the simulation.
 
It's also a feature that's relatively cheap for the authors, because the shape of the windshield in 3D was something they had already built for us. They were already building a 3D model and were reusing it. This is very cheap. When they do the windshield wipers, we need the path of the windshield wiper in the 2D space of the particle system, which is a hard thing to get, because the geometry of the windshield wiper is potentially very weird. That mechanism could be almost anything that people engineer and every airplane is a hundred, or the million, or billion-dollar engineering effort, so there's no reason why they wouldn't just build everything bespoke.
 
We looked at this and said, there isn't a way for us to get this information in real-time from the 3D mesh that's even remotely performant. We bake it. We have the path of the windshield wiper in the 2D space of the particle system as a color gradient, where the different grayscale levels tell you these pixels would have been affected by the wiper at this timescale. Now, we're starting to make it a problem for the artist, because you have to give us this special, magical, exactly right formatted texture to make it easy for us at runtime.
 
What we did do is we built a script into blender to produce this automatically. You could start with your windshield wiper animation, which is something you already built. You do a very small amount of annotation. You say, go bake, and then blender can sit there and go chunk, chunk, chunk, chunk, chunk. Move the wiper, find the pixels, fill them in, and export them. We can still try to protect our authors from having to build something by hand, but in this case, it's going to be done ahead of time.
 
We're looking at the authoring cost for the entire feature and asking, by making this possible, by making it something our users expect, are we driving up the cost of making add-ons, or can we give the authors tools to do this in a reasonably affordable manner, and now their airplanes are better, but they're still able to produce a product whose price point isn't really changing with hopefully, a similar amount of man hours.
 
[0:20:04] KB: It's fascinating, because it's an area I hadn't thought about as much, so much of software, right? The downstream dividend of things getting more powerful is you can write software cheaper, because you can skip steps, you can use scripting languages, you can do all of these different things. In graphics, the expectations have risen faster than the tooling.
 
[0:20:24] BS: That's right. I think it's telling that in software, what have we done with Moore's law? We made our own jobs easier. My father was in hardware for his whole life. He scowl at me and say, "You kids with your virtual functions. We didn't make the chips faster, so you could just waste all the cycles." The truth is that was the most useful thing you could do would be to make software cheaper to make, or to make software more reliable. I think the dominance of managed languages is telling us something about what people need out of software, right? It's very expensive to have humans write bug-free, unmanaged code. Maybe we use Moore's law and not do that.
 
[0:21:03] KB: On that topic, what is the underlying software stack y'all are using for X-Plane?
 
[0:21:08] BS: X-Plane is coded entirely in C++. It's pretty much a big monolithic pile of C++. We use a smattering of library code for the things you would expect. We've got libpings, so we can open pings. We've got racknet for a networking sublayer. We have things like that. But it's very much ours, almost surprisingly so. I've had people ask, why are you not using SDL2 for your cross-platform layer? Why are you doing it yourself? The answer is, well, the code is 30 years old, and there was a time when the thing you're asking us why we didn't use didn't exist.
 
At some point, I think we've erred on the side of keeping what we had, which was already in production, which is a trade-off. Maybe our solution isn't as good as a new thing that came out as library code. Also, there's often a lot of client code on top of a library, which is why it's often cheaper to rewrite the library, or shim it, than to go rewrite all the client code. We were OpenGL-based for a very long time, because OpenGL was the first hardware API that Austin used for rendering. There was a time when X-Plane was software rasterized on the CPU. But by the time I got there, it was OpenGL, and OpenGL went on multiple platforms.
 
When we wanted to move to Vulkan and Metal, to modern low-level graphics APIs, we had to make a decision about whether we were basically going to just shim Vulkan and Metal under what we already had, or we're going to go rewrite a chunk of the software stack. We chose to do the hard thing and rewrite a chunk of our own code. I think if we just wanted to say, "Hey, it's on Vulkan," that could have been an eight-week skunk works project, because the API is very narrow and you just shim under it. All of the architectural contortions that you pick up from being a long-lived OpenGL application would have been still there in X-Plane. We would have had a great tool in Vulkan and gotten none of the benefits, because we'd be holding it wrong pretty much always.
 
Instead, I think we spent three or four years of continuous work on this, Sydney, Houston, I. What we got in return was a real rewrite of the rendering engine itself under modern principles. Vulkan lets you understand the performance cost of everything you do. Whereas in OpenGL, you make a call, some things happen, some pixels appear later. That's basically the whole contract. It's very loosey-goosey. Because that abstraction is not well-suited to modern hardware and you can't fault SGI, they failed to look in the crystal ball in 1991 and predict what would happen in 2025. We'll give them a mulligan.
 
Because of that, OpenGL doesn't have good matching between what's expensive and what you do. Realistically, what happens in an OpenGL driver is the driver falls along, it squints, it tries to understand what the app is doing. It does a lot of work to reprocess the information. At some point, it has to go do a lot of work. A lot of the times, that's at the very first triangle you draw, they go, "Okay. Now we understand the whole problem. Let's go compile some shaders. Let's go make sure these objects are in the right part of VRAM and all this work that you wish you'd done at load time is getting done at draw time."
 
Our users, especially the professional ones, really want X-Plane to be like a metronome. Rock solid 60fps, don't stutter. Stutter-free operation is a requirement from the FAA for professional sims. A driver that will sometimes stop and go compile a shader, because on this particular hardware, that's required because of this state in a way that you couldn't predict. It's really hard to hit that 60fps limit. But we didn't have any code in X-Plane to go prepare the shaders in the background, because there is no prepare shader's operation in OpenGL. You can tell that you're building the shaders, but it's going to half compile them and then wait, and then do the other half later when you draw, because in OpenGL, the specification for a shader is incomplete. In OpenGL, you give it the source code, but you don't tell it the render target. You don't tell it the fixed function pipeline. It's missing most of the information. It goes and it does it later when it has the info.
 
We had to restructure X-Plane and not just switch to Vulkan to get that actual performance win. We went in and we rewrote a lot of code. It took a while. But in hindsight, it was worth it, because we have something much newer, much better matched to the actual hardware, and putting new features on top of that has actually been very straightforward.
 
[0:25:38] KB: I think that's an example of what happens when you have this very long-term view as well. You mentioned, I think, this might actually be an interesting time to dive into what goes into maintaining a project for 20 years. How did you go about that three to four-year migration in terms of planning, phasing, isolating things so that you didn't break stuff that was running? What did that look like?
 
[0:26:01] BS: Well, Austin has always said that he wants our time horizon to be infinity when we think about the efficiency of what we do. If we say, "Oh, there's this thing we can do." It's going to be really great in the first three months, but it's going to be inefficient in a tenure horizon. It kills him to think that we're wasting our tenure time to hit that three-year thing. He wants us to think about the long game.
 
That means we can think about a roadmap of where we want to move the code. When it came to the Vulkan port, I think it started by trying to understand the technology. We had a pretty good idea of what we weren't happy with. We could really see the pain points of what was already there. We came up with this refactoring roadmap for which pieces we would rebuild and when. We had to look at how are we going to do this in pieces? How are we going to never have the app so thoroughly broken that we can't test? Because at some point, if you do that, you end up breaking a bunch of things, never noticing. Then it's really hard to put Humpty Dumpty back together again. There's a huge value in never breaking the universe.
 
We also looked at whether we had to stick with OpenGL. One of the biggest arguments we had was, do we have to keep OpenGL working when we ship Vulkan? My answer was yes, and this is what we ended up doing. We ship Vulkan as a free update to version 11, while keeping OpenGL. What this got us was the ability to have a very, very long beta. My God, I think that beta might have been nine months, which is atypical for us. We just needed to have Vulkan out there and find out what it was like on real machines. Because something you get from a shipping product is, what's this like out in the wild? Because the wild is really different than your lab machines.
 
In the case of Vulkan, we had no idea what we would find. We could not have predicted what we would learn from that beta. Having the old system running at the same time meant there was no customer going, "I paid $60 for this and it's bricked now and I want a refund, or I'm going to fight with my credit card company, or God knows what." It gave us the time and the space to do the exploration. I think if we'd said, okay, X-Plane 11 is OpenGL, X-Plane 12 is Vulkan, it would have been an easier engineering problem. We would have said, hey, in this new thing, our requirements are simpler, because the union of OpenGL and Vulkan is a little awkward.
 
On the other hand, God, it better have worked for the first time when you hit go, because you just paid for the new version, right? There's no safety net. You can't go back to OpenGL. In hindsight, we did not hit that. We needed that nine months of beta to understand how Vulkan performs in the wild. What happens in Vulkan when you're doing memory management and the user launches Chrome and Chrome goes and puts stuff in VRAM, right? Vulkan tells you how much video memory you have. By the way, it might just cut in half, and you have to survive that, right? You have to write your own code to cope with that situation, preferably without exploding in a million pieces. We did not have enough information in the company. We didn't have enough experience with Vulkan before we'd written Vulcan to correctly predict that, so we had to try it in the field.
 
Knowing that, we could then say, okay, which pieces are we going to roll out? What we found is that there were refactors we could make to our code to use OpenGL in a better way, in ways that probably didn't make a huge difference for an OpenGL driver. But we're also fitting the shape of Vulkan. Once we'd made those refactors and shimmed all the OpenGL code into an abstraction layer, then we could rebuild that abstraction layer for Metal on OS X and Vulkan on Linux and Windows, and then we could hit the lever and see, can we shift all the trains over to the new track?
 
Since we were going to do Metal and Vulkan for Mac and Windows, we knew we would need an abstraction layer no matter what. Having it also handle OpenGL and OpenGL ES in the mobile market, which we thought we'd be on for a while, that was the natural path. It gave us a way to do the refactor in two pieces. First, making a nice API boundary, where previously the OpenGL and client code was completely fused together. Once you've completely cut out that boundary, then you can do a shim, and you can switch between them without recompiling the app. That makes the refactor tractable.
 
[0:30:20] KB: Yeah. No, that technique of introducing a boundary and new abstraction layer, decoupling things, and then introducing these two different ways is that, I found that to be highly effective as well. I love what you said about the long-term time horizon. I feel like that's something very few of us in the software industry are in companies that support that, but that's incredible.
 
[0:30:40] BS: Yeah. I think we're very lucky to have that freedom. It's one of the ways that we're different than games, where often a game's time horizon is, we're going to do one title. While I have not worked in the games industry, my brother does. We have people who work for us who have been in games. You sometimes see some pretty scary code, and people write it for good reasons, right? Like, we have to ship this, we have to ship it on this date. No one's ever going to come and refactor this, so give me the big roll of duct tape, and here we go, and it holds together and it ships.
 
In our case, we know we're going to pay for ripping the duct tape off. Sometimes you end up using it anyway, but we have to think about what it's going to be like in a week, in a month, in a year.
 
[0:31:20] KB: Yeah. I wish more software business models took that approach. I think it is definitely a healthier one for the code and for the engineers. You mentioned testing. Can I ask what your test stack looks like?
 
[0:31:33] BS: Our test stack, we have a couple of things, and it's never enough. I would always love to have more testing. We have Catch2, which runs unit tests, and we have partial coverage of our lowest stack of software. We've been catching up on test infrastructure, because when the company was very small and X-Plane was very small, testing was kind of, let's give it to the users and see what they say.
 
For a very long time, Austin was the only person working on X-Plane. Testing was, he coded it, he tested a little bit, and then he gave it to some people, and they emailed him at his personal email address if it wasn't right. That was everything. We've built that up a bit. We have Catch2 for unit tests.
 
We have a Python-based runner of a scripting language, because any sufficiently large application has a command shell in it, and X-Plane is no exception. The command shell besides providing strange debugging tools and things also provides a whole bunch of stuff for testing. We can write these automated scripts, and some of them will do things like, we have one that opens up every aircraft with the engines off and does the procedure by invoking commands to start the aircraft, monitors the parameters of the airplanes it starts and fails the test if something's out of whack. If somebody makes a change and now the electrical system is broken and the battery doesn't cause her to be amperaged on the bus, test fail. We have a suite of those. 
 
They can also take screenshots, so we can get image output and we have Daniella, our sound engineer does a lot of web stuff, too. She built this fantastic browser-based diffing program, so you can pull up the before and after and spot the different regions. We have black box testing. We have unit testing at the low level. We have some integration tests in the middle. We have a tool you can run on every airport, and it will catch invalid frequency conflict. So, when you import new airport data, you can run a one-time check.
 
I would say the struggle we have is that because X-Plane is a platform, you can make an almost infinite combination of aircraft. A lot of the bugs that are hard to catch is - well, it turns out that if you have an airplane with three engines and two are electric, one is gasoline, and you have two clutches then, and that particular combination is difficult to cover. What we don't have is a very, very, very wide suite of aircraft. X-Plane ships with, I want to say, about a dozen or so, really carefully built, handcrafted aircraft that our team does a great job on. But because they put so much work onto them, I can count them on my fingers. The test coverage value is less. At times, we've built simple airframes just to do testing. But I think that's an area where the coverage is poor.
 
We have two QA engineers, and they both supervise users with the automated testing resources, and they also do manual testing. We do have tests where there's a old school test plan, right? It's a dock with words for what they do. In some cases, you need a human to go do this because automating it would be too complicated. Most importantly, they just sniff out bugs. They're really good at that. We joke about, well, how many milliseconds is it going to take Danny to find the first bug? They'll come in and they'll shake it and get that spidey sense that a good QA engineer can get. That helps us catch things that would fall between the cracks, that the regular test infrastructure wouldn't catch.
 
[0:35:07] KB: There are people who just seem to have a talent for breaking things and finding what things are broken. It's uncanny when you watch them. Actually, brought another thing to mind when you mentioned the many different combinations someone could do in these third-party content, whether it's an aircraft, or airport, or something like that. Is there a correctness validation that makes sense in those files, or things that are describable in the format, but maybe not physically possible? What does that even look like?
 
[0:35:34] BS: There are semantic validations that we can do and that we do do. I would say my own work in the scenery code has in the past been pretty slim in terms of semantic validations, mostly due to schedule pressure. I've been trying to pay my debts off and add more validation. Austin has a fair number of validations in the physics system, because he doesn't want to deal with junk data coming in. There's also a gray area between, this is clearly wrong. You cannot have a stall speed of zero. There's got to be a stall speed. Versus, this is very silly. Are you sure you really mean this?
 
Sometimes we say, well, we saw this weird bug and we looked at it and the data was crazy. We made the validation a little more skeptical, and then we hear back that, "No, no. I built an RC plane." It's one foot long. all the numbers really are small. This isn't a typo. This is actually how big the prop disk is because it's made up of balsa wood. There's definitely a tension between how much we can validate the model and how open-ended X-Plane can be in a very strange stuff.
 
[0:36:39] KB: Yeah, for sure. If people are using it to test out things that they might build, like this can be built, but falls off out of the sky, is a valid outcome.
 
[0:36:50] BS: That's right. Yeah. Flies well is definitely not a validity parameter.
 
[0:36:55] KB: On the flies well side, another thing that you briefly mentioned is FAA certification, right? X-Plane is not just for hobbyists trying things. It's not just for people manufacturing. But people are actually using this software to certify themselves to fly. What's involved in that? What has to be true at the software level to enable that?
 
[0:37:16] BS: Surprisingly, less than you think, but more than you think, not in the areas you might expect. First of all, they don't necessarily certify software. They certify devices. We are a software reseller and we sell a professional version of X-Plane to an integrator. They're going to select the computer hardware, select a version of our software, build a simulated airframe, build the controls, maybe a fiberglass enclosure, maybe they're going to buy a motion platform. If this is a high-end system, they might have multiple computers network together running separate displays to get more visuals. Then they're going to take this entire thing to the FAA for certification.
 
The FAA is less picky than you would expect about the accuracy of the flight dynamics compared to the real aircraft, but they are often very picky about the operation of the simulator. This one always cracks me up. If your flight control hardware is not working, meaning your joystick is unplugged, your simulator has to not start. It has to log that as an error and not run. The inspectors will come and yank the joystick and out of the USB port. If we keep flying, they'll fail you. I've always thought to myself, like, who's going to go in and do flight training time, and the joystick isn't working, and they didn't notice? What world do we live in? That is a requirement. Because otherwise, it's clearly broken. We can get tested on that.
 
The specific regs are quite long. My experience with the certification process is that it's a little bit ad hoc, that different inspectors have different parts of the regulations that they will check. We sometimes get calls from integrators saying, "Oh, my God. We got called on this thing." It'll be like, we haven't ever seen that before. But sure enough, it's there.
 
We try to make the most useful, sane, generally highly functional simulator. I would say that the regulatory structure doesn't match X-Plane's business model, because we have a tool that's meant to simulate all airplanes. The really traditional simulation models we're Boeing. We built the 737. We spent about a gajillion dollars on the R&D. We're going to get a eight-figure fee every time we sell one. We're going to make a level D maximally certified 737 simulator that we have made exactly like the real airplane in every way, including using the physical flight deck in that airplane.
 
This simulator is a one-off device to simulate this one airplane, and it's going to have, God knows, seven, eight, nine-figure price tag, too. The certification process is aimed at that, I suspect. Here's a simulator of this thing. What there isn't is a way to say, here's software that's generally useful, we certify, that in the general case, it is a jack of all trades. We're more of an arms dealer. We're selling shovels to the prospectors in terms of this process. I don't mind that, because dealing with the regulatory requirements is its own complicated thing. We're happy to do the software engineering, which we're good at, and let somebody else do the certification process.
 
[0:40:30] KB: That makes sense. From a device support standpoint, does that introduce - these are custom hardware, but underneath it, they're just running devices that you already know how to manage, or what does that look like?
 
[0:40:42] BS: Yeah. We are on Mac OS X, Linux, and Windows in the desktop space, and Android and iOS in the mobile space. We're running on a lot of operating systems. Most of the integrations are using Windows. Occasionally, we see people using Linux. But the professional integrators are usually pretty flexible about doing what we ask them. They want to succeed. They want to have a system. They get certified. They don't want to spend a lot of time screwing around.
 
If they're using windows and we say, "Well, upgrade to Windows 11," they'll do it. If they're using Linux and we say, use a different distro, they'll probably do it. We can we can recommend hardware. In terms of the overall machine, usually, it looks a lot like an ideal machine in terms of the specs are very good, because the cost of the hardware isn't that great compared to the whole project. The operating system isn't crazy. It's hopefully not loaded down with a bunch of really crazy mods and addons and malware, or something. It's a pretty good starting environment. We use USB HID for our hardware input. It's relatively straightforward to wire up just about anything to X-Plane in terms of joystick, or control input.
 
We also have a DLL-based plugin system. We're starting to do REST and web socket-based communication with a sim as well. We've had UDP input for years. They have a couple of tools they can get that they can use to get custom data into the sim. But a lot of the times, their joystick hardware is going to just work. Whether they manufactured it themselves and got a HID board, or they bought something from another manufacturer. Well, usually, you can make a config file to further specify how we interact with the joystick, and you can tweak a lot of the settings in our settings, but it's pretty flexible, so that usually goes well for them.
 
[0:42:31] KB: That makes sense. We've talked about a few different places where you have this interface layer that can be configurable and can let you work across the different types of hardware, graphics, obviously being one input device. Are there other areas where you are ending up handling this diverse set of devices? There's a lot of standard libraries app for doing that, but you rolled your own thing. What does that look like?
 
[0:42:56] BS: We generally try to keep the platform-specific layer as thin as possible, because anytime it's platform-specific, we're coding it five times and not one time. For example, our user interface is rendered into the app in Vulkan and Metal using our own UI kit. We're not writing separate Win32 forms and Mac Swift UI code. Then we'd need something else on Linux, maybe it's GDK, or GNOME, or there's a lot of chaos over there. We do something else on Android. We're not doing that. We're willing to say, the X-Plane UI just looks like X-Plane. It's not going to be platform native, but we got to write it only once, so we keep that how very thin.
 
We don't need that much casing, because four of our five operating systems look a lot like POSIX, and Windows isn't that different anymore. The platform story has gotten pretty good. I believe the HID code we're using comes from an open-source library, which does give us separate HID paths on multiple platforms, because how you talk to the USB driver varies, but that makes it uniform. Vulkan and Metal look different from each other and are abstracted, but Vulkan plays on Windows, Linux, and Android. Metal is on all of Apple's platform, so we get pretty good leverage from that.
 
We have a thin set of file system wrappers, but we don't need much. We have a wrapper around the memory-mapping primitives to get files in. Again, it can stay pretty thin. We're not doing tons and tons of file operations. Our needs are relatively straightforward. Racknet abstracts the network stack, and we also have a pretty thin socket wrapper, but Windows looks enough like sockets.
 
Overall, I would say the platform layer, we can keep it thin, and it's been okay. We need a way to get a native Window and get some mouse input off of it. Once you write that once, the amount of change in that space is pretty limited. Windows code really pretty much runs the same from. when I started, it was still Win32. I mean, people just like on Microsoft, but they have kept a platform running since forever. I think people don't realize how much they're holding up by doing that. We feel it a little bit, because we've been on OS X where they've changed the ABI, maybe four or five times. They've changed the UI kit several times. They've changed the language multiple times. You have to rewrite your stack, and your users aren't going, "Oh, I'm really excited to get the exact same features, but rewritten differently." You don't get anything for that. As much as we sometimes say mean things about Redmond, I do recognize the value of having the thing keep working so we can focus on features and not just rewrites.
 
[0:45:34] KB: Yes. No, they definitely understand they're a business consumer, I think, better than just about anyone else in the space. You mentioned features. What's on the roadmap? What types of features are you all working towards?
 
[0:45:48] BS: Hopefully, by the time people hear this, but as we are recording, our newest free patch 12.2 should be coming out. That's a patch that's very graphics-intensive. We try to have the patches be in a theme area. Marco, our product manager, looks at things and tries to gather up the features and find things that will be good for users, but also hit a theme and limit the testing. If we touch everything, we have to test everything.
 
The most recent one is a graphics one, and there's changes pretty much everywhere. I don't think I could just rattle all of them off, because there's been so much. This is the end of a pretty long, probably even a year-long, effort to refine the rendering and lighting stack of the sim, and really do write what we prototyped in X-Plane 12.0. X-Plane was a difficult release for us because Microsoft had come into the market with flights in 2020, while we were finishing the Vulkan port, and we were completely taken by surprise by this. They really stayed in stealth mode right up till their EA announcement of the new product. It was just, how soon will you guys have the new version, and how many features can you have in it? That's a pressure cooker for a release.
 
We did all new photometric lighting and clouds, and quality was something that got thrown on the floor to get that out there and just show that we were in the marketplace. People use it, and they could see the potential, but there was also a lot of weird stuff. Now, the team has had the time to go in and really do a great job of working all the bugs out of the lighting and going back to the real-world references and using new algorithms. It looks like a whole new sim. It's really a big upgrade. People who have used it in the company, you go back to the shipping branch, you're like, "What happened?" Now, it just looks flat.
 
It's funny how in graphics, everything looks fine until you see the new thing. Then we hedonically adapt so quick, and you go back and go, "We thought that was good." I know there was a time when we first saw X-Plane 11 and went, "Wow. That looks so much better than we could have imagined." But then, once you've seen X-Plane 12, you go back to 11 and go, "Yeah, that doesn't look right at all. Look at all the things we didn't do right."
 
The lighting, the color saturation, the clouds have gotten a big update. The interaction between them has been reworked. It's going to be a really nice visual update. Behind that, what I think could be 12.3 for us, but your mileage may vary. We have a number of features for systems and weather that are nearing completion. Better modeling of how clouds work, better interaction with the real-world data when we can pull it in. I believe we will finally ship the new weather radar, which is a very accurate simulation of the radars that are on aircraft, so that the pilot can see storms ahead that runs on the GPU and probes that weather data, but then simulates the returns and the occlusions.
 
You get a very accurate view of when there's a storm cloud ahead of you, and it's big enough, the pilot doesn't really know the weather behind it, because the radar doesn't penetrate it, right? We simulate that. Or, if there's a mountain in a way, you actually get a radar return off the mountain. You can't see the storm on the other side, but you do know there's a mountain. Phillip has done a fantastic job with that realism. We're going to ship that. There's new features for the FMC. I think we're going to finally ship the new network synchronization layer, at least one part of it. Pavel has reworked how we synchronize state over the network, because you can do land flight. You can do X-Plane plus external visuals.
 
For years, we only synchronized the aircraft, which is the most important thing. As X-Plane is grown as you of other stuff that started to be inadequate, and we get bug reports of, "Hey, I've got a three-monitor, three-machine setup, and the jetways aren't in the same positions across the monitors. They don't line up." It's like, well, that jetway is a simulated object that can extend and go to the airplane and move. The machines didn't know what the state was on the master machine. We have new synchronization, so it's going to be exciting to ship that for the first time. It's just one step in a roadmap for improving external visuals.
 
The team keeps working on features. We have releases beyond those, which I think Marco will slit my throat if I talk about. But we can see what's coming. The code isn't ready yet. It's in progress, but we can say this is going to be a valuable feature for our user. We've put shovel to the ground. We're engineering it, and we'll see in what order they come out. Because it can be hard to predict how fast things will be coded, because in software engineering, sometimes you don't know what the hard problem was until you crash into it.
 
[0:50:41] KB: 100%.
 
[0:50:43] BS: If you already fully understood the problem, it's probably because you already coded it, in which case, why are you coding it again? But there's just not a lot of repeat problems. When people build a house, it's not that different than the last house they built, and so the estimates can be very accurate, because you can use all that experience. Maybe if it's the first house with three printed concrete, that's going to be really harsh to predict. But if it's the six millionth two-by-four framed house, it's a well-solved problem. In the simulator, we're pretty much never solving the problem we already solved, because we already have the code to do that. So, it's always new. It's always a learning experience every new feature.
 
[0:51:21] KB: Awesome. We're getting close to the end of our time, and this has been absolutely fascinating. Is there anything we haven't talked about yet that you think we should touch on?
 
[0:51:30] BS: I think we've covered a lot of the landscape. I guess, the other area where we've seen a lot of change besides Moore's law drives us to use new hardware and new technology to try to give more to our users. I should say, I am very thankful to have that problem, because I started in video. When I was at Avid Technology in 1998, video was rapidly becoming a solved problem, and it was hard for the company to come up with new ideas that they could sell to their customers.
 
What happens is somebody comes into the market with a much cheaper product, using the fact that technology is faster and they've got a lower-cost engineering structure. You get eaten alive from below, right? The bottom will come up from below, and that's where you got to look. This was after my time there, but I think to a certain extent, their bacon got saved by HD. This needs to like, I'll upgrade all of this equipment." It's like, okay, now there's something we can sell to our customers that they really, really need. Other than that, once you solve the problem, you're in trouble.
 
We used to pay for programs to unzip files. That was the thing you paid for. Now, we take it for granted. It's built in, right? That's a market that's gone. That's great for all of us. That's great for users. But as a programmer, you have to look at what you're doing and say, what's driving this? Will people want more? We've been very lucky that our users want higher fidelity simulation, and we can go use this hardware and do a better job.
 
The other area where we've changed is new programming techniques and language innovation. Besides moving to Vulkan to use graphics hardware better, we've really changed how we approach concurrency. That's also driven by hardware, because when I started, dual-core machines were exotic. There wasn't really a point of coding for them, because no one had them. Then we reached a point where, okay, people have dual-core machines. Separating the loader from the simulator would be worth it. We could load in the background and not pause your flight. Now 16 cores, sure. 24 cores, 32 cores. People have massively concurrent machines, and fortunately, the language technology is growing, too.
 
We have access to coroutines now in C++. Structured concurrency has become something that people understand, so you can program with language support for these multicore machines in a way that isn't the more equivalent of go-to for a concurrent programming. As we develop new systems, we're also coding them differently, and we're seeing people use structured concurrency in the sim and be able to write concurrent code from day one that's pretty safe. Hopefully, I mean, multi-threaded code is always really hard, but this is at least not quite as dangerous. It's been interesting to see how our techniques can change based on not just better hardware, but better tools and better community learning, and that helps us to address that challenge.
 
[0:54:24] KB: Awesome. Let's call that a wrap.
 
[END]