EPISODE 1903

[INTRODUCTION]

[0:00:04] ANNOUNCER: Multiplayer games are among the hardest software systems to build, requiring developers to synchronize state across unreliable networks while maintaining fairness, performance, and a responsive player experience. Latency, cheating, server costs, and debugging distributed game logic all introduce complexity that single-player games never encounter. 

Dome Keeper is a minimalist tower defense game with roguelike elements where players must protect a fragile glass dome from relentless waves of alien attackers. The game was developed with the Godot Engine and released in 2022. More recently, the development team embarked on the challenge of adding multiplayer to the game. René Habermann is the founder of Bippinbits and the creator of Dome Keeper. Chris Ridenour is the founder of KAR Games, which is a Godot-focused studio that developed Drift: Space Survival. Chris is now working with the Dome Keeper team to bring multiplayer to the game. René and Chris join the show to talk about the origins of Dome Keeper, developing the game, and the process of adding multiplayer to a Godot game. 

Joe Nash is a developer, educator, and award-winning community builder who has worked at companies including GitHub, Twilio, Unity, and PayPal. Joe got a start in software development by creating mods and running servers for Garry's Mod. And development remains his favorite way to experience and explore new technologies and concepts.

[INTERVIEW]

[0:01:42] JN: René, Chris, welcome to the show. Thanks for joining me today. 

[0:01:44] RH: Hello. Thanks for having us. 

[0:01:46] CR: Hello. Glad to be here. 

[0:01:47] JN: So, let's kick off with, I guess, an introduction from you two folks for anyone who's not familiar with your work. René, do you want to start us off? 

[0:01:54] RH: I did actually once upon a time study computer science and then work in IT. But my heart really lies in game development. Eventually, I always did this on the side while working in IT. Eventually, I made something that was at least impressive enough to publish our Raw Fury to fund the project, which is when I immediately left the IT industry and went making games full-time with the company I founded, Bippinbits. 

We made the first game, Dome Keeper, was quite successful. And now we are working on some new titles, and the biggest one announced is PVKK, Planetenverteidigungskanonenkommandant, which is a nice name for English-speaking people to pronounce. 

[0:02:33] JN: Yes, we'll probably get to talking about PVKK. I will not attempt to pronounce the full name. I'm glad that we've got that in there at least once. Editors, you can just paste that over every time I say PVKK going forward. And Chris, how about you? What's your journey? 

[0:02:46] CR: Yeah, I spent about 10, 12 years in the startup world doing various startups as kind of founder and CTO, living in that web world. But like René, my heart was really in games, doing it on the side. And a few years ago decided to just make the jump into full-time indie. We have one game on early access called Drift: Space Survival. But then I got a chance to work with René on the Dome Keeper stuff and took the chance to help with that as well. 

[0:03:11] JN: Amazing. Yeah, that leads us very nicely into what we're here for today, which is Dome Keeper has been out since 2022. I'm a big fan, and we've noted it as it's one of the Godot hits, which is always great to see. But we wanted to put this podcast together, particularly today, thanks to your talk at GodotCon, Chris, which was revealing that you folks have been tackling what I think is quite an infamously cursed problem, which is adding multiplayer to a game after the fact. 

Before we get into that and all the hairy bits, I guess we should talk about what Dome Keeper is. René, do you want to kick us off on talking about what Dome Keeper is as a game and how it works? 

[0:03:41] RH: Yeah, it's a roguelike. It feels a little bit retro, and the things it does do feel familiar. You mine down into this tile map, a tile-based map. You find resources and bring them up to your dome, with which you landed on the planet. And from the resources, you buy upgrades for your keeper so you can mine better, carry more stuff. But you also build up basically your defenses because you are attacked by monsters cyclically on this planet. And that's basically the whole loop switching between this mining and combat. Maybe one minute, one and a half minute of mining and then 20 seconds of combat. And it has sort of this nice ebb and flow where the combat is a short intense moment and then you get back to mining, which is quite peaceful. But you also know you need to be back in time and you need to find some resources. Otherwise, you might not make it. 

[0:04:31] JN: Yeah, I'm startled to hear you describe it as peaceful. I always find the mining sequence very race against the clock to get in as much as I can before getting back up. And so I I know that Dome Keeper came out in 2022, but what was the origins of Dome Keeper? Where did it come from? 

[0:04:43] RH: In 2021, we participated in Ludum Dare 48, which is a game jam. And it was our 10th time participating in that game jam. And the 9th game we made was sort of a smart game, where I had this smart and fancy idea about the game design I wanted to try. And it did work out, but it's not like a lot of people enjoyed it or even were intrigued by the concept. 

And so we went into the next game jam going like, "Okay, let's make something very stupid, not innovative, and just do something that we know we can do and try to polish it." And the funny thing is we took two parts that feel very familiar to anyone playing games. It's very understandable. But in the combination, it still felt fresh to many people. So it wasn't as uninnovative as I thought. 

Yeah. And we kept working on the game in our free time because we noticed, okay, people are really digging this. We were working on a bigger project that we basically paused in favor of Dome Keeper. And then it went quite quickly that the publisher of Raw Fury was interested in this. We signed a deal and then spent only 9 months in production and released basically in 2022. And that's over 3 years ago. 

And during that time, we did continue to make content updates, bigger and smaller ones. The last content update is over one and a half years ago already, because the big focus after that content update was really getting started on multiplayer. 

[0:06:11] JN: Yeah. Which tells us something of the journey, I guess, we're going to go on shortly. 

[0:06:14] RH: Yeah

[0:06:15] JN: Before we get to that, I played Dome Keeper the year it came out. I haven't touched it for a while, and then jumped back in. The anticipation of this. I was quite surprised how much stuff had been added in the menu now. It's like interactive and in the mind, all kinds of neat things. Can you talk about the things you've added and why you chose to add what content and where the expansions came from, I guess? 

[0:06:31] RH: Yeah, sure. Dome Keeper, as a roguelike, is very expandable in any case. So, that always made more sense. You can just add a new dome, add a new keeper, and that gives you new toys to play around with. But for me, a key thing was that I had these nine months of production planned. But then I spent nearly half of that time basically for marketing stuff, for preparing a demo somewhere, for preparing an expo build. What we released with was way less than I wanted to have in the game. 

Then I basically pushed the following years just to catch up and bring it to a level where I felt like, "Okay, this is close to what I had in mind," in terms of variety and how varied the runs can be, but also how much you can replay it. And once that was kind of settled where I felt like, "Okay, now it's in a good state," that's the time when, for me, porting the game to console got relevant, but also doing the big final thing, which is multiplayer. Because I thought about doing multiplayer earlier, but then I was like, "Okay, if we do it basically immediately after release and we are thin on content, it's not super interesting as a mode." You can maybe get 5 hours out of this, and then multiplayer sort of starting to be slower. 

I always wanted to add more content, make the best possible game first, before we tackle multiplayer. Because my hope is that multiplayer already has this big impact and brings back a lot of people, and also brings in a lot of new players just because of, yeah, the promise of that. 

[0:07:56] JN: Awesome. Yeah, that makes sense. And then I guess last question before we come to the multiplayer. Obviously, I noted that Dome Keeper is Godot. Was the Ludum Dare version of Godot? What led you to choose Godot? 

[0:08:07] RH: Yeah. I think the first six or five Ludum Dare games I still made with my previous textstack, which was libGDX, which is basically a framework for making games in Java, which does sound strange. Why would anyone want to do games in Java? And it's not a fully-fledged engine. You don't have an editor or anything visual. It's just a framework, but it was okay. 

But then I knew it was by that time, I think in 2017, just not the best tool to make games because I was seeing those fancy engines with editors and just making - if you have a framework and you want to build a UI, you program a bit of the UI, launch the game and see if the UI is how you expected it is, and it never is. So you have to do that process 20 times and then hope eventually you punch it into the right behavior. Compare that to making a UI in an engine editor. That's a whole different world. 

By then, I knew I wanted to switch. And I'm always a kind of late adopter. So I tried a few different engines. I booted up Unreal, Unity, and Godot. But Godot immediately clicked for me. It had this component system, basically. It's how it's structured with scenes and nodes. And you can compose nodes within nodes under nodes. Each node can have a script. And that is incredibly powerful because that's the huge thing about composition over inheritance. And Unity didn't have that. Unity, it just had these prefabs. You can pre-make a thing, but it didn't have this strong composition aspect. I really love that, how powerful it felt, how easy it was, how clean it felt. You can compartmentalize features. 

Also, just how fast Godot was. It boots up in two seconds. It's 100 megabyte large as an executable that you download. It's not 8GB installer or something like that. And then you launch the game, and the game is running in the 300 milliseconds and not after compiling, after 15 seconds, or worse, something like that. 

There were so many things that I loved about Godot. And Godot was mostly good at 2D. It did have 3D capabilities, but those were not as advanced and not comparable to Unity or Unreal. But I wanted to make 2D games anyway. I think back then, it was the best 2D engine, at least for me. And yeah, we started with it, and I never looked back. I'm super happy with it still. 

[0:10:30] JN: Nice. It was awesome. And I guess it would have been 3.0 at the time. 

[0:10:33] RH: Yeah, I think 3.2, something like that. Yeah. 

[0:10:36] JN: Awesome. Chris, I'm aware that you have - you mentioned your game Drift. And I believe there's a bit of a journey into how you came into this project with that, right? You got some multiplayer experience on Godot through Drift. Is that right? 

[0:10:48] CR: Correct. Yeah. Drift was my first kind of commercial game project. So I decided to use the alpha version of Godot 4, make it 3D, and multiplayer. It was a lot of fun, but it ended up working out. But I did obviously have to pick up that multiplayer mindset and have to learn all the quirks of multiplayer, specifically in Godot, which was really beneficial in this project as well. 

[0:11:14] JN: Nice. Yeah. I guess the alpha 4.0 at that point. I know there's some new multiplayer features in 4.0, but I've not done a whole bunch of multiplayer on Godot. I'm not entirely clear on them. But I guess you were dealing with features that weren't completely cooked as well at the time. Would that be accurate? 

[0:11:27] CR: Yes, I would say so. Yeah, it was worth it for a lot of the rendering improvements for 3D, moving from OpenGL to Vulkan. And the new multiplayer nodes in 4, they are so powerful for prototyping. Just be able to say, "Hey, I want to sync these properties between every client connected." And you just drop in that node in that composition way that René mentioned. You pick out which properties you want to sync, and it just will automatically sync it across any node. 

And there's the equivalent spawner. You say, "Hey, anytime we spawn one of these, spawn it on everybody." They kind of break down, I think, in a lot of large-scale projects. So we end up not using them in the long term. But in terms of just getting things moving, getting things out the door, those nodes are really valuable. 

[0:12:09] JN: Awesome. Cool. Now I'm going to ask a series of questions that's probably awkward, given that René is here. As someone coming in, I think the setup's really cool. You've got two indie studios and one studio bringing someone in who's got expertise to help. I think that's just like a really refreshing way to see indie studios work together. It's super interesting. But I'm really interested in your perspective, Chris, as someone coming into this hugely successful project from the outside to do what is already infamously a hard thing. What was that like stepping into the code base and stepping into Dome Keeper? 

[0:12:35] CR: Oh man. Yeah. So I never went super deep into Doom Keeper. So I did not realize the depth of content in there. There was a little bit of regret that first time I opened a project and saw how many different gadgets and how many different types of monsters there were. And it was just like, "What have I done?" 

Yeah, I think I ran one of those lines of code. And it was like 60-something thousand lines of GDScript in the project, and they had just transitioned from Godot 3 to Godot 4. And so you had a lot of just cruft from that as well. But it was a lot. But it was really interesting. You could definitely tell in the codebase that they had been trying to think about multiplayer from the get-go in some of the data structures and things like that. And as you mentioned, how the menu system, you're now moving around the cave. That was done with multiplayer in mind. There are definitely some things there that kind of give a launch board for us to kind of dive in, but there's obviously a lot of things that just weren't designed with multiplayer in mind with some of the data structures. 

[0:13:36] JN: Right. Yeah. So, I guess let's get into that. If we could start in an ideal world, if you were designing these data structures in multiplayer mind, what does that look like? If you're approaching a multiplayer, if you were doing like a 2D Dome Keeperesque game from scratch, what would be some of the design or architectural decisions that you think we're not here? 

[0:13:54] CR: Some of the biggest things - I'll take a step back because I think what was really difficult for me when I was getting into multiplayer networking, it's thinking about what you're actually sending back and forth. Because it really took a mental shift for me. Coming from the web world, which we all kind of did, especially from a backend perspective, I think about things in a request-response. Or when I'm on a front end, everything's responding to an event. 

Now, obviously, some people, I have friends who used to work on rendering at Figma. They had to think about frames per second in the browser and things like that, but we never really did. It's for most web developers. But in the game world, you're thinking about what is happening every frame. The game engine asking 60 times a second, "What am I drawing? Where is everyone?" And so it's simulating this world that is finally ending up as pixels on your screen. 

And so for multiplayer, you now have two computers that we're trying to get to simulate the same world. And what we're trying to do is just send enough data between those computers that they end up with the same simulation. So with that in mind, things happen. Physics simulation in Godot is not deterministic, and it can't be based just with the way they've built the engine. 

And so in Dome Keeper, when you're carrying iron and different resources, those are physics objects, and they're bouncing around, and it makes it so much fun, but it's impossible to get two computers to simulate that in the same exact way. So then we end up in a situation where now we have one computer has to simulate everything and then send all of that data down to every single client. 

And so those are the kind of decisions where things can't be deterministic just by their nature that make multiplayer really hard. But also, it's why that's fun. Carrying resources, which normally isn't something you think about as fun in a game is actually fun in Dome Keeper in a lot of cases, especially when you add a second player and then two people are trying to grab it and things like that. That is why that is so fun, but it also is a huge challenge for that reason. 

So, if I had a magic wand, maybe I would have had a separate physics system that was deterministic and Godot or something like that. But beyond that, it really comes down to some of these data structures. So how do we think about who's doing what? There's a lot of times where before we even get into versus mode, but there's a lot of times even just in co-op where it kind of matters who's doing what, and whether you're grabbing something, which keepers doing that. There's a lot of things that just weren't in there across these systems because it was a single-player game. You pick it up. You are you. You don't have to worry about anyone else. And so those sort of things can be really difficult. 

And so as we kind of approach each system, obviously, since we're adapting a single player game, which kind of has a right answer, which is like this is the game field, this is how the game played before. We kind of have to break the system, break the data structures down, and then rebuild the same features, which was probably still like 90% code reuse, but build those same features back on top of these new data structures so that we can have multiple keepers. 

Or a big one is we do things in a different order now. There's a lot of times, a lot of bugs as we go through QA where things break down because there was never imagined that this cave could come at a different stage. It could come after the player instead of before the player, or things that where there was just no defensive programming. And there's really not in games cuz you don't have really time to do that. And you just - why? Again, who's going to change the order of this and why? But yeah, all of a sudden, now we're waiting on network packets to come in and decide when these things are happening. And so it's that ordering and things like that that kind of can really break down on us. 

[0:17:31] JN: Interesting. Okay. I'm realizing as you answer some things that it probably be useful for us to talk a bit about how multiplayer works from a gameplay perspective in Dome Keeper before we get into some of the end of it. René, what is the game mode? How does it work? You've still got you still got one dome, I guess, is my fundamental question. 

[0:17:48] RH: Yeah, we went quite hard on what you can do in multiplayer. You can play the normal game you always play, just with a different person or multiple different persons. So they all get their own keeper. They get their own upgrades for the keeper, but you share upgrades for the dome, which already makes the upgrade system really complicated because you have all these different kind of types of upgrades and different ways to count stuff. Upgrade system is also the one where there's one function, whenever we touch it, we introduce three new bugs. It's really bad. So, that's one thing. 

You can also do prestige runs. That is like a high score attack with one other person. But you also have a versus mode where you actually have two domes and two mines, one mine below each dome, but the mines are connected, so you can reach each other and steal resources. And you can buy upgrades, so the other team gets stronger waves. 

You think about, okay, do I increase my mining and defense, or do I really pressure the other team, right? That was the mode I was most excited about for multiplayer. But that's the second layer to it. You have local multiplayer. Up to four players in split screen. But you also have online multiplayer, where you can have more people. I'm not even sure if we have a strict limit right now, but eight players definitely works. 

[0:19:10] CR: Let's say eight. 

[0:19:11] RH: Yeah, let's say eight. Officially, it's eight. Yeah. And you can also - it's one of the things I'm most doubtful about, if it was a good decision. Or more like I don't think I would ever decide the same thing again in the future for other games. You can also mix this. You can also have two local players playing online with three other people that are online. And this really, really makes stuff so complicated because the whole local co-op from a code perspective works so much different than online multiplayer. And now you have to mix this and also have it work in single player. And you also have teams suddenly where you not only share one dome, but it could also be that the upgrades for a dome are also on the other team. Codewise, it just exploded this sort of space or the state. How many different setups can you have? And that's really complicated. 

[0:20:00] JN: Yeah, that's really fascinating. And it's made me think of a question that I think to not jump the gun a bit. I guess let's start with talking about how you approach this from networking perspective. Are you doing this multiplayer all peer-to-peer? You're not getting a dedicated server set up? 

[0:20:14] RH: No. 

[0:20:14] JN: Okay. Awesome. 

[0:20:15] RH: Yeah. So, as we think about it, you can do like a LAN game and just play locally, which we use ENet under the hood for. You can do peer-to-peer, which is on Steam. We'll just use Steam's peer-to-peer system, or we have a crossplay system, which makes sense as Xbox just launched, and they're going to have crossplay with that. And that all goes through Microsoft PlayFab. So, there's three different layers of networking on top of it as well, just to add to more of the possibility space for bugs. Yeah. 

[0:20:45] JN: Yeah. So, I guess lead to my question, which was from your GodotCon talk, which is wonderful. I highly recommend listeners go watch the talk. A lot of what you dealt with, which we'll get into, is optimizing the bandwidth and how much data you're sending to represent the game state. How does that work when you've got multiple keepers locally? Because then that's obviously - surely, that's so much more data that you need to send to the online player. 

[0:21:05] CR: I would say the majority of the bandwidth optimization we had to do was in all with the resources. Because when you think about - even if you came into the game with four local keepers, you only have four people moving around. That's where you get into trouble, is like how often do you have to send these updates, because that's really going to impact your bandwidth. Moving a keeper around, we want to send as many packets as possible as fast as we can just to keep that smooth and keep it feeling like that person is really playing next to you. 

But the keepers, let's say you have four keepers, well, the same thing applies to the resources. And so if four of you are then carrying eight resources, that's usually where we get into trouble and having to send those 32 resources and things like that. Especially in the talk, I talk about how we went from descending naive data structures to going down to individual bytes. And then we're now at the individual bit level, thinking about, "Well, I only need a six bit integer to really represent all the positions this thing could be in." And then you can start to pack those together into - we get it down to like two 64-bit integers for each resource. 

I mean, we pack all the rotation data, the location, who's carrying it, all those things down into that. And that has really helped into it. Because, before, I say eight is probably really where we're still going to max out because that's really when you have to have a really strong host sending a lot of data to everyone, because it's going to have to calculate and send that data for every resource that someone's carrying at the time. 

If no one's carrying it, it goes to sleep and we don't send data about it. But that can really just start to balloon really quickly. And so for each person you add, you're adding more things that could be carried. You're adding more people you need to send that much bandwidth to. And, yeah, that gets out of hand really quickly. 

[0:22:49] JN: Right. Yeah, that makes sense. So that's the resources. How does it work on the monster phase? Because you can have a lot of projectiles and all kinds of stuff flying around at that phase. 

[0:22:59] CR: Yeah. So with monsters, we were actually luck - I don't know if lucky. We'll say lucky. A lot of what the monsters do end up being pretty deterministic, or it could be. So one thing we do is when the monsters are generated, we send their kind of UID with them, their unique ID, and we use that to seed a random number generator on everyone's client so that their behavior is kind of mapped out from the moment they're spawned. And then everyone can respond to the events of like the monster getting hit. 

But the monsters end up being very event-based, which is really thankful. Because if we had to sync positions every frame for monsters and projectiles and things like that, it would end up in a really bad place. But yeah, the monsters, when they shoot, it's kind of predetermined. And even if it's not, we still send a packet when they shoot anyways just to make - it's kind of like a chance to catch up. 

And so we have a lot of these full sync check-ins for monsters just to make sure that if someone's running a few frames behind or just generally running a lower frame rate, we're just making sure that they catch up and hit things in the right order. But generally, that's more of like an oh no rather than a requirement. 

But yeah, the monsters end up being pretty deterministic. And we're in a lucky case where - and I talked about this in the talk a bit, but even when you're playing versus or the leaderboard prestige, we made a decision at the beginning of this project that we're going to trust the clients. 

If a client says, "Hey, my laser just hit this monster and it took damage, and maybe the server is a couple degrees off, it doesn't agree. But we're just going to trust that the client made that damage." Because if you start treating the clients as kind of adversarial, that's when you absolutely need dedicated servers, and then our costs balloon. And it just doesn't really make sense for this game, this type of community. It's not that competitive of a community. 

I think that for this game, this player base, it made a lot of sense. It's friendly. Even when it's not friendly, it's friendly. If you have friends who want to cheat, then you should talk to your friends and things. Yeah. 

[0:24:56] RH: Even before multiplayer, we decided to not have the sort of public lobbies where you play with random people. Instead, you do play with your Steam friends, which really takes off the pressure of kind of, "Okay, now you have to prevent cheating." Because yeah, if your friend is cheating, yeah, you will call him or punch him when you see him the next time. I'm not sure. I think that was also just like a mandatory thing to decide to even attempt to do this. Because the super tricky stuff is, "Okay, if this needs to be like server authorative, I don't think it would have worked to hammer this into the existing single player game." 

[0:25:34] JN: Yeah. Yeah. That's a whole different set of challenges. We had a great episode with Gabriel ages ago. Listeners, go check out the real-time multiplayer conversation with Gabriel to learn about server authoritative stuff. When I saw you talking about the friendliness, and now knowing there's a prestige mode, because I didn't think that you had mentioned that in your talk. That is really interesting because I know that you've obviously - when I was poking about the menu, I saw the big warning next to prestige mode about people using cheating and keeping an eye on it. I guess you're just like, "Yeah, it is what it is for the multiplayer." Or - 

[0:26:00] RH: That's another thing I will never do again, like a public leaderboard with global high scores. The only thing I will do in the future is high scores among your friends because we always have cheaters. I ban people sometimes. And then it's okay, but there's just this infinite amount of people coming back as cheaters. And the effort to maintain this and just keep it roughly fair is stupid, or it's not fun. 

The prestige players are hyper competitive, and they really take that serious. And we had people do like 10-hour runs, although the game is more structured towards one to two-hour runs. And they would do endless grinds in those 10 hours. But it's only like this tiny fraction, like the 100, 200 people in the world who really compete on this. But it's a lot of work. Yeah. And for multiplayer, yeah, it's not getting better. 

[0:26:53] JN: Yeah. I mean, I guess that was - and I know this was a factor in your choice technology, but that was one of the things that was really interesting, I think, especially coming from an indie project. Obviously, it's been a very successful indie project but still an indie project. And there's so much discourse now about what happens to the multiplayer elements of games after the company has moved on. How did, I guess, the economic constraints factor into the technical decisions for you folks with choosing how you're going to set things up? 

[0:27:15] RH: For me, it was mostly about - I mean, multiplayer has this big potential, as I said. Bringing in a lot of new players and old players. There's definitely a reason to do this kind of big investment in development. I think that makes sense. At the same time, we cannot do this high server cost, where we would need to monetize the people that keep playing in some way. That's really the reason to go peer-to-peer. And now it's more hoping that our PlayFab bill is not is not exploding. 

Luckily, for example, the people playing on Steam together, Steam is not charging for their services. It's within the fee they get anyway from sales. That's generally fine. I don't think there are really complicated economic calculations going into this. It's a premium game in the sense that you buy it, and then you own it, and then you can play it forever for free, and you don't need to do more. But that restricts what we can do also. 

[0:28:09] JN: I guess you incur PlayFab bill. Is that only crossplay, or is that also like if a console player plays for console player? 

[0:28:14] CR: I think it's anytime it leaves the Xbox network. Xbox to Xbox is free. But any sort of crossplay will incur cost. 

[0:28:21] JN: Interesting. Okay, cool. To go back a bit, Chris, you mentioned that you found some of Godot's and built multiplayer stuff didn't go when you built your own. Can you talk to us about that? What challenges do you face, and where did you have to find new solutions? 

[0:28:34] CR: Yeah, Godot, it's really interesting. They have their kind of like high-level networking. But Godot doesn't really prescribe anything. You don't actually even have to use the high-level networking or the multiplayer peer system that they have if you want to use their kind of built-in RPCs. There's nothing stopping you from just making your own just network library in GScript, or C++, or C. And so I appreciated that. 

But what I did like about the built-in high-level networking is it'll let you just - it kind of highlights when you actually have an issue. One of the things in Godot is if you have an object in the scene tree and you want to call a remote procedural call in RPC on other clients, that object has to exist on everyone's computer or on everyone's simulation. They need to have the same object at the same place in the scene tree. 

And that kind of throws a lot of people for a loop. And it threw me for a loop a lot of times because like, "Well, it doesn't exist yet." But then the question is, "Well, if it doesn't exist yet, why are you actually trying to call this on it?" You're highlighting an ordering issue, a race condition that you may not had to think about otherwise. Otherwise, you might just be sending packets to people who don't know about it or aren't ready for that packet. And so I think you can end up in a lot of bad situations if you're just kind of naively sending packets about everything to everyone and let them kind of figure it out. 

The high-level networking Godot kind of forces you to think about ordering and think about when you're doing things, why you're doing things, who's ready for what. Especially with Dome Keeper and same with my other game, Drift, it was drop in, drop out. So you might have people at different stages of the simulation. They might be loading, they might be currently spawning the keeper, or maybe they're ready for everything. 

And so one of the things we did specifically in Dome Keeper is kind of create our own kind of network state manager for each client. And everything that talks to the network in Dome Keeper recognizes this and only sends these packets to people who are in the correct state for that. And it lets us avoid just a ton of different race conditions. Not that we have none because we still have them all the time, but it does let us think about like, "Okay, this person is still in the loading screen, or this person is currently spawning the keepers. So we shouldn't be berating them with a bunch of different packets maybe about resources and things like that." 

[0:30:43] JN: You've mentioned the drop-in, drop-out. That was really interesting. I thought how you got into the mechanics complexity of that in the talk was really fascinating. One of the questions I had was what the design intention was. Basically, were you seeing drop-in, drop out as like, "Oh, I've disconnected and need to reconnect ." Or is it truly like, "Hey, I've seen that my friend's playing dome keeper. I want to jump into that session midway?" Which way around was it? 

[0:31:07] CR: We had talked about both. Just for game design reasons, it's more of I want to reconnect after I've dropped out. Because from a technical perspective, it's both, but it makes it really difficult of are you scaling the whole game when a new person joins in? And there's just some considerations. Yeah, it makes it really difficult. But I think you could definitely get there, but I don't know that it was worth - the juice wasn't worth the squeeze. I don't know how often that's really happening to really have this whole very dynamic difficulty system that kind of can flex up and down when most people just want to get together and do a run together. It's not like a 60-hour run. It's one gameplay session for most people. 

[0:31:45] JN: Yeah, absolutely. Yeah, that's what came to mind, especially when you were talking about how it worked earlier, René. Individual upgrades. I was like, "Okay, someone's going to come in, stack all the individual upgrades, and leave. What happens at that point?" I guess one question I had for you, René, about this new loop. Obviously, as we kind of spoke about at the beginning, a big part of the single-player experience is balancing the monster. I want to say day-night cycle, but I guess that's not accurate. But the monster mining cycle when you got multiple players. And obviously, one could go up the dome. That feels like it fundamentally changes that. How do you see that working? 

[0:32:14] RH: The most difficult thing that we cannot really get around is normal single player. You go into the upgrade menu and you have this big, big list of upgrades. And the game pauses while you are in this upgrade menu. In multiplayer, it's not very fun if you play with seven people, and every time one person is in the upgrade menu, the game pauses for everyone. The game kind of needs to keep running, which has this balancing impact already. And also, it's a little bit of tension. I guess it's harder definitely if you play for the first time and never have seen the upgrades. That's one thing where we need to compensate a little bit. 

The cycles are just a little bit longer to make room for the game not being paused. At the same time, with two people, you are so much faster with mining that you don't need to scale one-to-one with the pause, for example. And generally, it's mostly down to how fast the monsters scale. Depending on the amount of players on a team, you get stronger monsters. 

It's definitely tricky to get right. Because in a single-player game, you have this variety on how the run can go. But now if you have two people, it's basically kind of multiplied. Because if you have two excellent people, it's multiplicative in some way. So they would need a way harder difficulty than two very weak players. It's a bit tricky, and we're still play testing, getting feedback, and then tuning accordingly. 

The balance is really key to get it right. If it's too hard, it's no fun because you just die. But if it's too easy, you don't feel the pressure, this time pressure. And then without any of the pressure, it's just work, basically. It's more of a chore. You might still enjoy the chore. Some people do that. Some people mine every tile there is in the whole mine. But generally, you need this bit of pressure for it to stay fun for most people. 

[0:34:07] JN: Yeah, that makes sense. And yeah, on the topic of the mining game bigger, is it the same map settings? It makes me think that the maps will probably need to be bigger for an eight-player game mode. Or is it just shorter sessions? 

[0:34:15] RH: Yeah, that's a good question. We haven't actually made the maps bigger in general, which just means we have these missions in Dome Keeper. If you go on the same mission, you will just be able to do it more quickly if you achieve it. But also, I mean, the monsters are still scaling up. Generally, it comes back to the same question about joining. You start with two people, then two people join. The mine cannot expand anymore, or something like that. 

[0:34:43] JN: Yeah, of course. Yeah.

[0:34:44] RH: It might be something that we also find during a future play test. We still have the ability to change map sizes. I haven't yet changed it, actually. 

[0:34:54] JN: Cool. I guess zooming out from Dome Keeper a little bit. One of the questions that this raises for me is if you were - well, I mean, obviously you are doing again. You've got multiple games in development. But if you were to do this again, would this experience of having added multiplayer on later change the way you design games in the future? Or is it working out well than you would - the vision worked? How would you approach this? 

[0:35:18] RH: Looking back, it really is kind of a budget and validation constraint. So, if I don't know if I have a game that works and that has optional multiplayer, do I really want to spend half a year longer, or maybe more, on the game before I get kind of this validation? Okay, there's a market for this game. People want to play this. 

And with Dome Keeper, it was much in that way that we just wanted to make this game as quickly as possible. It works without multiplayer. And there I think it was the right decision. And then it's more about, "Okay, how much trust do we have in the new game, and how complicated is it? Or is maybe multiplayer a prerequisite to the game existing?" And now that we had our first success, it becomes a little bit easier to also take a little bit longer before needing to validate. But still, if you make five games and they all take half a year and none of it ever works out, even with success full game out, you still go bankrupt eventually. It's still smart to not overdo this. Not to spend too much time in a thing you don't know if anyone wants to play it. 

Yeah. But generally, I like multiplayer a lot, but it really needs to be a thing for me that is essential to - maybe not essential, but really elevates the game. There needs to be this huge benefit from it. And I'm also more looking towards, "Okay, what are cheap ways to do multiplayer?" It doesn't need to be always the real-time shared world stutter-free thing. Sometimes it can be an async multiplayer of some kind that still elevates the experience and is way easier to do. 

[0:36:48] JN: Yeah, when you said earlier about the friends-only prestige mode, that - well, friends-only prestige scoreboard. That immediately made me think of Zachtronics games. I don't know if you ever played any of those. But I think that's a really good example of like a game that feels like it's got multiplayer when really it's just that casual scoreboard on. But you see your friends optimizing against challenges, and you're like, "Oh, I now need to go compete with them." Right? And it's not actually playing the same session, but it feels like a collaborative experience in what I imagine is a very cheap and cost-effective way. Yeah, it's a really interesting point. 

Oh, yeah. You've already mentioned it. You've been working on the last update. It was a year, two-year and a half. And, obviously, now you've been working on multiplayer for a while and you're in the QA phase currently. Is that correct? Bashing bugs. 

[0:37:30] RH: Yeah. Cool. What is your QA process? Is it just getting bunch of people in the game and hashing it out? Because as complex and many-headed Hydra that multiplayer is, I imagine it must be hard to even design a process where you can hunt down some of these edge cases, right? How do you approach multiplayer testing, Chris? 

[0:37:51] CR: Well, luckily we have a fantastic QA team through Raw Fury that knew the game really well coming in, which was really beneficial for me because there are things that I would break that I didn't even know were in the game, especially coming in. And so it kind of gave me a lot of liberty to come in and make the changes and obviously do some surface testing and make sure I didn't completely break the build. 

But otherwise, kind of put that out there, and they are able to kind of dive in and really find these edge cases and put that together. And so it's been really nice. We've had a public play test now in the Discord where people can come in and kind of find more of those cases. So definitely some of the more hardcore players who have very specific things they're looking for have definitely told us where we've gone wrong. And a lot of the technology behind the multiplayer build is actually behind the console builds as well. With Xbox out there, we're still getting more feedback and more testing underneath changes to the game that are giving us a lot of confidence for this as well. 

[0:38:46] RH: I think the testing process is not very different from before. It's just like the QA offered is like five times as high because you have all these scenarios you need to cover. And you can have stuff that works perfectly fine in co-op and works perfectly fine in online multiplayer, but it breaks in single player suddenly. You cannot make assumptions about, "Okay, if it works here, it also works on the other bit." 

[0:39:09] CR: Yeah, testing with multiplayer when you're developing is actually something where Godot really shines. One thing you can do is, I think 4.2 and later, you can go down to the debug menu and tell it to launch multiple instances. And you can use kind of launch arguments to make them automatically connect to each other. 

And so when I hit play in Godot, I have two or sometimes four windows pop up. They automatically connect to each other, or just dropped right into a game, and they're all connected to the debugger. And so when I'm testing something, I can hit pause. Everything will pause. I can dive into the scene tree of the specific instance that I'm looking for to figure out what's going on. 

And so that has been just so powerful compared to trying to do multiplayer on Unity or Unreal, where you kind of have like one that's running outside of the editor environment and then one that's inside and you have to like, "Oh, is this a post problem? A client problem?" With Godot, you just kind of get all of that right for free. And so since we have that like ENet, we can run it all locally, connect to each other. But then obviously we then move to Steam. 

Testing Steam multiplayer is very difficult. You need two Steam accounts. You need two physical computers unless you're doing some VMs, or sandboxing, or whatnot. But even then, you're still not attached to the debugger and everything like that. We try and do as much as we can locally. And then once we're going, then we - we have QA. But other people, that's when you annoy your friends, is when you just want to test the kind of the Steam specific stuff. Yeah. 

[0:40:37] JN: Yeah. The breaking things in single player is really disappointing. I'm trying to remember which game it was. It was a past guess, but we had a game at one point where their multiplayer was a completely separate build, and they added multiplayer on, but none of those changes, and made it back to single player. And I guess that's the main issue you avoid if you're doing that. Aside from the fact that it's a nightmare to maintain forevermore. Yeah. Really interesting. 

[0:40:58] RH: That's another reason why we put multiplayer and code supporting until after we were done with making content, right? Because making content in that setup suddenly is so much slower and more complicated. 

[0:41:09] JN: Yeah, that's when you mentioned the physics objects being the main constraint. I was immediately imagining if you had built this all out before you added the accessor, the gravity-slinging things, and how that would probably break a lot of assumptions made of just the engineer. Yeah. Cool. 

I know we're running close on time. Before we run out of time, I do obviously have to ask about PVKK, which I think a lot of folks are waiting with bated breath for. My first question is, honestly, what is it about bunkers, being in a bunker that's appealing to you as a game designer? 

[0:41:43] RH: The pragmatic answer is that limiting your game to a single room means you can make that room really nice. It's a smart decision in production. That's a thing. But also, it's part of the fantasy, and this appeal. And we also play with it narratively in that you are confined to this bunker. It's basically a point in the narrative that you are essentially a prisoner. And that's so interesting in itself, maybe more interesting than if you would be free to leave. And we can explore the whole facility, something like that. 

[0:41:43] JN: Yeah. Cool. And then techwise. Obviously, we spoke about you chose Godot for its 2D capabilities. Is PVKK - I mean, it looks very shiny. Is it a Godot game? Or you moved over to a different engine? 

[0:42:21] RH: Yeah. 

[0:42:22] JN: Awesome. 

[0:42:23] RH: Yeah, it's in Godot. We could not have made it in Godot 3. It's something enabled by this huge push by the thousands of people working on Godot to make Godot 4 a more capable 3D engine. Then it's really down to, "Okay, who's working on the game? What's their eye for beautiful things? And what can they do?" Of course, it's looks different than an Unreal game or something like that, but it still looks nice enough that as a player you can be happy with it and enjoy the graphics, I think. 

[0:42:54] JN: Yeah. No, I think it looks incredible. I saw a comment recently. Someone who had seen the trailer was like, "Is this real? This looks like a render. This looks really good." And I was like, "Yeah, I'm pretty sure it's real. It looks so cool." Awesome. 

Obviously, I desperately want to ask when we can play it. But I was just asking the cursed questions. And I guess the next addition thing I was going to ask is this about - this is somewhat not related to the game. But as someone who was playing Xbox during the Steel Battalion era, obviously I was very excited about the huge controller you took to - was it Gamescom? 

[0:43:24] RH: Oh, yeah. Yeah, it was at Gamescom. It was in Tokyo Games Show and in TwitchCon San Diego. 

[0:43:30] JN: How did you go about - what was the development process for that? When you first bought the game, you like, "Okay, we need to build the bunker?" Or is this something that came out just planning for booths? 

[0:43:38] RH: I think during the development, this idea of making something physical popped up 20 times independently of each other. It kind of is a natural thought, like, "Okay, you have this very haptic tactile feel in the bunker. Could you have a controller for that?" And also, a lot of people are still fans of Steel Battalion, the Steel Battalion controller. We always maintained the thought a little bit. 

And then towards Gamescom, after we signed with Kepler, we suddenly had this - it was a little bit of an, yeah, out there idea to actually build something significant. And not just like a single red button you press to fire up the cannon, but the whole big station. And with the help of Kepler, it was possible for us. I don't think we would have taken the investment alone to do this. Because if you only build a single one of a new thing, the price for the single thing is very high. If Ford wants to build a new car and they only build a single car of their own model, it will be a very expensive car. 

We were looking for partners to do this. Because at Bippinbits, we're not like electric engineers. Some of us can do some things but not in that like full-fledged vision. We got a few concepts from different people somewhere a little bit underwhelming where it's more like, "Okay, you have a gorge, but it's actually just printed on the panel. It's not real." And for me, the fantasy of PVKK is really that everything is real. 

And then, eventually, we found with Mother Ultimate and Rethrow switches, we found a partner to build it. We still needed to do a lot of integration work. The game needs to run on this thing, then it's not a given. But it was quite crazy. How it came out was really a highlight, I think, for many people, how much they got out of this. It's the same for us. It's incredibly fun to play. 

[0:45:24] JN: I hope after, at least, it ends up installed somewhere where people can play it. I would definitely travel for the experience. Well, I know we're running out of time. Chris, René, thank you so much. Obviously, René, we spoken about Bippinbits and Dome Keeper a lot. Folks, how to find you? Chris, where can people find your work? 

[0:45:39] CR: Yeah, on Steam. You can search for Drift: Space Survival. 

[0:45:43] JN: Awesome. Perfect. Well, folks, thank you so much. And enjoy the rest of your day. 

[0:45:47] RH: Yeah, thank you. Thank you for having us. 

[0:45:49] CR: Thank you, Joe.

[END]