EPISODE 1821

[INTRODUCTION]

[0:00:00] ANNOUNCER: Emulating retro games on modern consoles is a growing trend, and allows players to experience classic titles with improved performance, enhanced resolution, and added features, like safe states and rewinding. However, this process raises many challenging technical questions related to hardware compatibility, performance optimization, rendering, and state management.

Implicit Conversions is a company focused on emulating retro PlayStation games on modern consoles. Robin Lavallée is the CEO, and Bill Litshauer is the COO at the company. They join the show to talk about the engineering that's needed to emulate and enhance retro games.

Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent.Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

[INTERVIEW]

[0:01:15] KB: Hey, guys. Welcome to the show.

[0:01:16] BL: Oh, great to be here.

[0:01:18] RL: Hey, nice to meet you.

[0:01:19] KB: Yeah, excited to get to dig into the cool stuff you're doing. But let's maybe start with the two of you. Can you take turns, introduce yourself? Who are you? How did you get started in engineering and game development? And what is Implicit Conversions?

[0:01:34] RL: I'm Robin Lavallée. I'm a French-Canadian. I live right now in northern Philadelphia. I'm a software engineer, geek, video gamer, programmer at heart. I had programming when I was, I guess, seven-years-old when my dad bought me a Coco 2 Radio Shack TRS 80. The manual was in French. It was important. I didn't speak any English. I started coding Basic, and I guess, I never stopped. Then I got a job. I worked at Ubisoft. I worked at 2K Games. I worked at Twitch. I work at Meta, where I work on the Oculus Quest, on the Project Horizon there. At some point, I guess, I got bored.

What I did was, I was always late on my gaming. When the PS3 came out, it was like, cool, now it's time to buy some PS2 games. When the PS4 came out, I guess, I never bought it. The company just came from there, because I play old games, and why not? I must not be around doing this. Yeah, that's me.

[0:02:30] KB: Awesome.

[0:02:30] BL: Cool. My background is, I'm also a Canadian. I was born in Montreal and moved to Toronto after I graduated from Queen's University. Had a computer engineering degree there. Although I studied the hardware side of things, like IC design, I never, and I vowed I would never work in software, because I didn't like it at the time. My first job was in software, and I've never not been in software since then. I couldn't do anything with hardware now if you asked me to.

I worked at some digital signage company called Omnivex for a while. Then my first gaming experience was at Webkins. That's pretty retro for most of us. I was there from 2006 to 2009, and I was very fortunate to join there just as it exploded onto the scene. I think I was the 23rd thousandth virtual pet that was adopted, and when I left three years later, we ran 100 million. It was massive growth, and I got to be a part of that, and I was director of QA there. From there, I went to another organization called Earth Rangers, which helped kids protect animals. There was a lot of casual games there as well, too. I told I've helped produce over a hundred casual games throughout my career. I was introduced to Robin through a friend of mine, and we hit it off really well, and of course, I'm big into the old games, anything classic. We have similar passions, and we hit it off really well, and this is pretty much the dream job for me, and I've been here since, I think, I was the 5th or 6th hire. I'm COO here, so I manage operations side of things.

[0:04:12] KB: Okay, awesome. The company, Implicit Conversions, y'all are emulating legacy games, is that fair? What kind of game systems do you emulate?

[0:04:21] RL: Yeah. Mostly PlayStation, so PlayStation 1, PlayStation 2, PlayStation Portable, until we have started looking at other platform, like behind the scene, we did NES, so we have a Micro Mage that we actually help port and publish, and we had a networking support, so that's been pretty interesting. I used GGPO with the rollback networking library that's mostly used in the street fighting games. This is an NES game, and that was made by independent Morphcat, and then we added PS4, PS5 support to it and networking there, so we can play four players online. So, we did that. Then now we're doing, from those PS1 game, we just mentioned, now we're exporting them to Xbox, to the Switch, and Steam, GOG, and I'm missing something here, PlayStation obviously.

[0:05:09] BL: Yeah. PS4, PS5. Yeah, the big thing we get requests for PlayStation games, especially PS1, that is by far the number one request that we get from our clients, and they want it on Switch. That is the big thing. We have our own proprietary software, and we call it Syrup, so it's called the Syrup Emulation Engine. That allows us to integrate different emulators as modules, or plugins. Our PS1 emulator is called Pancake, and our future PS2 emulator might be called Waffle, and PS3, which I think, we may talk about a little bit in the future, might be called Benedict, because it's the fanciest of the three, but we just keep adding these emulators to it.

Syrup, what it allows us to do is add a whole bunch of special features to the game. For example, on our 2D, like Micro Mages, which is a 2D platformer that scrolls upwards as you're playing, we can add widescreen support, so that you don't get the two black bars on both sides. We can also add an interactive manual over way, which is really slick. We added a shader so that you can turn the pages, and you still, we live that experience of when you bought a NES game in 1990, whatever, and then you - I suppose, it would be a super NES game platform. But you would read the instruction manual on the way home, in the car ride. At least that's what I did. You're able to do that with our games now.

Of course, like Robin said, we integrated modified GGPO so that we could have net play supports, which had its own challenges. Whatever the request is, we're able to add that as a module into Syrup, and branch that out. So, we support Genesis, for example, as well too. We just have a request for that at the moment.

[0:06:50] KB: Let's maybe dive into what it means to port one of these games over. You mentioned the word emulation, so does that mean that you're essentially starting with an existing binary, and then running a simulated machine underneath it? Or, what does the stack look like as you're bringing one of these games from PS1 to Switch, for example?

[0:07:09] RL: Yeah, it's emulation, so we don't use a source code. For example, let's say, we have the ISO file, the ROM file from one of the game, right now we're doing something called Fear Effect 1. That's a, I think, a two-disc game, or maybe four-disc. I got it wrong. Anyway, so we have those files, obviously as part of the package. The stack looks like we have a common library, like all the low-level file and link, and stuff like that, read text, and so on. We have Pancake, which is our PS1 emulator. You can see it a bit like any PS1 emulator that exists in open source, except this one is proprietary. Then we put this on top of Syrup, which is basically a front-end based on retro arc libretto. In that case, that front-end is ported to console, because there's a lot of emulators.

You might say, why are you doing all this? Emulators already exist. If I go in GitHub, I can find hundreds of them, right? That's true, but they're all on PC mostly, and people are not buying them. The reason they're not buying them is because, well, we won't talk about it, but you know why. Then the issue is, okay, fine, oh, can I get a Switch, a legal emulator that I wanted in Switch, well, that doesn't really exist, because you need an NDA with Nintendo to do that. You need an NDA with Microsoft to do Xbox and so on.

We have, with my gaming experience that I had at Ubisoft and so on, I learned a lot about console development. Then from the emulation side, if you join those two together, now we get emulation for console, basically. Now I'm using my, I mean, it gets geeky now, but my business connection to try to convince a developer, or publisher to say, "Hey you got this game," or we got any game, I don't know, Silent Hill. We didn't do it, but let's say, Silent Hill. I want Silent Hill on the original one, and do it again on the Switch. Well, we can do it.

People are actually surprised. I don't think the publisher and developer fully understand sometimes that we can do port. We call it port, because in the business dev, it's what it is, it's a port, without the source code. I think that gave us a big advantage as a company.

[0:09:08] KB: Okay. As someone who's not a game developer, and some of the folks out there will be very familiar with what goes into game emulation and some will not, what does that end up looking like? You have this file. It's making a set of, essentially, system calls down into the game level, and you are providing an interface for those and mapping them in some way, but what does that stack inside of the emulator look like?

[0:09:30] RL: We have HLE emulation. Basically, the original game, let's go back to the PS1 game. PS1 game was compiled by a team 30 years ago, and it was put on a disc. That code that live on the disk has a bunch of game code, which is fine, because we got permission to use this game code from the publisher. But also, has NDA code, like the SDK code from the PS1 SDK. Okay. You might say, well, who cares? But we do care, because that code, technically, SDK is copyrighted by Sony Interactive, or Sony Computer Entertainment back then. If you want to run this on PlayStation device, that's fine, because, I guess, you're doing Sony to Sony. But if you're running this on the Switch, you're not supposed to be running SDK proprietary code, putting that on the Switch. You don't own this thing. You're not allowed to do this.

We have to remove that code. Okay, all right. Let's remove the code. Let's say, the code is, I don't know, it's code to save on a memory card. All right, so there's some function that they added, like we call mem save, or whatever slot, and then it does stuff. Well, okay, you remove that code, now you can't save anymore, not really. You need to hook at this point in the function, like it was calling this C function to save. All right, we replace that code, it's gone, we have a mapping, the temperature when it gets there, it says, okay, you're calling function X, all right, it's called our code now, the code in Pancake. It's a PS1 code. Now, we do our own stuff.

Well, I'm running on the Switch. All right, I'm going to put this on the buffer somewhere, blah, blah, blah, blah. Then when the PS1 is done saving, now I can use the Switch SDK function to actually save on what the Switch expected, from the Xbox to Xbox. Now, you got this little layer of the save for Xbox, the save for the Switch, the save for PC, for example. That's for saving. There was something else I wanted to say about this. Yeah, talk about the interpreter. You might say, well, all right, we got an interpreted simulator, you were just interpreting the CPU instruction on the MIPS computer on PS1. That can be slow, because you're always interpreting code every time.

If you have a tight loop, that's just waiting for vsync, for example, well, your emulator is just doing use as work. It's just spinning around, waiting for no reason. Some external event that doesn't really exist. We do what we could call AOT. We cannot JIT. A little big thing from different. PC emulator, you can use just in time compilation. That's not allowed on console, for security reason. I mean, the kernel does it. If you use a web browser, for example, on PS5, I believe it's based on the webkit. I believe it may have used JIT on the PS4. I think they never removed on PS5 for security. That's why many of the security issues are web-based on console.

Nevertheless, so we have to use AOT ahead of time compilation, which was if you go back in time, I believe when Flash was removed from iPhone, I guess, 10 years ago, people still weren't making Flash games. We're like, well, how do I target iPhone anymore? Adobe provided some compiler. I might be wrong, Adobo Hair. That use AOT, basically, with precompile your Flash bytecode into C code, or C++ code. Then you would actually build this thing and then run it on the iPhone. Well, it's the same thing we're doing. We're generating, we take the MIPS code from the PS1, we generate a bunch of C++ code from this. Then we run the compiler on that, bang. Then that's it.

[0:12:42] KB: Oh, that's interesting.

[0:12:43] BL: Yeah.

[0:12:45] RL: Yeah. There's nothing new there. I mean, it's been done before, but we have to do it. We have to do it efficiently, because you can't just do it. Yeah.

[0:12:53] BL: Yeah. I can add to that. One of the number one questions we get is, well, how can it be difficult to run a PS1 game, which was a very weak machine compared to the PS5, which is gargantuan? Well, you're still limited by the hardware restraints that existed by the PS1 to a certain point, right? When you're running your emulator, or Pancake as a PS1, the game's expected delays in certain parts, right? If you're reading from the CD-ROM, it expects that, okay, when I read from this sector, it's going to take X amount of milliseconds, or whatnot. Well, that doesn't exist anymore, right? We're reading from disc, which is almost instantaneous relative to the CD-ROM.

We have to account for those timing issues. Timing issues is one of the big things that we have to deal with when it comes to PS1, PSP, and PS2. That's one of the things we have to take into consideration as well, too. Like Robin said, performance is arguably the number one issue we have to worry about, especially on lower-end devices like the Switch. Ahead of time compilation, what we do when we first get a game is our QA team will play through it, and we profile it. We're using our tooling system, we're able to see what all the hot spots are, okay? By running through, say, the first level, we're probably hitting all the major functions within the game, almost all the major functions within the game. The profile will then generate files, store them in a library, and then those files will be called the next time you run the game, it's running that pre-compiled code as opposed to running it on the fly. That non-performant part of the game now runs really well. AOT solves a lot of problems when it comes to performance.

[0:14:37] KB: Got it. Are you ending up, actually, just to make sure that I understand, and once again, new to the game emulation space, personally, and trying to echo this for everyone else who doesn't have that background. You're basically taking this binary, running it through a real-time emulator that is then capturing what's functionality, and then using that to generate code so that you can actually, rather than real-time emulate, you're going to compile a new binary that will run against whatever machine you're targeting?

[0:15:06] RL: Yes. I mean, the AOT will only be treated once, because it's the same C++ code that will be portable. You have to compile it on each target. You compile that C. We create a DLL, actually, like a DLL. Well, yeah, dynamic library, the .ISO file, or whatever the format is on the target, but using Clang, for example. It just works. Some of the issue we may encounter is sometimes when you do AOT, you might end up like, you're trying to find a function, trying to find the beginning of the function in the end. It might be a very long function. Now, if you just have the C function here, and there's no interrupt when you're running this, whatever the system call were happening, or the real hardware was doing other stuff. What's interrupting you to service some IO stuff? We're not doing that.

You're just running the whole thing. Maybe we need to insert some delay there, or like, oh - Now you need to have it, the code to be re-entrant. I'll be honest, I didn't write - I'm management company now, so I don't have a background in software engineering. I'm a programmer, but all the details, I don't do that.

[0:16:10] KB: Okay. Let's talk a little bit about those timing issues then, right? If I'm understanding, let me summarize back a little bit, right? Because this older hardware had these dependencies on things like reading from the CD-ROM, it would build in a whole bunch of techniques for managing the fact that this is going to take three seconds. This is going to take however long it's going to take. If suddenly, that delay goes away, things may not work at all, or they may be really choppy, or just feel very awkward. Are those delays predictable by device, by function call? How much is this a, oh, we can map directly to based on the code that we have, the binary, and running our test file, we know where the delays need to be, versus this is an interactive, "Oh, this feels funny. I need to insert a delay here," like iterative process?

[0:17:00] RL: There's a bit of both. For example, we know from the CPU manual, how many cycles are supposed to take. You can simulate cycle count. Like, all right, it's been - you did a instruction load memory, all right, the 12-cycle. You did a add, one cycle. That's approximate, because the load memory, what if it's in the cache? What if it's not in the cache? Well, we're not really simulating the real, I don't know what it was, 64 ways association cache on the PS1. We could. But at some point, maybe it doesn't really matter. Maybe everything is in the cache.

Most game will tolerate that. But some, some might. Give you an example, maybe there's legacy bugs. Maybe the game would actually ask to read from the disc. Then immediately after, check if the read is ready, because you that, it would fail, right, because we'd do - Please read. Is it really? No. Then do some code. Now, if we do it better, and we say, "You know what? You want to read this 100 megabytes on the disk? Yeah, you got it." Well, now the code will enter a path that it never did in the biggest game, and now maybe dragons will appear. 

Maybe it will be fine, actually. Depending on how the game was coded, hit my question now. You should probably should be better really behaving like the old hardware. But if you do that, now you got long loading screen, for example. There are times where let's say, you had the loading screen, the screen is black, how about we just reduce the cycle count, or how about we just pretend the CPU is not running at 33 megahertz, we're now running at 1 gigahertz? And let's see what happens. Sometimes it works. Now people are happy. They say, "Oh, loading screen from 10 seconds, and now half a second. That's awesome." You're right. It's hard to know. Sometimes we have to patch it. Maybe AI we'll have in the future, but I haven't found it. I don't think we have found a good way to do it at scale.

[0:18:40] BL: What does happen over time is the success of an emulator is generally measured, well, how well the games run, but by the broadness of its compatibility. As you play more games, or you work on more games, you find more and more of these timing issues, or other bugs, and then some of them will fix many other games that use a similar system, or similar architecture. Over time, the more games you produce, the more robust your emulator becomes. You broaden the compatibility. You're looking for 90%-plus compatibility, so that you pop a game in your emulator and it just runs. That's the ultimate goal, and there's less work to do.

For PS2, for example, because it was significantly more powerful, there are performance issues. There are per game fixes that we have to do on a pretty regular basis with respect to performance. Sometimes with the GPU as well, too, there's graphical glitches that happen. Like Robin said, you change the timing of one thing and it will mess up something else, indirectly.

[0:19:47] RL: Going back with timing, sometimes there are legacy bug. I think there was a game on PS2 that would crash, I don't know, 1% of the time when you were doing this level. Well, it's possible that because the timing of the emulator just we change it, that now this legacy bug happens a 100% of the time. Now actually, I guess, you have the choice. Either you figure out, you fix the legacy bug itself. You're like, all right, this code was actually wrong. Let me fix it in our Lua patch. We use Lua as a scripting language for patching. Or, well, you figure out whether the timing is off animation and change it. Now we're back to that 1% occurrence. You're like, all right, now we're back to legacy state.

[0:20:25] KB: Yeah. I want to dig into that patching in a minute. First, so we've talked a lot about timing as one of the challenges when you move from a legacy system to a new system. Are there other classes of problems that tend to show up when you're moving systems like that?

[0:20:42] RL: I mean, there's from in less technical perspective, there's all the texture replacement we have to do sometimes. If you're going from PlayStation to PlayStation, that's easy. X, square, triangle, circle. If you're going for PlayStation to Switch, now you have to use A, B, X, Y, or Xbox, where A and B is reversed and whatever. Now, those textures, they might be in the game, they might be in movies, we got to replace them. How do you replace them? Do you replace them on the disk itself, like the run file? Do you replace them at runtime? What about if the game is modifying those texture dynamically, like it's slowly that X squared and doing, I don't know, some replacing the pixel there, or I think some effect there. There's all kind of challenge there.

[0:21:24] BL: Yeah. Especially with some of the older games too, there are things that are not necessarily appropriate for today's climate in certain games, or the license has run out on certain games, especially sports games, there might be an advertisement. In a tennis game, there might be an advertisement for Adidas that needs to be patched out. Then, of course, the famous example is the Red Cross. Healing packs in the old days were all almost always health - health patchable red crosses. Well, that's copyright of the Red Cross, the organization right now. We can't use that anymore. We have to patch that out with a different image every time that appears.

[0:22:03] KB: Let's maybe talk about then what that patching looks like, both from an implementation standpoint, like is that something that you do at the Syrup SDK level? How does that work? Then also, under the hood, what is actually happening? You're starting from this legacy binary, like where does the patched code end up?

[0:22:21] RL: Yeah, so most of the patch, or we have a Lua Runtime. Lua is very easy. If you have never used it, I suggest you do. I know it's less - just going on a tangent here. Python is very popular. JavaScript is very popular. For Lua, it's very easy to compile and embed in any program. It's like, I think, 5C file. It compiles very easily. That's why it's so popular with game development. If you play out with the Sims from EA, it has a Lua engine there, and many other games have Lua.

Anyway, going back, so we have Lua script file. Each game will have its own data file, and then there will be a directory with a bunch of Lua patches and implementation there. We also do our trophies and achievement in Lua. I think it's similar to patching, where the logic in Lua will run, and we'll say, well let's take a - I don't know what game. Micro Mage again. If you want to know how many boxes you have destroyed in the game, well, maybe there's a counter in the RAM that tells you how many boxes you have. It's a byte somewhere. Well, the Lua script will just have this address there and just read it. Like, all right, what's the value? If it's higher than 50, all right, then call this function in our SDK, unlock this achievement. Bang, you're done.

Those Lua file are actually, the programmer writing them. Just like usual skip file, and then they're being hooked. They're fun, simply-wise so easy to use, they're fun dynamically. At runtime we just look to the folders, find out the Lua file, load everything, and there's some entry points, then shut them, and then you just call the function there. That's for trophies. For patch, I think it's very similar. I think we just replace, like I said, I'm being a bit out of my - As far as I know, we either prevent memory from being written, or replace code, totally. Again, going back to the function I was telling you about, if the game called this function, then instead, run this code. Then in Lua, we say, well, this is the new code, blah, blah, blah, blah, blah. This is where we have fixed some incorrect, well, get technical, there's some PAL game, NTSC to PAL game.

A PAL game we're running at 50 frame per second, not 60. Well, there's no more TV running at 50 FPS now. TV's gone at 60, 120, and so on. What do you do when you convert from 50 to 60? Well, this depends on the game. Maybe the game was - and some games were incorrectly converted. They actually run slower on the 50. Can you make them run faster? Sometimes you can. Sometimes you can't. If the game is frame-based, then the whole game, I think you can just run it faster. But if the game is time-based, then depending on how it was done, it really depends. I mean, the physics would break, things like that.

[0:24:55] BL: I think one of the cool things to add just to the Lua side of things, especially when it comes to trophy implementation, is the reverse engineering aspect of it. When you're playing the game, you're trying to find out, like Robin said, where in memory a particular value is being stored. It's interesting how that's done. Do you remember, back in the Flash days, when I worked on Flash games, we used a tool called Cheat Engine. Cheat Engine allowed you to basically, find those values in memory and then change them. If you wanted a thousand gold coins, you could find that value. But the way you would find that value is you'd scan memory, you'd play the game and you'd see, okay, right now I have 52,300 points. Okay, so then you'd scan through memory for 52,300, and there'd be a thousand registries with that, or addresses with that amount. Then you would modify your score a little bit. Now, it's 60,000. Then you would scan again, and then you would whittle it down, until you eventually get down to the exact memory address, and you know exactly where it's being stored then.

Then from that, that's when you can use Lua to find out what the score is, or the number of deaths, or the number of hits, or whatever that happens to be counted. If it's counted. Sometimes things are counted, and it's very difficult to track those things. Incidentally, that is the most common thing that we get asked is trophy support. I mean, that is a number one request from a business development standpoint is everybody wants trophies, and it's huge. Super popular.

[0:26:25] RL: Yeah. I just look at the Lua script, and that's exactly it. We have a hook that we check with the address. Then the CPU will say, oh, and our temperature will be like, oh, do we need to - we made a table at first, so we know exactly when to call this. Then the Lua hook will just read this memory, write this memory, and then add some logic there. Like, oh, are you trying to, I don't know, this is the wrong font, or something, some texture was wrong, maybe can we do a optimization here? The game was doing a tight loop here. We mentioned the vsync at the beginning. Some game will just loop until vsync. I mean, on time. Well, we're just wasting our time here, because actually, we want to run faster, or something like that. Just remove that loop. If you call this, just pretend the condition is true, put back the memory there, return from Lua, and then the game begins, the game continues, and it flies.

[0:27:13] KB: That's fascinating. Yeah. Okay, so you talked a little bit about trophy support. This is one of the really interesting things here is you're not just emulating the old game as it was. You're now augmenting it based on requests from clients of different sorts. Can we talk a little bit about what the different types of augmentations you get asked for are, or what Syrup supports? Are there any that implementation-wise, or more than, here's the spot, insert the Lua code?

[0:27:42] RL: You want to go, Bill, just for the high level, and I can go to the technical detail after.

[0:27:46] BL: Sure. Yeah. I mean, the trophy support and achievements is the number one. Load time improvements are definitely another thing. Being able to speed up when the characters are talking, and it's just going so slow across the screen, and people just want to speed through it, being able to skip through that, skip through dialogue. That's a pretty popular request. The widescreen has been particularly popular. I think the most complicated one, which Robin will need to talk about, is net play, and adding that. We implemented that for Micro Mages. That was something that the original game did not support, and we supported it on PlayStation.

That comes with its own can of worms. With one player, the game is fine. Two players, it's not that bad. But then, when you get to three and four players, which is what we support, well, it gets very complicated. You have to think about all the TRCs, or technical requirement checklists that Sony will check for when they're doing their certification check before it can be published. What happens if player three disconnects their controller? What happens if someone joins mid game? What happens if someone - you have two players who are local, and two players who are network from different countries? What happens in those cases? Are the PS IDs showing? The PlayStation IDs displayed. How do you handle that? Is it a lobby? Is it a matchmaking system? There's all kinds of different things. The complexity grows very quickly. Even going from two to four players, it grows very, very quickly.

GGPO, we can talk about that a little bit, but I will say from a non-technical standpoint, GGPO is great for fighting games. That's basically what it was designed for, so that you have this rollback system, and it's for tight performance for fighting games, where performance is extremely important. We went with GGPO. In hindsight, it came with its own challenges, because we weren't using a fighting game. We're using a 2D scroller. We were working on something that's not perfectly well suited for that particular genre of game, so that made it more challenging than it maybe needed to be. Yeah, I think I hand off to Robin here to talk about GGPO, because he was involved in the last push to get it fixed.

[0:30:08] RL: I guess, one of the issues with GGPO is it's synchronized everything. Fighting game is easy in the sense that you have two-player fighting. If you replicate each people's input, and you do prediction. I don't know if the audience know about GGPO, but basically, you try to predict. If you're pressing the right button on your D-pad and frame X, it's likely that you can still be pressing it on frame X plus one, right? I think that assumption, you can, if you're a client, doesn't know yet if the other person has pressed the D-pad right, but you know he has pressed a frame ago, you can probably simulate that he has pressed it again, and you just go run the game with that.

At some point, you reset the packet of the network. It will tell you, hey, is a D-pad state of that player? If it matches your prediction, all good. So, you were able, actually, to save time and do it. If you got it wrong, then now the state of the game is wrong, because actually, you submitted that you were pressing right, actually the guy pressed left. What do you have to do? Well, you can't send and say, deterministic game. Now, you got to go back in time to the previous frame, re-simulate as if it was pressing left now, and then figure out the new frame that you just did.

Depending on how laggy you are on GGPO, you might see little glitches, things going weird. The fighting game is, I guess, it's okay, because, punch, oh, yeah. I guess, something happens. Obviously, if the lag is low and the latency is very low, it happens less often, but it will happen every time you press a button. The GGPO doesn't know about the game. It's not game aware. If you're about to jump, every time you're about to jump, the prediction will be wrong by a frame, or two. You will see yourself a little bit glitching on the jump. Things like that. But it works. At the end, the system is deterministic. It will always stabilize itself. It may get some weird issues there.

[0:31:57] KB: Just to make sure I understand. GGPO, which we're talking about here, it's a networking stack of some sort that allows you to synchronize game state across distributed systems.

[0:32:09] RL: Correct.

[0:32:10] KB: It's a predictive one, so it's saying like, in a lot of cases, we can optimistically assume that these continuous things, and that'll get us most of the way there.

[0:32:19] RL: Yeah, correct. It's not game aware. It's not even rendering engines. It doesn't have networking stack. I mean, it has a little bit of the UDP stuff. But I mean, it's just a, here's the state, here is a state I think you have, here is a state I think I have, here is the combined state. You got it right. You got it wrong. This is is how you fix it, and so on. Then the game is responsible for rewinding the state and going back. It's worked well for the NES game, but it forced you to run multiple frame in a single frame.

You say you got it wrong by five frame. Let's say, actually, you simulate the things for five frame and figure out, "Oh, geez. I was wrong five frames ago." Now I need to go back in time five frame, first, I need to save that state. Then, I need to reapply those five input there. I need to run those five-frame within one frame of the game. If that's running those five-frame, it takes longer than the time you allowed, you just start lagging behind forever. Never catch up and things will get really bad. Performance become important. You'd be able to run the original game five times the speed of the -

What about sound and vibration? I just talk about the game state, like RAM and things. What about controller was shaking? The enemy hit me hit me, and because of that, we started vibrating the controller that you're holding. That's cool. Turns out, I was wrong. The enemy did not hit me. I go back in time, it did not hit me. All right, cool. Now we're not vibrating anymore, but the game doesn't know about that. It's not going to send a stop vibration, because it never sent a start vibration. Now we need to have custom code to stop that. Anything that's programming that pure function, anything that's not pure, that's because it's leaking out will break. It could be, I mentioned vibration, could be sound effects, if they're on that way. A little thing like that. I mean, it's not too bad. Worst case, it just vibrated a little bit and it stops. Maybe you taught you got hit, but you did not. Okay. Silly thing like that.

[0:34:12] KB: I mean, I think this, this adding multiplayer support to me is fascinating, because you're adding it to a game that was not multiplayer aware. There's whole classes of problems. You're highlighting of rewind, restart and synchronization, that it just has no concept of. What is the span of that? That feels like a massive endeavor.

[0:34:35] RL: I mean, going back to the fighting game, I mean, the title screen, you won't synchronize it. Where do you do matchmaking? Well, how do you start the game with two players? We have a safe state, actually. We did it with we put the game - We ran the game with two player locally, or four players. We say, okay, this is the beginning of the level. Now, we made a safe state there. Memory snapshot of that. Put that in the file somewhere. Now, we do the matchmaking on PlayStation 4 or 5, find the players, blah, blah, blah. Okay, cool. Start the game. Everybody agree? Now, all the players will load the same safe state together at the same time. Now we start GGPO there. Then everybody's synchronized.

there's a bit of, again, see, GGPO doesn't know about this. It's just being told like, by the way, you're now synchronized. But we need to know, because the original game did not have matchmaking. The game did not have, find the game and find the players and what game you want. Now, we toy around, how do we add? Can we add easily networking to any games in the past that were co-op game? I mean, it depends on what the game was doing.

[0:35:44] KB: Yeah, okay.

[0:35:45] RL: Where do you put that?

[0:35:47] KB: Right. It is games that were multiplayer, but multiplayer locally.

[0:35:53] RL: Correct.

[0:35:54] KB: Okay. That does narrow the scope a little bit, because I was imagining you're taking a single player game and adding a multiplayer mode. I'm like, whoa, what is that?

[0:36:01] RL: That would be harder. No, you need to have support for multiple controller there. Then there's another set of challenge. What about games that say, that were multiplayer, but that you want to add multiplayer? We haven't done those yet.

[0:36:15] BL: Yeah, because they may have the wrong protocols. But we do rely on, we have to use PlayStation's SDK and Xboxes and Switches and we call into that, like Robin said, we call into that their matchmaking system. They have their own PlayStation IDs. You have to subscribe to get all that stuff, so you can have online play. Then it's sent to GGPO.

[0:36:34] KB: Got it.

[0:36:35] BL: You go from that.

[0:36:35] KB: That's still super cool. Okay, Bill, you were going to mention some other domains.

[0:36:40] BL: Requests. Yes. Okay. Robin mentioned save states, and this reminded me. How could I forget? One of the big features that our emulator has is rewind system. This is pretty common in a lot of emulators as well, too. As we have all these save states, the player, or the user is able to use a rewind system and go back to a save state that they want to. If they're trying to get through a boss fight and they can't do it and they don't want to keep dying, they can just rewind 20 seconds, or whatever further back they want to go. Replay that over and over again, which really helps with QA when we're doing our trophy testing.

When we do trophy testing, what we'll do is we'll get to a particular spot and then create a save state. Then we can reload that and start from that area, so you don't have to play through the entire game to unlock a trophy. It's like, beat the final boss. Well, no, you just have to get the save state from that particular point. Rewind is popular. Up-rendering is another thing that is requested to improve the graphical quality of the game. Sometimes post-processing effects. Let's say, you want to add a CRT filter, or an arcade filter to make the game look even more retro, adding scan lines. We can do that. Custom audio-visual, contextual button remapping. If you want to be able to remap. That actually, controls is a pretty big issue that Robin could talk about more, especially when it comes to PSP, because PSP had a different controller than a PlayStation 5 controller, which is very different from a Switch controller. There's that as well, too. Of course, adding rumble, like games didn't have rumble in the past and now they do, so we're able to -

[0:38:14] RL: I want to add something about aspect ratio. We're doing one game. It's a PS1 game. PS1 game, if you remember, right, they were four by three, all right. TV now are 16 by 9 nine. All right, so you're going to get a little better box. The game was also widescreen four by three. The game itself was rendering as if it was a widescreen game in a four by three. Now, if you convert that without thinking about it, you're going to get a black box all around, like [sounds inaudible 0:38:37], like a stamp. That actually looks terrible. Now, we can actually zoom it. We made some change to zoom the game. Now, we're back to 16 by 9. Except, one problem. The game will display sometimes its HUD under that black bar at the bottom. That would be hart to the screen.

Well, Lua comes to the rescue. We can use the Lua hook to just move that back up. It seems to work. It's like, we haven't finished the game yet, but now we got a PS1 4:3 game running 16:9 as if it was 16:9 originally, because they were learning 16:9 in the 4:3, converted back to 16:9. Anyway, it's per game stuff again.

[0:39:13] KB: Yeah, that's wild. On that note, are there particular types of games that tend to be more challenging to emulate, or port?

[0:39:24] RL: Yes. I mean, it really depends on - Some games or more performant-heavy. We have one game we're working on, and we're able to do enough optimization. It runs at 60 FPS now. We're like, all right. It's more tolerant on the code. It's using the MDECK. MDECK is the media decoder of the PS1. Then, the way it's using it, we're able to cheat. It's asking for, I think, I think, I don't know, a buffer, and we return it immediately. Then, the game is happy about that. Then, just ping the movie and everything is fine. Some other game we're trying, we did that and bang, the game doesn't like it at all. The game expect, probably that timing delay I was mentioning again. Now, we actually need to emulate the ring buffer of the MDECK properly, in which we did not do it yet. We're like, "Oh, geez."

We probably have to do this anyway, because I expect more games to require this anyway, but we're just looking at the first game. I think it's a bit like, you do one thing, you know a little bit about it. You do have them. If you do three, three prime number. You have three, five, seven, 11, at some point, you probably get that bigger coverage of everything that the game required.

[0:40:26] KB: That makes sense. Can you talk at all - We've talked about a bunch of systems, Switch and things, and you alluded, oh, PS3 might be one of the upcoming opportunities. Can you talk a bit about that and your plans there?

[0:40:40] RL: Good question. PS3 is a big platform. I mean, the way I see it, we got PS2. PS2 is running right now truly on PS5 and PS4. Maybe PS3 will be on PS6. Who knows? One of the big challenges I saw with PS3 is the memory. I mean, there's some advantage and disadvantage. Going back, PS1 and PS2, they were not modern rendering pipeline. They were like Lego blocks. I don't know. PS2 has its CPU, as a VU zero unit, V1, and then there's also their Reg E. There's no graphic pipeline on the PS2.

On PS3, you're starting to get close to an actual normal graphical pipeline. You get now as SPUs. There's eight CPUs on that machine. That doesn't really translate well to a modern hardware either. The other big challenge I see is that, I think, it has 512 megabyte of RAM. We're just going to save the memory of that every frame, we're going to be saving a few gigabytes of every seconds. That won't scale. Thing is, that most game don't change all the memory. They just change a few bytes here and there. I mean, that's a few bytes for them. Maybe we can use paging technique. Like, oh, those pages changes, let's swap them to memory. Now, you have almost a DPCM compression between the frames. Like a video decoder, in order to see frame X, you need to have the previous frame and then you apply the delta on that.

You can't do lossy compression like on video. If you do lossy compression on bits and bytes, you might get some weird effect in the game. I think this is a big challenge there. I think it's going to be easier to system software. The game where I am saving here. I am asking for networking. I am changing the resolution. On the old game, it was like, there's no HDMI. It's like, oh, yeah, I think this game is running in this resolution, because it sends some few packets and register here and the documentation is not clear.

I think the system calls actually make it easier. The separation between the OS and the user runtime makes it easier to inject yourself like, okay, the game is saving here. Let me save on my Xbox. We didn't talk about this media size. It's just a lot of data. We use Google Drive, use GitHub, but the PS3 games we're talking about, the PS2 games, they were a DVD size sometimes, and they're multiple regions sometimes. By multiple regions, I mean there's a European version, a Japanese version, an American version. You need to support all those versions. If you want to run the French version of a game, then the data for the French language was only available in that French ISO file.

If it's a PS1 game, for example, 600 megabytes times three, four, five, all right, a few gigabytes. It's a PS2 game. Now, we get the same thing, but at the 4 gigabyte level times something. At PS3 games, I don't know. Blu-ray. I haven't checked. Maybe a few 20 gigabytes sometimes. It looks easy, but we're developers. You need a hard drive, SSD, and so on. You're, "Oh, let's me test this game. All right, I need 20 gigabytes here, 20 gigabytes there." It adds up. Then you're testing with QA. Let's make package. All right, I'm going to make a package of this, now it's 100 gigabyte, put this on Google Drive, somebody has to download it. You need to have the gigabyte, give it for second connection. We do have that, but it puts a strain on the overall, now we're going on build system and Jenkins and GitHub. They need to swallow this data.

You can't just say, "Oh, yeah. Just mount the file system and copy it every time." People like to listen to the cloud. Use Docker and just spawn the OS, spawn this thing. Well, it's going to take a few 20 minutes every time and it can cost you a lot. Don't do that. So, we don't do that. We went and almost anti-cloud. We have custom physical build machine in the basement in people's room and then they're connected to GitHub. Then, those machines, I mean, and they cost a lot, and then they have to maintain them. But guess what? You can put whatever hard drive you want on do those things. There's no Docker and it's actually pretty fast.

[0:44:42] KB: Yeah, it is interesting, the scope of data that you end up talking about here when you're having to send this whole image over, and every time you want to test a change, you need to update things and all that. Yeah, that's fascinating. It would push you towards, how do I shrink the distance between my data and my processing machine?

[0:45:00] RL: The distance between the data, because we were talking about physically, we need development kits. You need a Switch, [inaudible 0:45:05] you need a Xbox, we won't get to work on this thing. Might say, well, cool. I'm going to put it in the data center somewhere and that works. Then connect remotely to it. That works, too. What if the data is on your PC and you're constantly changing it? A few bytes on that. Remember that patching we're telling you about the RAM, replace it with that X texture. Okay, cool. You do it in the 20-gigabyte file.

Now, that data center that you connected to needs to read that file, or maybe it needs to - Hopefully, you only need to read a few bytes of it. But if your tool is dumb, it's probably trying to open the whole file and read the whole thing, and that's pushing 20 gigabytes every time. Now you need to optimize this thing. It's just the things you don't think about, because when you're working locally, data is just there. It's actually a challenge. I think that challenge efficiently put us ahead.

[0:45:49] BL: One thing we could talk about is the auto testing system, because that's something really cool that we have to, when I was mentioning earlier about the compatibility and the broadness of compatibility, so how do we check if there's been a regression in the games that we've done? We have an auto-testing system that checks hundreds of games every night. It runs and it checks all the different games that we've produced against all the different consoles that we support. It can detect images. It can detect code changes, if it'll boot. That type of thing. Rob, I mean, you work on this a lot.

[0:46:25] RL: Yes. It's basically based on GitHub actions, could be based on anything, but it's based on lovely YAML file. No opinion here. We have runners connected to GitHub, and then we have like, I mentioned hardware, Switch hardware, Xbox hardware, PlayStation hardware, and they would just run test scenarios. Basically, most of the test scenarios are pretty simple. Like, run this game, take a screenshot at frame 20, or take a screenshot after five seconds. Now, we have reference screenshot. I mentioned, like the NES game are deterministic. We have all those reference image, and we have the image we just took, we compare them. If they're the same, well, cool, thumbs up, and Slack is happy. If they're not, then we actually create a HTML report, and it's on the JS file, you can click on it, you can see the expected image, the new image, and the difference, we just do at del tab, then you can see like, oh, something messed up.

Sometimes it works, and it doesn't work. It works, because it allows us to have good reliability. If you run 100 games, because you don't want the engineers to test 100 games when they make a change, right? It's going to waste their time. But sometimes there's false bulletin, right? They will be like, "Oh, this game is just like, I don't know, the emulator is a bit wrong there, or maybe the dev kit is down." It's actually physical machine, like I mentioned. By the way, the cloud is also made of physical machine, which we don't see them, but somebody's taking care of them for you. It's just physical machine at the end of the day.

The networking is also something to manage. First, we need networking to be easy, so we use Tailscale, which is based on WireGuard. That's cool, except we're not networking expert, and we also need to be careful about, this is exposed to the public Internet, so we don't want people to just join in and start hacking our kits and code and stuff. Yeah, the auto test runs. It runs on every commits we do. We try to make them fast. If you see there's a constraint on like, if you're on the Switch test every commit, and you only have three kits on Switch, and you need to add more kits. Other tests we're looking at is not just screenshot tests, but just scenario tests. There's a rewind system. Make a snapshot, and reload it, confirm that it's the same value.

How about the logs? The game start, and we expect a certain log, or search never be sent. Well, check for that. We have lots of various stick. I wouldn't say, people like to write pure code. I believe more in heuristics. If you do enough heuristics, because at the end of the day, humans will be interpreting those results. I mean, and then reading them. Oh, it fails because, okay, that's why. It's good to have it 100%, but I don't think it's very achievable. I'm more the case that heuristics a bit.

[0:48:56] KB: When you are writing these tests, are you able to essentially record play sequences as well, where it's like, oh, the controller is behaving in this way, run it through these different pieces and then see?

[0:49:07] RL: I believe, we have done that for a few there. Yeah. For example, going in that menu, going, make a snapshot, saving it and reloading it was where recorded. We're trying to get QA to do a bit more of those recording, either in Lua. Do we record high-level, or do we record input replication, almost like GGPO, but the file? We haven't did some OCR thing. It might sound funny, but on the PlayStation dev kit reboot of a PlayStation 5 you have, there's a screen that shows you that tells you like, "Hey, you shouldn't have turned up the power." Yeah, the hard power reset, it tells you like, "This is what's wrong. Press X." There's a way to bypass the screen. Before we knew about it. We use a AWS OCR recognition, just recognize the screen there. Oh, it tells you to press X, and then we just send input to press X.

We also had to, I just mentioned power cycle. We have no outlet, or smart outlets. The console tried to reboot it, tried to reconnect to it. If nothing works, what about, well, turn off the power and turn on the power? Now, we do that remotely as well. It's scary that to know that GitHub action is like playing with electricity. But it works. Then going back to AWS, when you go on Amazon and you reboot your instance, or the instance goes into some weird state, it's probably doing that, too. It might be sending power and just like, oh, this machine is just dead. We restart it, you just don't see it, but it's doing that.

[0:50:24] KB: One last topic we haven't talked about that is more on the business side of this, which is like, who pays for this type of emulation work? Who are you selling to? Is this end-user sales, versus you're pitching publishers? What does that business model look like for game emulation?

[0:50:41] BL: It's mainly business to business. But there's two ways we're looking at it right now. We have the work for hire branch of the company. We work with a number of publishers. For example, we're working with Xseed Marvelous on a game that's going to be announced soon. We're working with Limited Run Games on Pure Effect, and a few others that we can't talk about. We're doing their PS1 ports for them. They're paying us to do the emulation work and the trophy design work and trophy implementation and all the TRC checks, or compliance testing that's required.

What we also found is that we would get requests from smaller studios. Really small studios. They've sold 300 copies of their game, a NES game, for example, but they would love to have it on Switch. They only have it on PC right now, on say, h.io. Well, it's not really cost effective for us to be able to port that game for them. We get a lot of these requests, but it's not economically viable, as they say. What we've decided is to go the SDK route. The Syrup SDK is something that we're working on right now. It's in the office stage. We're going to give that power to the developers and to the smaller studios, so that they can put their games through our SDK and then launch, do the port themselves using our tech. That's in alpha and we have a company working with us right now. He's going to be launching nine games on the Switch really soon. They're all NES games. Within three months, we'll have our first games published through the SDK.

Typically, we are working with publishers, or studios to get the games done for them. Sometimes we will pitch the games ourselves. If we think that a game has the potential to do really well, we'll pitch the publisher and say, "Look, we think this game is awesome." It did really well in the past. It was, for example, only released in Japan. We think it should be released to North American audience. It has a current demand from the community. It's not in licensing hell, because that is really the number one blocker for all these games is licensing. Who owns the IP? Who owns all the licensing within the tech, so within the game itself? Who did the VO? Who did the music tracks? Who did the, all the different things, the artwork? Everybody owns little bits and pieces. Sometimes even the studios don't know if they own the IP or not.

We'll pitch it to them like, "Yeah, that's a great game to work on. Let's just verify the IP that we actually own it." Then it come back a few and say, "Oh, we actually don't own that. 5% of it is owned by the estate of a long-lost person in Japan. We're never going to get that permission." We look at a number of factors. We obviously look at the financial opportunity. We want to make sure that the game stands the test of time and that still works really well in today's market. Like, pixel art is obviously really popular right now and has been for a number of years. That works really well from a lot of NES games and SNES games and any of the 16-bit era games that work really well.

Those are some of the factors that we look into when we're working on things. In that case, if we're pitching a game, usually it's going to be a revenue share agreement. We're going to take a cut of the sales, the publisher is going to be taking the cut, and we're taking on the risk of the development of the game, because we believe so firmly in it.

The vast majority are work for hire. We're doing work. We reach out to a publisher, sometimes calls and say, "Hey we're good at this. Do you need it?" Then, they'll be happy to work with us. Other times, they'll come to us and say, "Hey, I heard you guys really know PS1 games, or NES games. Can you help us with these ones?" Then, we work on an agreement and deliver them.

[0:54:34] RL: Yeah. Our goal is really to push all those games that people think, Call of Duty. Okay, we cannot do Call of Duty. Plus, someone is doing that. There's a lot of low-hanging fruits game. They didn't sell heavy on copies, but they didn't sell 100 copies either. They sold maybe the 10,000 copies, or 50,000 copies. Nobody's doing those games. Those games are just lost right now. The only way to bring those games is to piracy. Then we're like, what if we could scale this to make it? What Bill was mentioning, we can talk with the publisher. They would be like, cool guys, we actually found the owner, we can do it, but only 50,000 copies. It's going to cost us more than legal agreement and stuff marketing. We're not doing it.

What if there was a way, like YouTube, Spotify, Airbnb, SDK way to scale this in such a way that people, either the IP owner, or developer could use this technology and then self-port the games and then everybody gets a cut. People makes money out of it, because people enjoy it. I guess, Netflix may be the example. I'm not saying we're going publisher route and going there, but it's just the vision is there where I believe, there's a big loss of retro games, and there's two markets for those games. There's the market of people like Bill and I, we would play those games in the past and want to play them again, because nostalgia, but also the new market. There's newer kids that didn't have - they're 20-years-old, 30-years-old, they never play those games, actually. Some of those games are actually good. They're still good. Some are crap, by the way. But some are good. They would pay for it if they were given the chance.

On the Switch like, "Oh, $5.99. I'm going to pay this." But they can't right now. The only way they can play those games is find them randomly on the Internet and download them and nobody's making a buck out of this. I think there's thousands of those games.

[0:56:20] BL: Yeah. Especially when you look into other regions. We're working on a game that was released only in Japan. You ask about features. Localization, of course, is a feature that we provide as well, too, both the audio and visual side of things. That's a very popular request as well, too. Anyways, we're working on a Japanese game that did really well in Japan, but was never released in Europe, or North America. We want to bring that. There are tons of those games that were local to the Asian market and never came across to our markets. There's a real opportunity there, because there's gems. There's hidden gems out there. We want to make them available to as many people as possible.

[0:57:04] KB: Awesome. We're right about at the end of our time. Is there anything that we haven't talked about that you think would be important to leave folks with?

[0:57:12] BL: Well, I have one thing. I think one of our things we're really proud of is our community. Our community works. They're incredible, what they do. Our head of community does an outstanding job managing that. You can find us at implicitconversions.com and we're on all the social medias. Really on Discord, we have a really active and engaged community, who really are passionate about retro games. We get a lot of ideas from them. They submit bug reports for us. They ask for features. They rank games that they want, that they're interested in. Like, Legend of Legaia, for example, is a game that has come up numerous times. Simpsons Hit and Run, that keeps coming up. They want us to do these games. That gives us some valuable information that we can then use to pitch games. Or, if we're in discussions with some of those third-party publishers, we can share some of that information. It's an outstanding community. If people want to join that, please do.

[0:58:10] RL: Didn't we mention about other up-rendering would work? For example, PS1 game ability, we're 320 by 240. If you just take this frame buffer and put it in the HD TV, well, cool, you got HD graphics, but still, the HD graphics of the old game there. If you remap the render call, like no, draw this triangle, draw this mesh, to actually a PS4, or a Switch, Open GL. Switch is OpenGL call, then you can use a different frame buffer size, different target, and then render the thing higher resolution. Now, you get those little hedge that would seem like thick pixels. They're going to look very tiny, like10. The texture would be wrong. The texture that was used, it was still be the low-resolution texture. Now, you get nice mesh with low-resolution texture, but surely, it's a bit better there.

They may also get some scenes issue. The game was rendering the floor and maybe the wall there. Then, now it's high resolution. Maybe there was, again, legacy bugs there. Now, the legacy bug gets a floating-point error and now you see through it. These are the some of the features, up-rendering feature that we want to do more. Sometimes we have to remove stuff. Games we're using blur back in the 90s, 2000, a lot of games use blur. It was cool. You're going fast in the car, use blur, blur, blur, blur, everywhere. That looks bad on modern hardware. You actually want to reduce the blur, because it's actually just remove the resolution, or just remove it all together, so going back to the UI patches there.

I think there's a lot of work, a lot of interesting work, actually. People interested in gaming and solving problems. Something else to add. I'm always reading. People sending us email, or CV, or resume, the target audience - Some people ask us like, "How do I get the job here?" Well, yeah, video game experience, console programming, but a lot of passion that of like, "I work on the simulator. I like to toy around in NES. I do hardware. I do FPGA." I've been like, I do like some version of NTSC, too. Whatever you do, if you're passionate about it, if you have the skill set.

[1:00:18] BL: That actually works really well at this company.

[1:00:20] KB: Awesome.

[END]