EPISODE 1933

[INTRODUCTION]

[0:00:00] ANNOUNCER: Autonomous drone delivery has long been the stuff of science fiction, but ongoing advances have moved the space from experimental to operational. Zipline is one of the leading companies in the space with drones that change between missions and fly autonomously to deliver packages directly to customers. Kyle Madonia is the VP of Application Software and IT at Zipline, and she previously spent a decade as an engineer at SpaceX. 

In this episode, Kyle joins Gregor Vand to discuss how Zipline software stack powers end-to-end autonomous delivery, the engineering challenges of managing drone fleets at scale, and how the team approaches software releases for safety critical systems. Gregor Vand is a security-focused technologist, having previously been a CTO across cyber security, cyber insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at vand.hk or on LinkedIn.

[INTERVIEW]

[0:01:14] GV: Hello, and welcome to Software Engineering Daily. My guest today is Kyle Madonia. 

[0:01:20] KM: Hi, it's great to be here. Thanks for having me. 

[0:01:23] GV: Yeah, awesome to have you here today, Kyle. We're going to be talking about all things autonomous flying, which I don't think we've ever had an episode yet on this on Software Engineering Daily. You are at the company Zipline, which we're going to get into in a second, as we like to do on Software Engineering Daily. I believe currently you're the VP of Application Software and IT at Zipline. You do have quite an interesting career path to getting to Zipline as well. I think that would be interesting just to hear about and sort of how it influenced, I guess, you then joining Zipline. 

[0:01:56] KM: Yeah, my background is all computer science. I studied computer science in undergrad, grad school. Started working as a software engineer in a few different companies, and ended up joining SpaceX in 2013. It was a company that nobody had really heard of at that point. I hadn't really heard of it, and I thought, "Oh, it's aerospace. It must be boring and slow. Why would I want to go there?" And then I visited the factory, and I was clearly wrong. And thought, "Well, this was a mission that I could really get behind." And I spent the next 10 years at SpaceX growing into leadership roles but really focused around building out the internal systems that we use there. 

I built out WarpDrive, which is the enterprise resource planning, ERP system, that SpaceX used that Tesla also ended up using. I built out a lot of the systems there. I built out the network operating center for Starlink, how we communicated with satellites, managed the fleet, the telemetry, a system that we had for that as well. Even worked on all kinds of random internal systems. For example, in Starbase, we built out a leasing system for how we handled leasing properties to employees who were now moving to Starbase. It was just anything that SpaceX needed, we were building. 

And after about 10 years there, I started thinking, I wonder if now it's time to see what else is out there. And SpaceX had grown so much and was so successful in so many ways that I wanted to see what else existed. That was really exciting. And it have to be around this time that Zipline contacted me. And I started looking at what Zipline was doing, and seeing the mission that Zipline had, where, for the past around 8-ish years, Zipline was serving medical products in Africa and delivering blood supplies, vaccines, nutritional supplies, all of these to really help save lives there and to help improve the health system in Africa. 

I felt like that was already such an important mission. But now, Zipline was at this inflection point of launching in the US and saying we now want to bring some of this to the urban areas. And really our goal is to be able to have a logistic systems that serves all people equally. And I knew leaving SpaceX, it had to be a mission that I really believed in because that's important to me. But I also wanted to be at this place where we weren't sure would it be successful. Because I think that's always such an interesting place to be, where it's not when are we going to be successful, but it's if we're going to be. 

It's the same kind of thing in 2013 when I joined SpaceX. You didn't really know if we were going to be able to achieve this. And I saw that the areas that Zipline really was growing in was something that I could help in. So I decided to make the jump and join the company. And that was about 2 years ago. And it's been a fun ride here now as well. 

[0:04:34] GV: Yeah, that's amazing. And I think it's very interesting how one huge mission effectively was taking humans off the planet, and how can we put them in a different place. And then this mission is all about humans on the planet quite frankly. And I really like that. Yeah. 

[0:04:50] KM: Yeah. That's actually what I was saying when I started just putting out some feelers was that I wasn't actually sure what mission was after, because how do you say changing your mission from making humankind multiplanetary. What is the next mission? I just want to do something here on Earth. I've spent 10 years - how do we go and explore and be able to get off Earth. But now I want to say, "How do we make things better for people on Earth?" And Zipline just had the perfect combination where we were trying to make things better for people on Earth, but using the skies. And I thought that was so key to being able to make such a huge impact. 

[0:05:22] GV: Definitely. We're going to get into the end-to-end experience of what Zipline is shortly, but just more of like a super high level. You have touched on it. Obviously, you've mentioned deliveries, for example. But what is Zipline just to get us understanding that one? 

[0:05:37] KM: Yeah, Zipline is a company that does deliveries with autonomous drones. We build and manufacture all of our drones here in the US. And then we deliver a variety of products. And the important thing for us is that we're able to deliver extremely fast and safely to people for whatever reasons you need it for. 

Part of our mission is always in the healthcare space. That's really our roots in launching in Africa and internationally with what we've done with our longer-range drone that we have, and now with what we call our Platform 2, but it's more of our high precision drone that we have now launched in the US. We're delivering retail products, food from restaurants, prescriptions. We still want to stay very close to the healthcare space. We're going to have more healthcare partners coming online. And really, again, it's about just delivering to people whatever they need. 

When we talk about why I joined, I think when I was talking to Zipline, I have two young kids. And my kids, there was one night where my husband was out of town and both my kids were sick. And I'm like, "I just need Tylenol." It had to be specifically Tylenol. If you're a parent, you know. And I didn't know how to get it to the house because I needed it fast. And everything I was looking at was going to take so long. 

When you have two screaming children, you know, you cannot take a long time. It just happened to be the next week when Zipline contacted me. And I thought like that would have been my solution. And I think that's what we're seeing as we launch. It's like there are parents who are ordering formula for their babies, for medications for their kids, the elderly who have a harder time leaving. That's where Ziplines is trying to solve these problems of just delivering quickly for whoever needs it. 

[0:07:13] GV: Yeah. And I imagine you've talked how the core marker at the moment, I believe, is in Rwanda and Ghana, for example. And what you've just mentioned, that's amplified so much in these countries, these regions. There is no, let's just take a name, Uber delivery for X. There's none of that, right? And so this is the only version that's not like a car having to trapes across a ton of landscape and that kind of thing. 

[0:07:36] KM: Yeah. In that model, we have centralized fulfillment. We do all of the storage of the blood, of other medical supplies. Even in those countries, they don't need to think about how do you store that, and how do you make sure that you have quality products there. But the difference between flying in some place like that is you can get somebody life-saving supplies in minutes, where a car would take hours just in terms of how you would have to drive there. 

[0:07:58] GV: Yeah, exactly. I think let's start to look at actually what happens. Set the scene more for when a user is actually using the applications that mean delivery comes in. Just to be clear, what we're going to be touching on today, and given that we're software engineering daily, we're not going to be touching quite so much on say the hardware piece or the exact avionics, this kind of thing. We are going to be talking all about what is the software used by especially the end user and for Zipline internally to make this possible at all. Because there's just so many bits to this that to me is super interesting, because the failure modes here are so different, I think, compared to many other software companies. Especially the outcomes of what could happen if something fails is quite different to just a server going down. We're talking like a whole plane, autonomous plane can go down. Let's look at the end-to-end experience. Walk us through. Where does this even start? Who is the customer, the user. Where does it start? And how does it all work? 

[0:08:58] KM: Yeah, I think if we're focusing on really where we're going with this Platform 2 that we've built out and the more high precision drone, the way how this works now, we built out a marketplace. And so from a customer's point of view, it's super simple. You download the Zipline app, you open it, you put in your address, you get to see what partners can deliver to you. That could be Walmart, if you live near one of those. It could be as different restaurants that we're serving. Go in, you figure out what you want to order, you place your order, and that is it. We want the customer experience to being extremely simple and familiar. People know how to do that. 

From there, we get the order into our system. And so, we have a set of delivery services that is then ingesting those orders through - we use Kafka as our QA mechanism, right? These orders come in, we look at it and figure out how we translate these orders into missions. Because on our side, what's unique for something like Zipline is we have the fleet of drones, right? From that fleet, we need to figure out what is the ETA that we give back to the customer. So they can see, "Oh, I'm going to get my medicine in 20 minutes." And just calculating that ETA is its own complexity. 

But we look at what other missions do we have and orders coming in. What are the drones battery levels? How far is this going to make sure we know how far to fly? All of those services are core to this whole experience, but it's all invisible to the customer. They don't see any of that. 

From there, if it is a restaurant, as an example, that we integrate with, we'll send the order directly to their point of sales system. So, it's very familiar to the restaurant. Also, one of our goals in the operation side is to keep it as simple as we can for the restaurants, too. They'll get the order, they'll prep it. Right now, we have Zipline employees will go pick up that order, bring it over to our drone, put in the drone. They have an app that they use to be able to tell the drone which order is in there, so the drone knows where to fly and deliver it to. And all of that is fully orchestrated in the back end. 

We are also now rolling out what we call our zipping points. And you can think of a zipping point as looking like a mailbox. And so the zipping point actually allows our partners to put the orders in themselves. So you don't need to have a zipline employee come and pick it up, put it in. They'll go to the zipping point, punch in a pin code, put in the order into that zipping point, close the drawer, and walk away. And that's all they're doing. Again, on the back end. That then communicates with the cloud. We know this order came in. We figure out which zip can come pick it up. We send the zip. Zip goes, picks up the orders, delivers it to the customer. And that really helps also get the time down and just how fast we can get these orders from being prepped into the customer's hands. 

[0:11:26] GV: Wow. A couple of follow-ups here. When we do talk about autonomous and fully autonomous, literally, how true is that? In the sense, is this autonomous flight vehicle, everything is software controlled. And this thing just gets its mission I think is the terminology you just used. And from that point to the delivery of the good and then, I guess, the return of the vehicle back to its base, I guess, that's all purely software-driven, I guess? 

[0:11:55] KM: Fully software-driven and autonomous. Yeah, there are no humans hiding behind a curtain with a joystick trying to control where the drone is going. That is all the drone. The drone knows. It has its sensors to be able to navigate to be able to deconflict with other aircraft, to know if there are any health checks that it is doing and any results of those. So in all of that fully autonomous, we have people who are more managing the fleet overall and just ensuring that the overall fleet is healthy, in a good state. There's a few different commands they can take, but the 99% of this, just the drones know what they need to do, and they're able to execute that themselves. 

[0:12:32] GV: Yeah. Okay. And then you mentioned the P2 platform. And I believe this sort of differs either from the previous platform or just a general image that comes to mind when we think about drones, that there's like this launch and catch system. Could you maybe just speak to that? 

[0:12:48] KM: Yeah, if you look at videos of our Platform 1 that is really operating internationally right now, you'll see that kind of launch and catch system. And it's really interesting when you watch how the Platform 1 drone works. Every flight actually it gets disassembled into four different pieces and then reassembled for the next flight. You can swap out a different battery, put in another battery. And you have humans involved in doing that. And that model worked super well for our use cases that we have there. And we're still able to do thousands of deliveries a day on that platform. 

But when we looked at Platform 2 and what we needed it to be capable of doing, we're not talking about, sure, thousands of deliveries a day, which we're already doing, but we need to now get this to be millions and hundreds of thousands. We want this fleet to be capable of million deliveries a day and more, and to do this all really at scale, extremely quickly and safely, and with that high precision. 

And so what we wanted to be able to do for our customers was just very different than what our current system was doing. And so with that, it meant we needed a different design of the drone. We needed to think about it from the beginning as being a very autonomous, safe system that can charge itself, that is fully contained in that way, and can operate in much more urban areas. It can deliver to that high precision area. We can deliver to people who have very little space in their yard. 

And we took a lot of lessons that we learned as we did as a company with Platform 1 and designed the drone. And we've flown millions of miles on there that we could learn from the design decisions there. And there are still essences of that in Platform 2. But ultimately, the use case we wanted to be able to do for our customers was just so different. 

[0:14:24] GV: I think you were saying, launch and catch was P1. Is that right? And then what is, I guess, the mechanism change then for P2? 

[0:14:31] KM: Yeah. For P2, what we have are - we call them are towers and docks. Very simple. Because it looks like a tower with some docks on there. And so the zip is sitting in the dock. And that dock is a charging station. It'll charge the zip as long as it is sitting there. When the zip takes off, it does a little maneuver away from the dock, goes picks up the package and delivers it to the customer. And then when it comes back, it just simply goes and redocks and charges. 

And so our dock structures, we're actually just rolling out now with our 12 dock structures. And our idea with this is we're actually going to really have them be behind the scenes. They're going to be charging infrastructure that people don't really ever see. They're more in the back areas where there's not foot traffic. You're not around those. And all you know is that the zip just came, picked up this package and delivered it to the house. But where they are is actually just charging on these towers, waiting for their missions. 

[0:15:21] GV: Okay. Very cool. Let's talk about - you did touch on it in terms of fleet management. And I think the scale here is something that maybe isn't even understood yet, which is Zipline has actually, to my understanding, flown 130 million autonomous miles already. I'm sure that number is even higher perhaps than that number I've just pulled. And no injuries and all this kind of stuff. 

I think it's just trying to understand. Let's just hear what is that fleet number progression look like. And mainly, how are we sort of look at what things break when you scale fleets like this? Because, again, when we think about software, I think a lot of listeners today can probably think through what it looked like for them when they were scaling more like a SaaS platform. A very simple zero hardware involved SaaS platform, "Oh, we had 100 users. And then when we got to a thousand, then we got to 10,000." And certain things moved around, and that was understood. But this feels completely different when you're going through these numbers. Could we step through that? 

[0:16:22] KM: Yeah, it's a really interesting space, I think, especially when it comes to that hardware in the real world tying into the software side is always such an interesting place to be. And then taking a step back from this, I'll say it feels so similar to when I was at SpaceX and we were launching Starlink. One of the first things that my team was doing there was we had our two test satellites that were up in space. And we went and sat with the satellite operators, and we looked at how do you actually send commands. And it was a very simple web interface that an intern had built at one point, and it worked for that. But we thought about where are we going, right? You're going to have thousands of satellites. What do you need to build to actually be able to manage that fleet? 

And you can imagine, that's how that's grown at the systems you need for that. And I think we're at the same place at Zipline now where we have hundreds of drones. But where we're going, we're going to have the biggest fleet of aircraft in the world. How do you think about the systems you need there? 

We are really designing this from where you move from a world where you think about drones as snowflakes, right? Each one's a little different. You understand these drones. And you have to move into a world where your drones are more like cattle. You manage them, but each one is not special on their own. That means the systems have to be incredibly smart to understand how to reason about each one and surface what is needed for humans. 

And so we're building out a network operating center which will really show you the health of them, any zips that might have any alerts coming up. We have the zips actually telling us when they need to be out of service. If there's something going on, the zip will take itself out of service and surface that to us. So that way, we can have maintenance go look at it if needed, or we can run health checks on it to reboot it and get it back up. But we want to make sure always safety first. And so that's the important thing with these systems is designing that, so you can really reason about a fleet and not just single drones. 

And I think as we're expanding to more and more metros, and we'll be expanding not just nationally but then globally, you're also thinking about how do I now surface this in terms of the level of detail that you need. Because at some level, some people will want the detail of specific sites. People want the level of detail at metros. People want the level of details at a state-level or country-level. And you have to think about like how do you build your systems to be able to show that, the most important information kind of top of mind. So we can ensure we're giving the best customer experience. We have as much capacity as we can give because we are getting quite a lot of order volume. That's really important for us. And that we can also get zips back up in service as fast as possible. 

[0:18:55] GV: Do you have any kind of - I don't know. Main example that maybe comes to mind, something that really did just break when you move from, say, the five first prototype. Or they're in production, I guess. But the five main precious aircraft. And then you move to this castle analogy. We can't be so precious about these because the volume of flying is more important than, I don't know, the quality of the trip exactly. Something to that effect. But what broke in that process? Or something that broke, I guess? 

[0:19:25] KM: I think for us, the important thing is that we always do still have that high quality, high safety bar, right? What really was an example of something that broke was when we had 5, 10 drones. You can have people monitoring each one, looking at the telemetry coming out from each drone, looking at the logs after each one, just being like, "Oh, were these missions okay? How did they go? Oh, this one seems odd. Let's take this one out of service and go inspect something there." 

And as the fleet grew, obviously like that was not sustainable. And so we definitely saw people getting overwhelmed with just being like, "This drone's not doing anything. Does anyone know why?" Right? And if you don't have the tools you need to do that. We built out what we call an auto discrepancy system. And so we were able to hook into the alarms that we had on the zip software side to say, "This alarm is going based on some threshold." 

We built a configuration system that you could set this alarm, this threshold. How critical is that? And when that alarm then hits, it fully is integrated into our maintenance system then. We built that all in-house. So we have the full control over these systems to then go and create discrepancies. And a discrepancy is something that will then take the zip out of service. You cannot send missions to it anymore. It's like your hard stop of being like this zip is down. 

And so once you have that, now we could see first-hand which zips were down. Maintenance then had the immediate signal of saying, "Oh, it's down for this alarm." And now instead of humans having to go and monitor it, it turned it around to say, "Well, now the fleet is basically monitoring itself and telling us when it needed something." And that's just one example of a solution that you need as it scales. 

[0:20:56] GV: Yeah, that's really cool. And I guess it's software. And I'm like maybe sort of "firmware" that sits on these vehicles. When it comes to actually the maintenance, I don't know, cycles. I'm quite into sort of traditional aircraft, if you want to call it that, in this context. And obviously the maintenance of aircraft is like just such a huge part of what keeps them safe and flying every day. The software piece. I don't really understand maybe the software piece of traditional aircraft. If we think of the software piece of these aircraft, what kind of maintenance cycles are we talking to manage? Again, thinking at the sort of fleet level. 

[0:21:30] KM: I think when you look at the maintenance software that we needed for this, obviously there's off-the-shelf products you could get for maintaining aircraft. It's very different type of use case than what we're trying to do for drones, which are very, you'd say, small aircraft compared to what you would picture. 

And since our model is really about something that's much more configured to what we are building, and given that we do fully own the entire vertical of it, right? We know the manufacturing, we know all the pieces on the build materials that went into this. We know any issues that it had. We had the full traceability. Just made sense to build our own then and to make sure that we were handling all the cases that we needed to. 

And this maintenance software then, it not only has the discrepancy piece that takes aircraft out of service. It has all the tools that our maintenance technicians need to be able to get them back into service. You have work orders that you have planned for. Whether it's schedule maintenance or unscheduled maintenance, where they can see exactly what are the instructions. What do we need to do? Because since we manufacture it, we also provide the instructions to maintenance of like what do you need to do if you see certain problems. 

And all of this, including the system, has gone through review with the FAA to ensure that we are also building for the safety requirements that the FAA needs us to be building for and handling all the regulations that we need to be operating in the US for that. And so a system where now not only is it the day-to-day operational system of the maintenance text to be able to get zips into services, but also our traceability system. Because now you know exactly what were the problems, what was the work done, which parts were swapped, how many flights parts have flown on. We make sure we never overfly parts, too. Because every part has their own life limits on that. And that ties directly into our manufacturing system that we have built ourselves, too. So we know what it has gone through even before it gets into commercial. That's examples of that system. 

[0:23:22] GV: Yeah. Wow. Moving on to the software that actually drives your operations to make all this work as well. And you did actually, I think, talk at the beginning about building an ERP inside SpaceX. And I'm aware that within Zipline, the same has happened again. And I think this is really interesting, building an ERP from scratch. This is not something I think that most companies would do. 

Very often, I'm sure many of our listeners are familiar with the term build versus buy. Especially nowadays, most companies, they need a bunch of software just to run their company, whether they're software themselves or not. And often times there will always be this question, especially at a certain scale, of like, "Are we buying this in? Or are we going to build it? Or did we maybe build a version of this, and now that's not scaling? We should just go buy off the shelf. Someone who does this bigger than us," for example. 

Talking about ERPs, first of all though, I'd like just to make sure we have do have a very diverse listener base when it comes to ages and all sorts. Let's just define an ERP to begin with. And then let's hear the story around why you thought with Zipline it made sense to actually build one internally. 

[0:24:35] KM: En ERP system, enterprise resource planning system, is the different tools you need to account for your inventory, think through your supply to manufacture. It's all the applications there. You have procurement, purchasing applications, warehousing applications, finance application, all of these things in one. And the important thing - and I look at bill versus buy decisions. And you're right. That is pervasive, right? Everybody has the same question, these same problems. What I've seen is when you build your own systems for the areas that are your core competencies as a company, that gives you such a competitive advantage. 

If you look at SpaceX and Tesla and what they've done about building those internal systems, how you do manufacturing? At their core, they're manufacturing companies. And so to be able to build the system there, now you can design the process that you know is the best because you can change the systems to match that process. 

When you buy systems, you have to change your process to match the systems. You don't have control over those. And that really limits just how far you can go. And so I don't think you should build every single thing, but you should look at your company and say, "Well, what are our core competencies that we would really benefit from being able to build custom from?" And what are our not our core competencies? And let's not build those. It's great products. We could buy for those. But these other ones really can enable us to go so much faster. 

And that's when you look at Zipline, I talked about it being a little bit of this vertically integrated company. I think that is such a huge advantage we have. The fact that we do build all of our infrastructure ourselves. We build all of our drones ourselves. That manufacturing is so key to this company. It always has been from Platform 1 to platform 2, or with everything we are going. We have great teams who understand how do we really design these drones and infrastructure to really be safe and low cost, and to be able to scale with where we're going. 

And so it made sense to say, "Well, we should just build the systems as well to ensure that we can build them as quickly, as cost-effective, as safely as we can, and have all of that information." Because one of the challenges still when you buy ERP systems, I think you'll find that companies tend to not love the part where data is not fully integrated across all of it. You sometimes will go to one application, you'll see stuff there. You're like, "I wish I had this data in this other application, but now I'll have to re-enter it over here to make sure it's there, too. And which one is my source of truth? And how do I - I just wish they were connected." And that's really a thing I see from building our own ERP system. You're really building all of this to be integrated fully together. 

And so that's when, say, "Hey, we know from manufacturing our full traceability of that. We know any issues that happened. We know our end of line test results that came out of that." By the time the zip goes into maintenance, it's not a separate system. They're all integrated together. You have all that information. 

If we were to just buy an ERP system and try to build or buy another maintenance system, you're now spending all that effort trying to figure out how do I connect these two when their data models could be very different, and not really have a great way of doing that. And usually if you build those kind of integrations, you're spending most of your time just troubleshooting why certain data didn't get to where it needed to be. And when we can own it ourselves, we're able to manage all of that and build it in a way that the data model from beginning to end does make sense. And so I was really excited that Zipline chose to really invest in this space. I think we're already seeing huge dividends pay off from it. 

[0:28:05] GV: Yeah. I, in a past life, helped a company. Oracle, for example, is an ERP provider that I think a lot of companies would maybe look to. And I wasn't super familiar with ERP systems at that stage. I just knew, well, some companies use them. And we were providing software more for like the consumer end piece. It was just this light bulb moment where they had kept telling me, "Well, but all the data is in the ERP. We just need to "sync" over to the consumer bit." And I looked at the two bits and said, "Well, you do realize that the way you structured the data in the ERP, it's like almost impossible that you're going to break out." We're taking line items, and you say, "Okay, you got this like parent light line item. But you haven't broken it out into the four variants." If you want to push that through to the end stage, you have to go back and redo your whole ERP basically to make this possible. 

And they couldn't get their head around this. And then the penny dropped with the person on their side. And to me that was just a huge - I was like, "Wow, you can be a company and buy an ERP and be led maybe through by Oracle experts in that example and yet still come out with the wrong solution in the end and be left with this." 

I'm not saying maybe building in their case was the best solution either. But they probably would have ended up at least with the right solution if they built it as well. And in this case with Zipline, it's like there must be just so many complexities in terms of data modeling, I imagine. 

[0:29:24] KM: Yeah. I think you described exactly the problem that people face a lot of the times. And I think ERP companies, they do their best at really building great general solutions for companies. And for some companies, those will work really well. The challenge is when you have the more bespoke company or the more bespoke way that you want to work or you want to manufacture or how your variance are, right? That becomes more challenging to figure out how do we make this general model work for this. And by the time, when you get through the implementation stage of it, a lot of times your company may have already changed. Well, no. It doesn't work exactly how we need it to. And then you try to figure out, "Yeah, how do you just mush them together?" That's not usually the most fun part for engineers to be looking at. 

I don't think it makes sense for every company at the beginning stages to build these things out. Zipline used a different ERP system for the majority of the time. And it wasn't until we really wanted to get into the scaled manufacturing side of things where we said, "Hey, now is the time that we should look at building our own." 

There are different stages companies go through to make the best decisions for that. And it's it's interesting because I think you describe, yeah, you didn't know what ERP systems were. I didn't know what ERP systems were before I started working on them. I think they sound boring in that way. And it was never my dream to go work on ERP systems. 

What I realized working on them, they are actually so key to how a company can function and work and be efficient at what they are doing. And the power of having good systems is huge for companies. And my passion has always just been building software that solves problems for people. Just love when people are really frustrated - I don't love when they're frustrating with something. But I love seeing when somebody is frustrated by a process, a problem, something. I know we can build software to help with that. We can build applications that will just make that better. 

When you see people using that and just, all of a sudden, now being like, "This is so much better. This is so much easier. I now have all access to the data I need. It helps me make decisions faster," and I can see the impact of that, that's so fulfilling. And a lot of times, ERP systems are at a core of that. 

[0:31:26] GV: Absolutely. I've also met someone who works in hospitality, and he had moved from a company that did have an ERP to one that didn't. And their reasoning to begin with was cost, which is often - Oracle, for example, is six figures potentially. Yeah, just that mental shift to go from not having an ERP to having to go back to spreadsheets, he just couldn't fathom that. And I think he did manage to convince to get the ERP implemented. But it's just this sort of fundamental data and process layer that so many companies need. And just seems crazy that you try and operate without one at a certain scale. 

[0:31:57] KM: Yeah, I think that is right. And I think, to me, I look at it as like it's also not just an ERP system. It is all your custom software that we're talking about building. They all get very tied together in this way. Even, you talked about like what happens when a customer places an order and the systems behind that. But even if you think about the orders coming in, they need to know about the zips to be able to then send the missions, and have all these backend services together. So it's still tied to the fleet that we have. 

And so the fact that from end-to-end, we are building out both backend systems, but also front-end applications, so that way we can manage certain things about our partners, or the fleet, or about zips, or any of those things. And when you can build all this software in a very standard way, with standard tech stack, with standard patterns, all of sudden, it just becomes so much easier to continue to build applications for what you need. When you see problems, instead of going to more spreadsheets, you can now think about, "Well, actually, I already have these systems that have three-quarters of the data in there, I could just spin up another one. And then how do I think about how that fits into this whole puzzle that we're building." So, it is important still to think about the systems that we're building. I think ERP is one part of this, but it really is just this platform that you can then leverage into then saying, "How do we build custom software for this whole end-to-end process?" 

[0:33:13] GV: Talking of building custom software, I think one bit I was super interested to think about when I knew I'd be doing this episode was just this idea of most software developers, we understand the concept of like you've got a dev environment, you then push to staging and then try out there, and then push to production. And that just seemed - I'm just trying to visualize how does this even work in the world of autonomous flying. And I'm going to take a guess. There's maybe some simulation that has to be part of that process, because loading this onto an actual aircraft for, say, staging, maybe that works, maybe that doesn't. And that's kind of what I want to hear about. What is that process? And how does a maybe fully understanding that - for example, the ERP is the piece of the puzzle, and it's fully integrated. But maybe looking at specifically the software that does actually have to run on vehicle, how does that look when it comes to a dev process? 

[0:34:07] KM: Yeah. I think there's two major pieces. And so I'll touch on one and maybe go in more detail on the other one, because one of them is more of the software that actually goes directly on the vehicles. And that is more of the autonomy, the flight software, embedded software side of the stack. 

With all of that, we have a release cycle. About every six weeks, we'll cut a release. We'll test that very heavily both in simulation as we're developing on that, but then also at our test sites. I mentioned safety for us is top priority. And so we have a certain set of validation that it has to pass before we will start rolling out a software release into commercial. 

Similarly, we're rising more on the cloud and application side, we have software that is directly integrated with the vehicles. And we have software that is not at all touching the vehicles. And so the validation story there is it's a little bit nuanced because it depends what you're working on and how mission-critical something is. And the way I think about things is if there's a human in the loop, you tend to not need as much verification as if there is no human in the loop. And so that's how you can think of different software. 

And in that sense, similar to what you're saying, you have your dev, you have staging, you push to production. We have those pieces, what we call an integration environment. And that is specifically for our test sites. We can push software there to our test sites first and actually test with hardware in the loop. And we've even set up a kind of commercial test site, if you will, at our test site. It operates exactly commercial. So we can try things internally before, again, we impact any kind of customers with that. 

But we are also now at this inflection point that we can go there, where with how the fleet is growing and how we're scaling, we now need much more of a simulation on the cloud side too. So we've had a simulation engine. We've built one out more fully now for the autonomy and flight software side. 

On the cloud side, we're building out a fleet simulator to simulate all the different parts of an order from like order to delivery. So we can actually test that with the scale we are going to. If today we're doing 3,000 deliveries a day, but we know by the end of the year we want to be doing, let's say, 25,000 deliveries a day or 50,000 deliveries a day, I want a system that can test that, right? I want to simulate that we are doing 50,000 deliveries a day because I need to stay way ahead of that curve and see what services break. And so, we're now building out that simulation platform to make sure we are testing all of those services at scale. 

[0:36:29] GV: That's like effectively load testing, but there are no playbooks for this in terms of autonomous. And again, building just - I'll just call it regular software, load testing. There's so many platforms out there now that you can just plug in a load testing platform. But yeah, I imagine there is almost no buy option on this, right? This has to be another build, right? 

[0:36:49] KM: Yeah, it's really tricky because it's like you want to simulate your services, which some things can do. But then we also want to simulate our zipping points, or humans, or what those process are like. And we want to simulate our zip software that we have also, so we play it all together. I think that's where, if you buy a product, I think you'll spend almost as much time as trying to configure it to make it work than you would have just building something. 

And when we build software, we try to build it very iteratively, right? You like build a piece. See if that works, so you have one service running in it. Then you start building the next pieces of your simulation platform, you get more services in there. And so it is definitely a continued approach to development with it. 

[0:37:25] GV: Moving on to, I guess, the actual human bit of this, which is the team and teams that do build all of this software. How have you thought about structuring those teams? I'd like to get also on to just what does effectively, like incident response, and being on-call, and all that kind of look like again in this context because it probably looks a little bit different. 

But especially, I guess, you've seen how it was all done at SpaceX. And I read a bunch of books around how that maybe looked different there to say a lot of other companies, and that's why they were so successful. This big mission that they had to go with. And I imagine it's a little bit similar here. You've had to think about building software teams maybe a little bit differently. Yeah. I just kind of want to hear about that. 

[0:38:05] KM: There's a lot of ways where it looks similar in some ways, where it definitely looks different. And I think when I look at how to really build software teams, I think making sure that people can really focus in some domain, have a lot of autonomy and ownership in that space, understand it deeply to figure out what needs to be built is key to having just really high performing teams. 

And when we're doing this, we make sure the engineers are also responsible for the product decisions too, because then they can be really close to it and understand with stakeholders, with customers, whoever is using it, what is the problem? How do we then build a solution that solves this end-to-end? 

And so we've structured the team to have three big areas. There's the commerce platform around the marketplace side that we're building. A lot of the integrations with our partners, merchants, with healthcare partners going to be all in that side. You have the delivery network that is more around how we orchestrate with the fleet. How do we always reason about the state of the fleet? The mission processing. That side that we've discussed is in there, as well as the maintenance side to just make sure you have a healthy delivery network. And you have your enterprise system, which we've talked in depth on. 

All of these is to be able to have enough depth within a team to keep it small enough where you understand that, but also really to be able to have quite a bit of ownership of what you're building. You touch on incident response. One of the challenges as we are scaling, in some ways, incident response can look similar. Sure. Yeah. You get paged, you go, you figure out what the problem is, you fix it. There's that. 

But as we're scaling, we're now seeing like how are we building our systems to ensure we don't take down, for example, all of commercial, right? If we are operating everywhere, you now have to think about how do you shard your databases differently? How do you have a disaster recovery plan? How do you think about operating in multiple regions? And how you structure this? How do you release software to have canaries and be able to roll back, right? 

So, you have all of those problems that in other industries you will have similar problems like that. I think the the tricky thing always in ours is you also now have hardware in the loop as part of that. And that always adds an extra layer there that, in your traditional SaaS companies or just pure software, you don't have to think about that. 

[0:40:18] GV: Is there like a sort of skewing in terms of the makeup of the team that has come in so far that have already had some aviation background, or does that not really actually apply here? 

[0:40:29] KM: It doesn't really actually apply. We have software engineers, some of them who have been passionate about drones. That's always a nice connector for people to be like, "I want to work in this space." But it's not at all a requirement. And that is the thing, we are building out this team of software engineers because there's so much we have to build. If you think about the scale of what I'm just describing that we need to build towards and all these systems that we have a lot of zero-to-one still to build, while we're also building them for the scale of what we are having today and what we plan to handle in the coming months and years. There's so much to do that we are looking for great full stack software engineers who can really jump in on building out these applications, these back end services to be able to do all that. 

And when I talk with software engineers, what I'm really looking for is that kind ownership and that kind of curiosity and passion to just be like, "I want to solve hard problems. I want to think about how to solve those. I want to own that end. I want to be really close to customers and have an impact with what I am doing and see that impact." And I think that's where we see the most successful people at Zipline who just thrive with that kind of thing and that kind of ambiguity. 

And so to me, it doesn't matter what language you've used in the past, and which domain that you have worked in. It is just like do you have that kind of curiosity and passion and hunger of how do I solve these problems and build robust systems for that? 

[0:41:50] GV: I always hate getting asked this question because it's sort of like why you're asking the question. But I think in this case, the size of the team, I think that's interesting. If someone's coming into this industry where it's already a new industry, if they're going to join into autonomous vehicles. But are we talking 100 engineers? Are we talking 20? What does it look for them? 

[0:42:07] KM: If you look across Zipline, we have a few hundred software engineers. But that is across the autonomy embedded flight software side of things. I'll say the majority of engineers actually sit there. Within the application or cloud space of things, it'd be closer to 40, 50 software engineers. It's not a very big team. And that's on purpose. We know we need to grow, but we also are trying to be very thoughtful with this. 

A lot of the time, you can build so much faster when you actually have smaller teams as long as you are giving them that real ownership and focus. And that is still something we are working on at Zipline. We have so much that we want to do that we want to do everything. So how do we really narrow down to what we just need to be able to execute on really well? 

And so regardless of the whole organization side, we want to make sure teams are working in even smaller groups than that. Because I think if you think about the times that people have done their best work. And in my experience, it has been when you are on a team of two to four people, and you just have this ownership of just getting these things completed, you know what does success look like. How can you help there? You feel you can move so much faster than if you're on a team of 10, 20, 100 people, and you're responsible for just a tiny part of that. And so even as we grow and as we scale, and we'll always keep that in mind, is like how do we keep that kind of size where you just feel you can make those decisions. 

[0:43:27] GV: Yeah, absolutely. Literally, just as we're recording today, something's just shipped where I work. I mean, I work across a few projects. But one specifically, it was pretty much just me and one other person, an engineer and me. And it's so interesting when you have just two people, and you can just ping-pong, ping-pong. And this thing, it's going to go out to quite a large scale. But it is so interesting how actually some people might think that was a 20-person project. It was two. And as you say, I think that's where some of the best work is done. It doesn't particularly matter like what the total size of engineering is per se, but how it then gets distilled down into the team sizes I think is super important. That's good to hear. 

[0:44:04] KM: Yeah, I definitely picked up a lot and learned a lot of that from SpaceX. There was a time that we were building out some of the warehouse management side for Starlink. Knowing where that was going to go and scale. And went to go talk to this other company about it, and they had a team of 40, 50 people. And they were like, "Well, how many people is on your team?" And we kind of looked at each other, we're like, "Well, it's the two of us?" I'm like, "Well, there's like a half engineer who couldn't make this trip, but just us." And that was okay. We were able to build it out still and what we needed. 

And I think when you do that, you're really forcing yourself to think about what is the highest priority? What is the highest impact thing I can do? Because we don't have all the bandwidth and all the people to do this. You have to make sure you can ruthlessly prioritize and get down to that MVP and being like, "How do I just build the next thing that I know will make a huge impact?" And you always have to balance. How do we keep growing to make sure we're not so lean that we can't actually get stuff done, but also that we don't become so bloated that you just end up working on things that are actually not useful? 

[0:45:05] GV: Yeah, for sure. As we start to land the plane off the episode. Pun, I guess. Completely intended today. But if we look at where is Zipline specifically going, and I guess where do you think that drone delivery generally is going. Because if we sort of flip this back to the start of the episode of where does Zipline started in more rural communities, I think, in developing countries. And, obviously, we've now progressed through it hitting metro areas, and cities, and so forth. Where does this go? How far does the web of autonomous deliveries? Where does it go country-wise? I guess, just geography-wise overall. I'm just interested. 

[0:45:43] KM: Yeah, I think if you look at just generally where the world is going today, right? There's so much more investments being made into the autonomous space because you're starting to see both with how AI has grown, but also just with what we can accomplish now is huge. I think people are always looking for ways that they can save time in whatever they need. And I think Zipline plays directly into that. 

And so with autonomous delivery, drone delivery, you'd see more companies trying to enter that space. And one of the reasons I'm obviously very bullish on Zipline is because I think we approach it differently, where we really think about this from the customer-first side of things. 

If you look at our drone delivery, our drones stay 300 feet up in the air. They're very high. We want to keep that noise as quiet as possible. We're always looking for ways to make that quieter because we know that's something that's really important to the communities that we do deliver in. And then there's a droid that comes down and just nicely puts her package on the ground. And I think when you see this, it feels magical, right? 

And what I love is seeing these videos, especially kids, it's close to my heart, but seeing kids see these deliveries and just be inspired by it and being so excited. And seeing that parents love it too, right? Adults are really thrilled by this. And I think if you look at some of the other players in the space, it is louder or noisier and not as of that customer-first type of thinking. And that's something that I'm really glad that we're doing. Because with where we're going and how we're going to scale, we see this. We have tens of partners right now in our marketplace. We're going quickly right now to hundreds every day. We have new restaurant partners that are on our app. We are going to be expanding into a couple different healthcares this year, more healthcare partners next year. 

We are quickly going to the place where we are now to in a few years being at a million deliveries a day, which is still a small slice of the instant delivery market. But being able to change the way people think about delivery. And how often you could get it, or how fast you can get it, or how easy it can be. And how it just fits into your life where you don't even notice, but just what you need shows up. I think it's going to be so big, because then not only nationally, but we're going to be doing this internationally, too. That to me, Zipline really is at this point where we are revolutionizing an industry. And there's only so many times that you get to do that in your life. And I feel really lucky to be part of this and where this scale is really going. 

[0:48:04] GV: I like when you're talking about, for example, how quiet vehicle can be, and customer-centric. And it's also kind of environment-centric as well. Because if people see a device like that landing, let's just say, in their neighbor's house, but it does give a certain sound or something. They say, "Oh, well, who the hell is this company dropping into my neighborhood?" I don't want anything to do with them. But if it's something where they go, "Oh, wow. That thing is like so quiet. I never even realized it was there until I just looked up." Because I live in high-rise in Singapore, and it's actually an area where people just fly recreational drones a lot. But the pitch of those drones is so high and quite disturbing actually that I get quite frustrated at these drone flyers. Like, "Ah, god. Could they not fly them somewhere else?" But then by focusing on the sound piece, that's huge. That's really great to kind of hear that. 

[0:48:50] KM: Yeah, I think that is one of the front of mind things always in our approach of this is just working with the communities that we're in, hearing feedback from customers or not customers, the neighbors of customers, hearing what they are seeing so we can keep making it better and better. Because when you do something like this, it has to just fit seamlessly into people's lives. If it is more of a nuisance, it's not going to scale, and it's not going to win. We see this as something that really could just enhance people's lives. And so we have to make it that. 

[0:49:19] GV: Well, Kyle, it's been great to have you on. This is just obviously an area of technology that is only going to increase as we've talked today. But I think just the fact this is probably one of the first times we've really talked about it in any kind of depth on software engineering daily is just great to have done this. And yeah, we're going to be following Zipline, I think, very closely. Where's best for people to look up Zipline? Obviously, we got a lot of software engineers listening. If they're being inspired and want to like think about applying for jobs, that kind of thing, where's the best place to go? 

[0:49:48] KM: You can go to our website and you look up Zipline. All of our roles are there. Look up software engineering roles. I'd be very excited to have listeners of this podcast come and contact us. You can also experience Zipline. Right now, we're really serving mainly in Dallas. And we'll be expanding to other metros soon as we talked about. You'll see this more happening more and more. But if you find yourself there, download the Zipline app and check out our delivery. 

[0:50:10] GV: Amazing. Thank you so much. And yeah, I'm sure we'll catch up again in the future. 

[0:50:15] KM: Definitely. Thanks for having me.

[END]