EPISODE 1961

[INTRODUCTION]

[0:00:00] ANNOUNCER: Building software that runs across many machines is notoriously difficult. Developers have to grapple with problems such as race conditions, partial failures, and message ordering. Notably, one category of distributed software has largely escaped these burdens. A distributed database can spread a single query across thousands of machines, handling the coordination, failure, recovery, and ordering internally. This raises a natural question of why general-purpose distributed programming can't feel the same way. This is a highly practical problem at AWS because the reliability of cloud infrastructure depends on getting distributed systems right at massive scale. 

Joe Hellerstein spent 30 years as a database and distributed systems researcher at UC Berkeley, where he pioneered much of the foundational thinking on applying database ideas to distributed programming. He is now at AWS, where he works to bring his research into production through Hydro, which is a Rust framework to bring declarative queries to general-purpose distributed programming. In this episode, Joe joins Sean Falconer to discuss how ideas from the database world could make distributed programming dramatically simpler and safer. 

This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him. 

[INTERVIEW]

[0:01:35] SF: Joe, welcome to the show. 

[0:01:36] JH: Good to be here. 

[0:01:37] SF: Yeah, thanks for doing this. I'm excited about it. You have very interesting background. I don't think that we'll even be able to cover everything, certainly, that you've done in your career, but I think we have some interesting things to jump into. You spent 30 years at Berkeley working on databases, distributed systems. Now you're at AWS. I guess, how would you describe this thread that has connected your research, maybe going back to the 90s, all the way to what you're working on today, which is Hydro, which we'll start to talk to about later? 

[0:02:05] JH: Yeah. Well, I think just to keep it focused, because you work on a lot of things over 30 years, I'll pick the through thread, like you said. I was trained up as a database person back in the 90s. I actually read out of college. I went to IBM Research in the very early 90s and got baptized in the database religion by the IBMers and then by Mike Stonebraker at Berkeley, who was my master's advisor, and then by the Wisconsin team, which was the database powerhouse at the time, Dave DeWitt and others. So that was my training. 

And one of the things that was cool about databases was that you could write a query and it would run on any hardware you liked. So today, you can run an SQL query, and it'll run on an embedded device like your phone, or it'll run in the cloud across thousands of machines, and it's the same query. And something that really bothered me as a software person was that programming wasn't like that at all. Programming computers was somehow different. And every time you got a new piece of hardware, you had to start over. And particularly if you wanted to run something that wasn't on a single node, but was on lots of nodes or parallelized, distributed- call it what you will- you had to start from scratch, and programming languages weren't there to help you particularly. 

I've been wrestling with that gap between declarative queries and imperative programming during the course of my career and trying to narrow, narrow, narrow that gap until we can really talk about programming the cloud, all the computers in the world, if you will, in a way that's portable, just the way we write queries that are portable. 

[0:03:28] SF: I mean, I think that's a really interesting point because if you think about if I'm running a query against some massively sharded database that's running all over the place, as the actual user writing that query, I don't have to think about any of the infrastructure. I don't have to think about the configuration of that. But suddenly, if I really need to run my application, all that complexity is thrown in my face. And I have to just deal with it. And I guess there's some bypasses out there that try to hide some of the subtraction. It was really, really hard to get away from, especially as applications tend to scale more and more of essentially what's under the covers becomes probably - maybe it's 90% of essentially the engineering time is just working on that platform. 

[0:04:10] JH: Yeah, I think that's the apologist's response to my question that I asked 30 years ago. It's like. why must it be so much harder to write general-purpose programs than queries? And the apologist says, "Well, there's a lot of details in there, man, and you have to get them right." And I'm like, "Well, what details?" Because that's what I'm asking. I'm exactly asking that question. Let's not hide this under the rug and say it's too hard. Let's stare at this thing until we can tease out all the little bits that we care about controlling as software engineers, right? That databases are glossing over and making assumptions about. 

And in studying that, we can bring more and more of this stuff to the point where the engineer maybe only has to specify that stuff when it really, really matters for performance, or maybe not even that. Maybe actually compilers should be better at this than engineers. So I really like to leave this stone not unturned. 

[0:04:58] SF: What do you think led to us being able to do this with databases, but struggling to do it with general programming languages? 

[0:05:05] JH: Yeah. So if you roll me back 20 years and ask me this question, I would give you a sharp answer for that. I'd say, well, databases very wisely chose sets as their fundamental model. So a database table is a set of rows. The ordering is not dictated. And the programmer knows that when they write an SQL query, unless they say order by, they'll get their output in some random order, and they're cool with that. And I would have said a bunch of years ago that everybody else was getting it wrong. We should all program with sets all the time. I think at some point in my career, I stopped trying to yell that from the mountaintops because it's annoying and nobody wanted to hear it anyway. And I applied my own medicine, which is, no, let's stare at this thing harder and say, well, okay, sequentially sometimes matters. How are we going to accommodate it in a way that doesn't require all this careful single machine at a time reasoning? And so that has been the story of, I'd say, the last 5 to 10 years, and what led up to Hydra was closing that last gap because it's not correct or reasonable to say everybody should write all their programs using these high-level, set-oriented languages. 

[0:06:04] SF: Right. I guess what motivated the decision to go from full-time researcher to industry full-time? 

[0:06:12] JH: Yeah, well, I think part of that was the fact that we were starting to really answer the final questions here. The Hydro project, which we'll talk about more, was the end of a long line of efforts that I had in my lab on trying to make programming the cloud easy to get correct, easy to make scalable. And I think we finally got it. And so part of the story was, well, I never felt we were ready for production with this research till the last few years. And now I do feel we are. And I think it can really change the way that we write safe, distributed, correct, high-performance code. 

There were other considerations, too, about after teaching for 30 years, that you still want to keep teaching and so on that are more personal. But I think technically, in terms of this arc of research, it's ready to put into production. And that's what we get the opportunity to do at AWS. I've been saying we should change the way we program the cloud. Okay, let's go to the world's biggest cloud and see if we can change the way they program it, right? Stop pussyfooting and writing research papers. 

[0:07:04] SF: Besides the personal stuff, kind of like the biggest thing is that by being part of AWS, there's just things that you can do there that you wouldn't be able to do in a university lab. 

[0:07:12] JH: I think that's part of it. I think a lot of it is also validation, to be honest. I could build toys on campus. Hey, here's a Twitter clone. Here, I implemented Paxos. I can do all these little feats of daring do, but who's really going to believe that, right? And there's always these questions like, "Well, but if you're doing it at scale and you're doing it for dollars, you wouldn't do it that way." 

So at Amazon, I can remove that objection. We can work with colleagues here at Amazon, retrofit systems that the world relies on with Hydro, show that it gives correctness guarantees that the previous code did not. And does not give up performance. 

[0:07:44] SF: And we're talking about distributed systems here. And I think a lot of times when people get introduced to distributed systems, they get introduced to, like, the CAP theorem. And that becomes the overall thing of like how they think about the world. And there's always this discussion around whatever the investment is or whatever they're building. What are the two things that they're guaranteeing out of the three? But I guess you've been a proponent of the CALM theorem. How do those things relate? And I guess how does CALM potentially change the conversation away from CAP? 

[0:08:10] JH: So many questions are embedded in there. So I'm going to postpone the specific CAP versus CALM thing for a minute because I want to get across something you were hinting at in the first part of the question, which is when people think about distributed systems. So the thing is, everything's a distributed system now, man. Your phone app is talking to something in the cloud. So there's at least two nodes in almost every app you use. Your desktop app is also probably always talking to the cloud. Nobody writes a single-node system anymore, unless it's pretty much a toy or an experiment. Most things are distributed. 

And bugs will just creep in. Race conditions, failure handling bugs, all this stuff. This is part of writing software now. And it's really leaking into every applications problem. Now, it may not be every developer's problem. If you're a front-end person writing JavaScript, you might not run across it, but I bet you somewhere in your product you are dealing with this one way or another. Having said that, front-end people are really good at asynchrony, so they actually get this stuff very quickly when you teach it to them. 

[0:09:10] SF: Yeah, and I would say a front-end, to your point, like everything being a distributed system, it's also a client-server, so there are at least two nodes there. 

[0:09:17] JH: Yeah, so actually, I don't mean to dunk on front-end folks at all. They're often quite sophisticated about this stuff. But the bottom line is everything is distributed pretty much, and that's exposed to software. It's not just like everything's distributed in your processor and you don't know it. It's like, no, really, you're programming a distributed system most of the time. 

Now, to answer your question about CAP and CALM, so let's jump into really technical stuff here. So CAP was an effort by my colleague Eric Brewer at Berkeley to try to articulate. And it was early. So he was really feeling his way through it. And I think you have to understand the historical context when he was wrestling with this stuff. So I was there with him at the time. They were building the Inktomi search engine, which is essentially the predecessor of Google, and trying to build it on clusters, not on a single big box, which is the way previous search engines had been built. 

And their first effort at this was to take a transactional database. I believe it was Informix at the time, which was at that time a very good parallel database, and see if they could build a search engine with it. And all the transactionality guarantees that the database provided were getting in the way of availability. 

So if a single node in the database went down, the database would say, "Uh-oh, it's not safe to run transactions," and it would lock you out. They couldn't build a always-on search engine that way. Moreover, a search engine doesn't really need transactional consistency. So Eric was trying to articulate in a semi-formal way, like what tradeoffs do you have to make between some notion of correctness, which transactions are giving you a particular notion of correctness, and the notion of availability. 

And CAP was his effort to codify that. So the words are consistency, availability, and then the P is partition. So think of the network partitioning your set of nodes into two groups. Maybe one node is outside, and the rest are inside, or maybe it's split down the middle. His assertion was you can only get two out of three: consistency, availability, and tolerance of a partition. So that's CAP. And it was not a theorem. It was a way of thinking about the world. 

Some folks at MIT decided that they would try to prove a theorem version of it formally. But what they reached for was a much narrower view than Eric actually intended. What they proved in the CAP theorem is that linearizability, which is a very specific model of correctness, exhibits these CAP tradeoffs. The problem then was that the universe was running around thinking about CAP as this bigger umbrella that Eric had defined, while the actual formalism was this very narrow thing. And that led to all sorts of confusion, and therefore a lot of discussion, which was pretty illuminating. 

But over the course of maybe a decade, there were blog posts of people saying, "I bypassed the CAP theorem. I have a system that's all three." And other people saying, "That's not possible. It's a theorem. Look, there's a math paper, and it's from MIT." Right? And there's just like this disconnect between the CAP of the spiritual idea that Eric had and CAP the Gilbert and Lynch paper that was published at MIT. And Eric subsequently wrote another paper that explained he didn't really mean the Gilbert and Lynch thing. He meant this bigger picture of like, "You got to think about these tradeoffs." But they may not be endemic in all definitions of consistency, right? The consistency that Gilbert and Lynch chose was linearizability. So that's Eric's journey. 

I come in in the middle of this and say, consistency, well, maybe that's the whole thing. Maybe the definition of consistency, some definitions give you this tradeoff where you can't have all three. Some definitions of consistency maybe allow you to have all three. And just saying consistency is too vague. So where CALM came out was an effort to try to identify exactly what is the sharp line in this consistency space between programs that succumb to CAP and programs that can avoid CAP. 

And particularly, I didn't really want to talk about availability anymore. I just wanted to talk about, like, will two nodes potentially get into a disagreement, like a long-term disagreement? Whether they're partitioned, whether they're not partitioned, that's really the question. Can you keep the system consistent? 

And so what we observed in the CALM theorem, the original conjecture, much like Eric, it started as just an idea, was that I know in database queries how this works. If you have operators that stream, so as they get more input, they produce more output, those operators you can run, and you can eagerly consume their outputs, which means that you can consume it faster in one place than another and not worry about correctness. Okay?

Something about these streaming operators, which includes filtering, it includes mapping, it includes joining two tables, which is that's most of database queries; that stuff is actually not subject to CAP. I know this because I've built these things, and they stream, and it's okay. It's cool. Everybody can see these streams at different rates. But there are things in the database that have to block. A classic one is sorting. If you're going to sort a table and you haven't read all the items, you don't know what the first item is. So you can't sort something until you've seen everything. 

Another one is aggregation. So think about a sum query that's adding up all the numbers in a column. So you see all the numbers. You don't know what the total is, right? So these are things that somehow required you to block and consume the entire input before producing an output. Well, it turns out that, taking this story down the road, this property that as more stuff comes in, more stuff goes out, but all the stuff from before remains true. There's a math name for that. It's called monotonicity. Monotonicity says if you give a bigger input to a function, you get a strictly bigger output than you would have gotten, which is another way to say that is in a smaller input, you could start emitting its outputs without worrying about retracting them if more inputs come in, right? So it's the streaming property exactly. It's just said in terms of, like, two fixed instances: one input, one bigger input, one output, one bigger output. Yeah, so that's monotonicity. 

And so CALM was a conjecture that you get consistency if your program is monotone. And you do not get it if it's not. And that's a sharp line. It's an if and only if. And again, a graduate student, much like with the CAP theorem, proved it about a year later in a very specific way, very much like Gilbert and Lynch. So he proved it for relational languages on these things called relational transducers. 

If you're like a data log geek, you might really love this paper by Tom Ameloot. And it led to like three or four more papers, much like the CAP theorem. So it was a little cottage industry. But still doesn't really apply to, like, "I'm sitting down in Rust and writing a program. How do I know if it's monotone?" 

So just this past six months, I finally decided it was time to clean this up. And I have a - it's like just an archive so far. But we have a notion of what I call complete CALM, and it captures in a way that you can analyze actually in a compiler. We do it in Hydro. For arbitrary programs, essentially, it says what kind of monotonicity we're talking about. You don't need a special language like a logic language or a database language. Any language can be used to implement a spec, but the spec is either monotone or not monotone. And if it's monotone, any language can be used to implement it in a way that is both consistent, so everybody gets the same answer, and also it's available under partitions. You can start outputting answers even if you're in a partition state without worrying about the future. 

[0:16:01] SF: And I guess, like tactically, like if I understand that my program does this monotone specificity, how does that help me as an engineer? What do I take away from that? 

[0:16:11] JH: Absolutely. So let me work backwards from what we have now in Hydro as a feature, and then we can talk about why and how it works. In Hydro, you can point at, like, an endpoint of the system. So think about a network endpoint in the system that's going to stream out outputs as it gets inputs, right? So requests are going to come in, responses are going to go out. So look at one of these response streams; Hydro will tell you what kind of consistency it offers. 

So Hydro uses the CALM theorem to say, "Hey, guess what? All replicas will agree on the messages that come on this endpoint, but they won't agree on the ordering." That's a sort of set-based consistency. Or Hydro might say all replicas will agree on the sequence of outcomes on this endpoint. That's a sequential consistency, okay? So it's a stronger one. But Hydro can use the CAP theorem to tell you what kind of consistency you're getting for each one of your endpoints of your program. And then you can decide what you need to worry about. 

If all of your replicas are getting the same outputs in the same order, you don't have to think about anything. You're sort of just living in your usual sequential world. If all of your replicas are getting the same set over time, they might get them in different orders. So you have to think about, like, is your downstream code that's consuming this endpoint sensitive to order or not sensitive to order? Commutative is the math term. But it's like very actionable. 

And actually, in Hydro, we can check these properties downstream for you too. So we can tell you, "Hey, this was set-wise consistent. But downstream, you have a thing that needs a specific order." And we won't compile those together because that's non-deterministic. You're getting a random order being fed into a thing that cares about order. That's broken. So the compiler can literally check this stuff for you these days. 

[0:17:44] SF: Yeah, so we mentioned Hydro. We've teased it a few times. So it's probably a good time to define what is this. What is Hydro? 

[0:17:51] JH: Yeah, thanks for asking. Hydro is a programming framework in Rust. So it works in standard Rust. It doesn't use any nightly features or anything crazy- standard Rust for writing distributed code. And its general-purpose, which is to say, if you're writing a really fast event-handling system, like it's running on network switches, you can use Hydro, and it'll be fast enough and low-latency enough. 

If you wanted to run like a big data analytics job, you can also use Hydro. So it can do things like Spark or Flink would do. It doesn't care about the performance shape of your events. They can be many or few; they can be quick or in volume, but it allows you to program a distributed system in a way that is quite general-purpose. 

And then when you program in Hydro, you get a bunch of kind of metaphors and then a bunch of correctness guarantees. The metaphors are that you're programming locations, not single computers. What's a location? A location could be a process, although we don't say where it's running, but it could be a single process. But a location could also be a cluster of processes. 

And the reason to have locations is to talk about their behavior. So you can say, for example, if you have two processes and you want to compare a value between them, you can't. You have to send a message, right? And so that's in the type system. So if you try to write down X is less than Y, and X has a location that's different from Y's, that's just a compiler error. The compiler is going to make you put in the networking and then think about the delays and all that stuff. 

So it's a global model, but communication and location are explicit in the model, and they're type-checked. If you're on a cluster, then sending to a cluster goes to a single member. And then you have to figure out how the members talk to each other. But you have in the program a very clear definition of what this cluster is and properties about it. Can I broadcast to the entire cluster? Or can I only send unicast? Or is the membership going to change under me? These are all things that are in the type system. They will help you with bugs. 

[0:19:42] SF: Is it a case that essentially you do a compile-time check to see could this thing talk to this other thing? Or are you making it so that those two things can talk to each other and I don't have to think about the underlying networking message passing in order to do that? 

[0:19:56] JH: It's a little bit of both. I mean, you do have to put in some syntax for networking. So the first thing you'll see is this doesn't compile. You're comparing two things at two different locations. But the syntax in order to get things from one location to the other is like .send. You just needed to add an extra word essentially to your program, and then it'll compile. So we did this in Rust in part because I think Rust programmers are used to the idea that there are small, subtle things you have to fix to get your program right. But once you fix them, you're pretty guaranteed to get correctness. 

In particular, we like to talk about Hydro as offering distributed safety, much like Rust offers memory safety. And so some of the kinds of bugs we prevent for you are race condition bugs. You can't write race conditions in pure Hydro. It's not possible. It won't compile. 

[0:20:39] SF: Right. In a lot of ways, you're putting in these guardrails around these known issues that tend to bite people writing and running distributed systems. How does this affect testing? Let's say that I do have a function that spans multiple machines, how do I actually write a unit test? 

[0:20:56] JH: Awesome. Great. So let's start with this like location stuff I teed up. Suppose you have a cluster and the code on the cluster, so this is the code that runs on every node in that cluster, says send to Sean. And Sean is a process, okay? Well, we know because the cluster is a cluster that the messages will arrive at Sean in a random order from the different cluster members. We can't control the timing of the messages. They'll come when they come. And on different runs, Sean may receive messages in different orders. 

The input port at Sean will be marked as a stream that's unordered of messages. Now, if you have downstream code in the Sean process that is order sensitive, like it's accumulating a list, or it's buying stocks, we will flag that as a compiler error. So you simply cannot connect this unordered stream of input messages to this ordered operator, okay? That's all guaranteed by the type system. There's no testing needed, right? The same way in Rust, you can't walk off the end of an array. You just can't compile that. 

However, as in Rust, if you really want to do that, you can, and you have to put in a keyword. So in Rust, the memory safety of the keyword is unsafe. In Hydro, it's non-debt, non-deterministic. So to get this unordered stream and shove it into this ordered thing and say, "I don't care, man. It's coming in whatever is coming in. I want to run this code." You will put in a non-det block. And at that point now, the compiler will let you run this thing, compile it and run it. But you have a non-deterministic system. And so now you want to test it. 

So Hydro comes with a batteries-included simulation tester or model checker, if you like that terminology. But it only checks the non-det stuff because everything else is correct by assertion with the type system. In particular, if you take an unordered set of messages and you put it in a commutative operation like sum, sum doesn't care what order things come in, right? When you're adding stuff up, you can add it up in any order. 

So if you take unordered and you stick it into sum, Hydro won't simulation test that at all. It'll just try any one order and say, "Yep, any one order is the same as any other. So I'll only test the one order." But if you put in a non-det, so you have random order going into a thing that's order sensitive, the non-det then in the simulator will try all the orders, run every single one of them, and you can see if your tests pass. In Hydro, testing is there when the type system doesn't save you. And the type system doesn't save you when you tell it not to. And then we'll do simulation testing for you. 

[0:23:14] SF: In the scenario where I have a single function that's spanning multiple machines, is Hydro handling the state management between those machines as well? Like a durable execution framework or something like that. And if some things go wrong, essentially, it's going to have to back up the checkpoint, right? 

[0:23:28] JH: Right. This is where I think a lot of libraries trip, actually. And why we've postponed a general-purpose solution for all these years is that if you start to answer those questions like you're asking, what about durability? What about replay of dropped messages or things? All these kind of specific problems typically lead people to specific solutions. 

If you look at Spark or Flink or Kafka or many of these frameworks, they only work in their narrow sweet spot. They're distributed, but they only work at the stuff they're good at, and they're really bad at other stuff. You would never implement Paxos with Spark. That would be a terrible idea. And the reason is they answer the questions like the one you just asked. They say, "You know what, we're going to need a way to make things durable. Let's bake one in and force people to use it." And it'll make something super easy. It'll make other things completely too slow or inaccessible. 

In Hydro, we don't do that. All that stuff that you want to do, like durability, you just write some more code or you plug in a library because we don't want to be opinionated about architecture. We just want to be opinionated about correctness. So you get to build the architecture that works for your application. If your application does well by spooling things to disk and then reading them back, cool, do that. But that's going to be too slow for a packet handler. 

For a packet handler, you might instead send a negative acknowledgement to the sender. The sender's job is to send it back. There's all sorts of ways to do failure handling. There's all sorts of ways to do durability. It can be done with replication. It can be done with spinning iron. These are decisions a language should not make, in my opinion. 

[0:24:55] SF: Mm-hmm. I guess what is the best way to think about Hydro in reference to maybe these existing solutions that people might go to who have been building these types of systems historically? 

[0:25:09] JH: Yeah, great. So I partition this world of distributed frameworks into two. There's the high-level frameworks. So think of things like Spark and Flink and Kafka. And durable functions is another great example. And then there's low-level frameworks that are really convenient for programming, like Actors or RPC. 

The high-level frameworks tend to be very opinionated. They're really good at one thing, and they're quite bad at lots of other things. That's sort of the Spark, Flink, Kafka thing. Those are essentially applications. They are great at what they do well, okay? But they're not programming frameworks. They're high-level programming frameworks. SQL is another example. I really shouldn't be trying to convince the world to write all software in SQL. It's a terrible idea. So not general-purpose programming. So that's one bucket. 

And the reason those things don't generalize is because they make these strong architectural assumptions. In the other bucket, like take RPC as the canonical example of a programming tool for distributed systems, it's completely unopinionated. It just says, "Hey, here's a way to send messages," which is great, or even particularly invoking code on the far side by sending a message and getting a response. But it doesn't offer very much help in assembling a program, and it doesn't offer very much help in checking if the program's correct. 

And the same is true with actors. Actors are basically an event loop around event handlers. Cool, that's handy. But it doesn't really help you get through, like, "Is my Paxos doing Paxos?" You know what I'm saying? It doesn't tell you if you have race conditions. It doesn't tell you how it deals with failure. It's just a little event loop. 

So there's been nothing akin to a real language compiler for distributed systems. It's general-purpose and solves for you the hard things about your language. So Rust memory safety is such a beautiful example. We've all internalized that. It solved a hard problem we thought we had to enforce at runtime using garbage collection. And it said, "No, you know what? We can solve this at compile time." And we want to do that now for all the hard things about distributed systems; failure, parallelism, ordering, retries, all that stuff. And I think with Hydro, we've demonstrated that that's possible. 

[0:27:07] SF: And when you talk about making this simpler, what does that mean for, I guess, end user? Is that simpler in terms of less bugs, less lines of code, maybe both? How do you sort of frame the simpler part of that? 

[0:27:18] JH: I would argue mostly less bugs. Maybe less lines of code. That's not always a virtue, right? You can write some really dense code that's hard to read. That might not be a good idea. I have used that argument in the past, but I think it's not really a very honest argument. No, fewer bugs. 

And in particular, in the era of agentic programming, you've just generated a distributed program. How do you know it doesn't have race conditions, man? In your own code, you might have been, "Well, I assert that it doesn't have race conditions because I'm really good." And I know people who can do that pretty reliably, but they make mistakes. 

AI agents are really bad at this right now. They make tons of distributed systems mistakes. They don't even, in my experience, really know what tests to write for distributed systems yet. They'll get better at that. But really to know that your code is correct in an era where you're not reading the code or writing the code, you want your compiler to make sure it's correct. Right? That's kind of where we are with this stuff. This is why it's better to use Rust than Python when you're using an agent, right? The Python could have all kinds of bugs and you really can't know. 

[0:28:18] SF: I guess if I'm going to use this, if I'm using Hydro as a developer, what is that process like? How do I kind of plug it in? Where do I start? 

[0:28:25] JH: Yeah. So there's a Greenfield version of this where you go get the Hydro template, you say cargo generate with your Rust tool chain, and it generates you a project and you start writing Hydro. And that's basically a dialect of Rust that looks a bit functional. So it's a lot of maps and filters and things like that in big, long chains. It looks actually quite a bit like Spark or Pandas. Except instead of handling data streams, you're handling requests, packets, stuff coming off the network or off the I/O channel or what have you. But there's also sort of a brownfield use of Hydro where we don't need to control the network endpoints. You can just have input-output streams, essentially, of events. And then you embed inside some Rust code. And the Hydro bit is Hydro-y, and the Rust bits are still plain old naked Rust. 

And the main thing to think about is Hydro does have this global programming model, which is kind of nice. So where do you want to apply it? And I'll give you an example. A lot of distributed systems have sort of roles defined for them. If you know your Paxos, there's like proposers and acceptors and learners. There are these different roles. And we write a different piece of code to run on those different roles. They each run on their own machines, right? So we write some code for the acceptor. We write some code for the proposer. In Hydro, the code's distributed. So instead, you write functional units. You don't write roles. You write like a task. And the task might be something like take a vote, which is a key part of these agreement protocols. 

Take a vote involves many machines. One machine says, "I would like to take a vote and broadcasts it to many machines." Then it collects the votes from the machines and counts them up. And then it announces the result. That could be just a single Rust function in Hydro. So the modularity then becomes around like reusable distributed components. And that's a nice thing to think about when you're retrofitting your system with some Hydro is like, well, what are the functional units of this distributed program? And how can I tear them out of the little programs I hand-wrote for each location and turn them into a reusable service in a library? 

[0:30:22] SF: Would this only work if you built this in Rust because of the properties of how Rust works? 

[0:30:27] JH: I think it would work quite well and arguably better in a language like Scala, actually. One of the things that we often are missing in the Rust ecosystem is dependent types, which is a programming language nicety that I'm not going to get into the details of, but it does make some of our Hydro internal code kind of hard to write. Hydro would have been easier to write in Scala. 

There are many advanced languages that would be amenable to this kind of programming. The key issue is something called quoting. And the idea is you write some code and we tear it up into code that's going to run on every machine, right? So you wrote a function for vote. It really eventually does have to be binaries that run on every individual machine. That code that you write, we're going to need to kind of take it apart. And that taking apart business in our compiler is easier in some languages than others to do it natively. 

[0:31:15] SF: But to use this, am I only basically building a certain part of my application in Rust using Hydro, and I can build other parts of it using whatever language I choose? 

[0:31:27] JH: Yeah. I mean, keep in mind, it's the distributed systems world. So you can always take a piece of your system and turn it into a service. And then the endpoints are GRPC or REST or something like that, right? They can be cross-language. Certainly, that's always an option. You can write a microservice or tear out a piece of your system and turn it into a service and write it in Hydro even though your service was written in something else. 

Other than that, you're doing the moral equivalent of that in-memory. Say, Java native interface is an example. You can take Java code and embed Rust in it and the rest could be hydro. But if you're going through JNI, it's a bit of a drag. 

[0:32:02] SF: Yeah, I see. You talked a little bit earlier about this simulation environment for testing. How does that kind of work at a high level? Where am I running that? 

[0:32:13] JH: Yeah, great. It works on your laptop. So when you write unit tests in Hydro, you'll write them as simulator tests. And so you'll write a unit test that says, "I want the sum of the numbers that I put in to equal the sum of the numbers that I put in." You know what I'm saying? But the way it's going to get simulated is all orders of the messages in your program to do the summing that we talked about will get run by our test harness. And you'll get an answer not just to one run of that, but all possible runs of the system with all the orderings of the messages. And it'll check every one of them for your invariant. But it's just doing it on your laptop. It's a simulator. It just looks like a unit test. It's just a super great unit test. 

[0:32:55] SF: We were talking about Rust and you were mentioning that some of this stuff might be easier in Scala. I think Scala runs on JVM. Java still dominates the data infrastructure world. But do you see Rust potentially displacing some of that over time because of these key properties that Rust has of these compile time guarantees? 

[0:33:15] JH: I mean, I'll say for the record that using the JVM for infrastructure was a terrible idea. Terrible idea. It was a movement that was going on in the big data world because I don't know why, because it was a terrible idea there too. But it kind of was happening around the same time the cloud was emerging. 

And so what ended up happening was the early clouds got polluted with a ton of Java services, and they're slow and they're ugly. It's nobody's favorite language. The garbage collection, despite decades of trying to make garbage collection free, is not free. And when it runs at an inconvenient time, it looks like a failure. 

And distributed systems, the number of "failures" your distributed system deals with because the JVM is going to be much higher than in a non-JVM language. And just the overhead of most code in Java by the time you're done, and it's written by normal developers, not by super duper Java experts, it's slower than Rust code. 

I was brought up as a C programmer. And the downside of C, of course, is bugs are just all over the place and you don't know where they are. Rust has been a beautiful bringing together of how do we get memory safety and a high-level language with a nice type system that will really help you find your bugs. And I just see no justification for writing anything over the JVM anymore at all. And my experience at most of the companies I've talked to, AWS among them, is they're in the middle of Rustifying as much Java code as they can or C-sharp code or what have you, because it is a language with fewer bugs. And the agents can do this semi-automatically. So let's give the agents a really tight guardrail language to re-implement things in a fast language. 

[0:34:54] SF: Yeah, I mean, I think Google's kind of, I guess, Goified by their backend. But I wonder if part of that is like the product of when a lot of these systems were coming up with where the state of programming languages at that time. My hypothesis would be like Java gave just an easy way to run these applications across variety of different types of machines and operating systems at that time. And we didn't have as many good ways of doing that. 

[0:35:18] JH: I mean, Oracle used to run on everything. It was written in C. It was expensive to do that, but it's not hard. I mean, it's hard in ERGs. You had a lot of programmers typing a lot of code and testing it. But portability, I think, was a sham.

[0:35:35] SF: Yeah. 

[0:35:35] JH: I can look at it that Hadoop is what made Java so successful because it was so wildly hyped. And it was just a mistake. The Google people didn't write MapReduce in Java. Certainly not. They wanted performance. They had to deal with all the world's data. They weren't going to do it in Java. Hadoop was intended to be kind of an experiment. It just kind of took off and the hype began. And I think Java really got a huge boost from that. Similarly with Scala and Spark, by the way, but it wasn't a big enough movement to make a language as hard as Scala popular. Scala was like pretty hard to approach. Java was much easier to approach. 

[0:36:09] SF: Yeah, I mean, I think if you look at the history of some of the languages that are now the most dominant languages in the world, Java, JavaScript, probably Python, their beginnings were not focused at all on the types of workloads that they're handling today. And I think you can make pretty strong arguments that they were not the right - hindsight being 2020, the right languages to put basically the entire internet behind. 

[0:36:31] JH: Yeah. I'm sensitive to this. A lot of technologies emerge from lo-fi, right? Because they get adoption, and adoption drives usage, and usage drives improvement, and we iterate to a better world, right? It's just that if you do that without counting the consequences, you can really drive yourself into a corner. And then somebody has to, at some point, say, "Can we do this better?" Right? 

This sort of lo-fi to medium-fi movement is one that I hear a lot. It might be a good thing for certain startup companies, for example. You start at the bottom of the market with an easy to adopt low function solution. Somebody else will probably start at the top of the market selling a very high function thing for a lot of money. And then very often the market just bifurcates. There's just two winners. There's the kind of high-end winner and then there's the low-end lo-fi winner. 

But it's pretty rare in my experience that the lo-fi thing evolves into a really good solution. And so when we talk about system infrastructure like the cloud, I don't know why you embrace lo-fi technologies. Seems crazy to me. 

[0:37:31] SF: Going back to Rust, Rust clearly has some great things about it, but what have you found the hard parts of actually building in Rust? Where does the language kind of fight you when you come from maybe some of these other programming paradigms? 

[0:37:44] JH: Yeah, I mean, I suffered with the Rust learning curve pre-LLMs like a bunch of people. I had a master's student who tutored me, became a software engineer on my team. I hired him. And we paired for a year or something where he's really, really good at Rust. And I'm an old guy. I learn slow. But I was learning to write pretty fancy traits in Rust. And we were doing some pretty fancy stuff. 

And the borrow checker difficulties actually didn't come up so much in the kinds of code we were writing. So I'm still pretty bad at the Rust borrow checker bugs. And then along came LLM coding agents. And none of those frictions really matter anymore. It's like, "Oh, you forgot to put a lifetime on this variable. That's okay. I'll do that for you." It's just a lot of those learning curve hurdles get smoothed out by the LLMs. And now the challenge is not so much is Rust hard to write, but will you take the time to learn how to read it? 

[0:38:34] SF: Yeah. I even think with a lot of the - Historically with programming, we've invented a lot of different layers of abstraction, different frameworks and things like that to hide away some of the details. But if LLMs are really eventually at the details, I would presume a lot of those types of things go away because it's like you're going to probably get better performance if you just write it natively than - if you use the front-end example, if I could natively write everything in JavaScript, not having to go through some buffer of like a React framework or something like that, it's probably going to be more performant. The assumption is that I'm not a good enough programmer to do that in JavaScript without tons of bugs or poor performance. So I can rely on React to kind of do that a way where people have already solved those problems for me. 

[0:39:18] JH: Yeah, I think that's a reasonable hypothesis as a starting point. But what we're learning now with agentic programming sort of 12 to 24 months in is that maintaining that code, even with the help of LLMs, is hard. So if you write very verbose, very detailed code that doesn't use componentry, you can get it right. But keeping it right as you grow it gets increasingly difficult for the LLMs. 

Modularity and elegance and simplicity actually still matter quite a lot, I would say. Language dialect, less so. But you can get an LLM to write you some great assembly code, right? You just don't want to maintain that. And probably, if you ask the LLM to change it, it'll introduce bugs. 

I think you know the key components are is the language going to keep the LLM on the rails, right? Guardrails are just so important. And is it evolvable? Which is the second part. And that part, I think, does still benefit from nice design patterns, libraries, all that good common reuse stuff that we use as humans. 

[0:40:16] SF: With this project, Hydro, where it started at Berkeley, now you're doing this at AWS. I guess how does it change when you take a research project and you turn to making that essentially like a production? Those are pretty different. I was a researcher. I know a lot of the stuff that I built in research. I would now build it the same way if I was building it as a product. So is there a lot that has to change in order to do that? 

[0:40:40] JH: Yeah. But I think you have to see it in context. Hydro is the result of 25 years of work in my group, something like that. We were not ready to just go to industry 15 years ago. We were ready to go to industry three years ago. So it took a long time to do research. And part of that was I went off and did our data wrangling startup that had nothing to do with this whole agenda. And so I lost the better part of a decade to other interesting topics. But it was still many years of work. It was sort of three iterations of languages, I'd say. 

Now that we sort of had ironed out the model and we really believed it, going to AWS was eye-opening in ways I didn't expect. I expected we'd get beat over the head with performance. No. We perform just fine. In fact, we're usually way faster simply because we're Rust and we're replacing Java. We're using the Rust compiler. I will say the fellow who taught me Rust, he's an incredible Rust programmer, and he's generating the code in Hydro. So he's generating code that the Rust compiler is very good at inlining. And so our Rust code is actually very, very fast. 

Performance, we could think about that on campus, and we could measure cycles and all that stuff. What we didn't think about was, oh, there's some weird brownfield code on a router, and they want to use Hydro, but they need it to interoperate with QUIC, which is a networking protocol, which is used in a minority of settings. But if you really care about stuff, it's used. It's like, oh, well, we had baked in the networking layer into Hydro, and we never thought about it running separately from its own networking. We have to strip out the core Hydro stuff from its event loop and its network handlers. 

There's stuff you learn about, I would say, interoperability in the real world that just wouldn't occur to you. Sitting in an office, just designing something beautiful. I think that's the biggest lesson is really interop. And it's not, in this case, just interop of protocol. It's interop of my code sitting inside your code. It's like compiler interop inside a program. That's been a lesson. 

Deployment would be the other big lesson. We started out thinking, "Eh, we'll write some shell scripts and deploy some stuff." And then we're like, "Okay. Well, if we were in the real world, we'd use like Terraform, Kubernetes, something like that, ECS on Amazon." But that's just because those are a little easier. 

No, actually, when you start looking at deployment, you really do want to think about it very carefully and it changes what you do. So I'll give you an example like off the bleeding edge. Hydro Simulator now allows you to simulate multiple versions of your program at the same time. So one of the things about deployment that's true in the real world is that you do incremental rollout, which means you have multiple versions of your code running simultaneously. 

Okay, now all the bugs you worried about in v1 and all the bugs you worried about in v1.1. What about when they're talking to each other? Have you worried about the bugs in V1 and V1.1 working together? And we will simulate that for you. Again, we'll just go through all the possible configurations and run a whole bunch of tests for you automatically in our simulator. So you can do deployment tests before you launch that incremental rollout. You can sim test to see if it's going to be safe. That's not a thing compilers or tests usually did. And it's not something we thought of in the office. We only thought of it sitting at AWS doing real stuff. 

[0:43:46] SF: What about advising? You've advised a number of students over the years. You have some very successful students. Three of your students won the Jim Gray dissertation award. I'm sure many others are now in prestigious positions. Is that something that you've been able to translate into your role in industry as well? Whatever your approach is to advising and helping these students be successful, is that a skill that's been transferable as well to industry? 

[0:44:13] JH: I hope so. I'll say a couple of things. First of all, the success of my students is largely biased by my having had the good luck of getting a job at Berkeley. And the students at Berkeley are tremendous. So many of these successful people are successful because they're amazing, not because I mentored them. So I just want to flag that. 

But he other thing I want to flag is that a cool thing about this AWS setup is they think it's beneficial for me to stay at Berkeley advising PhD students. So I actually still have four students in a research group, in a lab, in an office, on campus. And we continue to push the envelope there on things that are not on the time horizons that AWS operates on. And so they're essentially giving a little bit of my time to have a longer horizon in the open source world of research. So that's cool.

[0:44:52] SF: That's fantastic. 

[0:44:52] JH: Having said that, there's a lot of mentorship that goes on in companies, as you know. And I do think having mentored students is helpful. I also think I'm learning things about it's different when you're building together than when you're just kind of educating and PhD students are exploring ideas because they'll often go off and do speculative stuff. And it doesn't have to be solid. It just has to be interesting. So there's just different criteria that you're optimizing for on campus than you are in production. And learning the craft of software engineering for production is a bit different than learning the craft of research. 

I'm definitely learning about that as I go. My startup years, which were many, helped with that, but I didn't run engineering at my start-ups. And that's kind of where this kind of mentorship that I'm seeing at AWS is, is inside of engineering. I got a lot to learn, but I hope I can give back some stuff. 

[0:45:43] SF: Last question. I know you're a jazz trumpeter. You've played with some incredible musicians. Does any of that map to how you think about distributed systems? Has that been a transferable skill for you? 

[0:45:54] JH: To distributed systems? I'm not so sure about that. But I think to an outlook on the balance between structure and creativity, Jazz sits in a particular place because it's so improvisation-focused about you have a skeleton of a structure, but you don't worry about the details till the very last minute. You know the chords, but you haven't picked the notes yet. That's your typical jazz solo, right? You haven't picked the rhythms. You haven't picked the notes. You've just got the chord changes, the harmony. 

I do think that has something to do with my way of doing computer science, which I'm pretty intuition-driven, and I'll skip the details for a while. I know people who are much more disciplined up front. They remind me of classical musicians. You have to play every note exactly the right way, and that's the value of classical music in some sense. It's, of course, not. It's the table stakes for classical music is, at minimum, you play the notes correctly. Maximum, obviously, you're making music. In all of these forms of music, the important thing is just to make music, connect, create experiences. But jazz does it with less attention to setup. 

And so I do think there's a role for that in research, for people to be thinking kind of atmospherically and letting the details fill in as they do. Perhaps my best example of that, my master's advisor was Michael Stonebraker, who won the Turing Award. He's the Postgres guy, among other many feats. That guy could not code his way out of a paper bag. He did not know how to write software. He didn't use LaTeX because he didn't use LaTeX. He was too old for LaTeX. He used nroff. The guy did not program. And he was right like most of the time. And as a graduate student, it drove me bananas. How could this guy who doesn't know how to program be right all the time about these architectural decisions? And he just is really good. 

So I think there's room for that in the world. And we give Turing Awards to people who behave like that. So it must be one of the things we have to do in computer science to do well. And I do think there's a connection to the variety of ways to make music and the variety of ways to do computer science. Distributed systems, I don't know. 

[0:47:54] SF: Awesome. Well, thanks so much for being here. 

[0:47:56] JH: Yeah, this was a pleasure. Thanks for letting me be here. This was great. 

[0:47:59] SF: Cheers. 

[END]