EPISODE 1807

[INTRODUCTION]

[0:00:00] ANNOUNCER: Modal is a serverless compute platform that's specifically focused on AI workloads. The company's goal is to enable AI teams to quickly spin up GPU-enabled containers and rapidly iterate an autoscale. It was founded by Erik Bernhardsson, who was previously at Spotify for seven years where he built the music recommendation system and the popular Luigi workflow scheduler. In this episode, Erik joins Sean Falconer to talk about the motivation for founding his company, the market gap in ML and AI tooling, optimizing container cold start, Modal's interface design, and more. 

This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him. 

[INTERVIEW]

[0:00:55] SF: Erik, welcome to the show. 

[0:00:57] EB: Thank you, it's great to be here. 

[0:00:58] SF: Yeah, thanks so much for being here. I was diving a little bit into your background preparing for this. And so it seems like you spent a lot of your time working in data throughout your career, which also kind of matches my own experience. You previously were at Spotify for a number of years. You were the CTO of better.com. Now you're the founder and CEO of Modal. Were there certain things in these prior roles that led to identifying some sort of need for Modal? What's the story behind essentially going off and deciding to start this company? 

[0:01:26] EB: Yeah, for sure. The answer is yes. And the long story is I was at Spotify for seven years, built the music recommendation system. But as a part of building that, I also realized there's kind of a general gap in the tooling. I ended up building a vector database called Annoy that we use today, and also a workflow scheduler called Luigi that very few people use today. But generally realized, as a part of building all of that stuff at Spotify, I also did a lot of other stuff, that this is very little tooling in data AI machine learning. There's more today, but I still never really felt like much later, like in 2020, 2021, when I started thinking about building a company, I realized there's kind of a gap in the market for a tool I always wanted to have myself. That was kind of the genesis of Modal. It's almost like building selfishly for what I always wanted to have throughout my years at Spotify, building the music recommendation system, and also, to some extent at Better, I was a CTO, a little bit more like general role. We spent a lot of time thinking about platforms and data and stuff like that too. 

[0:02:22] SF: Yeah. I mean, sometimes people talk about how the discipline essentially, or like the tooling and the things that are available for data engineers, lags somewhere five years maybe behind more traditional application development. Where would you say that something like ML engineering from an applied sense of being able to actually run these things in a company that maybe user facing lags behind what we think about from traditional application engineering? 

[0:02:50] EB: It's definitely behind. I don't know what the number is in terms of number of years. To me, it all comes down to developer productivity and how much time is like wasted on like tooling stuff versus actually delivering business value. And the only way I found that I think it's like somewhat correlated with developer productivity is like fast feedback loops. How fast are your feedback loops, right? And it's not like necessarily a perfect metric, but I think it sort of speaks a lot about like developer productivity. When you ask developers if they feel productive, a lot of it, I think, it comes down to like having fast feedback loops. 

And when I look at other types of software engineering disciplines, if you look at like front engineers - I don't know if you do a lot of front end, but I kind of enjoy it because you have editor in one window and you have it browsing the other windows. And today when you're writing frontend, it's like so crazy fast. You save the code and it just like automatically just like reloads the other screen and you see it right away. You have this like sub-second level feedback loops. 

And I think there's something similar about like backend, maybe it's a little bit different, you run unit tests or whatever. But with data AI machine learning, all of that stuff, I would almost argue like we're taking a step backwards in that. As much as I love the cloud and I'm a massive proponent of cloud and it gives us tremendous power to do stuff, for whatever reason, using the cloud. And suddenly, you have to build like Docker containers, you have to trigger things, you have to mess around with infrastructure and stuff just being this massive friction in that loop when you're working with AI and machine learning on the engineering side. And that was like my sort of frustration. A lot of like what Modal came from was I just want it to be fast. And maybe I have lack of patience, but I want to write code and then run it in the cloud in less than a second. That was like my kind of starting point is like, "How do I solve that problem?" 

[0:04:32] SF: Yeah. To your point about frontend, there is something - I don't know. There's just immediate gratification. I can go in and change a string and I see it. And I think that's probably why you see a lot of people who are starting to get into engineering probably start with frontend because you have sort of that immediate feedback loop. And also, even from languages, adoption of languages, there's sort of a more immediacy of getting to that like aha moment, those feedback loops, perhaps using something like Python versus using a lower level language where you might struggle a bit more to just kind of get started. 

And even older languages, now, even things like Java and C# have tried to reduce sort of that barrier to entry of doing that simple "Hello world!" essentially equivalent application because of how easy it is in some of these other languages that have severely simplified it and then went to a high adoption curve. 

[0:05:21] EB: For sure. And kind of is a side topic, one of my beefs is Rust. I love Rust. It's a great language. I think one of the things they got wrong is like it's just - they made a language that makes developers very productive in certain ways, but then you have the compiler getting in the way. If the compiler was just fast, I would actually really, really enjoy writing Rust. Now it's like I enjoy it. But to your point about the older language, I used to write a lot of C++ and having that compilations thing take minutes every time you want to just run. It just takes the joy out of coding. 

[0:05:54] SF: So, I mean, can you break down a little bit more about what are you essentially doing with Modal? You want these super-fast, essentially deployment times four ML workloads. But what's that end up looking like? 

[0:06:05] EB: Yeah, totally. Yeah, and maybe we should take a step back or talk a little bit about making developers productive and happy. But what is Modal, right? I tend to think a lot of running data, AI, machine learning stuff today is about building code and shipping it into the cloud and scaling it up, running it on GPUs, mapping over large data sets, etc. That's originally what I set out to build. And as a big part of that, as I mentioned, I wanted the feedback loops to be very fast. 

We spent a couple of years focusing on that platform as kind of a core concept. We felt there's such a huge opportunity. This is also during the ZIRP. We had no pressure to make money. But two years in, where we started seeing a lot of traction was with GenAI applications. And so a lot of our use cases today that we see that as driving a lot of the growth of Modal is various types of GPU inference use cases in GenAI. Large scale, particularly video-audio music, people who deploy models, often proprietary models that need to scale up to sometimes thousands of GPUs and they don't want to handle the infrastructure or they're running very large spiky batch jobs. 

Modal is both sort of an infrastructure provider, in the sense that we have a big pool of compute in the cloud, a lot of GPUs, a lot of CPUs. And then kind of the separate side of Modal is we have an easy to use Python SDK that lets you iterate very quickly. And in a way where you don't have to install anything, you don't have to configure anything, you just have to write a little bit of code. And then when you run that code, it runs in Modal. And that makes it very easy to take a particular machine learning code and scale it up. 

[0:07:41] SF: And then in terms of this big cluster of GPUs, essentially, your customers are sharing. 

[0:07:46] EB: That's right. It's a multi-tenant model, which is different from, I think, traditional applications. I tend to think it's the future of the cloud. There's so many benefits of us having a shared multi-tenant pool. Just to give you one example, if someone needs 100 GPUs, we can often get you that in a few seconds because we can pull all this variable demand, it means we can run - do capacity planning at a very large scale. 

And so there's been many benefits of having this shared tenancy model. Not to mention also the fact that there's zero installation. Well, you have to do pip_install Modal. But then once you do that, you can immediately start running code because we manage the infrastructure in the cloud. 

[0:08:24] SF: For people who are managing their own GPUs, do people end up typically underutilizing the GPUs they have available? 

[0:08:32] EB: Totally. Which is another thing that Modal solves beautifully is that, because of the multi-tenancy, we have a model where everything is pay as you go. It's all usage-based. So we only charge for the time the GPUs are actually running. There's no capacity planning with Modal. You just start running model and then we charge you per essentially GPU second or CPU second. Actually, I should say, there's many use cases where we don't use GPUs with Modal. 

And again, kind of going back to the multi-tenancy model, part of the reason why we can offer this is we can pool a lot of people's very burst workloads and run an underlying shared compute pool. The other thing I'll say I spent a lot of time on is doing things very fast container cold starts. Because in order to have a usage-based pricing model, you need to be able to start containers and stop containers very quickly. 

Interestingly, that is the problem we have to solve in a separate context, which is we talked about this previously. I wanted the ability for users to write code and immediately run it on the cloud. And that's actually the same problem, fast container cold start. That is another thing we've had to spend a lot of time on. We built our own file system. We built our own container runtime. We built our own scheduler. And all these things lets us have A, very fast feedback loops, B, a fully usage-based pricing, and C, I guess, fully managed infrastructure running in the cloud so people don't have to think about it. 

[0:09:48] SF: Can you break that down? What's happening essentially behind the scenes? I've done pip_install Modal, I write some Python code, presumably I tag it in some fashion so I know what I want to run within the cloud. I run it. But what's essentially the magic that's happening between my machine and being able to run this on Modal? 

[0:10:07] EB: Yeah. Modal is an SDK, which means you import Modal. And basically, the way to think about it, I think the easiest mental model to think about Modal is function as a service is similar to AWS Lambda, if you're familiar. The idea is that you can take any function in Python and turn that into a function that runs in the cloud on invocation. You can have all these functions in Python. Then you can say, "This one should run on H100. This one should run on whatever, T4." Different container environments, different even drivers, right? And you have to specify that in the code using decorators that you apply to these functions. And then you can have these functions call each other. And so you have this function as a service programming model. 

And under the hood, the way that works is we take the code, we're able to build a container image and launch that container image in the cloud in about a second, right? And more for very large images. If you have an image that's 100 gigabytes, it might take a few seconds. But we spend a lot of time on how do we take the code on the local computer, stick it in a container in the cloud with arbitrary dependencies and launch that on a worker in the cloud that might have GPUs or many workers? Maybe you need 100 GPUs that we need to spin up that container 100 times over on different workers in the cloud. 

[0:11:18] SF: Does each function that I'm specifying - let's say I have function one, function two, but I want to run them on essentially different GPUs, do those end up being separate containers that have to be deployed? 

[0:11:29] EB: Yeah. Every function ends up being a separate container. In fact, every function is auto-scaling. If you start issuing many requests to the same function, we will just auto-scale it up to as much as it's needed in order to serve all those requests. 

[0:11:42] SF: In some ways, it sounds like as a developer of these, it's almost like you're coding a monolith, but then the deployment is able to automatically sort of scale this out as more of like a market service architecture. 

[0:11:53] EB: Yeah, sort of. Yeah. And I think that's one way to look at it. Like every function ends up being its own kind of container. I mean, you can have like a lot of code running inside a single container, but you can have very isolated pieces of code. You can have one container running Python 3.9 and another container running Python 3.12 calling each other, just like a normal Python function call. You have this convenience of it just feels like local code, just functions calling functions, including handling tracebacks and exceptions and all these things, right? It just feels like Python, even though it's like running in a distributed way in the cloud. 

[0:12:28] SF: Right. How does that function calling happen behind the scenes? Essentially, are you using some sort of like gRPC call? Or what is essentially allowing you to call functions across these containers? 

[0:12:40] EB: Yeah. Under the hood, we use gRPC just for internal communication. And also, the client library talks to the server using gRPC. There are a few different layers. But also, within Python, we use cloudpickle just serialize all the payload and send it between containers. But again, that's not something that a developer necessarily has to think about because it just feels like you're just calling a function in Python. We handle the serialization, all the exception management, all of that stuff. But yeah, there's a lot of work, obviously, under the hood. Basically, it's all complex queuing theory and scheduling and doing that fast at a large scale. I mean, I think we're serving something like 100,000 requests a second right now across the scale of Modal. Managing the state management and the scheduling and constantly scaling up and down. Obviously, there's a lot of work to build all that infrastructure. 

[0:13:25] SF: Yeah. How's the state management and persistence in these distributed applications work? 

[0:13:30] EB: I would say Modal right now focuses on compute. There is some level of state management. You can create simple distributed. We have a primitive to create a distributed key-value store and also queue. Most of what people use Modal for today GenAI, where GenAi is actually kind of like a weird use case in many ways because it's extremely compute-intensive, but it's actually kind of low IO. If you think about stable diffusion for instance, you send this tiny piece of a text to a GPU and then that GPU does like a trillion operations and then it sends your little JPEG back that's like a 100k, whatever. And that's like the type of applications that we do really well. It's like very compute-heavy stuff, but not necessarily like super IO-intensive. We're not necessarily trying to compete with Spark in that sense, which has these operations, you can wrangle very large data sets and stuff like that. There is some state management in Modal. 

We also have a distributed file system you can set up in Modal, and you can attach to all the containers basically just like a local POSIX compliant file system, which means you can basically interact with it just using normal file system operations. But anything you read or write is globally distributed across all the other containers. That's another way you can also work with large datasets in Modal. But we don't necessarily handle the sharding and partitioning and shuffling. That's not something we've built yet. 

[0:14:46] SF: When I think about something like an IPAS for traditional applications, a lot of times people run into these challenges eventually where essentially the abstractions start to get to a place where they don't work. They want to get in there and sort of adjust things, tune it to their specification. Let's say we use a certain scale and it releases this graduation problem. Is that a challenge that people building a Modal in this world face? 

[0:15:10] EB: I think so. I mean, I think that's something I've spent a lot of time thinking about. It's like the right abstraction sacrifices a little bit of power, but makes the remaining stuff so much easier to do. I think any obstruction layer you add, of course you're going to sacrifice a tiny bit of capabilities. But by doing that, it turns out you can actually do the remaining stuff make it so much easier to build. 

I think the bad abstractions, they basically sacrifice 80% of the capabilities and then turns out they're very limiting. How do you make Modal fairly general-purpose so that we can do almost anything in Modal? I think that's very hard. And that being said, I mean, I think so far, this is something I'm obsessed with, by the way, thinking about the right abstractions and thinking about the developer experience and making sure Modal is fairly general purpose. That model is not necessarily super frameworky. I think there's sort I think I don't like about a lot of modern framework where either they're very config-based or they kind of lock in a certain way. Modal is opinionated. But I want Modal to be like a bunch of Lego blocks, and then you can like take those Lego blocks and build whatever you want. 

And I think to a large extent, it lets people do that. We've always thought about programmability. We've always thought about making it possible to run any code you want. We're thinking a lot about doing non-Python stuff, for instance, as one example. I don't know. I mean, I'm obviously biased. But I tend to think that Modal is a platform that doesn't sacrifice that much capabilities. You can do almost anything in Modal that you could do just using lower-level cloud primitives with 5% of the effort. 

[0:16:52] SF: Yeah. What are the typical GenAI applications, workloads that people are running on this? Is this primarily people running their own model and they want to be able to run inference across these GPUs? Or are there other types of workloads that are also there? 

[0:17:05] EB: Yeah. One user that I love, one of our customers is a company called Suno. They do AI-generated music. They run a lot of their inference on Modal, very large scale. Basically, what they run on Modal is GPU-based. they have their own proprietary model and it generates music. Super cool application. 

I think a lot of our customers fall into that bucket. It's like there is a proprietary model. It does some very cool magic stuff on a GPU, typically in audio-video, image stuff music. It's been kind of the domain we've seen most traction. There are other use cases too. We've recently seen a lot of traction coming from computational biotech, which I think is super exciting. Protein folding, multiple sequence alignment. There are all kinds of medical imaging, processing like very large data sets applying computer vision to assays. I don't know too much about the field. But I find it incredibly exciting. 

And kind of in that vein, we've seen people use model for like geospatial analysis, physics simulations, like turbulence stuff. And then there's a lot of just like kind of people who just like the developer productivity and they don't necessarily run big things, but they have a little web scraper running in Modal or a little web server. There's a lot of that stuff. The goal was always to build a fairly general-purpose platform. 

We've found initial kind of core product market fit in the GenAI. That's been the main use case for us. But there's just so many other things that people always surprised me. I was just talking to someone the other day who's running a chess engine on Modal. I don't know why, but it was really cool. 

[0:18:37] SF: I mean, some of those examples you gave are pretty high compute examples, which, yeah, I think make a ton of sense. Are there certain workloads that don't make sense? I guess you mentioned things that maybe require a high IO are probably maybe not the right fit. 

[0:18:50] EB: Yeah. High IO I think - but we're pushing into that. I think it's going to be a big focus for 2025 to also handle. Similarly, I would say very low-latency things is something we're very excited about. Right now, there's an overhead in the system of like 100, 200 milliseconds of every function call. A little bit less if you run it in the same region. But that's enough for a lot of use case, especially in GenAI, like stable diffusion. No one really cares if it takes 200 milliseconds. Because the inference in itself takes a couple seconds. That's pretty negligible overhead. But it's not enough for something like real-time streaming of audio or real-time video. That's like another thing we definitely want to push into over the next year is how do we get the overhead of the system down to 10 milliseconds, five milliseconds, whatever. 

[0:19:29] SF: In terms of getting to a place where you are able to run these former things like real-time video-audio streaming, what do you see as the main technical hurdles that you have to solve in order to reach that kind of performance? 

[0:19:41] EB: Today, it's mostly about geolocation. Or basically, we run - 

[0:19:45] SF: Shortening the distance. 

[0:19:46] EB: Exactly. The speed of light is pretty high. As you may know, it doesn't take that many 100 milliseconds to send something to Australia and back to the US, 200 milliseconds or whatever. But it is a challenge when you're doing real-time stuff. A big challenge for us is while we have a distributed data plane, all our workers run in many different regions, many different cloud providers, our control plane right now is not distributed. 

One of the things we want to do in 2025 is decentralize the control plane so that we can use smarter ways, basically, kind of route to multiple edge. I feel like the word edge is overused. But you're running Modal in many different regions across the world, also the control plane so that we can route things and execute it faster. That's a big rearchitecture. 

[0:20:31] SF: In this multi-tenant architecture that you have today, you have control plane, you have your data planes. Your data planes are distributed across these regions. Is the primarily the sort of reuse of resources happening in the control plane? 

[0:20:44] EB: Yeah, the control plane makes all the decisions, right? And it sort of has the global state of the world, which is like another thing another thing to some extent. When you have a very large worker fleet and many different scheduler running in different regions, I think you also need to change the truth of the system to be owned by the workers themselves because they always know the latest. Because, again, speed of light is not always as fast as we wish. There's a lot of that state management. Where do you make the decisions? Who has the authoritative view of which worker is running which containers? And how do you propagate that information? I mean, these are like hard, technical challenges. But luckily, we like those at Modal. 

[0:21:22] SF: Was the plan from the very beginning always to build this out as this like multi-tenant structure? 

[0:21:27] EB: It is challenging sometimes because I think people are just not entirely comfortable with that model yet. But I don't know. I've been coding for 30 years. And I remember when the cloud came in 2007 with something like that. AWS launched or at EC2, I think. Something like that. 2006 maybe. And my first thought was like, "That's insane. Why would I put my code on someone else's computer?" And then a few years later, I was doing it. I was like, "This is kind of nice. I like this." It took a few years. I mean, it's still taking a few years. Many people still run on-prem, right? But I think it's just been, obviously, people are seeing that the cloud makes sense for a lot of stuff, even though it's arguably a shared resource. 

[0:22:08] SF: If you look at a company like Snowflake, they were like, "Hey, we're going to build a cloud-native database." And at that time - which maybe doesn't seem that crazy now, because everybody's doing that. But at that time, it was like what do you mean you're going to take this thing that you run in, I don't know, on-prem Oracle today and move it to the cloud? We're not going to that. 

[0:22:28] EB: Yeah, it's funny, you took words out of my mouth because I was just going to talk about Snowflake next. It's funny. I actually interviewed with Snowflake 2012 and they told me the idea. I'm like, "I don't think this is going to work." And then I turned out the job offer. It was obviously a very terrible decision. But I think what Snowflake showed is like beyond just the cloud vendors. Also, Snowflake showed that you can be infrastructure as a service and host people's data. And eventually, people will be comfortable with that. 

I don't know. I think security compliance is like shifting. I think people, a lot of customers today, they don't necessarily worry about the fact that something is multi-tenant. They worry about like best security practices. And we take that extremely seriously. We think a lot about how do we encrypt all the storage? How do we encrypt all the data in transit? How do we run the containers in a way where like it's impossible to break out of containers? Those things are very, very important. Exactly where things are running in terms of networking or in terms of VPCs. And I don't necessarily think those are the prime concerns. And I think over time, there'll be even less of a concern. Multi-tenant is definitely like a change in mindset. It's a change in how security and how compliance operates. But I think the trend is our friend here. 

[0:23:40] SF: Yeah. I also think for a security perspective, when you look at things like breaches or other security vulnerabilities, I don't see a lot of reports that are a result of some sort of compromise, multi-tenant, cloud offering. It has a lot more to do with just, a lot of times, simply human error of like, "Hey, we accidentally committed our API credentials to GitHub and now global online. Of we have an unencrypted log file that has the social security numbers of all our customers in it or such things like that. 

[0:24:10] EB: Totally. And even network segmentation, I think, is kind of to me an outdated model. There was like a hack a few years ago where Target got hacked because it turns out their HVAC system was running on the same network and there was a vulnerability and whatever. The HVAC system, someone was able to get into. I think network segmentation to me is like a never a strong security model. And I'm kind of glad like people are not - there's like the BeyondCorp and Zero Trust. That to me is very clearly the future of cloud security. 

[0:24:39] SF: In terms of things like capacity planning for inference, even outside of Modal, is capacity planning for inference and GenAI applications a fundamentally difficult task for businesses today because it's hard to know in any given time how many tokens you're going to be generating and what the workloads actually look like? 

[0:24:57] EB: Yeah, I think so. Because there's so many sources of noise, right? I mean, first of all, you have sort of a daily variation, like kind of a sign curve over the day. But then often you have kind of a noise. I mean, it's like a plus on distribution at any point in time. I have always like a little stochastic noise. But then there's so many other things, people running batch jobs. Suddenly, you launch something and it goes viral in Hacker News. There's all these different sources of noise. The distribution is extremely fat-tailed. And so it's fundamentally hard to plan, do capacity manage for inference. 

I think for training, it's a little bit easier. When you're training a model, you can just like buy 1024 GPUs and just like kind of make sure the GPUs stay warm. But for inferences, it's much harder because you can't fundamentally plan. And I think the thing that also exacerbates that is sort of the only way, in the past at least, up to very recently, to get high-end GPUs was to make big reservations, long-term reservations. Three year or whatever, right? And that's kind of a hard matching problem. You have this unpredictable demand where you have to go out and buy fixed capacity, which means that a lot of people are running things very poorly utilized. 

However, I actually tend to think resource pooling is kind of a free lunch in many ways. If you take a lot of people's noisy workloads and you aggregate them, you can run the aggregate at much, much higher levels of utilization. That is one way we can save a lot of cost for people. Even though, frankly, our prices is sometimes higher, we can still save money because they run things at effectively 100% utilization from the point of view of paying Modal. 

[0:26:31] SF: How do you see some of the landscape around AI infrastructure evolving over the next couple of years? Maybe we're still very much in early days. How do you think that's going to continue to change Modal? 

[0:26:42] EB: Yeah, we're in the super early days. This is so incredibly hard to predict. I think it's also - there's so many different layers. There are so many different boxes. I don't know. I'm very bullish on high-code tools. I think at the end of the day, looking back at my 30 years of coding and going even further back, I think the story of software engineering has always been better tools drive more demand. And then making engineers more productive is always fundamentally where the value is created. I'm very bullish on building better tools. I'm very bullish on the infrastructure layer. Making it easier for people to build these applications. Because clearly, there's a lot of demand. That's what Modal focuses on. 

[0:27:25] SF: In terms of the types of things that people are doing today and industry to build AI applications, using things like vector databases for building RAG applications. And there's been all kinds of takes now on RAG. I see a new three-letter acronym with databases. But what are your thoughts on vector database as a dedicated storage for AI? Just given that you've spent a lot of time thinking and working in space, do we need that? Is that sort of the right form factor for building some of these applications? 

[0:27:55] EB: I don't know. I mean, I think it's needed on some level, right? I started using vectors and built my own vector database at Spotify back in 2012, I think something like that, and open sourced it. And for a while, it was called Annoy. it's still called Annoy. A lot of people actually used it. Funny, actually, when Twitter open-sourced their recommendation algorithm a year or two ago, I was looking through the source code and, apparently, they were using Annoy for some of their stuff. 

Anyway, talking about vector databases, I think there's definitely a need for a vector databases. I think that being said, when a space is new, I feel like no one really knows what's the ultimate abstraction and the right interface boundaries. With vector-based databases, I think a valid criticism is like maybe that should just be a part of Postgres. Maybe Postgres should just do vector. And I think that's valid. But I also wonder if, in the long run, I don't even know if we know where the boundaries are going to be. I think right now, it's easy to look at Postgres and say, "Yeah, we should just put vectors in it." But in the long run, things kind of end up redrawing themselves. 

I could see a world where, for instance, in one direction, you can say maybe people shouldn't even think about vectors. Maybe people should think about a database. You can insert text. You can insert images. And then you can search for which images and text are close to each other, whatever. Vectors is kind of a low-level primitive. That's like one way you can think about it is maybe the interface shouldn't even be vectors. Because right now, it's kind of tedious. You first have to embed it. Maybe the embedding should like sit in the data - I don't know. That would be one direction. 

Another thing I've thought about a lot is LLM sort of kind of in a way like vector databases They store very large matrices. And the way they store these matrices, this would be like the other direction. And they do that, the state lookup, through very expensive matrix multiplications. Maybe there should be a differentiable vector database inside every LLM. My point is just, I don't know, we're still kind of early with vectors. And I don't know. In the long run, abstractions, interface boundaries, all of these things may change. And categories never look the same when you look at it. I think it's too early. It's too early to say. 

[0:29:53] SF: Yeah. I mean, it is a little bit strange that currently. And I think this is just a sign of the times and things being early that you have to think about actually generating a vector or even think about what a vector is and then to search a database. It's a little bit having to really understand. And there is some value to this if you're like a DBA or something. But really understanding the underlying tree structure that's used in indices to think about optimizing my lookup and so forth. For most people doing simple application development, they don't necessarily need to be digging into that level of detail. 

[0:30:27] EB: Yeah, I don't think that's the interface people necessarily want in the long run. And for that reason, I don't know if the shoe warning vectors in the Postgres is going to be the right boundary. But we'll see. We'll see. 

[0:30:39] SF: What are your thoughts on sort of the role of things like lakes and warehouses when it comes to building GenAI applications? I think in like traditional machine learning, a lot of times we're aggregating specific data down into a particular location, going through a process like feature engineering to build like a bespoke model that we need to deploy. And maybe we don't update it that often. But in three GenAI applications, we have a very general model. And then we're sort of massaging it for application-specific behavior by adjusting the prompt during prompt assembly, which is a lot less about sort of the old world of batch updating these models, but really sort of real-time updating the prompt in order to generate some sort of behavior that's going to be relevant for the user. I'm curious about what are your thoughts on that? Is this sort of a shift in the way that we need to think about the role of lakes and warehouses? 

[0:31:33] EB: I'm not sure. Maybe this is like my boomer perspective. I feel in a way, maybe we're throwing out the baby with the bathwater. When I look 5, 10 years ago, if you look at a search application, right? They would have a multi-stage retrieval process and then like a re-ranker. They use an ensemble. They would have all these like feature stores, generate a lot of features. And in the end, they would run some sort of XGBoost to aggregate all those features and then rank based on that. 

My feeling is - and recommendation system is kind of similar, like searching, ranking, all these things had this kind of complex up and it was kind of hard to build. And then my feeling is like there's so much demand for these applications, but they're kind of hard to build. Then when like LLMs came, people are like, "Actually, let's just turn this into a bunch of prompts instead." And doing that, I kind of feel like it's almost like low-code retrieval. 

My feeling is, doing that, we kind of threw out the baby with the bathwater. In the long run, I wonder if a lot of those prompt engineering things will just become features into like a multi-step retrieval process that will also combine other features. The pendulum will swing a little bit back towards those more traditional models. LLM ends up being one feature, a very powerful feature. But you might have different prompts like generating different features. But in the end, you sort of go back to sort a more traditional view of like a multi-step retrieval. I'm talking specifically about those types of applications. There are many other types. But like in general, I wonder if LLMs may just become like one feature out of many other features. And that will be a step backwards towards the traditional sort of feature stores and all of that stuff that has been a very powerful paradigm for many years. 

[0:33:17] SF: In some sense, I mean, I think you could think about like agents to this sort of multi-step process where you can weave in traditional ML models or even other types of workflows. And a core component of that of course is the LLM or the foundation model, but it's not necessarily - it doesn't have to be responsible essentially for all behaviors in there. You can use a combination of things to get essentially the behavior, the output that you want. 

[0:33:41] EB: Yeah, for sure. I'm very bullish on the high-code approach. I think LLM is a little bit the low-code. But over time, clearly, there's a lot of demand for people wanting to build AI applications and machine learning applications. I think there's going to be 10X more machine learning researches in the long run. They're going to use LLMs. They're going to be very happy with it. But they're also going to use the underlying core models as well. 

[0:34:02] SF: What are your thoughts on the energy consumption required to do model training? I think the latest open AI model consumes more electricity than the city of Pittsburgh? 

[0:34:12] EB: I don't know. I think it's kind of overstated. Part of why I think it's somewhat overstated is actually the cost of a GPU over its lifetime versus the energy consumption of its lifetime. The main cost of training a model is actually still on the GPU side. Building the GPU is a lot more expensive than the energy required to run the GPUs. Even if you run a GPU at full capacity for its entire lifetime, the cost of the GPU - which to some extent I think speaks to just the cost of building GPUs. I don't know if run - it's actually the other way around for CPUs. CPUs, if you look at the cost of operating CPUs, the energy cost is higher than the cost of buying the CPUs if you run a full utilization for a few years. I don't know. I tend to think we focus a lot on the energy consumption. I don't know. Humanity always needs more energy and they will find uses for more energy. I'm an optimist. Otherwise, I wouldn't have started a company. 

I'm very optimistic when it comes to things like climate change and energy consumption. I think we're always going to find more energy sources. And GPUs, the energy consumption is going to come down. And the GPU costs will come down. And all these things will be easier and do more better. And in the end, they're going to create a lot of value for humanity. 

[0:35:25] SF: Speaking of sort of having this optimistic view of the world and that being sort of a core part of surviving the hardship of building a company, when you were thinking about Modal originally and you had this sort of vision of being able to do these deployments at less than 100 milliseconds for running ML workflows. But were there parts of that that you weren't sure you'd be able to actually solve in order to realize that vision? Were there certain things, technical challenges that you were really scared about whether you'd actually be able to be successful with? 

[0:35:58] EB: For sure. And I think a lot about like one model I have of a startup is you have to pick a problem that's like hard enough that you create a lot of compelling value, but it's not hard enough that you can't solve it, right? And look at a lot of AI startups. I feel like there's a lot of companies in all three buckets. There are companies that I think are doing solving too easy problems. And for that reason, they're just kind of rappers and they don't have a lot of pricing power. And it's like you can sort of doubt that they're going to have the competitive advantage long-term. 

Then on the flip side, companies that I think have way too ambitious goals. And they're like, "We're going to do AG or whatever. We're going to do this agent thing that can act autonomous for everyone." And that sounds very harsh. I think the trick of a startup is picking a problem that could conceivably be solved in about three years. I think three years, four years maybe, five years is a good time frame. Yeah, I've spent most of my career in infrastructure. For me, looking at this problem, I knew all the components would be possible to solve. I knew, looking at containers, I was like, "Yeah, it's possible to start containers quickly because Docker works this way and Kubernetes works this way. And they're doing a lot of unnecessary stuff." 

That being said, I mean, a lot of my early VC conversations, people are like, "Why aren't you just using some existing system?" But I was kind of adamant. I'm like, "No, we're going to build our own thing." And I'm very happy I did that. And it took about three years. But I think doing that means now we have a pretty strong competitive advantage. And we have a very unique set of infrastructure, the primitives that lets us build things and deliver much better developer experience than anyone else. 

[0:37:34] SF: Are there particular performance optimizations that you did that you're particularly proud about? 

[0:37:40] EB: So much. But I mean, I think at a high level, a big part of it was building our own file system for serving container data. As it turns out, Docker, I mean, as much as I respect Docker for introducing a very new paradigm, it's quite inefficient in how it stores images and pulls and pushes images. What we realized looking at what happens when you start a container is that most of the data is never read, that is in a container image. And the data that's read is highly redundant between images. 

What if we can switch to using a content address system and then we built a few space file system that caches all the data under the hood? And then we then spent two years figuring out how to optimize the page cache and all these things, right? But that is a big part of why we can start containers very quickly is just optimizing the hell out of just the file system side of it. How do we send the data very quickly? 

[0:38:33] SF: Got it. So as we start to wrap up here, is there anything else you'd like to share? 

[0:38:38] EB: What else? I mean, we're working on distributed training. I think that's going to be really exciting. We're hoping to launch that pretty soon. And the focus is not super crazy large running thousands of GPUs. But for companies who are training models and maybe it takes too long to train in a single GPU, we make it pretty easy to scale up to eight GPUs right now. But we want to go beyond a single box. I presume we'll make it really easy to scale up to 16, 32, 64 maybe, maybe even 128 GPUs and get this super-fast feedback loops, just kind of was always like the core value of like Modal. Now you can also hopefully get that up to 100 GPUs. I think training is going to be super exciting. 

What else? I mean, I think throughout the year, we're spending a lot of time on security compliance. We're kind of moving up market and focusing a lot on like enterprise customers and serving their needs. The whole range from SSO to SOC2, building custom telemetry integrations and stuff like that. That's another area I'm also like super excited about. 

[0:39:35] SF: Awesome. Well, Erik, thanks so much for being here. 

[0:39:38] EB: Yeah. It's great. Thank you so much for hosting me. 

[0:39:40] SF: Yeah, cheers. 

[0:39:41] EB: Thanks. 

[END]