EPISODE 1968

[INTRODUCTION]

[0:00:00] ANNOUNCER: Over the past couple of years, companies raced to get their employees using AI, often encouraging them to lean on it as much as possible. However, as AI coding tools spread through engineering teams, many companies watched their AI bills explode. That kind of runaway cost growth can threaten a business, so the new challenge is capturing the productivity gains from AI without the spending spiraling out of control. Databricks is a data and AI company that hit this problem early, running AI tools across thousands of its own engineers. Rather than clamp down on usage, the team built a set of practical ways to manage the cost. From taking advantage of steadily falling model prices, to automatically routing each request to the cheapest model that can still do the job well.

Patrick Wendell is a Co-Founder of Databricks, where he leads engineering teams and has driven much of the company's internal AI adoption. He recently wrote a viral blog post on managing AI coding costs that reached the top of Hacker News. In this episode, Patrick joins Kevin Ball to discuss why AI costs suddenly exploded, the main levers companies have to bring them under control, the trade-offs between managing costs on the server versus in the developers' tools, and why the next big challenge is giving AI the right context about how a company actually works.

Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through latent space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

[INTERVIEW]

[0:02:00] KB: Patrick, welcome to the show.

[0:02:02] PW: Thank you so much for having me. I'm excited to be here.

[0:02:05] KB: Yeah, I am excited to dig in. Well, let's start with a little bit of your background. Can you tell our audience a little bit about your background, how you ended up where you are today and in this conversation?

[0:02:15] PW: Yeah, sounds great. My career history is pretty simple, because I've only really had one job in my career. That job is that I was in grad school at UC Berkeley in 2013, and a few of us that we're working on - we were working at the intersection of machine learning, then machine learning, which was not really a lot of deep learning yet, but it was scaling up classic ML, and we were working at those problems, and then they were very tightly coupled with big data problems, so dealing with really large data sets. Almost a little small progenitor to the scaling laws that are happening now. It was also true then that you saw, if you just kept scaling, a lot of stuff started to work better.

We were looking at research problems in that area, and then we left to start a company that's called Databricks that's still around. It was initially around some open-source technology we had built around these problems, the large-scale data and machine learning. Then at Databricks, I wear a founder hat. I'm one of the founders, so we have many different jobs. You never really just have one job as a founder. But my day job is working quite a lot in engineering and helping support several of our engineering teams. In that capacity, I've also been focused a lot in the last few years on our own use of AI in the company. For a company our size, I think we were pretty early in terms of being quite ambitious about adopting AI internally. I've been the one most internally responsible for a lot of those efforts working with our team.

I think I have quite a perspective to share around, we have thousands of engineers and we're trying to give everyone AI tools and get them to be super productive. At the same time, we're worried about security, costs, our own intellectual property, being careful around that. I think it's an area a lot of people are interested in and I'm happy to share. I don't think we have all the answers, but we have learned quite a bit from what we've done, so I'm happy to share that here with you.

[0:04:04] KB: Yeah, absolutely. Well, and I feel we could do multiple episodes. I'd love at some point to just dive into the history of Spark and all of that, which really, honestly, changed the direction of data analysis as you enabled streaming data analysis and all of that stuff. But let's focus on AI for today. It is the topic of the day. You recently wrote this blog post about managing costs. I feel like this is a transition we've gone through over the last year where it was like, first everybody's like, "Adoption, adoption, adoption. Throw everything at the wall." Then suddenly, people were saying, "Oh, my gosh. This is expensive. How do we rein things in? Can you give us the TLDR of how that's gone at Databricks?

[0:04:40] PW: Totally. Yeah. The reason I wrote that post, I do encourage people to check it out. You can just Google Databricks, say "AI costs" and you'll find it. Is that I had talked to so many of our own customers who are also many of them tech companies that are trying to roll out AI. They were saying, "Hey, what have you tried? What works? What doesn't work?" I also have talked to a few other peer companies. Actually, there are some credits in that post. We had talked to Stripe, Uber, I think Ramp and a few other. We just trade notes with other tech companies and see what everyone's doing. Remarkably, folks are looking at very similar problems and solutions. That was pretty cool.

I wrote that post to dump everything out of my brain, so that at least it's one data point people can look at, since I think many of us are facing similar problems. But the origin of that problem is something you just said, which is that there was this period and we got there actually pretty early, but it was around 2023, 2024, maybe beginning of 2025 where companies just were worried they couldn't get people to use enough AI in the company, especially bigger companies. It was all about pushing employees, try this, try that, incorporate these into your workflow.

Something changed roughly, I think, around Opus 4.5. It was when the AI models start to get really good at coding and it was started with Opus 4.5 and then many others came, combined with these really high-quality harnesses, like Codex or Claude Code, or Cursor, or there's others as well. Suddenly, the problem wasn't getting people to use them. The problem was that people are using them a lot, and around the same time, all of the foundation model apps switched to consumption prices. Instead of a seat where, okay, I'm going to pay $100 per month for developer, it was a consumption you pay as you go kind of thing. Then those two combined to at least at Databricks, start to encounter what looked like exponential growth in costs, and exponential growth is not a super sustainable, like in any cost in a company, unless you're exactly getting exponential increase in revenue that's commensurate and exponential growth in cost is just eventually going to be a huge problem.

We started looking at this when we were modeling the cost and saying, "Huh, that's funny. It's like doubling in this two-month period and it's, now you're going to get the two-month period and we have thousands of developers, so this can start to accumulate really fast. I would say, oh, I don't know, maybe beginning of 2026, around that time, or late 2025 is when we realized - Again, we wanted to have an attitude of people use tools. We want you to use tools and be productive, but they're just some law of physics, where when costs start to accumulate at a certain rate, it's going to eventually, any company that doesn't address it, it will eventually bankrupt that company. Just law of physics.

[0:07:16] KB: Well, and I think that's why they went to this usage-based pricing, right? Because the model providers and then Cursor and the harness providers are like, "Oh, we're paying per use, because we're fundamentally shuttling electricity through GPUs, turning it into intelligence in some form that costs money." As the models got good, people's usage of them skyrocketed. They said, "Oh, we can pass this along." They can hook their revenue into their costs, but that's hard for the rest of us.

[0:07:40] PW: I mean, I'm not on the inside. We do partner with a lot of those companies. I actually think, many of them wanted to have a fixed price state. But if they have a fixed price thing, then they're in the business of managing how much capacity and goodness you can get out of that fixed price. I actually think some of those companies got a bit ahead of their skis, where they were offering fixed price deals. Then when you did the math of how much token capacity you were getting, the foundation lab was losing a lot of money on certain people, at least I am one of those people, because I would look at it and -

[0:08:09] KB: Oh, 100%. Me, too.

[0:08:11] PW: Yeah. I think, they themselves had a sustainability challenge, and then understandably, they're like, "Okay, well, let's switch to consumption pricing, where then our customers," which is a company like Databricks or others, "can then internalize, okay, where is the ROI for us? How much are we willing to spend compared to how much we're getting?" That has a lot of benefits to the ecosystem, but it has one problem, which is it puts the onus on you, the buyer to figure out how to do cost management.

Incidentally, a lot of the solutions we looked at at Databricks, we ended up building also products that our own customers, we productized basically a lot of the stuff we built, so others are welcome to use those products that Databricks now offers. It's our AI gateway. But before we productized it, we just looked at how to actually solve the problem. It passes the responsibility for ROI to you as the user. We thought a lot about then, how do you as the user handle that responsibility, basically?

[0:09:03] KB: Yeah. In your blog post, you identified a few different levers that are available. Do you want to take us through that high level and then we can dive into each one subsequently?

[0:09:13] PW: Yeah, definitely. The real world is obviously very complicated. But I had to make this simple in the way that I thought people could understand. If you really simplify things, at least as it came to the short-term mitigations that were available to us, we looked at four things. Number one is that the single biggest lever that I've seen, and this has actually become even more true since we wrote this post, is that there's this beautiful thing happening of constant price reductions in models. Meaning to say, now, even at one moment in time, there's models that are more expensive and have better performance. There's models that are less expensive and have worse performance. There's always that I often called a Pareto curve, for those who are familiar with economics. But it means to say that there's this continuous tradeoff of quality versus cost.

Nothing is more amazing than when that whole curve shifts in one direction. Meaning to say, a new class of models have come out that you could view it either way, either you fix the complexity of what you're doing and then that got a lot cheaper. Or if you have a fixed budget, you can do a lot more. It means to say that there's a deflationary thing where costs go down. The way that that happens in the AI space is that some new model will come out that expands the frontier would be the economic term, but it dominates in some ways, the models that were there before in terms of usually, what people target is, "Hey, it can do roughly what we could do before and it's cheaper."

When you're driving a massive token volume, even getting 20% off a huge cost is a really meaningful thing. But there's a lot of secondary things that we can get into when we go through the detailed list around how do you put yourself in a position where you can take advantage of that? Do you want to go into that now, or you want me to finish? I was getting too excited about this, so I wanted to talk about it.

[0:10:52] KB: No, no, no. This is good. It's a really interesting one, because classically, you have this with a new technology where it's pushing the boundaries of something new and people keep chasing after that. I keep expecting us to hit a point where we saturate our need, where it's like, "Oh, you know what?" I mean, in some ways I thought, when we hit Sonnet 3.5, even before we got to Opus 4.5, Sonnet 3.5 was the first model I played with that could write solid code reliably. I was still driving it by hand. I didn't have the agentic harnesses we have today. I was like, holy smokes, models could get no better and this would change the world of software engineering. And models keep getting better, and Sonnet 3.5 you can't even get anymore. But if you could, it would be so cheap, it's unbelievable. I thought, okay, well, are we going to keep going?

It feels like we keep chasing after that frontier, right? People keep adopting. "Oh, why would I use that model? I could use Opus 5 now, or what have you." Where is that efficiency frontier? Why have we not saturated that for at least coding?

[0:11:47] PW: Yeah. I mean, it's a great question. There's different things to tease out in your question. One question is like, how come they can keep making these models better? Isn't it getting harder to make it better? the truth is they're just getting better at training them. First of all, software engineering is a highly verifiable domain, which really helps. Because they can throw RL at it and they can get into these self-improvement loops. They also continue to throw more compute at the problem. Data is, I think, pretty saturated from what I understand, although I'm not in those companies, so I wouldn't consider that authoritative. It's just between throwing more compute, clever techniques, optimizing heavily the inference and the training stack. Companies are even looking at building their own hardware. They just keep squeezing, squeezing more out of this quality of model per dollar.

There's almost some Moore's law equivalent thing happening here that I don't know if it's been measured, but the quality cost ratio just seems to keep improving. That's why it's so beneficial to be having agility to switch between models. Then I think a second part of your question was, by the way, I realize we're doing depth first search, because technically, I'm supposed to be going through the four ways to save money. But the second part -

[0:12:55] KB: Yeah, following interest. I think let's go depth first and then we can pop back up the stack.

[0:12:59] PW: Okay, cool. The second part of your question is about, or maybe what I heard maybe was, aren't we as developers at some point, aren't they good enough for the things we're trying to do? There, I think what's remarkable is that developers are just able to hand off more and more complex tasks. In some sense, they were good enough for the old stuff, but we were thinking of new stuff to give them. That also seems to be - there's this famous quote. I don't know if it was Bill Gates, or someone was like, why would anyone ever need more than 1,000 kilobytes of RAM, or something? It was this famous.

[0:13:33] KB: Yeah. Yeah.

[0:13:34] PW: Do you know who said that? I don't even remember. That is a quote. Something like that is a real quote. Someone told people about -

[0:13:39] KB: Well, you have the old IBM of who would need more than five computers in the world? We keep making this mistake.

[0:13:45] PW: We keep making this mistake, right? When you look at other analogs, like how much bandwidth does your mobile phone have now compared to 10 years ago? It's been an exponential increase. People just find new stuff to do, like Facetiming with another human, with near instant, super high-def video communication. We never would have even thought of that as a thing you would do on phones 10 years ago, or 15 years ago. When capabilities improve and prices go down, people just find new ways to use them.

[0:14:16] KB: Yeah. That's definitely true. I do wonder, and maybe this pops us up the stack and goes to the next one, right? Some tasks don't need all of that horsepower. Are there ways that we can decompose the types of work we're doing, so that we can keep those costs under control a little more?

[0:14:31] PW: Yeah. Definitely as an example, just to move to a different context, before these enterprise coding stuff started really working, the big hit in AI was personal chat stuff. That was itself a very disruptive and innovative product form factor was like, oh, I can just pull up my phone at any time and I can ask any question I want. It turned out that although the models keep getting better for that, the set of things people want is saturated. Even very low-cost models can answer the vast majority of individual questions, because most questions are like, where can I get a hotdog in my neighborhood?

The frontier labs are really excited about proving new math, or those problems and stuff, but most people don't ask those questions day to day. In that capacity, in that form factor, my view is there has been a bit of saturation, where I actually think the main thing happening now is that costs continue to go down, rather than that there's lots of higher quality models that matter for the types of questions people are asking.

Then the second big wave in gen AI has been this coding and other enterprise workflows. As I said, I don't think we've hit saturation. I think there's far more difficult coding stuff that they still can't totally handle, but especially when it comes to architecture, high level system design. But it is absolutely true that for routine coding tasks, hey, go refactor this thing. Please add this simple feature to a JavaScript code base. They're much lower cost models. Even the last generation of models will perform identically to a very, very expensive frontier model. There is this cost gained from fitting the model to the problem. There's different ways of doing that fitting. One way is the user themselves can do fitting. You can ask the user.

Now in a small company where people are policing their own costs, maybe that works. In a big company like Databricks, where no one is themselves paying the bill, understandably, it's a little hard to convince people to constantly switch to crappier models. My incentive as an individual developer is just turn the dial to 10 and get out of my way. We do a little bit of downshifting on purpose, or how you can control this when you're doing things at scale is the default model maybe could be not the most expensive model. If you want to shift up to it, you can.

We also prune the set of models, meaning to say, even the frontier labs continue to release updates to their models that are not better than the old models. If you look at how they train these models, it's like more art than science. I mean, they put a bun in the oven and they're seen, does it look like it's burning on this side? But they're not quite sure they're doing evals as it's being trained, but they don't quite know what they're getting.

[0:17:08] KB: Well, and they're very spiky, right? Soul is phenomenal for mathematics, apparently. My engineers are like, "I don't want this for coding. It over-complicates everything."

[0:17:17] PW: By the way, we had one of the Opus - I think many people were using Opus 4.6. Opus 4.7 came out and we deployed it. Well, actually, I think first we did some evals. We did some benchmarks. On our own benchmarks, we're like, "Huh, this seems like it's more expensive and lower quality." It's the inverse of the parade. It was like, both things got worse. Then we tried it with the pilot group. By the way, I don't want to get in trouble. I don't know if exactly 4.7. I think it was. I mentioned it in the blog post. It's since got better with the next model. But there was one that came out that was worse along both dimensions we cared about.

There, it was actually an easier decision. It was like, "Oh, we're going to skip that one." We just didn't make it available to our users. When our users asked and we just explained, "Hey, it's not so great." Then there's other models where like, okay, that model is good in certain circumstances, or different complexity of task, but it needs to be this fitting thing that we were talking about before. There, the levers you have are like, ask people to do it on their own, but increasingly, what we're using is dynamic routing, basically, where we try to decide what model to use and there's a few different flavors of that. Sometimes it's done at the task level. We might for a whole session, for instance, put someone on a particular model, or there's certain interfaces where someone just submits the unit of work. It's not an interactive session, so then we can decide based on that thing they submitted what model do we want to use.

There's also much finer grained routing, where even request to request you might divide across. This looks like traditional load balancing in a server system. That one is trickier to get right. It's more powerful in some ways, because the end user is even further abstract. They don't even know what model is really being used. It's trickier in that, although we can model inference as a stateless problem, there actually is some state in the server.

[0:19:03] KB: I was going to say, the key value cache feels like, it can be a tricky thing to optimize.

[0:19:05] PW: The cache is really important to economic performance, or to economic cache reads are like, I don't know, 10 times less or something than non-cache reads, or something of that nature. You have to be very deliberate about routing. When is it worth it to develop a new cache to get a much better fit for the economics for this particular problem? We already have some efforts in this area and our gateway supports it, but I think this is a very rich area of research, to be honest is like, how do you decide when to switch based on a variety of how much you can save and how much will it degrade the performance for the end user, if at all, and how much am I losing that cache benefit? You need to keep all these into account.

[0:19:42] KB: Yeah, it's fascinating. Well, and how are you classifying what types of tasks? Is it going to be good enough anyway, even if the economics work out?

[0:19:49] PW: Yeah. That's also a hard problem. First of all, even humans aren't good at that problem. This is the number one thing that people say when we're like, "Hey, choose the cheaper model." They're like, "I don't know which model is good enough for this." That's why it's really tempting to just stay on the most expensive model all the time. The answer to your question is we build models to do that classification. That classification of which model is good enough for this problem is itself an AI problem, and is itself amenable to training models that are classifiers that are also LLMs, basically. They're very different type. I do think it is in principle, learnable, not with 100% confidence, but it doesn't need to be 100% confidence. It just needs to be a high enough hit rate and it is in principle, learnable.

[0:20:29] KB: Well, especially in software development, because as you highlight, it's very validatable in terms of outcomes in a lot of ways.

[0:20:34] PW: Exactly. This validation is a little trickier, because it's like, oh, was that model in practice good enough for this task? That can sometimes be measured just completely objectively like, oh, the pass rate on some tests, or we ask a much smarter model to evaluate the solution. There is a little bit of eyeballing that still happens here. In my view, one of the things we focus on at Databricks a lot is we're not in the business of ourselves building these massive, massive foundation models, but we are in the business of how the models get used in real companies.

I think routing and coming up with models that are good at routing is a very important problem. It's one that we're looking at and have some ongoing research in. But I think I believe most companies should take AI into their own hands and have different models they're choosing from and different vendors they're working with. Part of that is being able to know how you can switch in between them, basically.

[0:21:32] KB: For sure. Staying on routing for a second, I feel like there's a couple different types of routing that you have alluded to here that might be worth digging one level deeper. You've talked about gateway level routing, which is like, here we have a particular inference coming in. We can make a decision in using a classifier, or something else and route it to a particular model. Then there's harness level decision-making, where you're actually - something is deciding, "Oh, I want to explicitly route this in this way." That's something might still be an LLM, or sub-deterministic logic, but it's in the harness layer that knows more of the surrounding context, rather than just, here's my current inference request. From what I could tell in your blog post, you're looking at both of those. Do you apply them in different ways? When are they relevant? How does that look?

[0:22:12] PW: Yeah. We use both. If you want to ask me in two years where we'll end up, will almost everything be a fact client, or will there be proxies? I don't actually know. It's going to depend on how well this stuff works and various other factors. But we use both. The nice thing about the proxy, or gateway-based routing is that your client is out of the loop and doesn't need to even be routing aware in some sense. It just needs to be the case that this proxy speaks the same inference API to that client, and you can just swap it in with no assumptions about your client. You may even have an older client that doesn't know how to do routing, or whatever.

Also, it depends on your control model, but in a company, your end users are mostly trusted, but for things like cost controls, you might want to have ability to coerce activity and therefore, doing that in a server rather than the client is a bit easier, because it avoids creative employees realizing that they can tweak settings on their client and get around the rules. For instance, things like, entirely, let's say, removing access to certain models, because they're not efficient models and we don't even want them in our set, or maybe some of our users accidentally run runaway spend. They have a loop going. They didn't even realize they left it on overnight. It's not doing anything useful. We have a thing that will torpedo that and it lets them reset it if it was truly intentional. Those are much easier to do on the server side. It is transparent to the client and in some sense, the client just doesn't get to make the decision, which is sometimes a feature.

However, there's a lot to be said about doing this on the client side, too. One is, as you said, the client knows much more. It may know much more about what's going on. It has tighter access to the whole history of the session. For instance, it may have other context on what type of job it's being used for. I do think potentially, you'll see a lot of intelligence happening at the client side as well. There's also this further evolution of client routing, which is more like dispatching between harnesses. This gets into a slightly nuanced thing, which is that there's increasingly a coupling between the foundation model and the harness, the choice of harness. When I say one model is better than another model or so, I'm sort of was simplifying a fact of the world, which is that model harness pairs are really what you need to look at. A particular model with a particular harness may be very expensive, or very cheap, and so on, so forth.

Another class of thing you can do on the client side that you by definition can't do on the server side is you could even dispatch between different harnesses. There's a project and open-source project we've released called Omnigent at Databricks. It's a multi harness dispatcher, basically. The idea is you do your coding against a fairly uniform interface. Behind the scenes, it can leverage other harnesses. Sometimes it's even beneficial to have an adversarial relationship between your harnesses.

[0:25:08] KB: Absolutely.

[0:25:08] PW: One of them writes code and one of them reviews code. It tends to be the case that when you review code from a different model than wrote the code, you get way better feedback. Models are very bad at reviewing their own output. They give their own output high marks and it's not surprising, because it's all a matter of what statistical distribution they come from. For even other reasons than just cost, there's nice benefits to having these multiple harnesses that are playing with each other on the client side. It's not even just client versus server side. It's also, like as I've said, it's server side versus a client where maybe within one harness you're routing to different back-end models. Then there's even this next layer up of a meta harness that's routing amongst harnesses. I don't know in the very end of the day, if you fast forward, where most of the logic will be. Right now, we're playing different types of logic in all of three of those places.

[0:25:56] KB:: Yeah. No, it is fascinating. I remember, I think it was the early GPT 5 series. Folks were trying it in the context of Cursor and seeing incredible results and trying it in a different harness and finding really running into trouble. I've recently been on Pi and comparing that to different things. The harness makes a larger impact than we like to think.

[0:26:17] PW: Totally. For instance, I think, Databricks also regularly, we do a lot of evals of our own, because we have this problem of like, I told you some models come out, they're actually not better, it turns out. Well, the way we used to find that out is we would roll out the model, stall these developers, like thousands of developers. Then the complaints would start to come in. By the way, everyone always has some complaints about every model. You'd have to wait until you have enough signal to noise that's like, okay, this actually is really a problem.

The cost, we get an AB test that's pretty quick, because you'll just see costs go up or down based on the cohorts that move. But the quality is hard was initially quite hard for us to gauge. The downside was that we would waste weeks with a crappy model. Then eventually, someone said, "Hey, come on. Look at all these people complaining. This model really does stink." Then we roll it back and we would remove it from our working set, basically. The way to fix that we found was you just need to have good evaluations you yourselves can run. We ended up building our own evaluation suite based on our own code, basically. It literally asked these models to go contribute code to Databricks, the code base. We obviously have a point in times snapshot of our code that's before we went and actually added all those features.

Then we run these offline evals, and we want to get to the point where within an hour of a new model coming out, we know if this one is a candidate to try internally. That offline benchmark rules out a surprisingly large number. There's so much. I mean, every week now, we're running that multiple times, between open source models and proprietary models. Every week we're running it. We want to get down to a set of three, four, five models we're using that define that Pareto frontier. We needed to get the speed of evaluating whether models were even worth considering down, and so we ended up making a benchmark. We also published the results to that regularly, although I don't want people to over extrapolate. That's based on our code base, so there could be some fitting going on.

One thing we found when doing that benchmark is the model plus harness pair needs to be benchmarked. You'll see even drastically different results. By the way, Pi has suddenly become very popular at Databricks also. I guess, you guys are using that. There's this constellation of open ones that are getting pretty good.

[0:28:26] KB: I mean, I'll just say, Pi is incredibly hackable, so you can customize it to yourself and build extensions and it knows about its own extension system. But that compounds your harness plus model problem, because now it's model plus harness plus the set of extensions, or other skills that you've incorporated, etc.

[0:28:42] PW: Yeah. We have to cut it off somewhere. I'm sure that there's probably some minor regressions, where someone actually gets worse for someone, because we didn't test their particular configuration. We do quite a bit of this offline testing, mostly to save the human time of it's just becoming practical to test each one with humans. There's too many of them.

[0:28:59] KB: For sure. I want to dig in on an implication, or a thread. You mentioned these are on Databricks code, and so it's different than other things. One of the things that I've definitely seen is code organization in its style has a dramatic impact on how effective LLMs are writing that code and how cost effective it is. I'm curious, is that something that you're able to see in data in terms of like, oh, if we use a service-oriented architecture, or this language tends to be more efficient in terms of things, or stuff like that, where you can have even that meta level of feedback of, hey, if we write services primarily in, I don't know, I'm just going to pick Go as an example, that tends to be much more cost effective than writing a monolith in Python, or something like that.

[0:29:44] PW: Yeah. So, that's a great question. We don't have systematic research benchmarks that look at that problem. I think the problem, if I understand it, is are there ways you can write your code that make it more amenable to being highly efficiently modified by models? The only answers I have for that are of the form of my intuitions, which I do have some. Certainly, the distribution of code written in different programming languages on the Internet is almost a power law. I think there's some extremely popular programming languages and then there's a bunch of very obscure ones.

Databricks started, we have a meaningful portion of our code base is written in Scala, which is not on the very extreme end. We've actually over time more and more moved those pieces of code to Java. But we started Spark in 2013, or 2010 when Scala was super exciting. I still think it's a cool language, but some of the useful features that have actually been moved into Java at this point. A lot of our go forward work is in Java and then also we have a huge amount of Python and Rust and C++ in certain areas.

I can tell you, LLMs are not amazing at Scala, but that's not a useful fact for most of your audience. I think the general principle is more useful, maybe that the biggest and most popular languages just tend to be the best, because they have the most training data on those languages. Python is probably the single best. For instance, we did have an episode in one of our own products, where we have certain internal parts of Databricks where you can call it via Python, or via SQL. Behind the scenes that gets translated into exactly the same representation. We just have two front ends for it. Wouldn't you know it when we switch that the code generated portion from SQL - this is not ever exposed to our own users, but it's just an internal representation. When we switch that from SQL to Python, everything got way better, even though - just because the AI is better at generating Python than it's a generic SQL. I would say, SQL is a fairly popular language, too. I think Python dominates in terms of quality of generated code, but it's getting so good now. They're probably all pretty good.

Then in the teams that have started new code bases where they're able to also innovate on how the code base looks, I have seen them focus much more on interfaces, behind which they could build things that were highly verifiable. I realize it's a very abstract statement. But in some sense, the key to being able to trust the LLMs more was make the scope of their changes smaller and more verifiable. In a big legacy code base, that's difficult. In a brand-new code base, you can think about modular design principles, where you have - by the way, this is probably what would have any ways been good for humans.

[0:32:15] KB: This is one of the fun things here is like, what's good for LLM coding is actually the stuff we probably should have been doing anyway.

[0:32:20] PW: Yeah. the ROI from doing it for LLMs is even much, much higher. You'll see some very flat types of organizations where it's like, okay, I actually built an interface and there's 50 different implementations of this interface, because each one was able to be super easily verified when for humans, you might have imagined that same thing would have been written with a slightly more monolithic structure. This is in the territory of my intuitive takes. We don't have robust research findings in this area.

[0:32:47] KB: Yeah. I feel like, this is one of the fascinating things to me of like, what are the characteristics? Some of the characteristics that make something like Scala, for example, good for humans may not actually be there for LLM. One thing we didn't talk about, but a thing that's true of both Java and Python. I've seen it in Go. I think it's true in TypeScript. Is it's very easy to trace where a particular logic is coming from. Java, if you go heavily into dynamic dispatch, maybe, maybe not. But I think Go is a great example, where there's a very linear dependency path, where the LLM knows exactly where to pull the context in to understand what something's doing. It doesn't have to dive through a deep dependency train, or load a whole bunch of context to be able to do it. I think that linearization.

[0:33:30] PW: I do agree. Scala code has these implicit things, where it's like duck typing and it will become a thing if it meets the signature of that thing. I actually suspect what totally dominates all these factors. Is just, how many lines of code on the Internet are in this language.

[0:33:46] KB: Fair enough. Yeah.

[0:33:47] PW: I think that's just how these LLMs learn. They're pretty good at learning almost anything. They weren't very efficiently compared to humans. They just need to see a lot, a lot, a lot of examples. My hunch is that that would be the thing that differentiates the most. But there's certainly languages where there's more implicit stuff happening and that probably also makes it harder for the models to learn.

[0:34:06] KB: Let's dive into some of the other things that you've learned in terms of rolling out AI at Databricks. Because I feel like, every company right now is trying to figure out how to effectively roll out AI. As you highlighted, there was this top-down push. I think you were generous, honestly, and saying, enterprises were going through that up until 2025. I keep hearing of enterprises that are still in this top down, how do I get people to use the tools space? What have been the important challenges and what's been really effective in terms of getting these things to work well for your org?

[0:34:37] PW: Yeah, there's many facets to it. The types of things that we have thought about is first of all, how many different AI tools do I even give my employees? And, can they use any tool they want, or I should just give them a few, or should I choose two? Then, do I give the coding tools to the non-coding people? Because even my marketing team wants access to Codex, or something. They don't know what to code, but it can build these apps for them that they're super excited about.

Also, my inference data, I'm attaching these tools to all my most sensitive data and then that inference is getting logged by some third-party company. Now they have all my sensitive - There's so much around not just cost, but which tools you deploy, how to deal with data security, and where is your data going by virtue of these tools? Who gets what tools? Do you go all in with one vendor, or do you have multiple vendors? I can share maybe some insights we've had, but I don't think we have all the answers, but we felt strongly from the beginning that we wanted to be a multi-model, multi-tool company, at least for R&D, which is where I focus a lot of my time. We really did not want any scenario where, by the way, it doesn't necessarily mean that an engineer can find some AI startup that's existed for one day and buy their tool and give it all of our sensitive data. That's not what that means.

What it means is that we have a set of pretty popular tools. We let engineers vote with their feet and use the tool they want and we want to provide internal choice around tools and models. By the way, Databricks, our product is really for data users. It's not a general coding tool. But we also in our own product have capabilities as well that are consumption-based, and you're also seeing also a lot of embedded AIs now inside of different products, including ours. Our thought was like, let our internal users choose what's best and what's best seems to be changing every month. That's one reason to this.

Where we insist on uniformity is we do have a narrow waist in the company, and that narrow is this thing that we built internally called our AI gateway. Basically, all tokens go through that gateway, so we can see every inference that's happening from any random tool that's in the company. All data access to internal systems also goes through that gateway. It acts as a proxy, basically, to an MCP proxy effectively. You want to access Google Drive. You want to access Salesforce. You want to access anything. It's going through that proxy where we're doing authorization. We actually log everything. Also, we insisted on zero retention for the people providing us the models.

The whole thing you do with these AIs is you give your most important data, so we can help you with your most important problems. The inference request that goes to the model foundation provider includes all of your extremely sensitive data. You might have customer names, or PII, or credit cards. We're a serious big enterprise. The rules we came up with were the end user tool choice, we want some flexibility on. We want to have a set of models that we can always be adopting the newest ones and making them easily used by our internal users. We need to have some sanity. The way we do that is we run everything through this thing we call an AI gateway. They're sometimes called LLM gateways. There's a few, but now they're expanding in scope. This is an emerging architecture, I think.

For any company that's big enough, you're going to want a central control plane for this stuff. We ended up productizing ours, called Unity AI Gateway. There's other companies that also offer them. But whether or not you use the Databricks one, I think at scale, this is necessary. If nothing else, it's necessary for being able to quickly adopt cheaper and better models. The truth is the Pareto curve, if you look at the Pareto curve today, I don't even have the most recent one, but we run the emails every week. On there is probably one open-source model. One model from Anthropic. One model from OpenAI. Occasionally now, even there's - the Meta ones are starting to get good. The Grok ones are starting to get good. We're in this period where there's more and more supply and it's a huge cost. Just basic economic theory dictates maintain option and suppliers.

[0:38:49] KB: Absolutely. Yeah.

[0:38:51] PW: I think for us, that was a principle that really mattered, was like, tool choice, not amongst any random tool you can put in your credit card, but we'll try to basically provide the popular tools to engineers and let them choose. Even day-to-day they can choose. We do insist on like, the access needs to go through a gateway, so we can control and observe costs and do model routing and stuff. Any access to our internal data needs to go through our proxy basically, and then we insist on zero retention for the model provider. Those were the three things that I was like, that's my religion. Unless, I get a really strong argument, that's my religion. Almost everything else is let's experiment. Let's be flexible. How we organize the code base, how we organize teams, those things are all ongoing experiments, basically.

[0:39:36] KB: Let's talk a little bit more about that control plane and what goes into that, because you mentioned a few things. You mentioned, all tokens are going through that and that then allows you to do routing. It allows you to do audit logs. It allows you to do cost analysis, a bunch of other stuff. And then, internal data access. Is there anything else that's flowing through there? Or are there any other mechanisms of control, or places you're either locking things down, or titrating?

[0:39:58] PW: First of all, data access is quite nuanced and we have very sophisticated policies. The way I think about it is like this. We already had the problem in enterprises that sometimes I would want to allow a piece of software to act on behalf of me as a user. That's already been a thing that was a problem. It was solved with something called OAuth. I mean, there's different approaches, but really, the approach that dominated was OAuth, which was a delegation model. It says like, I'm okay for this app to be Patrick. There's relatively reasonable token exchanges and so forth to avoid long live tokens. It's a lot better than I send that vendor my password to that thing.

[0:40:36] KB: You can ask for particular capabilities, though I've noticed the foundation model companies love to ask for everything.

[0:40:40] PW: Yeah. Well, I'm getting there. We're still in the old days, okay. OAuth has a thing called scoping. It's up to the vendor to implement scopes, but most reasonable applications have some notion of scopes. They have tended to get better over time. I don't know if you've ever set up OAuth with GitHub. That first, they just had very - It was a read and a write, or something. It's like, okay, well, I want much more fine-grained scopes than that. But it's actually very hard if you implement OAuth as a vendor, which we also have. It's hard to implement new scopes. You need to propagate that throughout your whole app. OAuth scopes tend to be fairly coarse, but they've gotten better over the last few years.

Now there's this other thing called an agent. It's not quite the same as a vendor app. I want that agent to do things on my behalf. But OAuth is like, it's necessary, but not sufficient. Because the truth is, if I have 100 agents running around, I don't fully trust them as much as I trust myself. I don't think the company should trust them to be exactly equal to me. They need to be put in much more of a constrained set of capabilities. The problem is we're stuck with OAuth scopes, which are okay, but they're very coarse-grained and they're also static. It's like, there's an OAuth scope that I give to all of Claude code, or Codex, or even the Databricks agent, or whatever, but it's not a thing you can dynamically adjust, or adjust in a super easy way. Agents look more like people.

The way that historically this problem has been solved, like ignore software systems. Okay, here's an analogy. There's credit cards running around, okay. This credit card is a delegation point that I've said like, this card represents me and my bank account. But sometimes people steal those cards, and they start to do bad stuff. How do credit card companies decide whether to authorize a transaction and know like, okay, is this a legitimate thing that the person wanted to have done, or it's a thing that they didn't want to have done? Well, it turns out, they have a highly dynamic and sophisticated modeling system. If it's like they risk score every transaction and if a few risky ones, some might be so risky that they just block it, some might be risky enough that they put it on warning. If another one comes, they block it, they might dynamically decrease the credit limit. These agents running around need to be modeled more in that way.

One thing we do in our gateway is we're dynamically looking at what agents are doing and looking at risk associated with their actions. Because just saying, oh, you can have read access to Google Drive, that's a static, or entitlement that will sit around for a long time. I don't think they use the word entitlement. It's a scope, I guess, would be the right word. But what if I deploy an agent that suddenly starts scanning our entire Google Drive and downloads 10,000 documents? That is a risky behavior that warrants attention, and that warrants attention of the form of cut off that agent and email the person who deployed it and ask if it was intentional, or not. I get on my phone, the credit card like, "Did you mean to buy this gas in Nevada, or something?" I'm like, "No, I didn't mean to do that." You need to profile the behavior.

If you want to have good security, it's not sufficient to just say, "Oh, I have a lot of scopes. And so, I'm done." You need to look at what they're doing dynamically. That is a function that we enforce at the gateway layer as well. You can even set policies. You can set policies like, you can have access to this particular Google Doc, but you can't just - because OAuth is too broad for that. OAuth doesn't support - the protocol doesn't support these extremely fine-grained entitlements, or this dynamic risk scoring-based approach.

[0:44:03] KB: You're like laying out all my frustrations with working with agents and Google Docs right now. Yeah.

[0:44:08] PW: By the way, a lot of them don't even support read-only access. That's the most simple one you should have had from 1995. Your question was, what else do you do? Is this just gateway for just cost, or something? I would say, the two really big functions are cost up and security stuff. Security stuff actually gets very, very nuanced. My view is if we want to tell people to go nuts with agents, we need to make sure that things are being done safely. I think it's very hard to ask every individual person in a big company to conceptualize what safe means. We can ask them to do some things, but some of it needs to be enforced in a central location.

By intermediating all access, we have the ability to log stuff. We have the ability to start looking at potentially weird behavior. I sleep better. I tell them it's my blood pressure, founder blood pressure project. I sleep better knowing that when inevitably - these agents do unaligned things. They might not do really severe unaligned things. They're not going to go launch a cyber-attack, hopefully. But they might think, oh, someone wants to find something on Google Drive, let me download everything in the Google Drive to a public URL, paste bin or something. They'll post stuff on the Internet that we don't want posted on the Internet, because they'll think, "Oh, it's very easy. I can copy something to this public endpoint and that's the easiest way to accomplish this task."

If you start to really observe what the agents are doing in practice, although there's not severely unaligned behaviors, there's medium unaligned behaviors, even in a normal company to play in frontier agents. Frontier, meaning to say, this is from OpenAI, or Anthropic, or Google, these are highly safety post-trained models. Those occasionally do stuff that you would be bad security practice kind of thing. The only last thing I would mention, which is it couples with the gateway, but it's not quite the same piece as the gateway is we do also network sandboxing and other types of sandboxing of the agents. Not only do you have to go through us to access our enterprise systems, you can't go anywhere else. You can't just call out to the Internet easily from one of our development environments, and we can strain what they can do. In many cases, they're running in physically isolated, which by the way, the AI companies can't seem to figure out how to do a network sandbox. I don't know. Because they're having their agents escape them all the time. To our knowledge so far, none have escaped our own development sandboxes.

[0:46:27] KB: Yeah. I feel like, sandboxing is one of the big pieces here. So that I understand, everyone, all of your development environments that are cloud-based, or you're shipping around these little sandbox environments?

[0:46:38] PW: Yeah. First of all, we primarily have cloud dev boxes at Databricks, which is that we've had it for a while. It has a lot of benefits in terms of being able to more elastically scale, or let people do. Maybe I'm compiling my code base in four different branches and I don't want to - my laptop to get too hot. We do have some people that will, for certain things, develop on their laptops, in which case we actually have a local sandbox that we use as well that ships with on the gint. But most of it is happening in our cloud environment, where it's already running in a virtualized environment, so that thing is just hardened with pretty restricted network access.

[0:47:12] KB: Well, and so, we've talked a lot about current layer of challenges and what you're tackling as you look out. I don't know what the right forecast length is these days, because as you highlight, the Pareto frontier is shifting every week, but say, the next three to six months, what do you see as the big challenges that you're focused on in terms of AI usage at Databricks?

[0:47:31] PW: Yeah. I think, this is an excellent question. I think first was, by the way, I don't want to sound like some Debbie downer. AI is helping us in immense ways at Databricks. It's awesome. We're moving way faster by every observable metric in our development stack. We're shipping more code. We're shipping more features. Our revenue growth has actually accelerated in the post-AI era, which we've talked about publicly. A lot is going right. But as the leader, when the stuff is going right, that's like, I don't need to do anything. I need to come help when there's challenges.

[0:47:58] KB: Yeah. That's not what you need to worry about.

[0:48:00] PW: Yeah, exactly. I should qualify all this by saying, it's been an abject success, our adoption of AI. There's just a lot of tricky things to deal with, which are the ones we're talking about here. In terms of upcoming challenges, I feel that we - the costs are feeling much more under control at Databricks based on this set of techniques we've done. I would say, that's an evolving situation always. So far, what's happened is that the set of things we want to do, just keep getting cheaper, which is great. If some new set of capabilities comes that has some really high costs associated with it, but we see huge ROI, then we'll have to reconcile with that, but so far, that hasn't happened.

A lot of what I focus on now is making sure these teams are actually as productive as possible now with AI. It's quite interesting. The models themselves are just amazing reasoners now. It's very rare that they will make any error in that they have a premise and they have conclusions and the premise does not fall from the conclusions. They will never make that form of error. In fact, I would say, they're smarter than me in many cases. I think they're smarter than our median employee, which I guess, implies that I'm smarter than our median employee. But let's say, I'm in the median. Let's say, I'm at the median, okay? They're smarter than me.

It's quite interesting, because despite that, even though you have in principle, you have an army of in theory, we could have 10 times more employees than overnight, because we have these people that are just as smart as the people that we have at Databricks. What is actually not quite working where we're not getting immediately those gains that we can't just add 10 times as many agents and we have 10 times as much output. What is broken? That's where we encounter every other type of problem. I would say that the number one issue is that the models not having the context that they need to get the job done.

It's funny, because people thought we're shocked that models were good at coding, because coding is a very specialized skill in the world. For instance, there's way more people that can click around on a computer than can write code, but the models are really bad at clicking around the computers and they're good at writing code. That doesn't seem quite right. By the way, they're getting better at the computer thing. Even Databricks does this - we haven't built an AI in our product that's not primarily focused on coding. It's focused on asking questions from your data. In many ways, asking questions from your data should be easier than coding, because there's so many more people that know how to use Excel, or whatever, and then know how to write code, but it turns out, it's really hard. It's hard, because it gets into the set of things that can't be verifiably hill climbed for the models and it involves a lot of surrounding context and nuance that just can't be baked into the model, because it's very specific to each company. Coding, although it's intellectually difficult, it has this amazing property. A code base is self-describing. It's a bunch of files that are self-describing. It's designed to be hermetically sealed.

[0:50:45] KB: We've spent decades figuring out how to deterministically validate the outputs of non-deterministic inputs are coding.

[0:50:52] PW: Exactly. We handed this to the LLMs on a plate. The other nice thing is like, ever since Git came around, no code bases have ACLs on them. Code bases have already one big problem in the data side is like, not every piece of data in a company can be seen by every person, okay. You can't feed all the data to one AI and then suddenly, it's telling - it's leaking something from the CEO to the intern that just started yesterday. There's this ACL problem.

Now merely by coincidence, by the way, perforce in these older source code management tools did have ACLs in them. But just merely coincidentally, this tool called git took over and git has no ACLs. The only ACL git has, you're either in or out of the repo. That actually means that this whole problem of who can see what doesn't happen in the coding space, which also makes it a lot easier for the AI models. Although coding is intellectually very difficult, or at least for humans, it's a thing that not everyone could do, it was considered a specialty, anything intellectual the models can run at, what they need is they need all the context to be hermetically sealed and searchable and they need to have verifiable testing and they have all that stuff for coding. I find it actually unsurprising that they got really good at coding.

It turns out that most people at Databricks, even though we have a lot of engineers, they don't do coding for their day. They're doing other things. The thing they're doing, they might require less extremely sophisticated individual reasoning, but it requires a lot of context. The thing that I'm most working on now is how do we at the scale of a big company - I think there's this other layer base. The layer I just talked about was this gateway layer that will deal with security and tokens and all this stuff. There's also, I think, a knowledge layer. Someone call it an ontology. That's what we call our products at Databricks, but you call it whatever you want.

There's a layer that's going to exist in any company of a certain size, where it's not trivial to just stick all the information from that company in a very small amount of context. That layer is going to be the thing that helps the AI have the context they need to do the job. Designing that layer is hard. It's not like a code base where there's a very hermetically sealed thing. There's a sprawl of a million random things in a company. There's data in Slack, there's data in the structured data warehouse, the data lake, that's what we do at Databricks. There's data in docs and there's data in people's activities and there's implicit knowledge that seems to only diffuse through conversations. How do we get this all in one place and it respects access control and it's not creepy? I don't feel like I'm in the company I have people listening to everything I'm saying all the time. That's a big, big problem in enterprises. I think it's the next frontier, and we're solving it in very - there's very small ways in which it's being looked at.

For instance, you can create skills and you can share those skills. Those skills are supposed to encode some knowledge of this enterprise and what we're doing. But when you have 10,000 people, that doesn't really scale. By the way, the scales themselves become the same thing of they're messy and some of them are wrong and they contradict each other, and so they just look like every other thing in an enterprise. That's the big other area I'm personally focused on is this, how do we build this knowledge layer and how do we make it so that the AIs are never stuck on, they're just not smart enough. They're never stuck on that. They're stuck on like, they don't have the right information to get the job done. They don't know how that job is done, or they're missing access to a certain thing, or they aren't able to search and understand. The user is asking them.

We have, in Databricks the unit of pricing we have is called a DBU. Databricks unit. It's an opaque thing that means like a gold token of Databricks. That's just how our pricing works. Every different product we have normalizes into that unit, okay. People will ask AIs like, "Hey, how is our DBU growth in this sector, or whatever?" A lot of the models are going, what DBU means? That's an internal thing that we have. By the way, there's slightly nuanced different versions of that we use internally. Finance uses one thing and product uses a different thing. Actually, finance and product don't even totally agree on the definitions of every derivative metric from this thing. That problem is very difficult to solve and very messy.

As a data company, we're very interested in that for our own products and for us internally, but this is the thing that I'm most focused on next now that we're not going to go bankrupt is like, how do we actually make these AIs better? The thing that they're stuck on often I'm seen is the context, basically.

[0:55:12] KB: No, it's a fascinating problem. You mentioned skills, right? Skills are probably the best example right now of this technique of progressive disclosure of context, where it's like, okay, I'll give you a little snippet that helps you dive in. But I feel like you need nested ontologies of how do you build the map for the AI to even know what context it should be getting?

[0:55:31] PW: Well, yeah. Skills have this hierarchical - the problem is there's a context budget in these AIs.

[0:55:37] KB: Exactly.

[0:55:39] PW: Skills have - they do have this nice property that - and the way you write them is increasingly to do this progressive disclosure thing. In my view, everything that a skill is accomplished in today should in principle just be learned by the AI. First of all, skills have the same problem with a lot of other things, which is they're human created and they're - human better keep it updated.

[0:56:00] KB: No, they go stale really quickly for sure.

[0:56:02] PW: They go stale, like every other document inside of it. We need to solve the problem of knowing what's stale and knowing what's not stale. We already need to solve that problem, because we need to understand what's in all these Google Docs and all these other things. In my view, by the way, I think skills are very powerful. They're way better than nothing. But I don't think the end state is all of us curating lots of things and desperately - it's like a wiki. Half the things are out of date and they're doing more harm than good sometimes. I think the end state is that there's a new learning problem, which is learn how this company works. It's like what you have to do when you join a company, and for the first year, you're trying to figure out where the light switch is.

[0:56:36] KB: Totally.

[0:56:36] PW: Ais in principle can also learn that.

[0:56:39] KB: Yeah. Well, and as you highlight, from what I've seen, AIs are really bad at understanding time. They're very bad at understanding, oh, this is actually still relevant, because this was updated recently, and this one thing is actually probably less relevant, because it was last touched two years ago.

[0:56:53] PW: Yeah. They need help. I mean, this gets into traditional search type stuff. How do you know Google search had to solve this? There's a lot of data on the Internet, a lot of it's garbage. There's things that contradict each other. There's ways to do this. You look at authority, you look at recency, you tell it very deliberately, prefer things that are more recent, prefer things that are from users who seem to be authoritative users in this company. If it's a question about finance, maybe prefer something you saw from the finance team. These are implicitly often called signals and search ranking, or the search space. The same thing needs to happen here. I think skills, or other types of explicit annotation are like, you remember the Yahoo directory? It was like -

[0:57:36] KB: Yeah, exactly. Yup.

[0:57:37] PW: Yeah. At some point, that thing didn't scale. It was like look, I'm dealing with a long tail thing. That's not in any one skill, or that thing is outdated. Google just came and was like, "Oh, we're just going to go dynamically on all this stuff," and it was way better.

[0:57:51] KB: That's a great summary. We're in the Yahoo directory era of LLM for enterprise.

[0:57:56] PW: With respect to skills. The other stuff LLMs do are complete magic, and I don't think ever should be compared to Yahoo directory.

[0:58:04] KB: Well, I feel like we could keep diving on this for forever, but we are just about at time. Is there anything we have not spoken about yet that you want to make sure we cover before we wrap?

[0:58:11] PW: I do want to do one minute to plug. We have our AI gateway that we made into a product. We have thousands of customers using it way more than developers. We actually also have this product called ontology product, which is very similarly. It's totally open. You could use it with other tools. But we've thought a lot about this problem of how to encode that enterprise data. Yeah, give Databricks a shot, if you're interested. Also, there's tons of DIY versions of these things, too, if that's a better fit for people, so I'm not overly trying to plug Databricks.

Then, I would let people know that everyone is just figuring all this stuff out and I put myself in that category. I think we've learned some things that I think will stay relevant for a little while, even the things we've learned might become irrelevant. The only way that we've been able to learn is through experimentation. Just trying different stuff, seeing what works, seeing what doesn't work. Finding colleagues and other companies, asking them for their advice. As I said, several other folks helped us when we wrote that blog, and I think that we'll just figure this out one step at a time. But it's a very exciting time right now.

[END]