EPISODE 1960

[INTRODUCTION]

[0:00:00] ANNOUNCER: Historically, databases were responsible for storing data and returning exact results in response to queries. However, AI is now bending that contract in a new direction. Applications increasingly expect structured and unstructured data to come together. This is pushing databases into territory that looks more like search, where relevance and ranking matter and results are no longer strictly exact. Agents are also beginning to write their own queries and even propose their own schemas, which raises new questions about how data should be structured, governed, and trusted. 

Sailesh Krishnamurthy is a VP of Engineering at Google, and in this episode, he joins Matt Merrill to discuss his background, how databases have evolved over the past 50 years, and where the field is heading as AI reshapes how data is queried, structured, and trusted. 

Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in backend development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at DEPT Agency. You can learn more about his work at code.theothermattm.com. 

[INTERVIEW]

[0:01:34] MM: All right. Hello, everyone. I am Matt Merrill, and I love databases, so I'm very excited to be here with VP of Engineering for Databases at Google Cloud, Sailesh Krishnamurthy. So before we start getting into all the geeky stuff, I was prepping for this interview and reading through your background, and I thought it was quite incredible. So can you tell folks about your background and leading up to what you do at Google? And then we'll kind of dive in after that. 

[0:02:00] SK: Sure. Thank you so much, Matt, for the opportunity. I have been in the database industry for most of my professional career. I started in the mid-90s. I worked at IBM, worked on Db2 and enterprise databases. And then I did something a little crazy. I decided I liked databases so much, I went to grad school. I went back to grad school, second attempt at grad school. This time I went to UC Berkeley. I did my PhD on streaming databases. I ended up starting a company around to try to commercialize my PhD work. It was an interesting time to start a company, late 2000s, a lot of challenges, lots of ups and downs, a lot of interesting learnings. In the end, we sold our company to Cisco. 

And it was a very interesting opportunity. I learned about networking, a whole new space I didn't know anything about. And then I went to Amazon. I worked at AWS and I worked on a system called Aurora. And I was there for about five years. And I came to Google in 2019 and I've been there since. 

[0:03:01] MM: And so with a little system called Aurora, I think a lot of our folks probably know that too. And that was with multiple different flavors of databases or was it with one particular one? 

[0:03:12] SK: In the end, my eventual role, I ended up leading both Aurora MySQL, but all the other MySQL engines. So there was also classic MySQL, managed MySQL, part of RDS, and other systems. 

[0:03:25] MM: That is fascinating. I think it's suffice to say a thing or two about databases. And so what are you doing now at Google? 

[0:03:33] SK: Database is probably the only thing I know at this point. But yes, I like to say I have two jobs at Google. I lead transactional databases for Google Cloud. But I have a second job. I also lead operational databases for all of Alphabet. And so whether it's Gemini or Gmail or YouTube, all of these run that same database infrastructure on the same set of systems, Spanner and Bigtable. 

In many ways, my second job is very much like the interlocutors of my first job. People who build and run infrastructure teams, they run it on behalf of various businesses. Sometimes businesses listen to them and sometimes they don't. And that's very similar to my world for my internal job. 

Again, we have a lot of legacy infrastructure at Google. It's a 25-year-old company. Sometimes, of course, we do know that we have to keep modernizing our infrastructure. One of the beautiful things about working at Google is through the years, we've been really good at taking long-term bets. We've been willing to take the hard risks, willing to try out things, which may not be very clear when you're taking the bets. 

And so the reality though when you have so much infrastructure, you can't afford to take bets for everything. And so sometimes you go and run a complicated migration to modernize some old infrastructure, and sometimes you can't afford to do it. And this is in many ways very similar to what people in the enterprise world would do. 

[0:04:55] MM: Absolutely. 

[0:04:57] SK: They would like to modernize every single system, but sometimes you just can't. 

[0:05:02] MM: Yeah, that's pros and cons. Oh, boy. I don't think when I was prepping, I fully appreciated that that was your "other job". So we might dive into that a little bit too. 

[0:05:11] SK: For what it's worth, when I got hired, no one told me about that other job. I came and learned about it when I came in here. 

[0:05:17] MM: Oh, boy. Okay. Just a little thing. Just run the databases for Alphabet. Yeah. Okay. Just so folks know, how long have you been at Google now? 

[0:05:25] SK: About seven years, basically. 

[0:05:27] MM: Seven years. Cool. So let's dive in. So you said you did your PhD on streaming databases. You turned it into a company. I think not many people have researched databases in academia and at a startup and inside a company as large as Google. So over those 20 years, what do you think has changed and what do you think has stayed the same? 

[0:05:47] SK: You know, in many ways, there have been a huge amount of change. But in other ways, there's not been that much change. Fundamentally, I joke that with databases, you have one job. Don't lose the data that's being stored. Give the exact results that you're asking for. And this stayed true for a remarkable amount of time, right? For all these 50 years of databases. 

In fact, it's interesting. A couple of years back, I was giving a talk at Berkeley, my alma mater. I think it was on May 5th or something in 2024. And I became curious. I was wondering when the original SQL paper was written. It turned out to be almost exactly to the date, maybe two days off in 1974, the year I was born. And so it's easy to forget, but this remarkable industry, the system, the set of technologies has been around now for 50 years. 

But the truth always was about building reliable systems, making sure you get exact results for your data, and something I like to call database dogma. Database dogma is really being focused on specify what you want, having a declarative notion of what you want and where the system goes and figures out how best to do it. 

And there have been all kinds of excursions around this. You had different kinds of systems, non-relational systems, various other kinds of things. But in the end of the day, database dogma has remained the same. Try to build systems where you have declarative interfaces, where people can ask what they want, and the system focuses on producing it. Just to make sure, like I said, don't lose the data, small detail, and produce exact results that you can rely on. 

One of the things I'd like to say when you say don't lose your data and being highly available, availability itself is an interesting idea that it's not just about if the system is on or off. A long-duration performance event effectively becomes an availability event. Because if you start seeing latency excursions, effectively your site may go down. A long-duration availability event effectively becomes a durability event. So all of these are deeply connected. 

So in many ways, the crux of how databases were built, what we are serving to our purpose in life, has not changed. But in other ways, they have changed a lot, especially over, I would say, the last three to four years in the AI era. But even if you zoom out from that, one of the biggest changes I think has happened maybe over the last 10 years is how sophisticated enterprise IT has become. It used to be the world that people built in corporate IT. People would buy off-the-shelf applications and, in the process, install a database. 

Enterprise IT in the past was not necessarily as sophisticated. A lot of the excitement, a lot of the change was sparked by open source software. It took a long time for the best practices to make its way to enterprise IT. That's not true anymore. Enterprise IT, I talk to customers all the time, just incredibly creative. And so the idea that enterprise IT would just take software and run it, it's not true anymore. They're innovators. And so they are demanding a lot more from these systems. There's been a lot of experimentation. There's been traditional databases where scale-up systems. 

And so while they evolved coming to the cloud in a very evolutionary path, there were also these what I call revolutionary systems, scale-out systems. Bigtable was one of them that we built in Google about a little over 20 years back. And Spanner is another one of them. And these set the tone for a whole industry where people approach this data management problem from a fundamentally different way. 

And so a lot of things has changed, I think, over the last 10 to 15 years. But I want to zero in over the last three years or four years. I think what is happening with AI is taking things to the next level. What we're seeing is it's not just about producing exact results on exact data. What we're seeing is the confluence of structured and unstructured data. And what that's leading is applications, workloads, where people are expecting that these things get put together. 

And now the database world starts to look a little bit more like the search world. Or in academic terms, it looks like information retrieval. And so it's no longer just enough to not lose your data and produce exact results. But when you're asking to combine structured and unstructured data, you need to provide the best results. And so now you have a mindset shift producing exact results to starting to produce inexact results. And so the relevance and the ranking starts to get important. And this underscores a more secular change, even when you go from databases to systems at large. I think we have to think about how you build non-deterministic systems in a fundamentally different way. 

[0:10:41] MM: As you started talking, that's exactly what I was wondering about is the exact information retrieval, because it definitely seems like that is something that's changing. Also, I might be going out on a limb here. Of course, the database's goal is to not lose any data. But in an age of big data, you know, if we lose a couple of click events or something like that, it's not quite as big a deal as losing a financial transaction or something like that, right? I'm curious if you see that at all or if that's built into any systems. I would imagine it's not, but - 

[0:11:13] SK: I mean, there are certainly systems which can choose to have a different constraint. It becomes at some level a cost issue, a performance issue, a latency issue. I think it's true that it's no longer one size fits all. It used to be that all databases looked roughly the same. And I think that's not the case anymore. 

[0:11:32] MM: That's a good lead into Spanner. So if I'm being honest, I don't know much about Spanner. I've heard of it. I've never actually used it. So for those of my fellow Spanner noobs, why would I reach for something like a Spanner instead of a Postgres, my favorite, or MySQL or something like that? And then we'll go into this new product you have called Spanner Omni as well. But let's start with Spanner itself. 

[0:11:56] SK: First, I just want to preface that Postgres is amazing. I love Postgres. 

[0:11:58] MM: Me too. 

[0:11:59] SK: As a grad student, I wrote code in Postgres. And it's really taken the world by storm. And it's really a testament to the power of an open source community. And that's a whole separate discussion we could have. And there are many, many good reasons to use Postgres or MySQL. And there are people who will use commercial databases. There's a role for Oracle. There's a role for SQL Server. And we support that too. 

I think some of the biggest differences that are there in Spanner is first extreme scale. So you are not limited to what you can run on a single node and you have the flexibility to scale out, scale in. And so that's pretty remarkable. In the past, as I mentioned before, the evolutionary systems, people came at the problem space of needing more scale than you could on a single node. And given that they were building more bespoke applications, web crawling and things like that, the requirements were different from what needed to be served by traditional databases. And so a whole era of these revolutionary systems started getting built. Bigtable, various other efforts trying to build something like it. 

At the same time, there were many things that were amazing with relational databases. Strong consistency, transactional semantics, secondary indexes, a whole bunch of things that application developers take for granted. And prior to Spanner, no one had figured out how to make these two worlds work together. 

In fact, the origin story for Spanner goes in Google. The ad system used to run on sharded MySQL. Because when it ran on MySQL, because it was so big, you had to shard it. And so that was a complicated process. And roughly, it was something like every year, you would have to figure out how the system was going to grow, and you'd spend a huge amount of time sharding it. 

And I think one year, I think it was sometime circa 2008, 2009, they came to the conclusion that in the time they would take to shard it, the world would have changed again. And this is a hamster wheel that they had to get off. And at that point, Google leadership, again, as I mentioned before, one of the amazing things about Google was those big bets decided this cannot stand. We've got to do something better. 

At the same time, there was also recollection that if you look at ads, it's very different from Google search or photos and so on and so forth. By the way, some of these consumer services that run on Spanner, you can look at the DDL file, like sets of tables, and you can stare at it and make sense of it. 15 tables, 20 tables. But if you look at ads, it's a classic enterprise application. Thousands and thousands and thousands of tables. And not just one, many, many applications. 

So if you look at the ad system, you needed the scale that was there in Google, but you also needed the functionality of enterprise applications. You needed secondary indexes, you needed transactions, you needed all that other stuff to build enterprise applications. And so we set out a build Spanner to solve this problem, that you wanted the functionality of enterprise applications, you wanted scale. And then there was an additional twist, a very uniquely Google twist. You wanted systems to run globally. And so while people had figured out some ways of trying to address scale, no one had figured out how to solve all these three together. And so that was how Google built Spanner. 

Now, you asked a slightly different question, though. You asked, "Hey, what should I reach to when I was used as an everyday developer?" And I think if what you're trying to do is something that will run in a single node, Postgres is just fine. Even if you need to run a bunch of replicas, Postgres will work for a lot of situations. 

I think what we've found with Spanner is there are enough and more people who need some combination of things. Sometimes they need global resiliency. They need to have a multi-regional configuration that's synchronous. Sometimes they need scale. Sometimes they need a multi-model database. So one of the big things in Spanner now is you can have beyond just a relational schema, you can have a graph database, you can have full-text search, you can have vector search. It's all integrated into one system. 

Having said that, I would say that's probably we're just a little ahead of the times there. If you look at Postgres's evolution, you'll see graph databases technology coming to Postgres, full text search coming to Postgres. So at some level, you will see all of these databases are seeing the same things. They are seeing this emergence, going back to inexact processing, structured and unstructured data coming together. That's the need for AI. And so in terms of that kind of functionality, I think you will see it in many other places. But scale, being able to run on multiple regions, being able to scale up and scale down trivially, these are things uniquely that you will get with Spanner. 

[0:16:44] MM: Nice. Okay. Now, that all makes sense. And then you just released Spanner Omni, which lets people run this on-prem. 

[0:16:52] SK: That's great. 

[0:16:52] MM: Which is really interesting to me based on everything you just said. So you need to run globally. I mean, obviously it's a little different, right? But why is that a big deal for Spanner? And really, how does that work with being able to pull something out of the massive Google infrastructure and allow it for somebody to deploy it on-prem? 

[0:17:14] SK: The engineering work that went into building Spanner Omni is really actually an inspiring story in its own, right? We've taken runs of this problem before and certainly knew it was going to be a hard problem. And I think the key was the technical leaders who had talked about it before decided, "We're just going to go do it. We're not going to worry about how hard it is." Again, classic Google. We'll take the hard problem. We'll start working on it without figuring out an end date and delivered it. I think it was two years. Just remarkable work, remarkably fast. 

And that's a whole separate story, and maybe that's a story or appears. Let's talk about why this is important and what does it mean. I think there are two or three broad themes. One, the nature of Spanner, many of these customers are backs. They're highly regulated. They are people who have extreme requirements. And either they needed to have some kind of business continuity are regulators in some industries. And some jurisdictions mandated that you need to run in a world where you also have the optionality to run in their data center. 

For a stress-to-exit scenario, one customer of mine who works at a bank was telling me when they were moving a traditional legacy system, and he was complaining to me about the regulator's position. He said, "I know that I'm considering moving from a system which gives me two and a half nines of availability to a system which gives me four or five nines, whether it's a single region or a multi-region configuration. And the regulators are asking me to justify it. This makes no sense because I'm actually getting to a better situation." Right? Everything is better. 

[0:18:58] MM: The regulations always have to catch up with tech, right? Yeah.

[0:19:01] SK: Right. To be fair, I think I can understand the regulators' positions. I think you'll see a lot of different positions are there. And having continuity of data in multiple places, I think is important. So I think this is one of those big requirements that really was there for lots and lots of customers that we talked to. And in many ways, it inhibited their ability to come on Spanner. So many people would say, "I want to come on Spanner. I want to take the bet. But how do I get optionality?" So that was one. 

The second, I think, big theme is customers in this AI world are scrambling for capacity and infrastructure everywhere. They are not in a unidimensional world anymore. In fact, I think two things that have happened here. One, it used to be that it was a unipolar cloud world. Now there are multiple cloud service providers. There's neoclouds, there's a whole bunch. It's a very thriving ecosystem. 

And second, it used to be that cloud vendors hated the word hybrid. It used to be that you could not even use that word if you worked at certain cloud vendors. But that's not where customers are. Customers today have a lot of infrastructure. Sometimes it's because they want optionality. Sometimes it's because every non-trivial business grows through acquisitions and integrations and acquires different businesses. And so diversity in their infrastructure is a reality. 

I was at a customer advisory forum a few weeks back and one customer passionately explained that their crown jewels will remain in their data center. So this is, I think, the reality of 2025, 2026, that it's not a secular world where everyone is running, picking a single cloud and their world is simple. The world is complex and the world is messy. And we have to react to that messiness and that opportunity. And there's a few other reasons. Sometimes the customer wanted something, desperately wanted Spanner because, look, the truth is there's nothing quite like it anywhere in the world. And so between the mix of all of these things, it became very clear for us that making Spanner available on other infrastructure, on other data centers, in other clouds was actually important for us. 

Now it comes to maybe what you're trying to get at. How does this work? How do you peel it apart? What is the fun technical part? So there's, I think, two or three separate streams here. One, the way we operate Spanner is a little different than how we are packaging Spanner Omni. In the cloud, we get operated - as you would imagine, there's lots and lots of different microservices, different piece parts, different things that we run separately. And that's organically evolved. And it's the way we operate it. We know how to operate it. 

But when we are making it available for someone to run that infrastructure, we knew first and foremost, the packaging was very important. We had to actually simplify the operation, right? You could not expect people to go and run Spanner Omni just the same way we run Spanner. So one of the important packaging things is we collapsed it all into a single binary that you can go run on a deployment. So all of the different microservices that we have, you don't need to worry about it. Download a single binary, you can run it on your laptop, you can run it on a deployment, you can assign a set of nodes, and you're off to the races. It may sound simple, but it was actually, as you might imagine, a really hard thing to do. 

[0:22:19] MM: Oh, the simplest things are often the hardest, right? I think that's incredible. 

[0:22:25] SK: It is actually a really incredible achievement, right? Then what is the infrastructure that it's running on? Certainly, storage is important. And so how we run Spanner inside Google, we benefit from Colossus, which is this really amazing system that no one else in the world has, which I think so key to how we operate inside Google. 

[0:22:46] MM: What is that, like a raw store item? I'm not familiar with it. 

[0:22:49] SK: It's a storage system, basically. It's a distributed storage system, and it's a zonal service. Spanner depends on it. One of the reasons Spanner and Bigtable are so much better than any other scale-out system is because we leverage Colossus. And so when we need to split or merge, when you have to scale out, unlike everyone else in the world, we don't need to copy data. We only have to copy metadata. It's a huge superpower. No one else can do it because everyone else keeps copies of data. And then they sit copying the data back and forth when they have load. So we have to model how we deal with storage. And so there's a whole bunch of innovation and clever tricks that we came up. 

And then the thing that everyone wants to know is true time. So one of the things that marked Spanner when we first came up with it in having a globally replicated system is to tame time. And so in order to make transactional semantics work, to make distributed transactions work, it's very important to be able to reason about time. And what we did at Google relied on special hardware, atomic clocks in the data centers, a whole bunch of things that I don't actually understand, but to be able to talk about what we call the epsilon, the delta between these different times. 

And Spanner's transaction semantics essentially depends on what the delta is. And we are able to impose things like how long you wait on commit, various aspects of the transactional system by looking at the different participants in the transaction and what is the notion of time they have. And with our understanding of the actual epsilon, that is the deltas between the time. And because of these atomic clocks, we are able to keep the delta very low, which makes Panner run like a champ inside Google. 

Now, the question was, what do we do when we go to other environments? A few things I think that helped. One, the fact that we talked about true time and other things have meant various other systems have done a better job at providing better time primitives. And so the state of the world in 2025, 2026 is very different from the state of the world 15 years back, in no small part because of what we did with Spanner and what we helped teach the world. 

And so there are better time primitives that are there. Still not as good as our hardware clocks in our data centers, which we still think are the best, but they are probably actually not meaningfully that worse than Spanner than our world in 2012. I talked to you about ads. 2008, 2009 was when we had our come to Jesus moment, right? 2012 is when Spanner went production inside Google for the ad system. 

And the world of what you can get with time outside Google is certainly not as good as what we have in our data centers. But it's ballpark how it was in our data centers 15 years back. So the world outside is kind of like the world inside Google in 2012. And Spanner was built for that time. So we built something called Software Through Time that we have available that you can run in other cloud providers and other legacy cloud providers and data centers. 

And with Software Through Time and with Spanner, we're able to provide still a very quality experience. There will be aspects of it which will not be as good as if you run inside our infrastructure. But we believe that for many, many use cases, it will do just fine. That's the secret sauce. 

[0:26:24] MM: That makes a lot of sense. So basically, what I'm hearing is, especially with TrueTime, the level of accuracy needed to scale at Google levels, you need that level of accuracy. But when you're at lower levels of scale, which that's not undercutting what some of these major customers probably need, but it's also not Google scale. 

[0:26:45] SK: It's not Google scale. And in many situations, it depends on the actual workload. You can accept a little bit of that extra - maybe it's a slightly different latency implication or something like that. In many, many situations, it's just fine. And by the way, I think other interesting things have happened. There have been customers who come and talk to us and said, "Hey, what if we install the same kind of atomic clocks that you have in your data centers? What if we do it in our data centers? And my answer is, let's talk. I'm very happy to support that. 

I think it's one of those things, in many ways, how data centers are built today are because the work my colleagues have done over the last 20, 25 years. And so each of those lessons from commodity infrastructure, from things like atomic clocks, I think they will make their way to the wider ecosystem. And as a labs researcher, leave aside my role as a Google employee, I think that's a good thing for the industry at large. 

[0:27:41] MM: Yeah, that's fascinating. If you can share, the idea of that single binary fascinates me, first of all, because I love the idea of a single binary. I don't need to install anything else. It just runs. What is that written in? Is it C++? 

[0:27:55] SK: A lot of C++ code, yes. A lot of it is primarily C++ code. 

[0:27:59] MM: All right. Cool. All right. Good old-fashioned C++. 

[0:28:02] SK: Maybe AI will help us write it all in Rust or something. I don't know. 

[0:28:06] MM: I've been talking a lot about Rust lately with some clients. Graph algorithms natively just chipped in Spanner as well. And I'm having a little bit of a hard time wrapping my mind around why would I need that in a transactional database? So can you talk about a real use case and where this clicked for a customer and why this is really valuable? 

[0:28:29] SK: It's a great question. It really gets to the heart of why running these graph algorithms natively in an operational database like Spanner is so amazing, right? It's a real game changer. First, what we did with graphs lets you natively model the data as graphs without moving the data out. You could have your data in a relational schema. You can build an overlay on top of it. You have a DDL command that says create property graph. And you don't actually move the data. You just start querying it like a graph. It's amazing. It's really, really, really powerful. 

[0:29:01] MM: It feels like a graph view. 

[0:29:03] SK: Essentially. That's right. Now, you can also come and say, "I don't want the relational view. I just want a pure graph interface." We support that too. What actually happens when you create a graph view? Effectively, in a way, you're modeling relationships that are there. And then once you're viewing it with the graph prism, maybe you ask questions differently. You ask questions around traversing the system. By the way, it's still declarative. The database dogma is not lost. You still ask, find me all the nodes, which are two hops or three hops or four hops. 

And so where this becomes powerful is in any place where there are lots and lots of these relationships. And there are many classic use cases, of course, just talking about graphs. And we'll talk about algorithms in a second. Some of them infrastructure, right? You have ports and network switches and things like that, and you want to connect them. But a lot of use cases often come in the security world. You might have users and relationships between them and accounts, and you want to look at the trust graph, if you will. 

Customer 360 is something that is very big. That's a very big graph use case. You have a lot of information about customers. You have business questions that you're trying to ask. But customers are similar in many ways. Maybe they share certain purchasing profiles. Maybe they have interconnections in social media that you're able to find out. Another very important case is around fraud detection, where you were trying to understand relationships between users and accounts and things like that. 

But tipping aside, I think the biggest interesting thing is you can ask more complex questions on the data in your transactional database without running a messy ETL pipeline. We can all run pipelines to move data and ask questions somewhere else. But do you want to? Do you need to? In large part, what we're trying to do with Spanner, with the multi-model approach, is to be able to ask more complex questions in your data in c2 without having to move it around. 

And graphs are all about relationships. So the first version of what we did with graphs was made it easy, like I said, to define these graph views, as you put it, or a graph overlay and start asking questions, make these things perform like a beast. But then what we started seeing was, again, a little bit of inspiration from what we're doing inside Google. So inside Google, we've been building these graph algorithms for decades for solving many complex problems in the web space. But now we found they're all actually very transferable to problems that customers face when they think about graphs. 

So a classic question, financial fraud detection, right? It really clicks for many customers. Sophisticated fraud, like money laundering or synthetic identity fraud, often involves networks of colluding accounts. And the problem is that these fraudsters create these complex webs of accounts and transactions to try to obscure their activities. And they're always moving. It's an arms race, right? It's a cat and mouse game. As your systems become more and more complex, they're changing things, they're doing different things, which, by the way, is one of the challenges of having to keep moving the data of my operational system to another system and run it. Scans are moving very, very fast. 

So analyzing individual transactions in isolation often misses the bigger picture of coordinated fraud. So how Spanner Graph helps is that banks and payment processors can store customers and transactions in Spanner. They can model this graph, this data as a graph, like I said, as an overlay. Accounts are nodes and transactions are edges. And you have very interesting community detection algorithms like weakly connected components, modularity clustering that you can now run natively in Spanner because I have this big monstrous graph. I want to keep finding out what are those sets of related people, right? What are these communities? What are the clustering things that you're trying to do? 

And once you detect these suspicious clusters, some of them may be suspicious, some of them may be not, now you want to zero in on who are the most influential people within that suspected cluster. Now you can squint at this problem and say, "What is this like?" Actually, it's just like PageRank. In fact, it is PageRank. You can run PageRank on the nodes within that cluster. And so although PageRank was originally for web pages, it's actually really very, very good at identifying who are the most influential nodes or the central nodes in the network. 

So not only can you figure out here's the colluding set, you can now zero in on your actual problem space. So I think this is kind of a great, great example of how you could actually run these things. And it's really valuable to be able to do this in c2 without having to move this to some other system. 

For example, I think we have a customer, DANA Pay. It's an Indonesian digital payment solution. And so they are using Spanner Graph for their anti-money laundering use case. And by running these algorithms directly within Spanner, they can do a whole bunch of things. They can detect suspicious networks much faster, again, without having to go through an ETL cycle. They can identify key players within those networks. They can potentially flag or block transactions. And this is where the real time comes in. Sure, you can go run this in an analytic system. But if I can flag and block the transaction, I can prevent financial losses and really avoiding these unnecessary operational overhead. 

So maybe in general, the average transactional database doesn't need PageRank. But what you really need is fast, actionable intelligence on relationships within your fresh operational database. And native graphs on Spanner deliver exactly that. 

[0:34:32] MM: I can see the value now. It's a proximity thing. It's built in right into the data. So that makes a lot of sense. And I would imagine even some places not at scale that can build from the ground up can take advantage of that and not have to worry about ETL and things like that. 

[0:34:47] SK: Right, right. And yeah, it just makes it easier. I think SoundCloud, for instance, they were running graph algorithms and batch more for years. I mean, a lot of times people may have been thinking, "Well, that's the only way to do it." And I think one of the things I told you earlier about how impressed I am, how inspiring it is to work with enterprise IT and developers who are so smart. I think as you provide more things, we get surprised. They'll find things that we didn't even anticipate. 

[0:35:14] MM: I think it's fascinating. Something that I'm seeing like a pattern coming up is like you guys dog food this amazing technology with this amazing scale problem that Google has. You productize it. And then it's out in the marketplace and there's also more creative things that happen. I don't know, it seems like a nice circle. 

[0:35:34] SK: Indeed. 

[0:35:35] MM: So let's zoom out a little bit. For a long time, the first hard decision in any project was which database? How do I structure my data? So now we have LLMs, which are happy to pretty much gobble up any data that you might want to throw at it, structured or unstructured. And we have agents writing their own queries. And so what actually has changed in the last few years about how you should structure data? And is schema design still relevant? Is it not relevant? I think that that's a little bit of a rhetorical question. But I think what still remains important about schema design? 

[0:36:12] SK: By the way, it just occurred to me, there's, I think, a famous Dilbert cartoon strip, which says the first thing you do is you design the database. I don't even know what I'm doing. I'm going to design the database. 

[0:36:23] MM: I think I remember that one. Yes. 

[0:36:25] SK: It's a very interesting question you're asking. And I think, just to be very candid, I don't think we know everything that is going to change. It's changing very fast with AI. Having said that, there's an enormous amount of applications, bespoke applications, off-the-shelf applications, that I don't think will disappear overnight. These applications still exist. 

Now I think there's an urgent need for them to be more easily used in agentic applications. So even before you think about schema change and what needs to happen, I think you have perhaps a more urgent pressing problem, which is how do you activate and light up the existing database for these systems? 

And so in our vision of the agentic data cloud, we think there are three important pieces. One is the AI native infrastructure. And I alluded to you before, things like graph, by the way, are very relevant. Vector search, full text search, all of these things are very important as you imagine the world of structured and unstructured data coming in together. So that's the first. 

I think the second thing we also talked about, we think in this world of agentic AI, a lot of gaps, friction needs to be eliminated. And that's a big part around maybe a borderless world. You have these lake houses which are spanning clouds. You have things like Spanner Omni. We have AlloyDB Omni for Postgres applications. I think that's kind of the second thing. 

And the third thing, perhaps starting to get very relevant to what you're asking, is all about trust. And trust really means in order to light up these databases, you need to have the context. You need to know what's in these systems, what's in the data. And you talked about schema design. Yes, it used to be you have to design schemas. But it turns out for agentic applications, just the schema itself is actually not even enough. Forget getting to how schema changes. 

Back in the day when I first worked on Db2, almost 30 years back, I was wondering, why does the SQL standard let you say add a comment? I don't know if you know, when you say an object create table, you can add a comment to it. It seemed like an - 

[0:38:32] MM: Extraneous. 

[0:38:33] SK: Theoretical thing or something. Now that kind of stuff is actually interesting because it now tells you what could be used to record what is the meaning of something. But more realistically, I don't think we're going to depend on a comment in a schema. You're going to try to gather information about the schema. 

So while the structure of the schema is an interesting question we should talk about, I think perhaps the more interesting question is where's the context around the schema? I'll give you a very simple example. If I have a table with a billing address and a shipping address, It's two columns. If one of them is null, there is some implicit knowledge that these two are the same. That's not there in the schema. You need to extract that information. That's the critical context. 

So for LLMs to activate your operational databases, and I think before you even think about new greenfield applications, this is the pressing need to light up those databases. You need to have the metadata on more than the metadata and the context about your data in order to have agents go ask questions. And it's not just about things like that. If I have a column called city and I have an agent which is going to use an LLM to generate a SQL query to query the database, somehow something needs to tell the agent that the values in the city column, are they in lowercase? Are they three-letter airport codes? Again, this information about the data is super important. 

Now, I think so one stream is, yes, schema and context. Schema is important. What's more important is context. I think that's the first piece. I think there's a second piece, which is also interesting, which is, is there a world where you have citizen developers just generating apps? And I was reading some stat about how the number of applications on number of repositories in GitHub is exploding. 

We have a partnership with Google DeepMind with AI Studio where you can go and say when I'm building an application in AI Studio, you can say, "I want this to be a persistent application." And you can choose either Cloud SQL to store the data in relational format or Firestore to store the data in document format. More practically, you ask the AI, the AI will decide. We are just actors on the stage, I guess. 

So in things like that, an interesting question, is the AI going to define schemas? Well, the truth is AI is going to define schemas the way humans have been designing them because that's what streamed these AIs. So I actually think even when you look at that, AI models will tend to pick this kind of schema structure that made sense. 

Now, I think it's very valid to ask, is that the right one in a world of AI? Someone is telling me about a situation where they realize they don't care about abstractions of code as much as they used to because they're like, "Yeah, sure, I'll just have AI go copy it and make changes." I don't know. I mean, surely that's a path towards more chaos. I'm not entirely sold that the general rules for defining and structuring schemas is going to be fundamentally different if AI is generating the code or humans are generating the code. 

But I think there's a third piece. The third piece, in my view, is there is a world here with new kinds of applications where the question of what is an application changes. Today, an application has what I like to call four walls. It has a boundary, an API surface area. I think we're going to see a new world where applications are ephemeral workflows. The application as such doesn't exist. 

Maybe today, one morning, a knowledge worker comes into work and says, "You know what? I have system A and I have system B. I want to create some more complex purchase order that combines them. Well, what would they have done in the past? Well, maybe they'd have done a contract with a large system integrator, paid a lot of money. We have got some results after six months. Whatever it is, right? But now in this new world, they're just going to ask the AI to go do something. 

Now, where is that information that connects all these things stored? It has to be stored somewhere. So you need to store these trajectories on the data, on these agent interactions somewhere. But systems also need to be more flexible. For instance, if you travel internationally, you're probably well advised to call your bank and put a fraud alert so that your credit card keeps working. Well, what if you decide to chuck your job or go and spend a week in each country in Southeast Asia and come back after three months? And so in the AI world, you might call your bank and say, "I want a fraud alert. I want one week in Vietnam, next week in Laos, the next week in Cambodia." Well, the bank system is probably going to choke because people who designed the system didn't think about the world where you have setting 20 different fraud alerts. But I think in a new world, people are going to expect that. Expectations are different. 

So I think one of the interesting questions is, should the schemas itself be more forward-looking, to be more flexible? I think maybe this is really the spirit of your question. Sorry, I came to it in a slightly long-winded fashion. 

[0:43:22] MM: No, it's actually not what I was angling at, but I think you're getting at something much more interesting. So please keep going. 

[0:43:28] SK: But then there's also this interesting question that when we look at - think about agents on data. We're all being very conservative right now. We are trying to say, "I want to be doing primarily reads because everyone's generally scared." right? No one wants to have the Bobby tables thing going and deleting records from a database. And we've seen the horror stories already. Bad things happen if you don't think about security and trust really well. 

In general, I think rights are going to be a complicated, challenging space because you're now going to try to think about how agentic applications or these ephemeral applications are going to do rights. How are they going to do schema changes? It's not fully clear to me what you'll need. 

For instance, even writes, maybe here's a proposal that when you're trying to make a change and modification to the database, you don't actually make the change. You make a proposal for a change. It's a proposed change, and it doesn't actually make it right there. It may happen through some set of steps. 

So I think the entire way in which data has changed and schema has changed, I think it's all subject to revision. Applications as we know it, I think will change. But I think the existing applications are not going to disappear overnight. Business logic, which maybe applications may be reviled, but business logic lives in applications. And so we have to activate them. 

[0:44:46] MM: Yeah. You said the key thing, the data has to stay somewhere. And I think I've talked to a lot of colleagues and other people. And I tend to agree with this. And maybe I'm the one that's behind. But I'm certainly willing to let agents do a lot of my application code. But where I kind of draw the line is at the data model. 

[0:45:07] SK: That's right. 

[0:45:07] MM: Okay, you could take a cut at it, but I'm going to review the heck out of that. And I'm going to make sure it makes sense to me and that I can pressure test it based on my own experience and where I think things are going to go and creative thinking and things like that. It sounds like it's very much in line with that. 

All right. We already said this. I think we're both Postgres nerds, which is great. It seems like every AI startup stack is Postgres plus PG vector. And so Google has AlloyDB, which is basically like a souped up Postgres. What breaks when people try to do serious vector search on vanilla Postgres? And what did you have to change under the hood on AlloyDB to fix it? 

[0:45:45] SK: There's, I think, a few different parts to this. One, PGVector is awesome. As you put it, I think it makes it very easy to use. And in fact, there's a whole side thing about the Postgres ecosystem and extensions, which let the experimentation go and PGVector took off. And so for many, many use cases, PGVector with the HNSW algorithm works really well. 

Now, at Google, we've been working on vector search for, I guess, over 15 years now. And it's at the heart of a lot of things that we do, like YouTube and search and so on. And so we have this amazing technology called ScaNN. And it scales to over 10 billion vectors, delivers very much more performance in terms of latency than just HNSW. And one of the things we did was we used ScaNN, which is a tree-based algorithm, and made that available inside AlloyDB. 

And we tried to make it look and feel like PGVector. We also offer the Vanilla PGVector so that the applications shouldn't have to change. And applications that work against PGVector will also work against ScaNN. So that's the key part. Now, there's a lot of challenging work in making that happen inside Postgres. And also various things that we've done on the side. How do you go build these indexes? How do you go maintain them and update them? 

And there are many, many reasons how we run AlloyDB with ScaNN to really get faster performance. So in our tests, we can get 6x faster vector queries than HNSW, PGVector, and standard Postgres. We use 4x less memory, which anyone who's run a database knows. It's kind of always a good thing. Because to use less memory, you can keep more for your buffer cache, which makes your performance better. But we also support things like distributed index builds. And so in general, for many, many use cases, for the most demanding use cases, we have a big advantage with ScaNN. 

We also have support HNSW and have taken some of our improvements in ScaNN and also made it available in HNSW. And the reason we've done that is - 

[0:48:00] MM: you've contributed it back to the libraries or is that a product? 

[0:48:05] SK: I don't know what we've contributed back, but we certainly have made it. We are very happy to contribute this HNSW. I don't want to say something that's inaccurate. I'm not entirely sure if we've contributed or not. 

[0:48:16] MM: Totally fair. 

[0:48:17] SK: The interesting observation from us here is vector search feels a little bit like what we'd call an ensemble problem. There are different domains where there are different systems that are better. There will be probably some kinds of systems where some kind of regimes where HNSW is fine and maybe it's a simpler approach. But what we tend to see is in the most demanding workloads, when you have high scale, you really need something like ScaNN. Well, we see value in both. And so we want to make sure both really work well. 

But I think there's a whole set of things that is more than just vector search. I think a lot of people were very focused on, "Hey, I want to get vector search." As you said, I have the simple stack I get going. What we realized is what gets more interesting when people go beyond vector search? Rarely do I see people asking queries which are only a vector search system. They are more typically asking queries which involve multiple predicates. It might be vector search and another predicate on my regular table, just a relation predicate. 

Classic example, I have a catalog and I have price. Price is a value that is in my table. And I want to ask a question which combines the two. We call this filtered vector search. And in filtered vector search, my query may have two predicates, a semantic search predicate, which is on the vector index, and a predicate on price, as an example. Now, how do you know which one to probe first? This is a classic problem in dealing with vector search. 

If my problem space is simple and I only have to go probe a vector, then sure, I go ask, check the vector index, get the answers. But in practice, I think what we see is this challenge that people want to ask more complicated questions. If you go probe the vector index first, maybe you get 20 results. And then you apply the rest of your other predicates and none of them may be satisfied with other predicates. Because I say price less than $50, I get nothing, right? And then I go probe my vector index again and say, well, get me the next 20. And then I go check those next 20 and I still don't satisfy. I have to keep going down this path, right? 

Based on the selectivity, maybe I'm better off first probing the non-vector index. And finding the set of results and then applying my nearest neighbor algorithm without even going and probing the vector index. But again, I don't know that. Suppose I probe the non-vector index. Suppose I do pre-filtering. What I just described was post-filtering. But now suppose I instead do pre-filtering and I probe the non-vector index, maybe I get a million records. And then I can't go and run nearest neighbor on a million records. I have to go. That would be a bad idea. That's why you built a vector search. 

And so no one else really managed to address this problem. In AlloyDB, we built this really amazing filtered vector search system where we are able to probe multiple indexes on the fly and figure out which is the right order in which to probe things. 

[0:51:13] MM: Interesting. How does it do that? I think this is an interesting point. One of the key techniques here is when you have multiple indexes and you have a query and they're all in the same system, we can actually probe both of them at the same time. And we try to figure out quickly which is the right strategy and change it on the fly. 

[0:51:31] MM: So it's a divide and conquer. 

[0:51:33] SK: Yeah, it's an adaptive technique where we are trying a couple of things. It's not quite divide and conquer. You're trying a couple of things and trying to figure out which is a better fit. And we are adapting the right approach based on what your actual situation is. 

It's an interesting piece, which is worth thinking about because you describe one stack which startups use. There's another complete stack, which is people say, "Oh, I'll go run a custom separate vector index, not in Postgres." Right? And that's also a path that people go. If you are doing a pure play vector search on just a corpus of text, data, whatever that isn't changing, maybe that's a reasonable approach. But I think, again, a few milliseconds after people come up with this, queries become more complicated. As I said. Developers are insanely inventive and creative people. They will find ways to ask more complicated questions right away. Our AI will do it for you, right? 

[0:52:28] MM: Users are very creative too. 

[0:52:30] SK: They do, right? 

[0:52:31] MM: Yeah. Customers are. 

[0:52:31] SK: So people will want to ask questions that straddle these worlds. It comes back to what I said. What has changed in a big way? People don't want to just look at unstructured data. People want to look at structured and unstructured data together. This all comes together, right? So when you're asking this, I actually think it's a really poor idea to go build a completely separate vector system because then you don't have a choice. You have to stitch these systems together at the application level. And that's incredibly painful because who's going to do this pre-filtering or post-filtering? And unfortunately, the pure play vector systems don't have a good solution here. And so when you're having multiple indexes under the control of a single system, you can do these kinds of very inventive, creative techniques that deliver tremendous value to customers. 

[0:53:16] MM: Very cool. Okay. 

[0:53:17] SK: And in addition to all of that, one of the additional things that we're doing is we're also combining high speed vector retrieval with full text search on operational data. So that gets hybrid search is now getting interesting. So your search is not just vector and non-relational but also full text. And all of these things are really coming together. 

[0:53:37] MM: Wow. What a world. It's a long way from select-where. 

[0:53:41] SK: That's true. 

[0:53:43] MM: So let's move on to MCP servers. So you all just released - I think it's called toolbox for databases. 

[0:53:50] SK: MCP toolbox for - 

[0:53:52] MM: MCP toolboxes. And I thought that was amazing. It's open source. Basically, MCP server on top of your databases. So you don't have to roll it yourself, which is fantastic. If I'm a dev, I'm going to take the thing that Google created and assume that they did it pretty well. 

And you've got those managed MCP servers for Alloy, Cloud SQL, Spanner, et cetera, et cetera. I think most people, most audience members here are engineers on the application level. What are the possibilities here? What are your teams at Google doing day to day with these? 

[0:54:26] SK: Just to be clear, my teams at Google are building databases and building infrastructure. So they're not necessarily actually using as much, but that is an area where we are. And this is for our user interfaces. So we're doing some really interesting work on something we call gen UI or generative UI. 

And here, if you look at Google Cloud's managed databases, we had a humble application, too. We store data in Spanner, the version of Spanner that we use inside. You use your Google Cloud, you create instances and databases, and that information is stored in the metadata system. It used to be that any time you built a new feature, you had to build a lot of complicated UI to help make that feature be available. 

One of the things that we are now doing is generating UI on the fly. And the way we're able to do that is by going and connecting to the MCP servers. Because each of these, if you think about the UI, it's a humble application that used the underlying API servers. And now what we figured out to do is, based on what users are asking, we're able to generate UI on the fly by connecting to these MCP servers. So this is an example of something that my teams are doing, right? And we think it's really exciting because it's not just about saving time. It's about creating these artistically crafted user experiences based on what someone wants. We can deliver something that is really awesome for them. So I think that is really interesting. 

Now, from an actual customer's perspective, as you said, there's both. You can use toolbox, you can use our managed MCP services. I think you'll see different kinds of use cases. The best thing about our managed MCP services is that it's enterprise grade, it's got governance, it's got all the bells and whistles. And so a lot of customers will absolutely use it, will gravitate to it. And I think it's great. 

I think it's also true that there's a set of developers who are on the bleeding edge. They will be needing more. And that's why we started building MCP Toolbox, to really learn what developers need. And they need a whole bunch of things. For example, they need custom authentication mechanisms. They may be running in their own on-premises environment. They're not using Google Cloud. 

And so our problem statement when we set out to build MCP Toolbox was how do you make it easy for developers to connect to databases irrespective of where they are. As I said, part of the agentic data cloud mission is openness. And so we want to solve for people no matter where they are. They can be on their data center and another cloud. We want to make it really easy to, as I said before, activate operational databases in agentic applications. 

Now, when they are in the Google Cloud environment, we also want it to be a very easy environment. And that's where you have managed MCP services, right? And so things like the GenUI application that my team has built is a great example of using those managed MCP services. But it could be various other things. I think classic example is on the control planes. I'm a SWE who is relying on a bunch of AI agents, and these AI agents need to go create databases, run deploy applications. For all of that, this managed MCP is perfect because it just directly connects to the control plane and so on and so forth. 

But there are also things like custom tools that we have where custom tools that people often are using in the world of MCP, where they're not just issuing execute SQL, where they may be doing something like search for a flight. And that might be a custom tool someone might try. And that tool ends up going and runs a specific query, select star from flight stable, things like that. And so for scenarios like that, MCP Toolbox, I think, is great. 

I think what you'll see in general is each of these pieces of functionality that we are learning by working with customers in MCP toolbox, if it's valuable, it'll also make it to managed MCP, right? And so one way of thinking about this is open source. We want to be where people are doing things. We want to learn from our customers, from developers. And based on what we are learning, that influences the roadmap of managed MCP. Both are valuable. 

[0:58:50] MM: Very cool. The Gen UI stuff, is that internal to Google? Is that something that is widely available? 

[0:58:59] SK: I think we have some version of it for observability in our consoles. We already have it available. I don't think it's called that. I might have talked out of turn and used an internal project code name. I don't think we give it a name. It's just part of the user experience in our database console. 

[0:59:16] MM: Yeah. 

[0:59:17] SK: That's certain kinds of things we are having. 

[0:59:18] MM: The reason I ask is it's a very interesting - so we had that conversation earlier about metadata and the context of data, right? The MCP server can help expose that and make sense out of it. And to be able to kind of, I guess, on the fly, create a UI is a really, really interesting thing. 

[0:59:36] SK: Powerful. 

[0:59:38] MM: Maybe that could be a follow-up or something like that. But if those two things paired, the average application developer would have quite a pair there to say inspect the data and show something based on what we found. 

[0:59:49] SK: Correct. I think our own observation was, if you looked at our UI infrastructure, we had a bunch of widgets that did various kinds of things. And so what we were pairing up was the managed MCP services for the underlying systems. And the ability packaging up those UI widgets also in some kind of MCP tool, now you have an AI model that's able to say, "I want to combine these things and put them together." That's an example. 

But you're absolutely right. What we did, any customer can do exactly that kind of work. I think the little bit of the trickiness always comes back to, "Well, what is read? What is write? How ready are you to go and let agents go right on to the operational database?" 

[1:00:39] MM: Just to correct myself, I searched there. I had heard of GenUI before and I wasn't sure. So that is an architectural pattern if you look it up. I would encourage audience members to look it up. At least I think it's just an audit pattern. Yes. And there's different implementations of it. There's an ecosystem. It's fascinating. And I think paired with that MCP toolbox, it's something very interesting. 

All right. Okay. An LLM with a live connection to your production database, it's a great horror story waiting to happen. With all this knowledge of database, how do you think about how to control the blast radius for something like this? We just led into this about talking about being careful with rights. What do you think is non-negotiable before anybody puts an agent towards prod and like not obvious stuff like lease privilege and things like that, but the stuff that really kind of keeps you up at night and that you're thinking about and that you want people to know to avoid? Maybe some anonymized horror stories from clients or customers or something like that. 

[1:01:42] SK: I'll talk about two or three different things. One of them I think is a particularly interesting challenge when you're trying to - if you think what you want to do is have an LLM or an agent generate a SQL query on the fly and go against your operational database. As a classic, "Hey, wouldn't that be an obvious win?" 

Here's an interesting challenge. If you're connecting to an analytic system, your persona is maybe a business analyst or a data scientist. The analytic systems, forget AI today. If a business analyst is connecting to your analytic system, they're connecting with their credentials, and there is access control typically built into the analytic system itself. These business analysts also typically have wider access to data. The nature of these kinds of BI queries are aggregate queries. 

It's a very different world for operational databases. In fact, it didn't used to be that different in the early 90s when you had client server applications, when you had clients which would take clients and connect to the database. But with the web, with three-tier architecture, maybe late 90s, that changed fundamentally. 

For an operational database, the end user's credentials are not actually part of the operational database schema. Databases have users and roles and groups and privileges, but the end user's information is not there. And I think one reason is because with the web, the number of end users went from hundreds to thousands, millions, hundreds of millions. So you can't really manage the data that way. 

And so with the web, with three-tier applications, you have the application connecting to the database as a service principle and figuring out what an individual logged in user can actually access using where clauses in SQL queries that are scattered in the application. That's the state of the world. 

If you want to have, let's say, an end user - and by the way, this could be an internal user or it could be an external user. Let's say you are in a car rental company. You have, I don't know, 2,000 branch offices locations, and each of them has a manager who can ask questions about the system. Now, obviously, security is important. They are allowed to only ask questions maybe about cars in their store, not about something else. Or maybe they can or maybe they can't. 

You can't just let the agent go ask questions on behalf of any user when that agent effectively has a service principal access to all of the data. So you have to solve this problem. There's no solution, no way out. We have a solution. We've built something called parameterized secure views. And this builds on some really awesome Postgres coolness. And now we're bringing it to all our database portfolio. 

Postgres has something called secure views. So you can define a secure view. And that has a very interesting property called a security barrier. And with a security barrier, it makes sure that you cannot engage in an information theoretical attack to extract information. The basic idea is, when you have this kind of an attack is to say, write a query, which will crash the system. You write a query with a divide by zero with a where clause. And if you don't have a security barrier, the Postgres optimizer will be too smart, will push this predicate deep into the system. And if you don't find anything or you find something, you may end up not getting a result, or it'll crash the system, or you'll get some error. And that error will tell you lack of information is something that's valuable for you, right? It'll actually tell you that. And so Postgres's crisis security barriers prevent that from happening because it will make sure no predicates can be pushed inside the table. 

Now, a parameterized secure view takes us to the next level where you can apply a parameter. Now, the kind of parameter that you might apply in the where clause here is essentially the same check for authentication. The authorized access, which I mentioned, is there in three-tier web applications, in where clauses and SQL queries scattered to the application. But we take that essentially from the application logic, and you define that in your DDL for this parameterized secure views. 

So now you have your PSVs defined for your schema. And when the agent calls the agent, it's part of some orchestration system. There's a logged in user. The orchestration ends up calling the LLM and getting the SQL query. And you can attack the LLM. You can convince the LLM, spoof it. It can generate whatever query you want. But when that query gets sent, it only has access. It's a special service principle for the agent. It only has access to the PSVs, to the parameterized security. So it doesn't have access to the underlying tables. That is easy to put in. 

And we take that logged in user's credential and have it as a side channel. It's bound to the underlying parameter. So the query can have whatever it wants, but the view is on the fly, only restricted to the user's credential. So it's a very important piece, critical piece of infrastructure. Without it, there's no way to get the kind of security you need for agentic applications. 

[1:06:57] MM: So that, I don't know if we call it role-based access, but that user-based access is stored alongside the data or managed alongside the data, I guess, maybe is the right - 

[1:07:08] SK: Effectively, what we're saying is don't change. You don't need to put roles inside the database. But you can take what was written as a query in the application and put it into the DDL. Now you can let the LLM do whatever the heck it wants. We can still guarantee that you don't access. 

One of the papers I read when I started my PhD program in Berkeley was a classic paper from 1975 called the End-to-End Argument in System Design. It's a fascinating paper. It basically says something like, if you have any kind of distributed system, and whether you want to do encryption or security, you need to solve this at the edges of the system, at the ends. You cannot solve it in the pipes. And that's a really interesting design primitive. And that's really the same primitive. The end-to-end argument is you need to have security right at the level of the data because we are living in a non -deterministic world where agents and other things can do anything. So I think this is one of the most important pieces in this whole topic of, "Hey, LLM with access to production databases." 

There's another interesting topic here, which is isolation. It's very similar to the set of problems with analytic systems, analytic queries on your transactional database. When you go run your analytics, if you run batch scans on your transactional database, you can impact your transactional database, which is one of the reasons why people offload sometimes data from the transactional database. 

And so we've done a lot of work over the years, done isolation and figured out how to solve this problem. I talk about horror stories. I was talking to a customer in a bank about these things like PSVs. And the customer said, "What are you talking about? We just spent years making sure batch scans can't run against your operational databases. Do you want me to run agents on it? Have you gone out of your mind?" Right? It's a totally valid question, right? You have agents and agent swamps that are asking questions. But I think the same lessons apply. 

Just like we learned how to isolate transactional databases, we need to be able to isolate agentic applications. I think some of the things that can be worse because it's not easy to control agents. Agents can run bad queries. Agents can run malicious queries. I think the lessons that we've built, we've taken from building transactional analytical systems actually apply to this world. 

And then other things, I think we need to have the right observability. You need to have the observability that tell you even little things. Can you look at your Postgres database and say, "Give me a list of queries that came from an agent versus from the application?" You need to build those kinds of guardrails around this. 

[1:09:44] MM: Yeah, it's almost like a little bit of real-time threat detection for what's happening to your data. That's fascinating, the comparison against how. Because I mean, I've never even really - I come from an era where I always just had transactions. And it wasn't always that way, was it? Right? And you had to learn that. Wow, that's fascinating. 

Okay. Let's close it out. I got two closers for you. And I try to make the closers kind of something that folks can reflect on their own roles. So you have said in the past that Google engineers can use AI as heavily as they want, but whoever commits to code owns it. You said that something like three quarters of new code is now AI generated at Google. How is that actually playing out on your teams? And I think for your database teams, what is the database engineer of, let's say, a year from now, two years from now, spend their day doing? What do you think? 

[1:10:36] SK: Look, honestly, I don't know what will happen three months from now, but - 

[1:10:40] MM: Me either. I was hoping you could tell me. 

[1:10:43] SK: Certainly, I think things have changed enough that if people had priors from six months back, they should at least revisit what is possible and what is not. So a lot more is possible. I think you have to be careful with numbers because they vary so much based on kind of what you're talking about. 

My gut sense is if you're talking about something like the distributed consensus protocol, that's Paxos technology, that's the heart of Spanner, my gut is that that's actually probably still quite hard for an AI agent to directly go and change in a big way. And we'd be very, very careful about it. But when we say AI code is generated, it means so many different things from refactoring code all the way to maybe code commit where in the extreme, so on the one side, it's just simple refactoring of code. And the other extreme is I have a whole piece of PR with a self-contained unit of code where the person who built it did not even enter an IDE, right? Did not enter Emacs or Vi. So that is, I think, one extreme. 

The other extreme is, I'm in the IDE, I'm just doing routine refactoring. And there's a whole continuum of steps between these. And so I think what you'll see right now, today, is a lot of variation, that there'll be areas where I think you'll see a lot more success and areas where you see less success. 

What we work with our teams, and I think this is probably useful for most people because of a lot of variance, I think it's very important to build the right harnesses and skills for each work group, each whatever group of engineers working on some coherent whole. And so like everything in the world of AI and agents, I think the key is evals. You need to have a really good objective way of knowing, "Is this thing mostly working or is it not working?" And so I think it's very important to continuously measure and figure out what the reality is. 

And another thing I think that is particularly important is the space for experimentation. I think it's really valuable to have pick a project, pick something which you go and say, "I'm going to focus on pushing the boundaries of what I can get with AI." And when you hit the obstacles and the roadblocks, which you will, I think the engineer's imperative and the right imperative normally would be, "I hit a bottleneck. I'm just going to drop down to the way I normally do it." Go solve the problem because that's what we're asking, right? Solve the business problem. 

But I think we need to have some set of efforts where we say, when you hit the bottleneck, fix the bottleneck, don't go around it and accept that it'll take longer to deliver. So I think these are some of the things that we do, which I think are probably the more useful in general. 

[1:13:35] MM: All right. Last one. What is the one piece of advice you want our audience to take away from today? I know I'm going to be thinking about data level authorization thing, but what do you want folks to walk away with today? 

[1:13:48] SK: You know, we've spent the last 25 years learning how to build deterministic, scalable systems. Being deterministic, being predictable is such a big part of how we've learned to build systems. We take it as an article of faith. I think we are living in a fundamentally different era. And in this era, as systems people, as thinkers, as implementers, as practitioners, we need to be ready to deal with the world where data is messy, where it's not deterministic anymore. 

And in some sense, I think most people who are engaged in this exercise have to go through stages of grief bargaining all the way to acceptance. And they're somewhere in that stage where first people start fighting about how do I make it deterministic? How do I make this AI thing deterministic? I think that's not a successful path. That's not the right path. 

[1:14:47] MM: I think most people hit it, and they hit it fast. Yeah. 

[1:14:50] SK: Correct. I think we have to embrace the chaos. And that means putting ourselves in this kind of weird spot, right? It's not how we've trained ourselves as an industry for - we're talking probably two generations of software engineers who've learned how to build things in this way. And so we have to accept that we are in a world of chaos and live with that chaos. 

And as part of that, I think evals, just what I said a few minutes before how we operate in software engineering applies to the kinds of systems we build. And I want to tie this back to where you started. You talked about what has changed. I think the fundamentally big thing that's changed for my area databases is that we're dealing with structured and unstructured data together. That we are talking about we're trafficking not in exact results, but the most relevant results. And so being able to have evals to understand the quality and the business outcomes I think is a very big part of how you can embrace the chaos and actually make sense of it. 

[1:15:55] MM: Yeah. That's something that I'm learning too, is there's no longer a right or wrong answer. There's a spectrum of relevancy. I think that that's - yeah. 

[1:16:03] SK: That's right. And we are all somewhere in that. 

[1:16:05] MM: We're all just floating in the chaos of relevancy. This was fascinating. I really enjoyed this conversation. It fulfilled all of my database nerd dreams. So thank you for being here. 

[1:16:16] SK: Thank you for the opportunity. I always love geeking out. 

[END]