EPISODE 1951

[INTRODUCTION]

[0:00:00] ANNOUNCER: Retrieval has become one of the central problems in building useful AI systems. The standard approach to grounding a model in one's own data has been Retrieval-Augmented Generation or RAG, where an agent searches a vector database for relevant information at query time. That pattern works, but it has limitations such as retrieving information that's not truly relevant, repeating the same lookup work on every query, and producing inconsistent answers to the same question. 

Pinecone is a vector database that's widely used to power semantic search and RAG at scale. The team recently developed Nexus, which is a knowledge engine that reframes context as a first-class precomputed asset rather than something reassembled on the fly. The approach borrows the database concept of a materialized view and curates context once into a versioned artifact that carries its own schema, metadata, permissions, and lineage. 

Jörg Schad is the VP of Engineering at Pinecone. In this episode, he joins Kevin Ball for an in-depth conversation about the frontier of retrieval technology. They discuss pre-compiled context, how context artifacts are curated and versioned, much like code, how metadata and semantic layers help agents choose the right information, and much more. 

Kevin Ball or KBall is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

[INTERVIEW]

[0:01:58] KB: Jörg, welcome to the show. 

[0:02:00] JS: Thank you so much for having me. 

[0:02:02] KB: Yeah, I'm excited to get to talk with you about this. Let's start with a little bit of your background. So, can you give us the TLDR of your history and how you ended up at Pinecone and Nexus? 

[0:02:12] JS: Yeah, sure. I would say I was really lucky I could follow my passion, which is probably database systems, where I started out what almost 20 years ago with grad school. Worked on distributed query optimization back in those Hadoop days. Really, really long time ago. Worked there on name nodes and basically how can you use data duplication and then route a different query. So that was really interesting times. And just looking back, it's super cool to see how it developed since MapReduce days. 

I kind of like followed along with that passion. I was over at SAP working on HANA in the early days. And then at some point figured out that large enterprises, maybe they're fun for a while but not for the rest of my life. And joined this startup back then called Mesosphere. So Apache Mesos. It was like an open-source project somewhere in between open source version of Google's org system, their internal cluster scheduler, and kind of like pre-Kubernetes. Built a lot of large scale systems across like Twitter, Netflix, Airbnb, where we can openly talk about. Also worked in the early days on Kubernetes. I recently checked all my code is out of the codebase by now, but those were like interesting times. 

Went back into the database space. Was a CTO at ArangoDB, which is kind of like a graph database. Interestingly, also worked on early GraphRAG graph retrieval when that kind of started up. We even built like our own vector store in that where we can maybe come back to a bit later when we're talking about vectors. And then I've been over at Nextdata also working on how can we connect large scale enterprises with data mesh, with data products to AI and agents. 

And I feel now this is actually all coming together in this one role, right? All those passions from data systems, over infrastructure management, over actually connecting creating end user value from agentic systems by combining it with data. 

[0:04:16] KB: Okay, that's awesome. Today we want to talk about Nexus and kind of the patterns behind Nexus, right? How we are designing data, data retrieval, data in different ways for agentic systems. But let's maybe start with just kind of the big overview of what Nexus is solving. What is Nexus? I saw it's described as a knowledge engine. But what does that actually mean? 

[0:04:38] JS: Yeah, a very good question. Let's maybe start with the history to understand the problem which we're solving, right? If we're following along, how do you - I mean, for me, the value of Gen AI is actually coming from by combining it with your own custom datasets which are either specific to the task which are specific to your enterprise. You might not want to share them for privacy reasons, or they might just be very specific to that task. 

I think this is where I've seen really big value being created. I mean those early discussions, like should I fine-tune my LLM? I think most value we have seen is actually giving it context and making it specific to my task. And I think this is where in the early days RAG came up. We mentioned that earlier in my introduction. I've been working on GraphRAG in the early days. Kind of explored that a little bit. 

And I think RAG is great if you're exposing information. It's basically that retrieval pattern, right? The LLM knows it can look up for a piece of information. We're going to find all the similar bits and pieces. This is when vector databases came up. We started having either dense retrieval for kind of like semantic search. We had sparse retrievals for kind of more lexical search out there. We added often combined with full text search in addition to that. We ended up basically building agentic retrieval system where the agent could retrieve similar amounts of information. 

I think just what we've seen then is that it's helpful but there is kind of that layer above. And I think this is where most people nowadays are using context for. Context is kind of a more curated version, right? What we have seen often when the agent or the LLM, GenAI, would start retrieving that information, it would have to do that over and over again, and it would then do operations on it. It would curate it kind of like ETL. And we just do that ETL kind of like on the fly and over and over again. 

I think this is one of the steps where we can actually come up with meaningful precomputed context. It's kind of in databases materialized views, right? I have a prematerialized view of that information which is pre-curated. And I think that's by itself enabling a number of different use cases. First of all, because we treat context as its own entity, we can put permissions on it. Similar is in a database system. You'll see I'll jump back to database analogies multiple times because that's simply my background. Apologies up front for that. 

But basically, I can treat them as like a first-class citizen of the overall system. I can give it permissions. I can share it. And so if I'm, for example, seeing that in an larger enterprise context, I can have my personal context, which is just really information context specific to me or to what I've been doing over the last hour on this particular task. 

There might be department level context. For example, for the entire finance team, for the entire engineering team, which is shared amongst them. And then there can be context which is actually shared for the entire company because this is company-wide knowledge. And then for a particular task, I can actually go in and assemble those different context together into a meta context if you want or a set of context. And this is then the set of relevant context for my particular task. 

I think by just starting to treat it as such, I'm getting the benefits of system engineering we have been doing over the last decades from permissioning to aggregation on that. Second aspect of that also with materialized views, simply I get more reproducible results, right? If we go back to what we said about RAG, if I'm doing that at each individual query, I'm doing it over and over again. And LLMs or agents therefore, they are just probabilistic systems. They are choosing something on the fly. 

And so if I actually care about reproducible results, if I'm asking my data body a question, "What has been the revenue last year?" I really want a consistent answer over time. I don't want it to be like fluctuating depending on which conditions are being used or what value is being taken for yearly revenue, for example. 

I think that kind of reproducibility is another important piece which makes it more reliable. And with that, I think I'm also getting in addition I'm kind of getting this might be more just an enterprise concern by itself. But I'm also getting this audit trails. I'm getting lineage trails. I can actually see this context because its curated asset has been generated with the dataset from two days ago, 2pm UTC, for example, right? I can basically go back there and I can also justify why is it in there. 

And I remember spending a lot of nights kind of like trying to rebuild this lineage tracing because we had to debug production issues and basically justify why was this here in the feature, right? If you're going back to feature stores, why is this information ending up in features? And I think by just having the option for this lineage train or being able to go back and connect it back to the source data, this is also I think from an observability governance perspective is a big asset of using a context over repeated probabilistic retrieval steps. 

[0:10:37] KB: Yeah. So there's a lot of different things to dig in here. I think some of the things you're highlighting are around these tradeoffs between determinism and non-determinism and what's traceable versus what's not. But I want to kind of start. I actually really like using the database metaphor. So, if we look at the concept of a materialized view, usually you're building that out because you have some sort of feature or functionality that you have planned where you need a different lookup pattern than was enabled by your previous database. 

One of the nice things about agents in general is they're so flexible. They maybe can handle many, many different lookup patterns. So how do you know what are the right conceptual materialized views to build out for agent context? 

[0:11:16] JS: I think one thing to add, I think it's different lookup patterns but can also relate back to different permissions, right? If I'm building a materialized view over my secretive finance data, which I don't want to go out to any LLM, but I'm building a materialized view where I'm making sure I'm not including sensitive data but only, I don't know, monthly aggregates, for example. And this is the interface where I can have my agents access that. This is I think another aspect to what you mentioned with the access. I think it's not just the data access or the data shape for that, but it's also just from a permission perspective. 

This is where we spend a lot of time, for example, with Nexus as well, is trying to find that routing to identify from the metadata of what is the relevant information to use, right? And I think that if you look back at this agentic development at, for example, MCP endpoints. What MCP endpoints, tools? What resource is so popular? It made it really easy. You just give it a description string, and then actually the LLM could decide, right? 

I think LLMs in general, they are great planners. You give them a tool, and that tool is being described that I can do this. And then if you're not giving them too many tools, they are pretty good at discovering the right tool to use for the right job. You might give them some help with progressive tool discovery, for example, or very narrow set to not overload them with too many tools. But otherwise, they're pretty good planning tools also for more complex pipeline steps. 

I think this is the nice thing you can do with context is you basically - description sounds now just like a very long string, but you can actually generate like also a more structured description. What is in there? What is the freshness of it? What is potentially even lineage information? And because you're often just generating context, right? It's not that with MCP tools where someone goes in and actually hacking a tool together, and it's a static asset. Think context, even though we compare them with materialized views, even the description can be dynamic over time. Even that description can change with different versions of that materialized view to stick with that analogy. And therefore, I can really make sure that the description is always up to date. 

Imagine, for example, you're trying to get the latest sales numbers from a context and you know that a context last has been updated two weeks ago. Then probably this is a sign that this is not very much up to data, and this meta information in the context metadata actually will give the LLM a really good indication it might not want to use that. 

[0:14:09] KB: Yeah, that makes sense. Maybe let's actually break down, when we talk about pre-compiled context, as contrasted to a lot of folks are probably familiar with some sort of Vector Database, or Elasticsearch, or other type of RAG lookup, what is it that's happening here? You have a description of the dataset with some set of metadata. And then what is the shape of the data? How is it loaded? How is it even generated? How are we doing the materialization? What does this look like? 

[0:14:34] JS: There are different ways. Let's maybe just talk in general what I want. And then I can talk a little bit how we do that for Nexus in particular. But I think there are several things coming together. And I think this is the nice part. When we earlier talked about how retrieval evolved over time, we talked about that there are different modes of retrieval, right? There's dense vectors, sparse vectors, there's full text search. And I think this is very similar what we are seeing in a context. 

For us, a context consists of, for example, a vector index. So we can look up and do similarity search in that. It also has schema. It can also have more structured information. I think one example we are, for example, often seeing as a pattern especially if there's some sample queries around it, and we know if this is a meaningful information, we would extract the dates, we would extract person names. 

Basically, one of the steps in context curation for us is we try to identify meaningful entities across it. Interestingly, just going back in my history, this is a lot when you're trying to build knowledge graphs. Right now, we are not using a graph database here, but I can imagine like a knowledge graph, for example, also being a very useful tool. 

[0:15:58] KB: I think there's something interesting there too. Because with similarity search and the ability to do embeddings on different fields, you can build kind of a fuzzy knowledge graph. It's not a strict graph, but you can have those relationships there. 

[0:16:12] JS: Exactly. I'll come to that in a second. Actually, we are building kind of - when we come to Nexus, we are actually building a kind of cheap version of knowledge graphs. Exactly. Exactly. Very similar to what you just described. 

But I think the key point is knowledge context is not like a necessarily single representation. It can be a combination of vector. It can be a combination of structured fields. It can be then also - as we just said, it could bring in a knowledge graph, for example. I think it can contain multiple modalities of data and potentially even the same information in different modalities depending on to support different query patterns. 

I think then the important piece next to that is that you have metadata associated with that. I think we already talked about kind of freshness metadata, lineage information. I think what we are also often seeing is just a reference to a semantic layer in there. Having very well-defined terms, so that I actually know if I'm talking yearly revenue, what is the definition of that field? What do we mean by yearly? Are we following calendar years or are we having like an offset fiscal year for some startups happen in the Silicon Valley? I think it's mostly startups going with that. 

But yeah, is actually calendar year equals fiscal year such kind of meta information which might be stored as part of the context or then as a reference to another semantic layer out there. But I think this combination of multi-modality representation of the data, of curated data, together with metadata, together with either an embedded or external. I think with scales usually seeing external, they're at least becoming more popular, is kind of that key part of bringing all of that together. 

[0:18:03] KB: And when you're designing one of these context datasets, how is it defined, right? Do you have a spec? Do you have a pipeline? What goes into like that materialization process? 

[0:18:15] JS: I think there are two different modes. Either I can create a general-purpose context. What we do for that, we analyze a dataset. We're trying to extract and identify the key entities. And then, for example, this would be where we kind of create a more structured view on that with a fixed schema, we can also check. That's kind of like a general - where we try to identify general terms. 

The second aspect is if we actually know in which areas questions might come in. If we either get emails or at least questions up front which might be answered, this allows the curation step to be a bit more focused, right? If I know I have to answer these questions, I can't ignore things which are a bit outside, and I basically have a direction. And then it's basically - I always imagine there's someone sitting there and curating documents kind of in the old days where you underline the important concepts or marks them in a different color. And then you actually bring them up and store them in another format. It's kind of it's that step nowadays by an agent obviously where we go through the set of documents. 

In step one, we identify the format we want to create. Both the unstructured plus structured aspects of it. And then we basically compile. We actually version kind of this step of how we want to curate it, right? The curate step, in the early days, it was an actual Python program. Imagine that just being generated. But having a version of that, it's actually another very powerful tool because the curation artifact by itself allows you to do that iteratively if the document - if your databases evolves over time, right? 

I think so far, we mostly talked about static datasets. But in reality, most of the datasets will be dynamic on one scale or another of time. And so I think by just having this two-step process of identifying how do we want to curate it and then actually apply the curation step, that allows us to get from a raw dataset to a curation artifact, to then the actual curated context. 

[0:20:42] KB: Yeah. Well, and it's interesting because we were sort of talking about this in terms of the different ways that agentic pieces play in, right? You just highlighted wanting to version the program that is doing the curation piece. We're sort of moving to a world in which you can write code is a primitive that is available. And code becomes like data where we version it. And it's just like kind of going through this. 

[0:21:06] JS: I mean, it was to a degree before, right? If you had like even ELT pipelines, or then ETL pipelines, or it was a Spark job, right? Even that was code you would actually version. And that was another kind of curation step. I think it has existed before, it just was someone either writing the code, or DBT generating that code, or some kind of function generating it. And now it actually moves to, yeah, more probabilistic tools, more agents generating it. 

And I think this is just why I think that versioning step becomes even more important because it might vary. And also having access to the old versions, it's fairly helpful for the agent to generate those tools. Ideally even generate, if it makes sense, kind of like eval sets to be used later on for training. 

[0:22:00] KB: You anticipated where my question was going to go, which is like around quality, right? How do you judge and iteratively improve quality of these curated artifacts? And I presume the curation process, as we've described, some of it's deterministic, right? You're writing code that's doing a set of things. Some of it's non-deterministic. It's got an LLM extracting or doing the highlighting that you mentioned. How do you get quality and iterate on quality? 

[0:22:24] JS: First of all, even the code generation, right? The curation generation is I think in most days right now it's also being generated. It's not someone writing code anymore in most cases. Ideally, I have a spec. I have a spec from where the code is generated. I think that's kind of the ideal standard, and this is also how we kind of see it. You have a well-defined specification. And then from there, you can generate the code actually doing it. 

In terms of quality, I think as mentioned, there is I think the two use cases we are mostly seeing right now. One is kind of like this general thing. I want to be able to ask any question out there. Often, it's a bit tough. We try to train the system. Wherever possible, we try to find like some ground truth in there. And then in many cases where users come in with ideas that they actually have a few questions they want to answer. And just by having like a few short examples, we can actually generate then a larger dataset for quality maintenance and iteration. And I think once you have that, it's basically an iteration step. Keep iterating, keep iterating, improving it, and move from there. Make sure that even with an updated dataset, this is still true, for example. 

[0:23:44] KB: Now when we talk about that check for quality, right? We have these examples. Is that at the level of the generated context or is it that at the level of the agent consuming it and using it to do the right thing? 

[0:23:56] JS: We see it as part the system. So kind of stored alongside the context. Because you might have like very different agents, right? You might have like different consuming agents on the outside consuming it from an interface. If you see that again in just like a larger context, you have your company-wide context. There might be a finance agent accessing it. There might be an marketing agent accessing it. I think that should be part of the general inside system and stored alongside. Whether it needs to be part of the context, that's a good question. But I think it needs to be at least stored alongside the context. 

[0:24:37] KB: That makes sense. You highlighted the question of datasets changing. I think there's some interesting things to navigate in terms of dealing with stuff like conflicts between different sources or drift between this particular data sources out of date or things like that. How do you think about finding truth with imperfect data and keeping it up to date? 

[0:25:00] JS: Very, very good question. I think to start out, I think this is something you can encode in that curation process right. For example, when we talked about this first use case where we're trying to identify the entities. When we identify the entities, again coming back into knowledge graph, in an ideal world, how I would imagine that not exactly what we're exactly doing right now. But I would imagine you can also generate constraints, right? 

In knowledge graphs, especially RDF, you're even separating that kind of schema layer from the fact layer. In the first step, by generating the knowledge graph schema, by identifying the interesting entities, you can also generate the constraints around that in the RDF world. I don't think we want to get into discussion around which representation is best, but there's like a long debate, SHACL, etc. You have a lot of options there to express constraints. 

But I would imagine this is probably that way. Right now, I said I think you start at the easy part you generate a schema, which is already a set of constraints, right? You can put on then constraints on that. Hey, you always know that I wanted to say - I mean, revenue needs to be like greater equal zero. I have to think about a number from a business context. But just to make up an example, you can for example say certain numbers need to be strictly positive because otherwise they don't make sense. And you know that because you have identified that entity. 

Number of attendees in a meeting, for example, cannot be negative to come up with a good example there. And others might be then related. And I think the more time you spend analyzing that data set and also the bigger it is, right? It might be something you actually discover over time when you get more and more data in, you discover new constraints. Again, there's a lot of work on knowledge graphs trying to identify these constraints. 

And yeah, separate that schema layer and constraint layer from the actual fact layer in knowledge graphs. I would imagine this is probably a way where you can get that next step of always adding more boundary conditions. And then if, for a conflicting data set, you discover them, you can actually call that out. And I think that's valuable information by itself if you can flag, "Hey, there's a conflict in the data set." Probably at the beginning, that will require human interjection. But if you're learning that like, hey, this one dataset, it's lagging behind, and that's why it's not reliable. You can actually start, for example, ignoring the last two days because you know that second data set you're joining with is not reliable for the last two days. And then you only generate it two days after if you have, for example, daily data ingestion. I think this is where this curation step can then learn over time from kind of constraint violations. And right now we're kind of dreaming. So I'm not aware of like any system doing that as of right now at least. 

[0:28:12] KB: I'd love to dream a little bit more on this because this is one I've been thinking about as well. And in particular, in my day job I do a lot of work with things that are like essentially information about people and relationships where there isn't necessarily a ground truth that's going to be correct. And so one of the things I've been pondering is can you create some sort of like Bayesian approach or confidence interval where you're trying to like including that in the metadata for each particular piece of context and then looking for ways to up your confidence or maybe even asking the end user. 

[0:28:43] JS: I mean, if we stay in the knowledge graph field, you can just have edges with confidence, right? You can basically have an attribute on the edge. To take your example, right? I imagine you would build kind of a network of people, and then they have some relationships. Know each other to start with our following to take like a Twitter/X example. I think there, you probably would have a ground truth. But I mean you can just add an attribute on each edge saying this is the confidence I have there. 

And again, this is something which might evolve over time, right? If you're getting like new information, Bayesian, as you mentioned, you can update your beliefs about that edge being true or not true. I think that, basically, if you view the context as that knowledge graph where you can embed the probabilities as part of that, I think that's probably the clean approach I would imagine at least in week as of now thoughts. 

[0:29:46] KB: Yeah. Well, and it gets to this interesting question that we sort of alluded to earlier of how do you design this for effective agent use, right? What is going to make the agent most able to take advantage of this? And it might be that you don't want to give it those shades of gray. You just want to like have a cut off. If it's above this confidence, it's in. And if it's below this, it's out. 

[0:30:07] JS: Yeah. And I think this is where agents are a bit different than humans, right? If we look at how is data consumed, also just from a general database systems perspective, we as humans or even like dashboards, we're going to send over SQL query or any query, we expect one result. And then we're going to fly with this result and just go forward. I think agents are both on the one hand empowered to use more because they can iterate, right? 

An agent, I might give that result back. I'm only 5% confident that this is actually true, and depending on which scenario I'm in. This might still be valuable information for the agent. And I think this is then where I think we can view that from two perspectives. I think, first of all, if we come from a governance perspective. And I think this is something we are seeing a lot with like Nexus as well. Actually, I want a constraint. Anything where I'm less than 50% certain don't even give out to the agent. 

From a generalist perspective, if I trust the agent, I just want to give it as much information as possible. Even this like, "Hey, I'm only 5% certain in that," can be something useful for the agent to just decide like what is the next step? What is my next planning step? Hopefully, it's not going to use the information, but it can use that meta information to actually take a different decision for its next step going forward rather than just getting like, "Oh, no information available around that." Right? 

I think this comes down a little bit to the different use cases whether you want to expose that information. But I think with agents compared to human users, because they iterate so quickly and they can form a new plan on a much - I don't like the term cheaper, but just kind of like faster level, right? Because I as a human, I start thinking about it in like a second latency range. But from like an agent perspective, if I'm getting the outcome, "Oh, it's only 5% certain in that," I can then turn around my question and not ask for a specific data. But I could ask what facts are you more certain than threshold, for example. I can actually change my approach for ring about data. 

[0:32:25] KB: That brings us to a question about like how do we expose to agents. And I'm kind of curious. I saw that with Nexus you have your own little dynamic NoQL language or something along those domains. But yeah, I was thinking about to what extent is this a pre-query that's preloading some things in context if we know it versus this is just a tool exposed to the agent. Or how do you think about the right ways to give agents access to this context? 

[0:32:52] JS: I think from an agent perspective, I need a combined answer of data plus metadata. For example, that answer we just talked about earlier, I believe the answer is five but I'm only 10% certain about it. First of all, if I can get both those facts back, this is something valuable. NoQL, for example - and also, we're trying to evolve and also get more standardization around that. This is, for example, the other thing we have seen which is super valuable for agents. If I can expose at least schema on the output I want, I want that result that should be US dollars. 

If I'm asking what is the price of X. And I as an agent, I can specify in a deterministic format, I really want to make sure that I'm getting it in US dollars as opposed to Euro, for example. And I can specify that as part of my structured query, right? NoQL is kind of that pair of structured query where I can also interject metadata. It's not just a SQL query. I can actually put more in there. 

And secondly, I'm getting the same on the response side. I'm not just getting the structure is the table back as with a SQL table, but I'm actually getting data combined with that metadata. Plus certain constraints are automatically enforced. So, we can automatically check that all of those are numbers. Because I said that price field, it should be a positive number lower than 10,000, for example. 

[0:34:37] KB: Got it. 

[0:34:39] JS: I think maybe just to step out. I think in that agent interaction, how do you think about it? Because in most case it's not going to be like a one-shot interaction. I'm not like getting one response back. I'm typically trying to give back and interact in this pairs of data and metadata because that allows both sides to actually give more relevant answers and understand the context, to reuse that term once more, around the data which has been given, right? 

It's not like one-shot interaction, one result going back. But it's actually the combination which makes it powerful. Of course, also on the downside those loops, they can get expensive if we just talk token usage. And I think maybe this is coming back to where we talked about earlier, like this curation. I think we want to keep that loop of planning. We are no longer super deep on that, right? The less layers we can have in there we have to do on the fly the better. 

I think if there is pre-curated, you have to do that only once. You have to do this loop of actually discovering. You do that once up front. That might be expensive. But then in the next interaction, you can actually benefit from it. And we already know what to give you back. And we have to do like one iteration part less because there's still going to plenty of that going on in the system. 

[0:36:09] KB: Yeah, I've heard it described as like LLMs are really good at things that they can sort of describe linearly. And the more layers of abstraction you have to handle all at once, the more they are both expensive. It takes a lot of their own internal expensive key value memory to track that context going through, but it's also less reliable. And so when you can like flatten those steps. 

[0:36:33] JS: Yes. 

[0:36:33] KB: For each LLM interaction, it's only having to track essentially a smaller linear interaction, but it may sub call out to things that are doing those depth. 

[0:36:42] JS: Yeah, I find your linear abstraction. I find that usually in my mind I'm still seeing that as a loop, but I know what you mean with a linear. I'm just trying to avoid like a sub-loop at like one step in that loop, right? If I can avoid that level of nesting of loops, the better. 

[0:37:01] KB: Yeah. Well, and you want to offload that to your context system, right? It's doing that sub-loop. It also has to deal with exactly one layer of loop. 

[0:37:08] JS: Exactly. I mean, sorry, this just coming to my mind right now. If you look at software engineering, it's kind of like it's an abstraction, right? In software engineering, I'm also trying to pull out common functionality into one function. Because, first of all, for us it makes easier to maintain code. I can reason about one sub part much better because I can also treat it independently. I can treat this like one sub loop which is focused on curation or preparing that independent of that larger loop of retrieving it. It's kind of like composability and software engineering as well. 

[0:37:46] KB: Oh, absolutely. Well, and I think one of the superpowers for dealing with LLMs and programming is thinking in terms of domain-specific languages, because LLMs are very linguistic, right? They think in language. And so if you can create a language that's the shape of your problem, LLMs are very, very good at utilizing that. And then to your point, you decompose that. Okay, now I have a new set of abstractions I need to build. I need to build those different functions that create my DSL. 

[0:38:11] JS: Yeah. I mean, I'm always a bit careful because you can go overboard with that as well and then make it too complex. But I think to a degree, the less we as humans have to. I mean, this is the power of abstraction, right? We don't have to keep everything in mind, but we can only talk about an abstraction here. And I think the same is true for the LLM. I think you described it very nicely. If you don't have to go in this sub-loop, which is lots of different context, right? Basically, the LLM has to change its role potentially and still wants to store kind of its previous context, but it's doing actually a different role. The more we can avoid that, the easier. 

Same as for us humans. If I'm looking at a code and there is an abstraction for a class with certain rules, I only have to kind of rationalize and think about that abstraction. And then at another time, I can actually go into the implementation of that if I want to. But from that high level code, I can actually stay at that interface level. 

[0:39:12] KB: I think a thing that's interesting coming back to what you're doing with Nexus is you're essentially doing that same thing for data, right? You're creating a domain-specific data object. A thing that is not only abstracting out the code layer that generates it but abstracting out kind of the form of data that's going to be useful for a particular problem domain. 

[0:39:31] JS: Yes. 

[0:39:33] KB: I guess a thing I'd be wondering about there, and this is more of an observational thing if you've seen a bunch, is how much variation is there in what makes for that shape to be useful? Obviously, the details of what's exposed, the permissions, those things are going to vary. But how similar are the shapes across these different types of context data sets? 

[0:39:50] JS: I think the shape, especially if it comes to the entities we're talking about, this is different between different datasets. But I don't think the recipe changes. If you had to represent it as like a class hierarchy of what's playing together, I think that underlying structure is going to be very similar. The schema of it is going to be similar. The actual instantiation is going to be different. What entities do I have? 

But usually, I'm having some core entities. And I think we said earlier, if we promise to come back to like that narrow knowledge graph representation, we actually do have and we do detect relationships between entities as well. I think this pattern of identifying a fairly small core subset of entities which are central to that, right? It's go going back to like document analysis, different scores of identifying what's relevant to a document. It's still like very much here. I think we're using different algorithm, different ways to detect that. 

But I think overall, we're still trying to identify what is core to this document? What is core to this dataset? And then from that, we can identify that small potentially related subset of entities. We parse it into that. And on the side, we're having the vector search for finding similar instances of these entities. We're having the kind of more structured representation. And then we're having the metadata next to it. I think that it doesn't vary so much. It varies just from the different instantiations. 

[0:41:33] KB: I'm curious. If I use the very particular domain I'm in, one additional thing we often find useful is having essentially quotes that support whatever the extracted entity is. I'm curious if that's a pattern that you see. 

[0:41:46] JS: Yeah. I think we talked about in the very beginning, that lineage. We always keep a trace back to where it's coming from in the original document. Kind of like a pointer back to the original document. Because yes, you're 100% right from multiple angles. I think first of all, in many cases we actually would recommend the system to retrieve the ground truth in addition, right? If it has a chance, if it's not too much context, it retrieves the ground truth as well because it might even give you more information. 

And I think this was already true in retrieval, right? I think a typical retrieval patterns, you find the similar vector, similar entity in the vector space. But then you still kind of retrieve the original data to back that up and feed in. And then I think this is still the case also with context. I think with context, because we already have a structured representation, I have to do that less. But definitely, we give back a pointer back to the original data, to the original place in your document. And again, coming back to then also the version document, right? If my data set is evolving over time, we need to make sure we are referencing the right version of that. 

[0:42:58] KB: Yeah, that makes sense. I'd love to dig a little bit into this knowledge graph concept. And in particular, the fuzzy vector-based or semantic knowledge graph. How are you thinking about those relationships in a world that maybe is not as perfectly structured as your classic knowledge graph?

[0:43:15] JS: I think the interesting aspect of this - and I think that's something we didn't talk too much into deep about yet. I think what is interesting is that even though imagine the data set being static, right? I'm creating one context artifact, I might create one context graph, one knowledge graph instance associated with that. 

Over time when I'm traversing that, I can actually keep modifying my knowledge graph. Right? When I'm initially creating my context, I have a certain budget of compute. Imagine every night, your GPU cluster lies idle, and you're actually going to show that on your previously curated context even though the initial data hasn't changed. You can try to find - I like your example earlier. Imagine in your knowledge graph you have certain probability values where you think this is true, that edge should be in the knowledge graph or not. 

And then overnight or whenever you got free compute time or idle compute time, you actually keep curating your knowledge graph and either collapsing it, expanding it, using graph machine learning to identify new connections, try to group things into certain clusters, and actually keep curating it. Potentially, also Bayesian approach with additional questions I've seen come in, right? I actually get more data not by the initial data changing but by actually just seeing the questions coming in. I know what I might want to answer. And that can help me curate that over time. 

And I know right now we talk a lot about knowledge graphs. I don't think that's necessarily has to be a knowledge graph. I can do the same curation over time by spending more effort on it. If it's in different representations, I can do the same if it's in a more relational format potentially with additional edges in between those with like semi-edges, right? I can always put a graph format into relational schema as well by encoding that as joints. Might not be efficient to retrieve, but I can do it. I think even there, it's true. Even if I don't have like a traditional knowledge graph, I can still maintain that over time and kind of keep curating my context artifact and making it better, especially if I've seen some of the questions being asked against it. 

[0:45:42] KB: That's interesting actually because it leads me down a path of like can you even change the schema of that context artifact? Because you see, for example, hey, we're reliably getting questions that end up having to query across two different sources of context or things like that. Maybe we should provide a combined view or what have you.

[0:46:01] JS: Denormalization, right? I remember that buzz. It like 20 years ago teaching the different normal forms in relational database systems. Most certainly, I think this is so nice that you have the abstraction of a context, right? A context is telling you, hey, this is actually the schema I have in there. From a consumer perspective, I'm not changing the interface even though I'm changing the schema inside. I just have version two, version three, version four of my context, but it can have a different internal representation. 

[0:46:36] KB: That's awesome. 

[0:46:36] JS: Maybe kind of like a container. I mean, I've worked on data products before. You could even see that as like a small data product combining the data and metadata into one entity. And then as a data product or in the microservice world, the Docker container, it gives you still an interface. You can change the internal representation and evolve that over time as well. Yeah, very good question. 

[0:47:01] KB: Well, and it's interesting too because you have some sets of data, and especially metadata, where you're wanting to do repeatable deterministic types of things on it. And so you don't want that to change too much. But a lot of this is prepared for an agent to consume as a programmer. You can treat it as a black box and change that internal representation. 

[0:47:21] JS: We talked about NoQL before, right? Kind of like our query interface there. I think this is one of the nice things there. Because in NoQL, I can basically also give you a schema I expect back, right? o I'm keeping that interface static. But again, because the interface is given by NoQL and we know how to map it, we can change the internal representation and might be more or less efficient to retrieve. But I have that flexibility through that abstraction. 

[0:47:52] KB: All of this we're kind of talking about a layer that is developing now that is on top of these graph databases, like Pinecone originally was. Or we talked about being on top of a relational database or something like that. Do you see this as being a separated abstract layer that there's going to be a set of different options out there in the world? Or is this something that's going to be tightly integrated and vertically stacked with these kind of underlying data stores? 

[0:48:19] JS: I think for us, it's actually we benefit a lot from owning the entire stack. If I look at our implementation at least, we benefit a lot from owning that end-to-end because we can just expose metadata. Freshness metadata for the underlying vector datasets, we know how to get them. We don't have to duplicate that potentially getting out of sync. 

Maybe just to give one example. I think for us, we actually benefit from implementing it really end-to-end, and I think this is giving us better performance. I think the other aspect where it's going to come in is governance. We talked about different access, permission levels, different service accounts having access to different things. In theory, yes, you can definitely develop an abstraction layer over that. 

I think that will take quite some time to identify what do we all need to expose there. What I would imagine is probably, at the beginning, different people will develop different solutions. And then certain standards are going to evolve. How we're thinking about, I think, for us, the layer which probably is going to standardize first is going to be a little bit on this query front. Of course, the NoQL. 

And then from there, let's see where it takes us, right? We can also probably standardize some of the knowledge definition, the knowledge spec. But I said, right now, also just from like a governance perspective, keeping track of lineage. It is a very beneficial for us controlling the entire stack. I would at least give it another two years of iteration. And then maybe we have identified all the patterns, and we can drive that out into a general spec. 

[0:50:14] KB: Yeah, that makes sense, right? We're very much in this phase of the tech isn't actually good enough yet. And so there's a lot of benefits to squeeze out every piece from that vertical integration. And we'll get to a place where, "Oh, okay. Now we've overserved that. And it benefits us to split it apart and optimize different pieces." 

[0:50:31] JS: I think it's going to start at the query front. With query, I mean, actually query and response, right? That kind of format. And then it's probably going to go down to that stack. It's just going to make I think some of the benefits you get from that. Except lineage is I think it's something super helpful to be able to trace it back to the initial days. That's going to get a bit tough if we go into a general abstraction to just get that through these different systems. But who knows, lads? I'm looking forward to talking to you in a year again and see how we've evolved then. 

[0:51:06] KB: Yeah, absolutely. Well, we're coming close to the end of our time. And we've covered a lot. But is there anything we haven't talked about that you think would be important to discuss before we wrap? 

[0:51:16] JS: No, I think we actually covered it very nicely, right? I think we kind of looked a little bit at the history, what actually led us to develop something like Nexus. I think that problem, agents actually retrieve how can we build a system for agent retrieval. How can we make that efficient? I think that's something very nice where we started. I think we branched out into many different areas. I think we really covered most of that. 

Also, we already covered what I had here in my last notes kind of the outlook. I think the outlook where this is going to grow is we're going to see more and more use of integrations with different semantic layers. I said, right now, we do that internally. We're going to see a lot of iterations. 

And I think the other side which we're going to see grow is just more people using it. And I think that's again when we said, "Well, let's talk again in a year." I'm really curious what use cases we'll have discovered in a year. Already, when did we launch Nexus? That's two months ago. We already have seen so many different interesting use cases which actually also helped us shape some of the internal implementation. I think that has been a super interesting learning experience for us. And I'm personally really curious how that's going to continue going forward. Right now, I can just say super exciting times. 

[0:52:38] KB: It really is. I mean, I think it's fascinating how we're all kind of trying to rediscover how do we package data for agents as a primary consumer. And we've talked about a bunch. I think, honestly, the coexistence with the metadata and having that be key both on the definition and retrieval side or query and retrieval, and how all of that tracks, that's a huge step forward. And we talked about semantic layers. I'm hearing all sorts of people were talking about, "Oh, we need much better semantics for agents," because people just put that in their heads, but agents need it right there. Yeah, discovering these patterns of what needs to be collocated now and what needs to be described. It's a fun time. 

[0:53:16] JS: Yeah.

[END]