EPISODE 1947
 
INTRODUCTION
 
[0:00:01] ANNOUNCER: Data retrieval is a fundamental challenge in AI systems, and the approaches for solving it are still evolving. Vector search was an early answer to the retrieval problem. But the rise of agentic systems has raised the stakes considerably. Agents issue queries at machine speed, decompose complex questions into parallel searches, and require retrieval infrastructure that can keep pace without becoming prohibitively expensive. 
 
Chroma is a company building open-source infrastructure for AI applications best known for its widely used database of the same name. The company also published the influential context rot paper which documented how model performance degrades as context window utilization increases. And recently released Context-1, a 20 billion parameter retrieval sub agent trained to do agentic search at frontier model quality but at an order of magnitude lower cost and higher speed. 
 
Hammad Bashir is the CTO of Chroma with a background spanning machine learning, computer vision, and data systems. In this episode, Hammad joins Gregor Vand to discuss the origins of ChromaDB, our current understanding of context rot, why a purpose-built small model can match frontier models on search tasks, the philosophy behind Chroma's open-source approach, and where the company sees AI data infrastructure heading. 
 
Gregor Vand is a security-focused technologist having previously been a CTO across cybersecurity, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at vand.hk or on LinkedIn.
 
[INTERVIEW]
 
[0:01:55] GV: Hello, and welcome to Software Engineering Daily. My guest today is Hammad Bashir from Chroma. Nice to have you. 
 
[0:02:02] HB: Yeah, great to be here. 
[0:02:03] GV: Yeah, awesome. So, we're going to be hearing all about Chroma. I think it's a product that probably a lot of our audience are familiar with and probably familiar with these types of products as well. But before we get into that, I'd just love to hear as we like to - Hammad, you've worked at quite a few interesting companies, I think, in the Bay Area and beyond. What's your quick path to getting to Chroma and all that? 
 
[0:02:25] HB: Yeah, I spent better part of the decade as a software engineer and broad expertise in machine learning, computer vision and data systems was most of what I worked on for the past decade. And before this, was spending a lot of time specifically on computer vision. And initially when we started on Chroma, the problem that we were solving is one of active learning for computer vision systems, which is the problem of every day you have a bunch of live deployed computer vision systems. How do you actively label the data that you're looking at in the wild and select which data is worth labeling? Because if you label everything, it's cost prohibitive. 
 
That process of choosing what data to label is a train time data curation problem. And as language models started to emerge, we started to look at that problem in the context of inference, which is how do you actually curate data for models to do inference on? Specifically language models. This is the problem of search. How do you actually find the most relevant data to enhance the quality of their answer? Which is the stack of tools that we now build at Chroma is a stack of tools to help people develop search for their downstream AI applications so that they can get the right information or the right context into their language models. 
 
[0:03:35] GV: Yeah. Nice. Yeah, interesting background leading into how - I guess everything that you guys are now doing at Chroma. Maybe just a sort of easy introduction. What is Chroma? There's a few bits to the product. But roughly speaking, what is Chroma? 
 
[0:03:48] HB: Chroma at its core is trying to solve the problem of getting models the context they need. And to that end, we build three main pieces of technology in our platform that people can put together to solve that problem. The core of it is the product that most people use, which is our core database. So, ChromaDB, which is now one of the most widely used databases in conjunction with language models. 
 
What it does is you can very easily ingest data into it and use a variety of different search techniques, vector search, full text search, metadata search in order to get information that might be relevant to a language model. In conjunction with that, we also have a part of our product we call sync, which is a managed ETL pipeline for ingesting data from various sources into that database. 
 
We found that most developers don't want to be building that sort of a pipeline themselves, because putting data into a search database requires a lot of processing. You have to clean the data, you have to transform the data in a way that makes it vectorizable so you can search it. You ideally want to enrich that data with metadata. And that process is quite cumbersome to do in a high throughput way. Our sync product solves that. 
 
And then quite recently, earlier this year, we launched a product we call Context-1. It's mostly a research endeavor, and we're in the process of productionizing it. What it is is a model that does agentic search. Agentic search is the process of instead of a language model application just doing a single search and then putting that search into the context window directly, a process commonly known as RAG or retrieval augmented generation, what you can now do is tell a language model, please take this query, perform a multiple set of queries, take some highle English query, turn it and decompose it into a set of low-level queries against some search system, and then repeat that process until you feel comfortable with your end conclusion. 
 
Kind of the way you and I would sit down and answer a query. We wouldn't type something into Google, look at the first 10 links, and then say, "My job is done." We would read some documents, author more queries, read some more documents, author some more queries, follow the links on those pages and do exhaustive research. The models now do that process themselves. 
 
However, it's quite cost prohibitive to do with large frontier models. Context-1 is a model that was trained using reinforcement learning to be just as good at frontier models at this process, but at an order of magnitude less cost and an order of magnitude more speed. Compared to Opus 4.6, we run at around 400 tokens per second, 500 tokens per second, whereas something like an Opus would run at around 40 to 80 tokens per second. The end result is that you get your answers at the same quality much, much faster and for much, much cheaper. 
[0:06:18] GV: Yeah. And we're going to go into Context-1 a bit later on. We'll get to hear all the interesting detail of, I guess, how that actually works in practice. I think just looking at Chroma generally. When I first came across this space, vector search was such a big thing for a while. And then at least in my framing, it kind of disappeared for a while, but now it's coming back in what I feel is quite a big way. If we kind of start there, I guess the journey of Chroma from its founding, I guess, or when you came into Chroma, what were you working on at the start? And how has that evolved? 
 
[0:06:50] HB: Yeah. As I mentioned, what we were working on at the very beginning was how do you choose what data is worth retraining models on? The problem of, "Hey, every day my models do some task out into the world." At the time, it was actually computer vision models. And some subset of that data is worth relabeling for the process of what's called active learning. 
 
And it turns out that that same process can be used for language models at inference time. How do you actually choose what data to put into the context window? Well, you do search and you find related information the same way that you might find related information when choosing what data to label at train time. That was the initial problem that we went out to solve. 
 
And actually, the way we went about solving it was using existing vector search solutions. We didn't set out wanting to build a database. But what we found is that every other solution on the market was quite poor in terms of serving the workload that AI applications had. Specifically, systems that were built for legacy search, not for AI applications, were really designed around the idea that you have one very large index. So, you might have a billion pieces of information to search over or a million documents to search over, but they're all in one index. And if you want to add another half a million, what do you have to do? You have to recharge your cluster. You have to add a node, subtract a node, rebalance across those nodes, which is a very manual and operationally intensive process. 
 
AI applications have an almost opposite workload, where instead of one or a handful of very large indices, you have extremely partitioned indices. You might have an index per customer, an index per team, an index per chat history, an index per agent. And that means that your system has to be able to store a very, very large volume of unique indices and be able to search over them with a high degree of accuracy. And if you're to store all of those in one index, one, you have to pay the cost of keeping that live at all times. And then two, you have to deal with the accuracy degradation in vector search when everything is in one index. Vector search doesn't do well with being partitioned. And so the best thing to do is hard partition the data. Keep it actually in logically separate indexes. You can't filter the search at query time. You have to do it at index time. 
 
And so if your data has this partitionable structure, you can build many, many indexes. And instead of needing to build one giant index, you get a much higher quality and cost trade-off. And so that's actually why we started to build Chroma, the database, is because nothing else solved that problem for us in a high-quality way. 
 
And the core of it was, at the time, object storage. Amazon has S3, and GCP has GCS, Azure has their own thing. All started to release support for conditional rights, which is a new thing, especially on AWS. It only came out as recently as 2024. And conditional rights allow you to actually build a database on top of object storage. And so what we did is leverage that functionality in order to be able to build our database, ChromaDB, on top of object storage so that the economics would be something that make applications for AI actually viable. 
 
Now you could actually afford to have indices for every customer, or for every team, or for every agent. Whereas before, you would simply run out of money trying to even build some products that people were trying to build on triply replicated pairs of discs or NVMe SSDs. By building a database solely on top of object storage and then providing a really easy to use user experience on top of that, we just built that initially for ourselves for us to be able to serve the workload that we wanted to serve, which was, "Hey, how do we choose what data to put into language models at runtime?" Because nothing else quite was doing the job. 
 
[0:10:24] GV: And if we sort of zoom forwards to today and we'll get into it in more detail, but the change to agents doing the searching here I think has had quite a big impact then on how that retrieval effectively has become even harder, if I'm not mistaken. 
 
[0:10:38] HB: Yeah, absolutely. And I think part of it is now the agents don't just issue one query, they'll issue five queries in parallel. And the agents don't query at a human rate, they query at an agentic rate. And so whereas you and I might issue a query once every 30 seconds at most, and that would be considered quite fast for us to be issuing queries, the agents will gladly issue a query every 1 to 2 seconds. And each of those queries will be five queries in parallel or six queries in parallel. This agentic process also puts a much more heavy burden on the read side of the database. 
 
And if you build a system that is fundamentally stateless in its core architecture in that you can just add additional nodes to supply queries because you have a shared storage layer so I can just add an additional node to serve reads, you can absorb that elasticity in a way that I think systems that are built otherwise can't. 
 
I think the ability to also leverage shared storage not just to deliver economics that work but also to deliver a performance envelope that works is something that actually we didn't anticipate when we started to be completely honest but we benefited greatly from after having built it. 
 
[0:11:47] GV: Yeah. Amazing. I'd love to go into the journey especially towards Context-1. And there was a paper that Chroma put out last year called context rot, which I think is become especially in certain circles very respected and understood for what it uncovered. This is back in, I think, June or July 2025. Could you just walk through the core of what the context rot finding was especially for anyone that hasn't read the paper? 
 
[0:12:14] HB: Yeah, I think Context-1 was our attempt to prove something to ourselves empirically and scientifically what most people already kind of knew on a vibes basis. Most people already knew of this degradation. But context rot was formerly an investigation into how does model performance degrade as context window utilization increases. As we add more and more tokens to the context window, does the model degrade in its ability to perform certain tasks? 
 
And we performed a series of very simple and targeted experiments where we do things like inject a repeated word a bunch of times and then ask the model, "Where are you seeing the word stop appearing?" Or we do simple reorderings of needle and haystack problems where we give it a bunch of documents and we say, "Can you find this specific piece of information in this haystack?" which is a needle in a haystack problem. And we perturb it in interesting ways. 
 
And by running these experiments, what you quickly see is a super sharp drop off in the tens of thousands of tokens. And this holds even in today's frontier models. It's a problem that has persisted even we released the report last year. And we've had three model generations or minor iterations of the models since then. And we still see that degradation in the models that, as you increase information to their context, they start to degrade in quality. 
 
And what this points to is the need for a disciplined approach to deciding what you're going to put into the context window. And whenever there's some new paradigms in computer science, people are very quick to ascribe terms like so and so engineering or so and so science. But we have, I think, been okay with adopting the term context engineering, which is the problem of how do you actually choose what information to put into the context window to maximize the chance of success in the next turn or the next set of turns of the language model. 
 
And I think that process is one that requires a lot of care in choosing not just what queries you make but also how you structure the data, how you prompt the model, what data you give it access to in order to not suffer from this problem of context rot. Because even if you solve the problem of context rot, because of the way that models work, it's quite cost prohibitive to just assume that you can grow the model's context window at infinitum. At some point, the matter of cost also becomes a concern, which is, well, if you can more carefully manage the context window, then you can also deliver much better economics. And as we all have learned the hard way, it's very easy to go crazy spending a lot of money in these systems in a very short amount of time. 
 
[0:14:45] GV: If I think back to - I think it was November '24, something like that, when RAG was suddenly like a buzzword. And people barely even understood this idea of, okay, if I add a document basically to my prompt and then tell it to do things. And nowadays, this context engineering is simply like the way it has to be done. There's no way you can kind of make progress on something without having multiple sources feeding into what you're trying to produce. Whether that's code related or, especially in my case, using a ton of work documents effectively to all kind of come together. 
 
Just going back to context rot for a second. I mean, this was tested, I believe, across - again, we're talking a year ago. I mean, things moving really fast. But 18 models, I think, this was tested across. I guess all the big models from the people we know like Anthropic, but a lot of openweight models as well. Is that right? 
 
[0:15:38] HB: Yeah that's correct. And you can also see very sharp differences at the time. I think the open-weight models have caught up tremendously in the past year. But at the time, you would even see a much sharper discrepancy between the open weight models and then the kind of frontier closed labs. 
 
And I think the thing that's been most interesting to me in terms of the open-weight models is how quickly the gap has closed. And we run the benchmark internally still just for curiosity's sake every time a new model comes out, and the gap continues to close. Where actually in some cases, on some tasks, the open-weight models actually can outperform some of the closed-weight models. 
 
The gap between the open-weight models and the closed-weight models at least in this very narrow and contrived benchmark context raw is obviously a very contrived set of experiments. But with the way we view it as a test bed or a gym to evaluate a very specific behavior, the open weight models have made a lot of progress in the last year or two. 
 
[0:16:30] GV: Yeah. And you touched on it there. The needle in a haystack problem. The findings were basically that a sort of shuffled incoherent haystack can actually help models find needles compared to structured ones. Can you talk to us a bit about that? 
 
[0:16:44] HB: Yeah, I think the general observation is that as we add distractors to the context window, the ability of the model to ignore distracting information or information that looks like it could be correct decreases. Models are quite bad at disambiguating between two facts that aren't in contradiction but that are about the same thing. Meaning I have two facts that are about the same thing and seem related, and they point to two different aspects of that same thing. And they don't necessarily agree or disagree. And they maybe distract from the core thing that you're trying to talk about. 
 
As a concrete example, if I'm asking a language model, "Can you find for me the population of Georgia?" And I give it a document that has both populations of the country Georgia and information about Georgia. And then I also give it things about the state of Georgia. And I've specified I want to know about the state of Georgia. It may confuse those two simple examples between each other. 
 
Now, that's a very simple example, but the idea is that like if you have things that seem on the surface quite similar and you mix them in, the ability of the model to discern between those degrades as you increase more distractors. And some models are quite good at ignoring information that seems relevant but isn't really. And then some models are quite poor at it. 
 
But universally, all of the models do suffer from this where, as information looks distracting, it starts to degrade its ability to reason if it just seems like it could be relevant. And you can really quickly start to imagine how that's relevant in functions of the real world. These are very contrived examples that we test on. Most of them are also synthetically generated. But when we look at the real world, you can imagine simple cases where a model retrieves documents about two people with the same name and you're talking about work that was done by one person, and it starts to confuse one person's work for another and points you to ask a question to the wrong person inside of your company. 
 
Or you can imagine when a coding agent looks at files, it starts to quickly discern the wrong area of the codebase to narrow in on because it seems relevant, because maybe you're asking about the authentication system, and there are two authentication systems that live in different places. And you're referring to one and it confuses it for another. It happens all over the place that the model can get confused between two things that seem similar but aren't actually related in the way that you're looking for. And so that's kind of what we were trying to study is how much are the models able to parse through and disambiguate between pieces of information that seem similar but aren't actually similar. 
 
[0:19:05] GV: Yeah. Super interesting. I guess let's go into Context-1 because this is - with that in mind, everything you just said, Context-1 is how to get over this, get over the context raw problem. Curious of where we'd like to start actually with Context-1. I mean, we could start, say, with the architecture. I think it's explicitly called a retrieval sub-agent. Could we just start there? What does that even mean? 
 
[0:19:30] HB: Yeah. The way that we design Context-1 to be used is as a sub-agent with a larger model. And generally, I think this is a pattern that you'll already see a lot of the coding agents do now, which is larger coding agents, Codex, Claude Code, Cursor, and then a lot of the open harnesses as well will launch a sub-agent which is just another language which model that then goes off and does some task. In this case, the task that we're interested in is, can you help me find all the information that may be relevant to doing the function that I'm trying to perform? Can you get me the context that I need? 
 
And so, Context-1 is designed to be a generic search sub-agent for any large reasoning model. And the idea behind it is that as in traditional computing, can we just separate concerns and give tasks to the right system at the right time? And in this case, our belief is that the narrow problem of search is one that can be solved pretty easily by a smaller model because it's mostly the problem of understanding user intent, decomposing that into queries, analyzing relevance of documents, and that performing that in an iterative fashion. 
 
And in most cases, you don't need that much raw intelligence as much as you need the ability to do a lot of that very quickly. And so for us, tokens per second is really the way that we think about it. And as we start to think about generations of context one beyond the current, we're even starting to think about how can you start to leverage parallelism and sharding to actually attend to very large corpora. 
 
And so it's used as a background sub-agent is mostly driven by this idea that we should separate concerns where it makes sense for performance reasons. And we dispatch this background sub-agent. It just finds the set of documents that it thinks are relevant to a task and then returns them back to a larger reasoning agent. And that larger reasoning agent can then go about doing its function. 
 
[0:21:14] GV: Yeah, I think exactly. That sort of separation of concerns is super important. Because at the end of the day, this is not something that will actually answer questions. This is a - as we're saying, it's a retrieval sub-agent that its sole purpose is to go and understand what to go and find and return that, but then hand that off. 
 
[0:21:31] HB: Exactly. And I think that you can even start to look at approaches that ask the large reasoning model to be more clever in its use of sub-agents and actually start to internally decompose tasks into multiple sub-agent calls and then get all of those things back in parallel. And so leveraging the large model as kind of an orchestrator and a planner over a fleet of smaller models is, I think, a paradigm that we believe in quite firmly. And it's already proving out to be how coding systems will work. And I think it'll find its way into almost every agent over the coming year. And so Context-1 is just trying to fit into that paradigm. 
 
[0:22:08] GV: I guess looking, again, what actually makes up this sub-agent. I think there are like four tools. They got vector plus BM25, hybrid search, regex grep, read document and print chunks. And there could be a lot to cover there. But I think more to the point of like why those four specifically. 
 
[0:22:26] HB: Yeah, I mean, the standard paradigm for any agentic search harness is you just want to give the model two tools which is, one, allow it to search, and the second is allow it to fetch results that it sees. Give it a way to get its 10 blue links from Google, and then give it a way to click the link. Those are the two things that you're trying to give it. 
 
And then the one thing that we were interested in is this idea of self-editing context or active context management. The predominant paradigm to managing long context windows is compaction. If you've used any coding agent at the end of your context window, it'll summarize or compact its context. 
 
Our belief was that for leveraging a small model, this would be a very difficult endeavor. It would be quite lossy. You would potentially lose valuable information. You might lose context if you're being dispatched by a larger agent in a narrow fashion. And so how could we avoid any lossiness in the small model due to its inefficiencies and its inaccuracies? 
 
The idea that we came to is can we just have it edit its context as it goes. So we give it this prune chunks tool. And doing agentic search is a very token-heavy process, because every document that you touch, you end up loading into the context window, and you very quickly process a very large volume of tokens very quickly. 
 
What we do is we give it this tool that allows it to go back and remove documents from its own context window. And this is a trade-off between KV cache coherency and the overall token usage. And what we found is the model is quite good at making judgments about relevancy and removing prior information from its own context window, which is not all that surprising in hindsight because the model already has to make that judgment in order to be successful at its task. 
 
The fundamental problem that you run into though is sometimes a document early on in the search rollout appears to not be relevant but then later becomes relevant. And so a lot of the subtlety of training the model becomes how do you actually train it on a wide enough variety of cases that have a wide enough variety of problems that have different amounts of relevancy in the documents that it returns. How can you perturb the documents that it returns in a way that you can create sufficiently challenging examples for the model such that it's trained to be very accurate in when it prunes? 
 
When we trained Context-1, the base model is actually GPT-OSS-20B. It's a 20 billion parameter model. And when we trained it, initially the accuracy of pruning I think was around 80%. 80% of the time it selected the right documents to remove. That basically makes the prune chunks tool for the base model kind of a non-starter. You're going to have an immediate 20% accuracy loss. 
 
By the end, I think we were around 95-plus accurate on a wide variety of tasks after training. And I think on our most recent checkpoints internally, it's like closer to 97%, 98% accuracy on the prune tool. And most of the loss in accuracy comes from the search itself. And the model itself is quite good at discerning what's accurate and relevant versus not. 
 
That ability to self-edit the context ends up leading to a very large reduction in the overall token usage. And what that does is it also makes it so that you can do inference a lot faster. Because we're keeping the context window relatively bounded at any given period of time, around 32,000 tokens, the prefill phase, which is the first phase of language model inference, ends up being relatively fast. And then the decode phase is relatively fast as well because we're keeping it fairly bounded at all times. 
 
We can afford to pay this repeated prefill cost because we're pruning the context and the model is small enough that we don't kind of feel that penalty from that trade-off. And there's obviously some subtlety there in like how much should you prune versus not, when should you prune versus not. And we're also starting to work on can you actually use latency as a training objective and let the model actually self-optimize around latency at train time? And so that it kind of learns the most optimal pruning strategy not from an accuracy perspective but also from a latency perspective. 
 
There's all these subtle tricks you have to do in order to make it work. But fundamentally, I think the core contributing idea on the harness side of the model, the thing the set of tools it has and the way that the model behaves is this idea of self-editing its own context and self-editing its own history. 
 
[0:26:37] GV: I think if we just look at what the result of this is, I think it was stated that Context-1 at 20B matches. This isn't that long ago. Opus 4.5, Sonnet 4.5, Gemini 3.1 Pro on most of your benchmarks. That seems pretty impressive. That's an understatement. I mean, what's kind of your read on why a purpose trained small model can actually get there, I guess? 
 
[0:27:04] HB: I think the core belief is if you start to decompose what a model does when it searches, the set of behaviors it has are more valuable than the reasoning that it's doing. So there's a program that you can follow or an algorithm that you can follow to do search. And a lot of the time, that algorithm is a set of very narrow tasks chained together. And if you can just figure out the right time to deploy those tasks and if you can figure out how to be good at those subtasks, then you can be good at agentic search. 
 
Fundamentally, I think the set of tasks is how do I look at a high-level user query and then decompose it into a plan of lower-level queries to issue against the search system. How do I look at a page and determine its relevance? How do I look at a page and determine what follow-up queries to make? And then how do I analyze the set of results I have at the end and determine what could be relevant, right? Which is a different relevance determination. 
 
I think that set of tasks which are fairly narrow you can optimize for in isolation through a set of training examples. And so, actually in training Context-1, we have a staged curriculum, where early on in the life cycle of training we only give it problems that are easier and need one to two hops to satisfy. And then later on, we start to expand the difficulty of the tasks that it has. 
 
And then we also do two phases of training. It's not only trained through reinforcement learning. We also have an additional supervised fine-tuning stage which uses synthetic data and data distilled from open-weights models. We take open-weights models and we distill their behaviors into the smaller model. 
 
And that initial set of behaviors alone actually shows you a very large performance increase. But the thing you're after is grounding the model in the right set of priors around how should I behave when I'm in certain cases. And once you get that into the model, it's pretty easy to optimize once you have those basic set of behaviors. Even just getting the model to tool call in parallel gives it a huge advantage in its ability to very quickly reduce the number of turns that it needs in order to discover information. 
 
And GPT-OSS-20B doesn't natively do parallel tool calls. And so if you just do some amount of SFT, or syn fine-tuning, on the model in order to teach it that behavior from the data, it very quickly gets better at that and leverages it even before you do any sort of reinforcement learning. I think the reason that you can get these small models to be good at these tasks is because they don't require a lot of reasoning capacity, and they don't require a lot of subtle decision-making. 
 
And we have done some work to show that - for ourselves, we've done some work to show that often times having world knowledge does help you. If you know information about a given topic and you've memorized information about a given topic, you're going to be able to issue better queries. 
 
And so a larger language model will probably always outperform a smaller one at steady state, but the smaller ones are cheap enough to fine-tune using LoRA that, on narrow enough corpuses, you can very quickly also fine-tune a small model on a narrow corpus and make it better at individualized corpuses and personalize the model for specific tasks. And I think that's where the advantage really starts to shine is can you actually just give it a map of the corpus or teach it the vocabulary of a corpus so it's able to search it better. The same way if you and I go and search our Google drives, we already know what's in there. So we're going to be able to search for it more effectively. You can see that sort of behavior in smaller models as well. 
 
[0:30:22] GV: Yeah, that's really interesting. I think just to wrap up Context-1, the bit that might not be clear or surprising even to some people is that you actually released the weights open source under Apache license. What was the thinking there? I mean, on SE Daily, we're always very pro open source that that's fantastic. But every company has probably gone through a thought process of why to do that. 
 
[0:30:44] HB: Yeah. Chroma is an open source company. Even though our core database is Apache 2.0, I think there's many reasons to do open source. Before Chroma, I wouldn't say I was the largest open- source zealot, but I've over the years become a card-carrying open source member or whatever you want to call it where I'm quite zealous about things being open source. 
 
I think there is some amount of just personal belief and philosophy inside the company that things should be open source where they make sense. And I think it's fundamentally, one, we wanted to learn very quickly. And more people are going to use the model if it's open source and tell us what's working and what's not working. More people are going to be interested in building with it. More people are going to be interested in integrating it into other places where we may not be able to integrate it. And also, the model is very small, so you can run it locally. 
 
And the way that Chroma initially grew as a product in the market was we offered a local version of the database. And so we're quite fond of solutions that meet developers where they are, which is on their laptops or on their machines at home, and let them tinker around and experiment with technologies that are close to the frontier in a way that doesn't cost them an arm and a leg in order to play around with it. 
 
And maybe it won't be the case that every subsequent model we release is open source, but at least for the time being, I don't see a reason for us not to continue to do things in an open source fashion. I think it's positive-sum. I think that we learn when other people release their work, we benefit greatly from it. And we obviously benefit from open-weights models both quite literally in the training of context one but then also just in our usage of them. 
And so I think mostly for us it's just a matter of getting people to use it, learning very quickly, and then contributing back to an ecosystem. And our belief is that our ability to grow that ecosystem. At this moment in time, AI is still very early even though it feels like the world is almost over or whatever people on the news want to tell you, AI is still very early in its life cycle. And I think that right now is not the time to be zero-sum in our thinking with respect to it. 
 
[0:32:41] GV: Yeah, that's great. I mean, we've talked to quite a few companies who have decided to move from closed source to open source over the last year, for example. And I think we're seeing more and more of that. Either the company just starts day one that way. I work for a company that started that way as well. Or actually, it doesn't look great actually if you're working on anything sort of on the frontier that is all closed source. And in Anthropic's case, you can be closed source and then somebody leaks the whole code anyway. So you might as well be working in the open. Definitely support that. 
 
If we just look at the product stack of Chroma, there's database, there's sync, and the agent product. Could you just kind of walk us through where does a developer start maybe? Or it does it not matter? It's just the right tool for the job. Or is there some classic arc you see of like how a developer starts and then adopts more of the product? 
 
[0:33:32] HB: Yeah, I think most people, their first interaction with Chroma to date is the local embedded version of the database. You pip install a package. And, all of a sudden, you're in a Jupyter notebook or you're just coding in Typescript and something is running locally on your machine. And then we have a CLI that lets you - once you're done prototyping, if you need to upload your data. Because unlike SQL databases or like traditional, the use cases that relational databases served, it actually does make sense to have this notion of pushing data up from staging to production. With these kinds of applications, you might curate some sort of knowledge base or curate some data set and then push it up. That's the place that most people start. And then they use our CLI to upload the data to our cloud product once they're ready to start going in production. 
 
There's two ways they can do that. Either they can use the database directly if they want to control their chunking strategy and their embedding model choice and choose all of these nuance parameters. But over the years, I think things have kind of coalesced onto a set of best practices. You should use sparse vectors with SPLADE. You should use a dense vector embedding model that's of a sufficient quality but not too large. Turns out you don't need 40x96 dimensions. You're very much okay with a smaller dimensionality embedding model, and you'll get just as much performance in terms of accuracy at a much lower cost footprint. 
 
If you want to just have those choices all made for you, you use our sync product and you can either dump your data in an S3 bucket, or upload a file, or connect it to one of our supported sources. GitHub, a website. And then we're soon releasing support for a bunch of other upstream connectors. And then we'll just automatically keep those sources in sync with your database, with your Chroma. 
 
And then that's kind of the normal arc. Context-1 then becomes something that people - once they start to think about how do I actually really get high quality search in a low-latency manner, they usually start adopting that part of the stack. I think agentic search is still new enough that people tend to be adopting it fairly late into their product development. 
 
[0:35:33] GV: But I guess do you see that maybe accelerating into the future? 
 
[0:35:37] HB: Yeah, now people are coming to us, especially after we released it. Now people are coming to us, and it's kind of the starting point for their product development. And that's been really interesting too, which is how can we - you actually start to build your product differently if you start with that frame of mind. If you start with "No, an agent is going to do the querying", you start to actually build your product differently. 
 
I've had people come to me and be like, "Well, the agent's doing the querying, why should humans do any of the editing?" And so actually they have products now where like users actually don't edit information directly. They're only allowed to edit through the model because they think that the model will be able to do a superior job of traversing the information space and making all the right edits in the right place and keeping all the information in line. 
 
I think it's also changing how people think about their product development. This notion of not everything needs to be a set of checkboxes and forms and sliders. Actually, you should just drive everything through natural language and push everything into the agent. 
[0:36:29] GV: Yeah, I definitely see the same thing where I sit. And it's definitely just an interesting time to be building products where every conversation is "Is this a human or an agent?" When you're talking about how it's being used, we're always saying is it the user or is it an agent. And how does that differ? And I just find that super fun and interesting that that's kind of where we are now when it comes to building product. 
 
If we sort of look at how a larger enterprise would want to use something like this, you guys have quite a bit of material on what's called BYOC. So bring your own cloud. Could you talk to me a bit about when in Chroma's journey did this come about and why? And small anecdote from talking to someone who is in the current YC batch, they were actually asking me and saying, "Oh, we're being pushed to you know by a certain big customer to do BYOC." And I said, "Really think about that strongly, because that's a big undertaking." And of course, they're going to say they want that. But don't take that on lightly. I think you can probably help expand on why not to take that on lightly. 
 
[0:37:32] HB: Yeah, I think Databricks was kind of famous for doing this like hybrid deployment model early on in their life cycle. And I think when they asked the CEO of Databricks, "What are two things you would do differently?" One of the two things was he was like, "I wouldn't do the hybrid deployment model. I would have just skipped straight to the other way of doing it," which is maybe kind of privileged hindsight, right? Maybe you wouldn't have succeeded if you hadn't done the hybrid deployment model. 
 
BYOC for us means we deploy our code into a customer's VPC and then we get back some amount of observability into that system in order to operate it and management. And we keep our control plane on our side. And then we're able to kind of monitor the health of the system, perform operational procedures, updates, upgrades, scaling, some basic logging and telemetry comes to us. And then you get the experience of a managed service, but you don't have to deal with actually running the infrastructure yourself. It's a very easy setup. You just kind of apply some Terraform and run a Helm chart and you're up and running. And then our team kind of handles the day-to-day for you, all the day two operations. 
 
And we actually had the idea to do BYOC from day one. I think if you don't choose to do it from day one, it's really, really, really difficult to bolt on because your entire architecture hinges on the fact that you're doing BYOC. And it actually was an immense pain to convince everyone on the team that this was worth doing because it's really painful to build. It's really, really hard to build a system in this way. It makes you have to do things in a much more difficult way up front. Greatly sacrifices your velocity. You have to draw boundaries in your system architecture that feel very unnatural to draw at times. Or you could argue. I'd actually say they're boundaries you should have but ones that you don't necessarily need right then and there. 
 
And I think that if you don't really need to do it, I wouldn't advise that you do it unless it's like quite core for your ability to grow your business. For us, it's really important because the customers that we want to serve that are at a large scale, which is where our product, the core database really becomes useful, is at a large scale. If you have a small volume of data, other solutions are fine, right? If you only have like 10,000 records, you probably don't need scale-out horizontal serverless search engine. You're probably fine just throwing all your data in a Postgres database, and it will work. There's no reason to use our cloud product at that point other than just ease of experience and just wanting something that really fits AI applications natively. 
 
But I think unless you really need to, I would not recommend it. And I think that my advice to this person in their YC batch would be make sure you're going to make a lot of money if you do that. Make sure that it's like something that you think you can repeat and deploy because it's a very large distraction. But I do think that if it makes sense to you, it is kind of an increasing trend. And I've noticed that buyers especially are pushing for it, where buyers have started to kind of learn that this is a deployment model. It gets through security. It gets through compliance. Ticks all the boxes. And that is probably reason enough to invest in doing it if it can result in a meaningful outcome for your company. 
 
That said, when BYOC started, I think it was pretty debated whether it was a deployment model that made sense at all. There was a lot of well-known people online who would tell you that this is the silliest thing ever. No one should do it. It actually makes absolutely no sense. There's blog posts you can go read about this. 
 
I think it's one of those things that's a little bit of an organizational tax, right? You have to make the compliance people happy. You have to make the security people happy. You want something that ultimately makes you able to sleep at night and give you control. And I think BYOC is great for that. Whether it's actually secure, more secure or less secure. I think that really ultimately depends. I think that at the end of the day, you still have to do your due diligence and you still have to check on the buyer side what's going on under the hood. I don't think it reduces that work or makes it go away. 
 
[0:41:11] GV: Yeah, I definitely agree with everything you've just said. On the control plane side, do you get any push back that the control plane does still live with you? And how do you, I guess, talk to companies about that? 
 
[0:41:21] HB: Yeah, I think there's various degrees of push back. When we initially started talking to customers about doing BYOC, there were some customers who were quite negative about having any data on our side, even telemetry. And that's just a matter of education. The product can't work if we don't have basic telemetry on the system. 
 
So at this point, everything that we store on our side of the fence is just metadata. It's what data exists in the system, what is the utilization of the system, what is the version and epoch counters of various pieces of information in the system. Nothing is directly revealing private information. And because it's open source, we literally just send people the code. We're like, "Here. Here's the control plane code. You can read the schemas. You can look at it yourself. And you can make your own determinations." As to what we're saying, you don't have to trust us entirely. 
 
I think that ability to just literally in conversation just send someone the code and just be like, "Here, this is exactly what I'm talking about. You can poke through it and read it." And now even with agents, people will just - quite literally, we have this Slackbot agent, and a customer will be asking us in Slack, "Hey, can you tell me what data is stored?" And we can just paste the code and then ask the agent, "Can you traverse all the files related to this?" And then compile a list of everything that is related. And you get like an extremely fresh and accurate grounded answer about what data we capture, and it just points to the code. I think that for us has been how we've alleviated people's concerns about what we're capturing and storing. 
 
And then the core of it is that we have the idea that no ports are open on the data plane. Everything is done through kind of a reverse tunnel approach where the data plane reaches out to us. We don't ever go in and reach into the data plane. That's also part of it is if you can just come and say, "Hey, it's your VPC. By default, no ports are open. Everything comes on a push-based model out." Even telemetry, we kind of buffer and then allow it to push out the telemetry. 
 
But then, also, operations. We don't reach into the data plane in order to do operations. There's kind of a service that runs inside of the data plane that we post a message. And then it pulls checks. Is there any operation that needs to be done? That gets pulled down into the data plane, and then that operation gets performed. And that's kind of like tax, right? Ideally, you wouldn't build it that way. That's a very strange way to build some sort of operations thing where you're like pulling for operations and then applying the operation. That's a lot of moving parts in order to do a very simple thing inside your cluster, which ideally you would just be able to issue a post request. And, boom, the operation is completed. Now you have to do it in this slightly different way. 
 
Although, because we're built on Kubernetes, you can kind of leverage a lot of the hooks that Kubernetes gives you to do declarative progression in the same way and benefit from that. But it's still requires a lot more work than you might otherwise want to take on. 
 
[0:44:01] GV: Yeah, I think that's a really good summary. And as you say, I think starting on Kubernetes is a huge help. As an infrastructure product as yourselves, it kind of seems like the only way, but there will be other products out there that don't start that way and does make something like a BYOC deployment even more complex. 
 
[0:44:18] HB: Yeah. If you came to me 6 months in and you were like, "We're doing BYOC," I'd have been like, "We can't. We absolutely can't. It's too hard to bolt on." 
 
[0:44:25] GV: Yeah. No, that's very interesting. I'll probably go back to this YC founder and give him some tips on this basis as well. Just looking, I guess, ahead the future. Obviously, Postgres, is having another moment these days. And I think people just say, "Oh, well, PG vector is good enough for things like this." Where do you see things going? Are we, I guess, asking the devil advocate here of like could things converge back to a frontier large model is actually going to get good enough at this stuff, and context rot isn't such a thing anymore. Or how are you guys thinking about it? And how are you thinking of building it into the future? 
 
[0:44:59] HB: I think there will always be a need for managing the context of language models. I think the problem of what should I know right now for a language model is a problem that will always exist. Language models today, if you just went to your Chat GPT and you were like, "Tell me my schedule for the day." Even if you hook in the MCPs and stuff, a lot of the time, they won't call the right tool. They won't say the right thing at the right time. And so I think there will always be a need for can you get the right information into the model. And that's fundamentally the problem that we're interested in solving. 
 
I think about it a lot, the CPU versus the file system. The language model is the CPU and then there's a disc off to the side. And there's a ton of complexity inside of your hard disk and the interface to that hard disk and the controller and the operating system that talks to it, and then the firmware on the hard disk. I think that that interface is the one that our company believes in, which is there's a language model. It acts as this new frontier natural language information processing system. And then there is a subsystem off to the side that decides what information that model should see and decides what information should be stored, and decides what information needs to be potentially updated and consolidated with existing information. That's the system that we, in the long term, want to build. 
 
Today, it looks a lot like a database. But then where we're going, if you can kind of tell with how we're building components on top of the database, right? Why is a database company training its own models? Is because our belief is that soon it won't be I have a database and I have intelligence. It'll be intelligence actually runs inside of the database. And a lot of the way that the database actually performs its function is by leveraging intelligent subsystems. 
 
And you need to think about those two things at the same time for performance reasons and also just for domain awareness. It is very cost prohibitive to move large volumes of data around and put them into representations for various downstream processing systems and do all of this conversion and ETL, and then do all this reformatting, and re-indexing, and reparsing. 
 
And so if you can build a system that manages that all in one place and makes it very cheap to move compute closer to the data by leveraging object storage, by leveraging a shared storage layer, such that if I want to, for example, just run asynchronous language model jobs on my data, I can. If I want to collocate the node that literally has the query indexes with the GPU, such that now my GPU can talk not to a remote service but talk to a local language model, I can. 
 
Because this year, I think it'll be very common for us to as consumers use language models that run at a thousand tokens per second. Last year we were all running with language models that ran at 100 tokens per second. And so it's not inconceivable that in a couple years we're going to have language models that run at 10,000 tokens per second. 
 
And so today, all of these search systems make the assumption that a network call and the latency to go to your search system is fine because, hey, the language model is slow. But I don't think the language model will always be slow. 
 
And putting these two systems next to each other such that you can put intelligence and data in a low latency path next to each other and maybe even in the future bypass the CPU entirely. Why does the language model need to traverse indices via the CPU? Can it just directly have a sub-kernel launch on the GPU that does the search of the index in line in a very low latency fashion and then feed that data directly back into the language model? I think those sorts of systems will exist. 
 
And our goal long term is to be in a place in the market that we can build towards that. I think that that's kind of what we've been building towards all these years is you should be able to put - there's some system that will need to exist that you can put any data into, any unstructured data. And then it can get any data, and then it can satisfy any query on the way out. And I don't think that system will look like a Postgres. I think that system will look like something much different. And I think it'll look vastly different than what we have today. But if we can get the right abstractions in place and get the right building blocks in place, maybe we'll get an opportunity to earn the right to build that system in the future. 
 
[0:49:02] GV: Yeah. Amazing. Well, honestly, with total sense, I think that's a great place to leave it. Two more questions. The first is a usual of just like up and running for a developer, but I feel like in this case, is it a website or should we be saying what CLI command do you start with? Or what would be best? 
 
[0:49:18] HB: Yeah, pip install ChromaDB. ChromaDB run is all you need to do to get up and running. But then if you're interested in learning more, try chroma.com. 
 
[0:49:25] GV: Great. And yeah, just a curious one for me. What's in the name Chroma? Where did that come from? 
 
[0:49:30] HB: That is a great question. I think it's just one of those things where it's short, it's easy to say, it's memorable, and then it has a nice ring to it. 
 
[0:49:38] GV: Yeah, you guys have a nice aesthetic as well around the whole product as well. Definitely do encourage people to go to chroma.com and check that out, because I think you guys have done a great job with that as well. Amazing to have you on. So much detail. Really learned a lot myself today. I'm sure the audience is as well. Really appreciate it. And we'll be following along with Chroma. 
 
[0:49:58] HB: Thank you so much for having me.
 
[END]