EPISODE 1685

[INTRODUCTION]

[00:00:01] ANNOUNCER: Machine learning models learn patterns and relationships from data to make predictions or decisions. The quality of the data influences how well these models can represent and generalize from the data. Nihit Desai is the Co-Founder and CTO at Refuel.AI. The company is using LLMs for tasks, such as data labeling, cleaning, and enrichment. He joins the show to talk about the platform and how to manage data in the current AI era. 

This episode of Software Engineering Daily is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him. 

[INTERVIEW]

[00:00:46] SF: Nihit, welcome to the show.

[00:00:48] ND: Thank you. Thanks so much for having me.

[00:00:50] SF: Yeah. Thanks so much for being here. I'm really excited to talk about Refuel and some of the cool things that you guys are doing over there. And I was thinking, as I was sort of preparing for us this conversation today, I think, generally, people kind of understand now that we're entering this like AI revolution. Everyone's talking about AI. And, particularly, generative AI and LLMs over the last year and a half. And we're entering this AI era. But, really, there's no AI without data. And, particularly, high-quality data. 

And I've often said, and I probably stole this from someone, but data is the love language of AI. And in a lot of ways, the less sexy headline is that we're entering massive quantity of quality AI data era. It's probably sells less - gets less clicks. But it's kind of the reality. Can you talk a little bit about why data for AI is so important? What are some of the challenges with accessing quality data today? 

[00:01:41] ND: Sometimes data quality is everything. It's a source of all knowledge and behavior that the model will learn from. That any new AI system will learn from. And it sounds the performance of any AI system is limited by how good is the data and how much of it is there. And how representative is it of the final application, the final kind of use case in which the system would be plugged, right? 

There's a few challenges when it comes to acquiring good quality data for AI systems today. And roughly, maybe I can walk through what the challenges are in each step of the pipeline. I think kind of at the top of the funnel there's the collection or acquisition of data. There's kind of wide sources of data to draw from. There's the public web data where there's challenges around just the scale of it, the freshness. Assessing the reliability of various sources. 

There's user data that's publicly available. But, of course, there's some challenges around things like privacy policies of various platforms, platforms like, say, Reddit, YouTube, et cetera. There's a lot of creative works of art kind of data; images, music, books, copyrights, et cetera. 

But, yeah. Broadly collecting and acquiring all of this data is just expensive. It's expensive to acquire them, to store them, to really kind of - and this is in some sense like really just the first step of the process. We haven't done anything meaningful with this data yet. We're just acquiring it. 

The next step often is kind of cleaning and curation. Here, there's challenges and questions around, "Okay, how do we ensure that this data is representative in some sense across geographies? Across cultural nuances? Across languages?" Broadly, a lot of the questions here tend to be focused on how do we ensure that the data that we're feeding to these AI systems is representative of the audience or like the user base? The kinds of applications people that will be using this in the future. 

And then curation is important just for efficient training. I mean, a lot of distributions of data in the real world tend to be lopsided. There's over-representation and under-representation of various kind of slices of it. And it's really important to do things like deduplicate, normalize it, so as to make sure that your model isn't wasting time in some sense looking at and learning from data that's quite redundant. 

And then there's like the last kind of challenge of just enrichment and labeling, right/ You've collected all of your data. You've cleaned it, normalized it, curated it. And now there's the question of, "Okay, how do we label this data so that the model can actually learn something meaningful from it?" 

Traditionally, all of data labeling has been very, very human operationally intensive. Manual labeling is both time-consuming. It's prone to errors. It's prone to biases. And there's like the question of how do we ensure that these human preferences again are representative of the entire user base? These are like some of the kind of challenges I would highlight.

[00:04:37] SF: Yeah. There's a lot to unpack there. Just going back to data quality, would you say that the quality of the data is also one of the things that's like sort of like a competitive edge? If we're thinking about like LLMs, in particular, there's a ton of models available. And is it really the sort of separator in some sense between, I don't know, a LLaMA 2 and a Mistral the quality of the inputs? Because this is kind of like garbage in, garbage out? 

[00:05:05] ND: You described it very well. In some sense, we can think of the two axes for improving performance of any AI system is the data access and the model access. And in the limit, broadly, the kinds of model architectures, the training schemes. Just, broadly, how do we get these models to learn? All of that, we see a convergence outright. Everything from GPT 3.5, to Claude, to LLaMA 2, to Mistral, all of them are broadly the same architectures. And it's the same architecture based on this paper that came out in 2017-2018, I believe, from Google that introduced the transformer architecture. 

[00:05:45] SF: Mm-hmm. Attention is all you need.

[00:05:46] ND: Right. Yeah. Yeah. Exactly. Yeah. Yeah. Attention is all you need. Really, the biggest access that we - I mean, we've seen kind of in our user customer based at Refuel and broadly what we see at large in the ecosystem is that data access. How do you, yeah, acquire, collect, clean data at scale? And then that becomes like the differentiator over time. Because that's what leads to differentiated performance behavior in the model. 

[00:06:13] SF: There seems like there's a lot of I guess problems with actually getting the data to a state where you would want to train on. Because part of it, it's like, "Okay, first, how do you get access to the data?" But then there's also you need to navigate potentially things - especially if you're scraping data from the web, copyright. I know, for example, there's been issues around pulling source code from GitHub and what open-source license is it on? And how does that impact code that gets generated? Is it under the same license? If it was inspired by that. Or if I pull information from a book that was scraped, is that copyright infringement and stuff? 

And then there's ethical issues on top of that. And then there's also all the labeling data and stuff. What are some of the things that essentially companies are doing today to try to like navigate the data collection, and cleaning, and labeling process? How is that done essentially without - are they using tools? Is it mostly like a manual process in some sense? 

[00:07:08] ND: Yeah. It's a good question. I would say there's a few categories of problems that we just highlighted here. Everything to do with collection of vast quantities of data, some of which is copyrighted, some of which has licensing issues, some of it like has kind of questions associated with them as well. I think a lot of this is in the domain of what we think of as language model pre-training. 

We're starting completely from scratch in terms of the kind of model parameters or model weights. And we're feeding it just trillions of tokens of these kind of, in some sense, human-generated data. Because that is ultimately kind of what the sum total of the internet represents. And we're just training the model to learn this representation. It's completely task-agnostic. It's use case agnostic. We're just getting it to learn our language in some sense. Hence, the language model. 

And, of course, I mean, I use the term language model like a little bit more - yeah, it is meant to cover other modalities of data as well. Audio, video, text images, all of them broadly work the same way of the transformers. I think there's a specific set of - well, there aren't that many companies doing large-scale model pre-training yet. I think those challenges do tend to be focused and limited to the OpenAI, the Anthropics, the Metas of the world. And I believe, yeah, it's, of course, like an active, kind of ongoing I think area of discussion and debate around what's okay, what's not okay. How should artists, writers, et cetera? Whose work is being used directly or indirectly? How should they be compensated in some way? Should be asked for permission, et cetera. 

And then there's like a set of problems that's a little bit downstream of that, which where the prevalence of that problem is much more widespread. It's pretty much every organization, every team that wants to use AI systems in some way, which is around how do we - once we take this pre-trained model off the shelf, chances are it's going to be great for prototyping. But then it's not going to be good enough to take into production as it is, right? And it typically needs some tweaking, some customization either in the form of what we call in-context learning, fine-tuning. Potentially, some reward modeling on top of it where the type of data that we need to label and collect is much more use-case specific. And there's, of course, a set of challenges are like a little bit different in terms of, "Yeah, how do you label this data? How do you ensure that the evaluation that we're doing is good and relevant to your task?" Maybe these are the two things I'd highlight. I'm a lot more familiar with the latter. Happy to dive into it more.

[00:09:41] SF: Yeah. I mean, I think you raised a good point there in terms of the building like a foundation model is probably going to be only like an activity that's taken on by very like specialized companies that are able to do that at scale. Have the means to do it. Kind of like public cloud. Not everybody's doing public cloud. It's like four companies that are doing that. Right? 

Eventually, there'll probably be even more convergence from the foundation models where maybe there's only going to be sort of four or five companies that are really doing that that have the means to keep that going and continually update them and do it at scale. And purchase the GPUs and all that sort of stuff. 

But a lot of companies are going to be starting with that as a base. And then they're sort of modifying them through fine-tuning or other means to build more domain-specific things that solve specific app application problems that are for their company. 

We've been talking a lot about label data and some of these other challenges. But for those that maybe are less familiar with the world of AI and how training works, can you give a little bit more of an explanation of like what is label data and why it's important for AI? 

[00:10:50] ND: It's best to think of these AI models or the architectures behind them probably that these are neural networks as like function approximation machines. What I mean by that is let's imagine kind of, okay, all of these AI systems fundamentally take as input some observations about the real world, about users, about customers, about something. And then they're trying to make some meaningful prediction from that. And this kind of relationship between what we observe and what we want the model to predict is in some sense - it can be - think of it conceptually as some function. This function could be very high-dimensional. It could be indeterministic. It could be - in some sense, it can't even be enumerated in many cases. But that is the function that we're trying these models to get to approximate as best as they can. 

And, really, the only way to kind of do this is for the model - for these AI systems to see tons of labeled data and then have some systematic way that they can learn from it, which is broadly what the training and the optimization for these models is about. And so, at the core of it, that's why labeled data, fast quantities of it and good quality of it is important. Because that is what they feed to these AI systems so that they can learn from it and approximate this kind of functional mapping and generalize it to unseen to unknown use cases. 

Conceptually, that's why label data is important. And there's kind of many kind of instantiations of it. Like you want to train a self-driving car? Okay, it has to learn from millions of hours of like humans driving cars and seeing kind of what is like the right next move. How do they anticipate the next action and like few actions down the line and so on? You want to train like a really good chatbot to reply to customer support tickets. Okay, yeah. It has learned that behavior by seeing tons of it in action. I mean, we want to build the best search engine in the world. Okay, yeah. At scale, that happens because Google has just billions and billions of search and user action data that it can learn from. 

I mean, what counts as a label I should say is like it varies a little bit I guess with kind of in the realm of pre-training versus fine-tuning. With pre-training, we're starting with kind of this very large corpus of what is unlabeled data. And then transforming it so that there's still some supervision from it. There's like a few different ways to do pre-training. But a common way to do this is to mask out some specific parts of that input, right? It could be words, characters, tokens, entire sentences. 

And then ask the model to predict what is the thing that is masked out. And in some sense, then you have the model's predictions and then the ground growth which is kind of what is actually known but just masked out from the model. And that gives - that is like some way of creating label data in some sense for the models to learn from. 

And then when it comes to kind of fine-tuning and kind of just training anything task specific, that's where a lot of the things like human labeling, and expert of human judgment, and data generated from that kind of comes into the picture. 

[00:14:03] SF: A lot of times, would labeling sort of fall into like I'm basically giving a categorization for something? If you take the autonomous vehicle example, and maybe I have, I don't know footage, of an accident and then I got to label that as an accident and essentially use that as a way to train the autonomous vehicle to maybe avoid accidents or those types of situations. 

[00:14:24] ND: Maybe, in that example, what labeling would mean is okay - there's labeling for specific objects and kind of parts of what driving on the road would look like. Here's the road. Here's the kind of pedestrian sidewalk. Here's a tree. There are other cars. Here's a truck, et cetera. There's a lot of that kind of labeling. And then there's like labeling for specific events or scenarios, which is kind of very much like the kind of thing that you highlighted. 

[00:14:52] SF: I want to start to talk a little bit about some of the stuff that you're doing over at Refuel. A lot of companies are sitting on mountains of data that - and they don't really know how to use it. It's unstructured. Maybe it's encrypted. Stuffed in an S3 bucket somewhere. And there's data lakes, and lake house, and so forth. But those take a lot of initial work and maintenance to actually get up and drive value from them. How do you get essentially the data into a form that's immediately useful without engineering and manual work? And what are some of the things that Refuel is doing to assist with that workflow? 

[00:15:27] ND: At a high level, Refuel is a platform to help enterprises, teams label, clean, enrich their data at scale with the power of [inaudible 00:15:34]. And if we can think of working with Refuel as, I would say, a three-step process where you point Refuel to where your datasets. It could be a database. It could be a data lake. It could be some set of objects sitting in S3. Typically, this data is unstructured. It's kind of just coming from either some production system that's logging that data there. Or it's some dump of data that you're getting from some external source. 

And this is typically the starting point for most teams that want to use, leverage Refuel in some way. And the first step there is define the thing that you want to do in natural language. It could be something as simple as classified the sentiment in this piece of text into one of these three categories. But it could be arbitrarily complex, right? 

Imagine you have a large taxonomy with hundreds of different classes and you want to make a determination for, yeah, there's three layers of the taxonomy to first categorize this input into layer one. Then depending on kind of what that answer looks like, maybe do something conditionally downstream. But, broadly, think of this as very much kind of guidelines that you would describe for a domain expert or like for a human reviewer if they were going ahead and labeling this data. That is what it feels like. Just define the rules.

[00:16:54] SF: How do you define those rules? 

[00:16:57] ND: Broadly, just natural language within the product. We have an interface where it looks, feels very much like you're writing guidelines for a human reviewer or like for a human annotator. Except the annotator really happens to be an LLM that they've trained, that they've customized for these kinds of tasks. But that's what it starts with, which is the assumption is that you, as a user, as a domain expert have a very good idea of like what do you want to do with this data. And so, just help us kind of codify some of that expertise in the form of a set of guidelines. 

And then we'll take these guidelines, we'll take the data set that the data that you pointed us to and Refuel, LLMs will start running the labeling job and produce a set of initial outputs. Along with this, well do a few things, like, okay, flag things that are potentially low-confidence. We'll flag things where the input is maybe weird, or outlier, or noisy and someway. 

Essentially, looking at - bubbling up things that would be good for you to review and provide feedback on. And once you give us this initial round of feedback, we use it in real-time to improve the model's predictions for the remainder of the data. Almost think of this active labeling type approach where you define a set of guidelines, we produce a set results. You give us feedback. You correct potentially some of the - you review some of the low-confidence labels and you potentially correct ones that the LLM might have made a mistake. We collect all that feedback and then label the next batch of data. 

And, typically, what we've seen is that within 30, 45 minutes of interacting with the system, we can get most teams, most use cases to a place where the result is at parity potentially better than human annotators. And in some sense, this is the first part of the kind of workflow where teams see a lot of value. Where, traditionally, if you're doing this with hiring, maintaining, training like a team of human reviewers, that process of defining initial guidelines, getting them to label it, reviewing that work, sharing some mistakes and getting - those iterations typically tend to be on the order of days two weeks. Whereas given that LLMs are many, many times faster, you can get that process down to something like an hour. That's the first part. 

And then depending on kind of exactly what the final goal is, typically, most teams going to do one or the other, which is sometimes like this task that they've built, they'll want to deploy that and just start using it online in some fashion and start collecting telemetry and usage data. And then at some frequency, they'll want to review this data within the platform. Again, provide it feedback. And this becomes this almost like this data engine where data is being collected in real-time. Some frequency, you're reviewing it. You're providing it feedback. And we're using all of this to improve the models output on an ongoing basis. 

And then there's like another set of use cases where just deploying this task with like a fairly big - kind of what ultimately is like a fairly big model, multiple billions of parameters. It has some implications in terms of just what we can do there in terms of latency, in terms of supporting throughput and so on. If that is not something that is feasible for many applications, then teams would want to distill broadly all of that knowledge into like a much smaller task-specific model. And that's where a lot of the kind of fine-tuning comes in as well.

[00:20:26] SF: As a user, what is the output of this process that I'm getting? And then how do I know - when I'm done, how do I know essentially like how good the resulting output is? 

[00:20:37] ND: The output at the end of this process is transformed, enriched kind of label data that's used that's ready to be plugged into some downstream kind of application that you have in mind as a user. It could be for training downstream models. It could be for powering a set of product features in whatever product that you're building. In some cases, it could be a lot of our kind of users customers are data providers in some ways where the data that they clean, enrich using Refuel's platform is valuable just as a product offering for them. 

How do I know it's any good? It's broadly like this area of evals as it's called colloquially in like the LLM ecosystem. I would say it's a fairly active area of discussion, debate, development. And there's a few reasons for it. I mean, A, just the ecosystem is fairly new where I think we're still beginning - just beginning to learn how do we evaluate LLMs. 

But at the core of it, it does have to be - at least in our view, it has to be some comparison in some fashion to what the expected output is that in terms of like humans should be the judge at the end of it, right? And there is the question of like, "Okay, what data do you evaluate it on? What set of metrics do you use?" Because there is a set of broad LLM benchmark that are publicly used. And we think like that those are often not the most helpful when it comes to evaluating how good is this model and the data that it did produce for my specific task and for my specific use case. A bunch of like different kind LLM leaderboards, benchmarks that are kind of used publicly. And those are good I think if you want to get like a very kind of high-level, low-granularity view I would say of just distinguishing kind of like the good candidate models from the not so good ones, which is fine. I think it's a good first filter. 

But then, often, they're not. Either the set of metrics that these measures are not super well-aligned with your specific task. Or they're non-discriminative enough, right? Like, okay, what does a performance difference of like one, one-and-a-half percent mean on the specific benchmark for my specific use case. 

Yeah, we're big fans of tasks. What we think of as task-specific evaluations where, depending on the kind of task that you're asking the LLM to do, is it a classification task? Extraction? Is it a free-form generation? Yeah, what is the expected output would depend on that task. And then there's a question of like what is a set of things that we should measure there? Typically, yeah, think of this as something that measures quality and then something that measures how faithful is the model. Basically, what is the likelihood that it's hallucinating? Those are like the two kind of important considerations for teams that are using Refuel. 

[00:23:29] SF: Yeah. And even hallucinations are sort of context-dependent. You're writing a children's story, hallucinations might be okay. But if I'm trying to pull Mark Twain quotes, then maybe it's not as okay. And, also, I mean, I think you're kind of getting into this. There's a lot of nuance in terms of like how do you actually measure quality. Because a lot of it is task-specific. And some of it probably dependent on like you actually need like human feedback in terms of does this meet the quality bar for the thing that I'm trying to accomplish essentially?

[00:24:02] ND: Yeah. Yeah. Absolutely.

[00:24:04] SF: Why can't I just take my data and use something like OpenAI's, like APIs directly to do something like this? 

[00:24:12] ND: It's a great question. To be honest, a lot of users, customers that come to us do start there. And I would say it's a very fine place to start. To some extent, it's a testament to just how good these models are out of the box and how easy they are to use. Just literally sign up at an API call. It's a really great place to really understand like, "Hey, is this LLM?" Within the realm of kind of potential solution candidate for like the use case that I have, right? Where if I want to prototype something real quick, that's often like a really great place to start. 

But I think kind of when you spend some time with these systems, typically, we have seen kind of users run into one or more of the kind of the following challenges, right? Which is there's a challenge of output quality. Okay. Yes, OpenAI, Anthropic, yeah. I think probably all of these kind of state-of-the-art but closed behind API only kind of models have more or less like the same strengths and weaknesses, which is they're very good generalists at thousands of possible tasks. But they aren't specialists at that one specific task or like a few specific tasks that you care about. 

And so, they're great to go from zero to 75, 80, 85% accuracy. But then how do you get it to 95, 96, 98% accuracy and reliably that you actually need to put this in front of your users. Or like to actually plug it into production. That's one challenge that we've seen. 

The other challenge is scale and throughput. GPT4, Claude 3, especially like some that are most powerful Claude 3 models, ultimately, like there are multiple tens of billions of parameters. Even with like mixture of expert-type architectures and so on. Where, yeah, there's just I mean no getting around the fact that it costs a certain amount of money to run them. And it has some implications in terms of latency and like the scale of throughput that they can support. 

And then like, yeah, I mean one other kind of challenge as consideration that we've seen is just around privacy and security, especially for some domains. Yeah, these are like a few kind of observation that we have seen in terms of like, "Okay, it's great to start there. But then, oftentimes, it's not enough just to do that." And you need like a set of layers on top of these kind of core LLM APIs. 

[00:26:30] SF: Yeah. I would think also beyond just the sort of tuning of the LLM for this specific task, you also have the workflow support. Sure, I can go to ChatGPT and even have it like help me write code rather than using a coding copilot. But it's like more effort go to the browser. Like it's a less integrated experience. And it's not really designed for that specific workflow of, in this example, writing code. Or in your example, I think the workflow is probably even more complicated where there's more sort of like tuning feedback loop. And, ultimately, I need to produce some sort of asset that I can actually use for going and then fine-tuning my model or doing whatever it is I need to do with it.

[00:27:11] ND: Yeah. Exactly. Exactly. Yeah. Absolutely. We think of kind of the core infrastructure in three layers. There's the core kind of base LLM that's where its interface is pretty simple. But it's very, very powerful at what it does. It's input prompt, output tokens. 

Then there's a data management layer on top of it that is actually doing this kind collection of feedback. Indexing it. Sampling from it in real-time for things like C# learning. It's doing the job of maintaining kind of this data set that is evaluation data set. And like a lot of this kind of - there's like the integrations into a bunch of external stores and so on. There is that layer. 

And then there's like the core kind of product of the workflow layer on top of it, which is how users mostly interact with us, which is that is where you define the task. You see these - yeah, you can iterate on guidelines. You can provide feedback. You can understand kind of which predictions changed from one version of the prompt to the next one. And so, a lot of this kind of just tooling that is catered and tailored towards the kinds of use cases that people want to use we fulfill.

[00:28:19] SF: Mm-hmm. What can you share about how the like LLM part of the infrastructure works? How did you - were you basically fine-tuning and more foundational model to be specifically built for data cleaning and labeling? How does some of that stuff work? 

[00:28:36] ND: Yeah. Absolutely. I would say there's two components there. We have our own LLM. And we can share some details about that. But at the end of the day, we are LLM kind of agnostic. We do support any of the state-of-the-art LLMs that people might want to use, explore, try out, including OpenAI, Claude, Gemini and so on. 

That said, yes, what we've found, what we've seen is that none of these models provide state-of-the-art performance when it comes to this very specific set of tasks around data labeling and cleaning, which is why I mean we have to basically go out and build our own model to do this well. 

And to do this, also, there's the quality consideration. But there's also the consideration of how do we get it to scale? And how do we build something that can then be customized further for specific customers and specific use cases? Yeah, this LLM, we call it Refuel LLM. It's something that we built and released a few months ago. We're training actually a new version of that right now. We'll see. Maybe by the time this episode goes out, it might already be released or might be on the verge of releasing. I don't know. 

Yeah. As part of building this out, we don't start model pre-training from scratch. We do start with a powerful base model. Think of a LLaMA 2 or a mixture of experts type architecture. But then we do extensive instruction tuning on top of it. Think of, yeah, the kinds of data sets that we've collected amount to - something like in the previous situation, this was I think about 2500 different kind of tasks data sets that are very much in this kind of domain of labeling. But then they're problem area agnostic. We'll have data sets from public internet, from law, from finance, from e-commerce, credit cards, et cetera. Most of it is publicly available. Some of it, yeah, we just had to go out and license that data so that we can use it. 

Yeah, that becomes sort of like the raw kind of bases. And then, of course, there's some amount of curation, cleaning, some amount of labeling that we do internally as well to create this data asset that we then use to tune this base model and then have it be purpose-built for labeling, enrichment, cleaning type tasks.

[00:30:44] SF: What's your sort of your toolchain behind the scenes in order to go from creating a new version of the model to actually pushing it to production? What is that like MLOps toolchain? Are you using combination of like existing stuff? Or you had to build some stuff to support the actual productionization and pushing these models to production and using them? 

[00:31:06] ND: Yeah. That's a great question. I think the answer is a little bit different for two workflows. There's the workflow for training and building Refuel LLM, which is not a kind of daily or weekly type activity. Well, at least with the size and scale that we're at, we'll probably do that like once every few months. 

And then there's the customer-specific fine-tuning workflow. That is very much - it's something that is part of our product where customers are using the data that they have collected within the platform and labeled, they'll want to use all of that to further customize any of the base models that they're using, including LLM. 

And so, that is like a lot more kind of much more frequent. I think the answer is like a little bit different for both of these. But probably like the latter is the one that's probably more relevant here. Yeah. For that, I mean we do rely a lot on just open source kind of tools to do this. Yeah. I've used Transformers from Hugging Face as the kind base library to be able to train these models. Along with accelerators like Deepspeed and FSDP. 

I mean, our training infrastructure is we use like a combination of training like GPU providers. But, I mean, all of our core infrastructure is on AWS. And, yeah if GPUs are available there, we'll train there. If not, there's like a few other different providers that we use as well. 

And then in terms of, yeah, actually serving these models, yeah, I mean use TGI, text generation inference engine, which is like an open-source project again by Hugging Face. We leverage that quite extensively. And then there's like a bunch of other tools around like monitoring, training fronts with tools like weights and biases. 

And, yeah, the thing that we have had to build out a bit custom is everything to do with evaluation. Because, yeah, it's the one that is very kind of task-specific. And, yeah, I mean there really isn't something that at least we found in the current ecosystem that we can just plug in as it is directly. And that's said, it is one of the more important problems to solve for on behalf of our customers. Yeah, that's something that we had to do a little bit custom. 

[00:33:15] SF: For the customer-specific modeling that you're doing, is there some like versioning of those models as well? 

[00:33:23] ND: Yes. Absolutely. Any specific application for which a customer might want to fine-tune a model. Think of like a lineage of models that we maintain. And these are snapshot and they're - yeah, with each snapshot, there is some understanding of exactly what data went into training these. What is their kind of performance? And as a user, you can decide to, yeah, roll back, delete, switch to older snapshots, et cetera. 

[00:33:50] SF: Okay. And then sort of like outside of exactly what you're doing, but just based on your experience and maybe some of the customers that you're working with, what are some of the trends you're seeing in terms of how companies are building with LLMs? Are most companies starting to use multiple models, private models? Are they sticking primarily with like public models like a GPT? 

[00:34:09] ND: Yeah. that's a great question. I'll say like we'll probably see two broad categories of use cases emerge that are powered by LLMs. There's the gener - and maybe I'm not sure this is - probably, there's better terminology for this. But I think of it as the generative use cases and the predictive use cases. 

Generative use cases are the classic kind of co-pilot for X kinds of a use case where there's typically a human in the loop. These are ultimately meant for human consumption. And it's for augmenting knowledge work is one way to put it. Think of coding Copilots. Think of writing Copilots. Things where it's traditionally the domain of knowledge work and we're supercharging this by having this very powerful assistant. 

The predictive use cases are almost kind of - in some sense, these are problems that are a lot more closer and relevant to Refuel. But these are typically completely automated. And almost like these used to be done traditionally either with like large kind of armies of human operators or some system of rules, rule engines, or even like traditional ML models. And we're seeing all of that converge to basically just be built on this new substrate of LLMs. And compared to generative use cases, these tend to be like fairly kind of high-volume. And they're typically completely automated. That they're [inaudible 00:35:33] if there were to be a human in the loop, for example, for reviewing every single prediction that comes out of this model type for powering that kind of use case. 

This is maybe one cut on kind of what we're seeing in terms of trends and how companies are leveraging LLMs. In terms of like the how of it, yeah, you started quite correctly. I think we're moving to a multi-model world where there's going to be always some powerful kind of frontier models. Think of the GPT4, Claude 3, Geminis of the world. 

And then at the same time, there is this kind of very rich and vibrant ecosystem of open-source models where that are still very good out of the box. But, really, like the value addition I would think of is like that they're a lot more customizable and you have a lot more control over them. Yeah, you'll typically start with something that is very powerful and then build on top of it for, again, leveraging your own enterprises and proprietary data. And, yeah, of course, it buys you things like control and potentially better cost and so on.

[00:36:32] SF: Isn't that a little bit though like running like installed software on-prem versus using like a managed service on the cloud? 

[00:36:41] ND: I think so. Yes, in some ways, for sure. At least on the access of control, I think there is like a very good way to look at it. I do think there is - at least with where the models are currently at, I do think the kind of customizability piece is quite important to get quality right in most cases where maybe we don't quite see that much need for customizability, let's say, for traditional software where the only access kind of to think about is like, "Hey, am I okay with like a SaaS hosted somewhere else kind of use of this software? Versus do I need it to be on my premises?" 

Maybe a few years out, when kind of these models get super, super powerful, we might see the need to do that less. But at least like where things stand today, I mean most of these models do need some form of customizability. Now it could be - I'm not saying kind of fine-tuning is the only way to do it. But it does tend to be a very powerful way at least today. There's, of course, other option around like what is thought of as retrieval augmented generation where we're not modifying any model parameters. But, rather, we're just focusing on supplying it like the right context. And it can reason to it.

[00:37:44] SF: Yeah. I think that makes a lot of sense. It's sort of like a byproduct of the immaturity of the market. I guess it would be similar to even in the sort of managed service versus like I need to get in and tweak parameters type of experience. Today it's like companies that are operating at a certain level of scale. Like let's say you're running a Postgres database, then running that on AWS RDS is probably going to get you to a certain level of scale. But at some point, when you reach beyond that, then you have to do something that's a little bit more custom where you can actually get in, like customize it so you can do like horizontal sharding or maybe even use something like Postgress extensions to extend the database for your specific use case or whatever. But you don't need to do that from day one in sort of the database world today because databases have been around for 50 years. So, they've done a lot of work to make it work for most people. But LLMs haven't been around that long. They've only been around for like less than 10 years. There's a lot more work to be done to get them to a place where they just kind of work out out at a box for most people.

[00:38:46] ND: Yeah. Yeah. Exactly. Yeah. That's exactly how I look at it.

[00:38:48] SF: With LLMs, they're basically kind of slurping up a lot of human-created content for training material. But now, LLMs are capable generating like a tremendous amount of content. At some point, the LLM-generated content that exists on the internet is going to dwarf the amount of like human-generated content. I was curious about your thoughts on this. Is this feedback loop where AI is trained on mostly AI-generated content going to be a problem at some point? 

[00:39:16] ND: Yeah. It's such a great question. Okay. Maybe let me answer the more kind of a slightly more constrained version of that question, which is, is there value in LLM-generated or like synthetic broadly to improve model performance? And, yeah, this is kind of again like one of these very active discussion debate areas within the LLM community I would say. But we have definitely seen good signs of this. 

And maybe to share a couple of examples, there's this paper that came out I believe last year or the year before called Textbooks Are All You Need. And, again, it's kind of a play-on kind of Attention Is All You Need kind of paper from a few years back. But basically, it made the case that like high-quality training data is important. And you can prompt it correctly. You can get LLMs to generate this high-quality data. And of course, it needs some amount of curation and post-processing downstream of it to things like removal of duplicates. Removing things that are very likely to be hallucinations. 

There's, of course, like some amount of expert kind of curation involved downstream. But this problem of kind of is there some value in generated kind synthetic data I think is like - I would say the answer to that is more likely yes. And, yeah, there have been subsequently like few other papers kind of that - just research efforts that broadly kind of point to this direction. There are like the self- instruct paper that kind of makes probably where the takeaway was very similar. And, yeah, a few other things as well. 

Even if that's the case, it's a bit hard to know exactly what is the impact of this at scale, which is kind of where the original question was, which is, "Okay, let's play this out five, 10 years in the future." Or, well, maybe in two years in the future where just creating content like used to be a lot more friction, a lot more effort and needed a lot more kind of just creativity and like human hours. Yeah, now that's kind of you know - that can be multiplied by factor of ten-hundred-thousand. And what does that do? I wish I knew the answer to that question affirmatively. I'm not sure. 

I think what's more likely is that just the ways in which we collect and curate this data, prepare this for subsequent kind of future training will need to evolve and adapt to account for this. Because, yeah, the distribution of data, the kind of properties that it has, the kind of strengths and weaknesses that it has like is going to be different compared to human-generated data. And like we do see that - I mean, probably, there was like this fun study I recently came across that kind of tried to look at - that try to study what percentage of peer reviews tend to be like ChatGPT-generated at these kind of academic conferences. And it was like quite evident by just a distribution of tokens, for example, that are peer reviews that, okay, there's this like this massive spike since like last year or so. 

Yeah. I mean, broadly, I think what this will mean in practical terms, at least that's my kind of assumption, is that we'll just evolve how we collect and kind of parse, curate the data. So that, ultimately, it does still end up being useful for LLM training. 

[00:42:15] SF: Yeah. I think like at the moment, anyway, based on my own experience, the thing that you were talking about with the peer reviews, if you're like a heavy ChatGPT user, I think you can see certain patterns in there [inaudible 00:42:26] length of a paragraph. Length of the sentence. Certain words come up more frequently than probably like a human would write. 

There are signals at least today. And who knows like in a few years from now as the models get better, will the variance level in terms of the output get better? But today, there's definitely patterns that are recognizable as like an LLM-generated piece of content.

[00:42:46] SF: Yeah. Yeah. Absolutely. As we start to wrap up, what's next for Refuel? And is there anything else you'd like to share? 

[00:42:53] ND: Yeah. Absolutely. I mean, we're a small team. We're eight people now. And, I mean, since the start of the [inaudible 00:42:56] we had something like close to like a thousand-X growth in terms of like just the volume of data that we've been processing on a monthly basis. Yeah, just like a lot of the team's efforts today at least on product and infrastructure side are focused on scaling, stability. And just ensuring that things - that we can manage some of this growth. And our users and customers don't have to bear the brunt of it, which occasionally happens. And, sorry, folks, for that. But yeah, we're doing the best we can. 

I think that's part of it. And we see that kind of just being an important area where the team invests in for like the next 3 to 6 months. There's a lot of research happening in the field of improving LLM output quality, reliability, and kind of training efficiency. Things like low-rank adapters, for example, that make training a lot more parameter-efficient. There's things around like reduced precision influence that basically where you can very aggressively quantize like the model weights and still get to a good output. Yeah, probably what can we learn and incorporate into our product and in our infrastructure is one other kind of ongoing area of just investment for us. 

Yeah, that there's kind of training feature and better versions of our own LLM that broadly powers a lot of our product use cases. These are like a few areas where I foresee the team investing in in terms of product and infrastructure. 

[00:44:22] SF: On the infrastructure side, are there unique like scaling challenges or maybe scale challenges get introduced earlier due to the nature of doing work with these AI models? 

[00:44:33] ND: Earlier compared to like building [inaudible 00:44:35]. 

[00:44:37] SF: Like a B2B, like standard non-AI based application. Or is it sort of just business as usual? Like we need to scale out our infrastructure. We're going to need more servers. We need to run in more regions or something like that? 

[00:44:49] ND: Yeah. Okay, I would say it is ultimately like the same kinds of challenges, which is like, yeah, just around resources and managing resources to like match with throughput and so on. But I think just given kind of how young the kind of broadly vague the ecosystem is, it does tend to be harder. And you do have to face it a lot more earlier. 

Because, yeah, for example, cloud providers have fairly good I think managed offerings for a lot of different software and infrastructure things, right? Like your database go RDS. I need Kafka queuing solution. Okay. I just use Kinesis. You need a massive scaling key-value store. Okay. Just use [inaudible 00:45:29]. Of course, if you're actually innovating as a business in those areas of course it would make sense to like not use those. With at least when it comes to supporting LLM training and inference at scale, unfortunately, there just isn't too many that are good out of the box solutions. There are tools that we can rely and leverage, which we do. But, yeah, beyond that it's - yeah, things like okay how do you even benchmark LLM throughput is not a very trivial question I would say. Because, yes, there's things at the level of requests. But then you have to account for like, "Okay, what are each of these types of requests like in terms of input and output tokens?" Because input tokens, it's much more easy to scale. Output tokens are very hard. Almost like your latency increases linearly as a function of like the length of the output. And then, yeah - anyway, that's like an example of it's like okay these are some of the best practices for how to do this. And how to do this well are still being figured out and written. And so, it's a challenge. But it's also part of the fun, I guess.

[00:46:32] SF: Yeah. Absolutely. Well, Ni, thanks to you so much for being here. This was a really interesting conversation. And I'm excited to see what Refuel continues to build and come out with. 

[00:46:41] ND: Certainly. Yeah. Thanks so much, Sean. It was really great chatting with you. And see you around.

[00:46:46] SF: Yeah. Cheers.

[00:46:46] ND: All right. Thank you.

[END]