EPISODE 1665

[EPISODE]

[0:00:00] ANNOUNCER: George Mathew is a managing director at Insight Partners, where he invested in weights and biases, Jasper, and others. He has over 20 years of experience developing high-growth technology startups, including most recently, being CEO at Kespry. George joins the podcast to talk about his path to becoming an investor, his data-first thesis about investment, the AI business landscape, his book recommendations, and more.

This episode of Software Engineering Daily is hosted by Jocelyn Byrne Houle. Check the show notes for more information on Jocelyn’s work and where to find her.

[INTERVIEW]

[0:00:48] JBH: George Mathew, welcome to Software Engineering Daily.

[0:00:52] GM: Hi, Jocelyn. Great to meet you as well. Pleasure to be here.

[0:00:54] JBH: We're really excited to spend some time talking with you personally, as well as for our show's audience, our technical audience. There's nothing hotter to talk about than AI, or data, I think. You have been a storied investor in this space, right? So many investments, so early on in weights and bias, Excel data, which is one of my favorites, Jasper. We're really excited to have you come and talk with us. But before we kind of get into the technologies that you're interested in, I want to learn a little bit about you. Were you just a young little boy out in the playground thinking like one day I'll be an investor in this space? How did you come to realize this is what you wanted to do?

[0:01:32] GM: Yes. I would say it's certainly wasn't a portion of my journey that I thought I was going to be investor when I was a young kid in the playground. I think originally, my intent when I was that young was probably to be a pilot. When I got a little bit more older, I realized that that wasn't going to be the right calling for me and that I really wanted to be a medical doctor. As I was studying, actually, for my MCATs, my junior year, I came across the NCSA Mosaic browser, which a certain individual named Marc Andreessen had shipped out of the supercomputing facility at the University of Illinois. I would say my heart skipped a beat back in 1996, and really wanted to understand what was going on with the emergence of a tool and technology like the world wide web, and the modern-day web browser in those days was pretty early in the overall journey. I just got really excited by that and followed a lot of where my passion was there to go find a job in California to work in early-stage company that was doing the first generation of E-commerce applications in the mid-90s, working over emerging technologies around the world wide web.

[0:02:45] JBH: It’s such an interesting moment that we're experiencing right now, because I do think, my two big sea changes that I've experienced was that Internet came in to replace client-server. Then of course, open source. These were like huge moments that felt like second industrial revolutions in their paling, in comparison to some of the excitement around Generative AI at the moment. Help me understand a little bit about what is your perspective on investing in Generative AI right now, because there's a lot of interest on the consumer side. But it's not clear which startups and what their strategy should be.

[0:03:17] GM: A lot of my perspective from investing in Generative AI has really come from being originally, a builder, and someone who's been around data and AI systems for a good part of a decade and a half. As I started to see this current generation of systems continue to evolve, it was just very clear in the enterprise, that the modern data stack itself was just an important underpinning for how this next generation of AI-based applications and systems would emerge. So, I really started to look at where the modern data stack was really headed when it comes to all the tools that were required, not only in data management, but data catalogs, data orchestration, data observability. And really, built my thesis on just a data-first view of how the next generation of AI and machine learning-based systems would emerge.

So, naturally from that layer of the modern data stack, started to look at machine learning operations, MLOps, as a necessary set of tools that would be required for a machine learning practitioner to be able to build a model and be able to bring those models into production. One of my first investments joining Insight about three years ago was in a company called Weights & Biases, which became, of course, one of the de facto tools for all the experiment tracking, hyperparameter tuning, version controls that were necessary for machine learning practitioner field to get their job done and effectively built models and bring them into production.

I think as you started to see the evolution of MLOps, then it became clear that there was this opportunity to take the data that was coming from these modern data stacks and merge them properly with the AI systems, which are course now, LLMs, and transformer-based AI systems and build these next generation of Generative AI applications. Of course, one of our investments about a year and a half ago was a company called Jasper, which was building a Generative AI application for how content writing could be more naturally be done with a copilot to support how content writers work on a day-in, day out basis.

So, the thesis for me was always starting with a data substrate, which at least in my case, in the enterprise was very much targeted towards the modern data stack itself, and building upon itself to go into MLOps and more recently, into Generative AI applications.

[0:05:45] JBH: There's a blog post on the website about the difference between MLOps and LLMOps. I interviewed Krishna Gade from Fiddler on this show as well. So, I spent a lot of time in that MLOps space. Help me understand what the – well, maybe we should sort of say like, what we say this MLOps or LLMOps, let's just set a business case. Why do people care about that? Then, let's kind of compare and contrast the two.

[0:06:06] GM: One of the things that we started to notice with emergence of this category of machine learning that was really focused on transformers and large language models, was that you were shifting from a model-centric experience to getting a model into production, and getting success out of the outcomes of model production, to almost a data-centric world of getting models into production. The reason I mentioned that is that the models weren't tremendously changing in terms of what was going into the algorithm, the models per se. You might see some shifts and the weights and biases surrounding the model, but less so around the underpinnings of the model itself.

What was changing pretty dramatically was the data that was coming into the models. This is where we started to see this current generation of transformers and LLMs, in particular, emerge that the more data that you brought into a large language model, the more human-like and reasoning you started to see the models perform in terms of when they were put into a general-purpose scenario, like what you saw with OpenAI’s ChatGPT. Or even domain-specific models that emerged, say, for instance, like BloombergGPT.

In all these cases, what became clear was the techniques that you were using to go build an LLM was going to be similar, but in some ways, quite different from the techniques that you would be using to build a model for the purposes of what would be a traditional use case around computer vision, for instance, or around a predictive statistical analysis that you might be doing.

So, we started to think about this at Insight. Where would be the delineation of where tools and platforms would evolve for the systems of record, versus the systems of prediction, versus the systems of creation? So, we've had a lot of history around the systems of record and the systems of analysis or prediction. But we haven't had a lot of history in terms of really understanding what these new systems of creation were. I think that's where we put a lot of our attention more recently, in what's basically known as LLMOps versus MLOps. In that regard, as we try to navigate what those differences were, and those similarities were, there were some things that really struck us as clear delineations.

First and foremost, if you'd look at a generative model, and compare it to historically predictive-oriented applications, there was this tremendous benefit that came out of transfer learning. The models themselves have this ability with very little bit of data, few-shot learning, single shot learning, be able to transfer learn quite a bit from what was previously taught to the model itself. So, that was really different from what was historically, the case, with a predictive model that was in market prior to whether you were using a neural network or any other algorithmic method.

In a similar way, you started to see a difference in terms of how the compute had to be managed, right? Because there was a tremendous amount of just compute required, particularly GPU-based compute, that was required to be able to not only do the heavy training that was required to build a model, with the number of parameters that we're now seeing in the model, reaching as much as a trillion parameters in the GPT forward style model. But also, in the inference itself. It was a compute-intensive experience to be able to handle the inference, even in a model like what you're seeing with OpenAI’s GPT-4, or Anthropic, or cohere.

The last thing that we saw as a difference is just the feedback loops. If you think about the use of RLHF, a reinforcement learning through human feedback, or an RLAIF, in all of these situations, the feedback loops were more important than ever just be able to improve models, particularly as they were in production, and you're starting to see this notion that the model is almost a living thing that continues to improve upon itself, using reinforcement learning beyond the initial training runs themselves.

So, those were the key things that really pushed us to push out our perspective on what was really importantly delineated around building for LLMOps versus MLOps. And of course, there were many things that were similar in nature. Privacy was very similar in terms of how you worked with MLOps versus LLMOps. Model governance, model security, which I’m sure you spent some time with Krishna and Fiddler, one of our portfolio companies in that regard. But in that regard, what we wanted to just really call out for anyone who in interested in putting this next generation of models as they emerged the LLM models, in particular, what would it take to build it, and how is it different from this last generation machine learning, and that's really what we tried to encapsulate in that article.

[0:11:22] JBH: Okay. So, in a couple of ways, just to summarize a little bit, what you're saying is like, on the left-hand side of the diagram, it's a lot of the same problems of data preparation, data privacy, putting the data from an enterprise perspective, you feel comfortable, it's ready to go in. But on the right-hand side, it's quite different. Because unlike ML, that's telling you about the data you've already got, it's generating new data, and there's a much more of a role for the human in the loop on that right-hand side of iterating and using the model. Is that a fair assessment?

[0:11:50] GM: Yes. I think, it's fair and I think there's some things that go into the compute management, which we've never had to really think about at the scale that we have to think about, particularly in both the model training as well as the model inference.

[0:12:02] JBH: We learned our lesson and cloud data. So, we're going to think about it early now. Think about our expenses early. I will just share and put in the notes as well, you have been a really amazing landscape of LLMOps. People are sending it to me all over the LinkedIn and my friends are like, “Hey, have you seen this?” So, I think it's terrific and I would definitely encourage the audience to take a quick look at it, because I think it has this notion of end-user management that I wanted to really double click on.

I've heard you did some other interviews, where you talked a little bit about how the ability to integrate human feedback is an asset in your mind. Is that what you mean, when you talk about end-user management as part of this?

[0:12:41] GM: Yes, it is one of the pieces, for sure. The feedback loop is pretty, pretty important. Because if you start to think about how model alignment occurs, building models that have values that are aligned with human beings, the only way to be able to do that is to have humans in the loop to be able to provide the feedback to models as they continue to be aligned for our needs. I think there's also another piece of the puzzle, which is when you think about beyond the model itself, and how you instill that into, say, for instance, an enterprise experience, you need more than just a powerful model with good feedback loops.

You also happen to need to get good private datasets, right? It turns out that those private datasets are the ones that really enable models to further target and focus, the task at hand, and hallucinate last, right? So, the more complimentary private data that you have to the task at hand, the less likely will hallucinate on things that we're basically not having enough data that was being folded into the model in terms of the model training itself.

The third thing I would also mention on top of that, from a user experience standpoint, is that you still need to think about this as an enterprise application. So, great enterprise applications have great UX and workflow associated with it. If you're building an enterprise-grade application, it's not just about the human feedback loops in the models themselves, but it's a great user experience for the application front end, and it could be as simple as a chat interface. It could be a more complex workflow, but nevertheless, a great user experience surrounding that is absolutely essential for any AI-based application to prosper in the enterprise.

[0:14:26] JBH: Yes, I want to get back to that. But first, I want to talk a little bit about this high-value private data, high-value datasets for train on to help give direction to the Generative AI model, LLM model. My experience has been that people who have the tastiest, most desirable private data are typically the ones who have the least capabilities often in building their own software, creating their own tools. Do you think that has implications? Because I agree. You have to have this like private rich data. Do you think that has implications for this whole debate around will it be proprietary open-source model that win overall? It's hard to form an opinion there, when you know that the organizations with the best data often are going to be reaching out to maybe a more proprietary organization, rather than building their own with, say, a bundle of open-source models. Have you given some thought to like, what does that sort of adoption route going to look like?

[0:15:17] GM: Let me kind of call out a few things in terms of what we've seen up to this point at the tail end of 2023, and then where things continued, or are will continue to evolve in 2024. First and foremostly, anyone who's gotten a model into production right now, it does seem like it's pretty much OpenAI, right? There's probably a few other things that are coming down the pike, including work that Anthropic and Cohere are doing, as well as number of the open-source providers, particularly Llama 2 is a pretty exciting option when you look at the compactness of the model itself, and just how it's commercially available to be licensed from an open source underpinning.

I think, for the folks who are building, and I mentioned that this need to have proprietary or private datasets, I don't know if it matters as much whether you're kind of working overtime with an open-source or closed-source model. I think it matters a lot in terms of what the model performance is, and matters a lot in terms of what the likelihood that the model hallucinates, as it is out of the box. How much you can either fine-tune it, or you can use retrieval augmented generation surrounding the model in being in production so that you can sort of coax it to deliver the results that you want it to.

But in almost all those cases, it's really going to be about model performance and feedback loops, and whether it achieves the objective that you'd want in terms of your model being in production. Right now, it does seem that the closed source and particularly, OpenAI is a closed-source model today, is the highest-performing model, and that's where most of the fine-tuning is happening as we speak.

I think over time, we're going to see some amount of diversity. It's not going to just be OpenAI close-source models only. When you think about the type of narrow AI use cases where you look at like trade settlement and clearing in a back office for financial services organization, some really massive opportunities there in terms of being able to introduce a generative model to help complement the work that humans are doing in that regard. But when you look at what kind of model would you use, do you need to have a model that understands 14th-century European history to be able to do trade settlement and clearance? Probably not, right?

So, I think this is exactly where a smaller form factor model, whether it be open source or not, it seems like some of the more capable, smaller form factor models are coming from the open source world, those models can be just as relevant for tuning and training a very specific private data set to be able to accomplish a very specific task at hand from a narrow AI standpoint. So, I think some of that's going to be coming to a theater near us in 2024. We just haven't seen it yet, mainly because it seems like most of the models in production, at least the Generative AI transformer-based models that are in production today are very much leaning towards a fine-tuned version of OpenAI.

[0:18:30] JBH: Mm-hmm. That's a really helpful perspective. Just recovering the media seemed like that was a more mature discussion. But I think what I'm hearing is we're still in the early days there in terms of which direction most enterprises are going to go for their enterprise adoption of LLMs.

[0:18:46] GM: Yes. I mean, if we kind of cast this at the end of the first inning, I would agree that there's quite a bit more of the game to be played out, and I think we're going to see more variety and diversity in models as they're in production. It just so happens, the first inning of the ballgame, at the end of it, there seems to be only one that, at least reached production level value for most enterprises. 

[0:19:10] JBH: Just from a product design perspective, if you're designing a product today and you want to incorporate an element of LLM, do you think there's going to be like impacts on the way that we actually even do product design?

[0:19:22] GM: I think so. As a former product manager, myself, and someone who's led product management teams prior to becoming an investor. I think for product leaders, you have a few things that you have to kind of think about. One is when you design software, historically, most software has had an underpinning of rules engine associated with it. These are the sequences of things that we encapsulate in our software, and as long as it does those things, it comes to a deterministic outcome. Here we are, we have a beautiful piece of potentially workflow-based software that follows those rules. 

[0:19:59] JBH: That's how it works every time.

[0:20:00] GM: Seems to have worked in software for 40-plus years. Now, I think what we are on the precipice of is systems that are less about these deterministic outcomes, and more about having a probabilistic set of reasoning associated with it, human-like smarts and reasoning associated with it, that is embedded into the software itself. So, imagine the rules engines of yesteryear are now being replaced by a generative model as the underpinning model that continues to improve upon itself. A model that learns and model that has reinforcement learning surrounding it.

So, if you're a product manager, thinking about the new products and services you're coming to the market with, you have to almost now introduce this mindset of what a probabilistic reasoning system could look like in software, either alongside a rules engine, or kind of completely reimagining your rules engine, one or the other. So, people in a funny way, sometimes ask me, “Well, what's the scale and impact of AI in the market?” Well, at the very least, it's going to be the total addressable market of all software, and they could be the total addressable market of all humanity. But at the very least, on the software end of the spectrum, you have this really interesting moment as a product leader, to be able to take everything that you have historically known in terms of building rules-based software, and replacing it with a probabilistic bottle that has human like reasoning associated with it. I think there's some powerful things that are coming about, particularly in enterprise software, if you take that first principle mindset to how you're designing the software of the future.

[0:21:48] JBH: Yes. I agree with that, right. It's a different mindset, different set of tools. I also think subject matter experts are going to be invited back into the design sessions more. We kind of over-indexed on like, let's ask the users. But kind of back to what you were saying about Mosaic coming out. When the Internet happens, apps were the pointy end of the spear. And to build those apps, you need those subject matter experts who deeply understood what are the expected outcomes? What is that workflow? Similarly, I think, for product design, there's going to be a requirement to have these Sherpas helped guide everyone through complex, very specific enterprise use cases.

Subject matter experts, process experts, folks who understand how the back-end workflows between humans and machines have historically worked. Those are all opportunities to reimagine it with Generative AI-based software as its underpinning.

[0:22:42] GM: Absolutely. So, I think that's kind of exciting in a way, because a lot of that is still broken. One thing I learned is I was looking at the MLOps space, I'm going to ask you some other questions. But in the MLOps space, one thing I thought was a funny quote is like, no model runs the same way twice, which is sort of the same thing that's happening in LLMs, right? There's a proliferation of training cycles in the ML world. In the Generative AI LLM world, there's like a proliferation of models, right? 

Let's say you've got your like controls around your data, great. Proliferation of models, you've got bundles of models running, not just one. How are organizations going to adopt that in a safe way? Is there going to be like a feature registry-type situation? Is it going to be an after-the-fact audit? What do you think is going to happen there?

[0:23:24] JBH: So, here's a few things that are happening today and there's a few things that will emerge, particularly, with some of the regulatory frameworks that are also coming about. A few things that are happening today is just as you called out multiple models running in tandem with each other. It really calls for more orchestration capabilities and I think that's a big need in the market, and you're seeing existing orchestration providers like Airflow now starting to jump in. So, we were investors in a company called Astronomer, which was one of the leading purveyors of Airflow. They're now entering into the space, as well as you have startups that are capable of delivering value from a model orchestration standpoint. There's some great examples there include both Llama Index, as well as Line Chain.

So, in all of those cases, it's the orchestration that's happening across the bottles and the data fabric that's underlying these models coming into production that's driving the need for orchestration tools. In a similar way, you're seeing the observability market continue to evolve and grow quite significantly. Again, you had Krishna on the podcast already. If you'd look at Fiddler's business, if you look at some of the other companies in that category model observability, their focus is really to be able to do the necessary model observability that lets you understand the model performance, understands model performance from an overall monitoring standpoint, and then introduces things like traceability and bias detection, and fairness into how the model monitoring is occurring.

So, that's a very, call it, internal view of how models are either observed properly or orchestrated, as two great examples. But there's also this external factor that's coming to a theater near us very quickly, which is the regulations themselves. What I'm going to really kind of see my kind of prediction going into 2024 is that there is a cast of regulations that are coming on a per-industry basis, where regulators are going to have updates in terms of what they're expecting models to do, and that's going to have to be managed from a software standpoint, from a governance risk compliance perspective, a GRC software standpoint. So, that's something that we haven't seen yet, particularly in the world of AI. But I think that's actually something to be really kind of paying attention to going into 2024 with particularly the executive order and additional other layers of regulatory frameworks coming into the fold.

[0:26:02] JBH: Yes, I work with financials who are already pretty highly regulated and this new wave of regulation is not only impressive, when you see like GDPR, EU suggesting expanding, you can elect not to be part of an AI-driven decision as your customer experience. As a citizen, I think that's great. As a technical person, it makes me like a little sick inside, because how do you do it?

[0:26:25] GM: How do you do it? It's like, okay, we know how to put humans and machines together and build systems and processes in the financial services world effectively. But now, if you're being asked after you've done that, and after you've done it well, how do you take the machine out of the loop and only have the human-based process? Tough.

[0:26:44] JBH: Right. We're kind of at the very beginning of that. We don't know who's the regulator going to be. There's competing sets of proposed regulation. One thing I do advise companies we work with is to get started with what you know. There's some no-regrets work, that should get started now, because waiting for regulators is a losing business and that can be tough.

So yes, we talked about regulatory risk. We talked a little bit about data risk and the need to orchestrate data and make sure you have a private reliable data source if you're doing an enterprise use case. What about – we'll just briefly touch on this, just because of you have an expertise in this area. But there's the other two kinds of concerns that slow down adoption and companies. Are bias, right? Just model bias. Then, some concerns around like just bad actors, prompt injection, that type of thing. What are you hearing and what are you telling people about those two areas of risk?

[0:27:25] GM: Yes, by the way, I want to hit that in a second. But I do want to just elaborate on the data question a little bit more? Then, we'll go into both those situations and detail those out. The data side of the world, and particularly privacy preservation around data, and how do you handle it, I think there's two really compelling things that I've seen so far. One is where you're using synthetic data to generate, like copies that are statistically significant for the value of a training run, for instance, or being able to just handle a privacy-preserved version of the data, particularly as you're doing testing, before by model goes into production, or an application goes in production.

So, there's companies like Tonic, that are doing a tremendous job in terms of just handling some kind of data generation. In a similar way, what we're starting to see is this notion of like, where do you keep all of your private data properly stored and manage? So, there's this idea of a privacy vault, right? And this is where Skyflow is probably one of the more unique players right now in this space, really focused on just a privacy preservation mode, or on keeping all the PII data in one place in a secure API that has one way in and out sort of door to get through, to get your information that's privacy-preserving, and everything else is kind of left in the hands of the application developer, but the privacy aspect of what you need to handle from a PII compliance standpoint, is delivered by a service like Skyflow.

So, knowing that is in place, then let's kind of talk a little bit about some of the –

[0:29:03] JBH: I want to hear what you have to say about that. But I cannot overstate what a huge sea change. It has been the embrace of synthetic data, the embrace of things like privacy vault and organizations. I mean, it is a major step forward for organizations that were pretty immature on that. So, I'm a big Tonic fan as well. We'll do another show on synthetics another day.

[0:29:22] GM: That's great. But I think it actually leads nicely to the question that you're originally asking, which is, “Okay, how do you really think about just the security concerns? And what are some of the underlying risks?” I think one of the biggest things that we continue to see in this space is that this is not only an opportunity for great things to happen, good things to happen, but also a veritable treasure chest, for nefarious actors to do unethical things. I think some of that is just in the fact that you have these models that are capable of doing many things and they can be applied in very unethical ways. Some of it's just, you're just less careful with the security of the system surrounding how you're building and putting things into production.

I think when it comes to the point of around cybersecurity, that I have been paying a lot more attention to more recently is the availability of unethical models. So, an example of this is like WormGPT. Turns out that a number of Black-Hat Hackers that had some very capable skills in terms of hacking, put that all into a generative model that they unethically trained and release that into the wild, so that any level one Black-Hat Hacker can have the skill sets that were now available to someone who is way beyond that capability and skill, all encapsulated in the copilot that does the work that a level five or six Black-Hat Hacker could do with the tools built into –

[0:30:53] JBH: Democratizing bad actors?

[0:30:54] GM: Yes. Democratize into a tool that's generally available on the dark web for anyone to take advantage of that wants to use it for nefarious purposes. So, I think that's one example of where the things that we are really excited by in terms of all this opportunity that's emerging, the scale of human productivity that's coming out of the emergence of Generative AI, there will be these dark corners, and these dark corners are going to come to a theater near us, and they're going to be pretty pronounced and there's going to be some really public incidences that'll come out in 2024 and beyond that, we just have to be much more aware of, and much more guarded about than ever before, because the use of this is now possible for not only some of the Black-Hat techniques that I've just mentioned, but also just impersonation, right?

[0:31:45] JBH: Right.

[0:31:46] GM: You can take three seconds of this conversation that we're having, and synthetically generate my voice or your voice, Jocelyn, and now, put that into an interactive voice response system and break through the typical IVRs that have been in place to secure someone's input into an online or VoiceBase account update for a bank. So, this is actually a lot trickier than all the rainbows and unicorns from all the opportunities that are coming. There's some really dark corners that we're still about to face as a society.

[0:32:22] JBH: We're in this awkward liminal space between incredible excitement about the opportunities and capabilities, but also, we don't have enough tools or thought work yet to mitigate some of these risks.

[0:32:33] GM: Yes. I mean, the scale of deep fakes right now that are coming to a theater near us is just unprecedented and that should be troubling for anyone who's thinking about this stuff today.

[0:32:43] JBH: I think I read that's one of the things that inspired Joe Biden to do this executive order, is that he saw a fake of himself. Did you see that?

[0:32:52] GM: It actually makes sense, right? Up to that point were like, “We'll get around to an executive order for AI.” And then finally, the President sees a deep fake of himself. It's like, “Oh, we should probably do something about this.” I did not know that. That's amazing.

[0:33:06] JBH: That's like such a great feeling. Because like, sometimes people are like, “Oh, you sent that email?” I'm like, “I did.” There's a million things you say and do that you don't remember. So, it can easily be fake.

Let's switch gears a little bit, because I wanted to talk a little bit about advising entrepreneurs – a lot of our audience are technical entrepreneurs, people who want to quit their day job and start a company. Maybe they're already in like a small startup. They're in that seed or A round space. People have asked me, what do you do if you're an established company? You got started as a startup, but maybe you don't have Generative AI in your pitch. Maybe you don't have it in your product? What do you do? What do you do to keep your investors engaged to keep customers engaged? Because it's almost required to talk about it now? I'm sure you've gotten a similar question, what are your thoughts?

[0:33:55] GM: We've been a little more proactive and inside about this, because we generally do believe that this is not just happy talk, and there's some genuine benefits of like reimagining your software business as a Generative AI based underpinned software business. In that regard, our Insight onsite team, which is really helping our portfolio companies continue to grow and scale in their specific journeys, whether it be in sales and marketing, or talent, or product and engineering, or finance, or business development. In all of those functions, you can kind of see two big opportunities emerge for Generative AI. One is just better products and services, and then more efficiently running your business.

In that regard, in the last six months, we've spent time with all of our portfolio companies to help them think through exactly those two levers. What are the products and services that you can be building with what you have today, that will enable you to continue to expand into the opportunities that you have in market because of Generative AI, as well as helping you run your businesses better, mostly sales marketing, it turns out. Now, it is in our view, a situation where I would be actually delighted to be a company that's already in market that doesn't necessarily have a Generative AI strategy. Why? It turns out that you have incumbency. It turns out you have private datasets. It turns out you have a market where there is distribution, because you are the incumbent in the space.

In all of those moments where incumbency, distribution, and data preexist, it's actually not that hard to build a generative bottle, right? You don’t need tremendous amounts of data science resources and machine learning resources, because you're mostly requiring data engineering, and great software developers, data engineering to get the data prepped in a way that you can tokenize and train the model appropriately, or tune the model appropriately. Then, of course, the development resources to be able to put that into an application in the right context.

So, what we're seeing is a lot of existing folks who have had very little capability and expertise, from an AI standpoint, now start to build some incredible products and services. If nothing else, look at how the incumbents in the space have been still capable of building great products. I mean, look at what Microsoft has done with GitHub Copilot as a starting point. Look at what Adobe has done with Adobe Firefly as a complement to being able to create a product that enables you to verify the sources of the creation of generative content and plug that into Photoshop. So, in all of these examples, what we're seeing is incumbents are actually not at a disadvantage. In fact, they might be at a relative advantage. Even if you are starting your AI journey very late, you can almost sort of catch up quickly by adopting some of the Generative AI capabilities.

[0:36:55] JBH: I love that. I'm an optimist at heart and you're absolutely right. You got the data, you've got the know-how, maybe of the customer relationships, those are the hardest parts. If you're already an established, you've got some of that in hand. Let's say we're – because you mentioned Microsoft, when you talk to entrepreneurs who are putting together their ideas, they're very early stage, there's a couple of items of common wisdom, which I'll share with you and you can debunk it or agree. The biggest players, Google, Microsoft, Meta are going to just sort of mop up here. They've already got so much money. They've got so much tailwind. Maybe you have to have a different idea process. What do you think about that?

[0:37:32] GM: Well, the one incumbent, which just literally emerged out of nowhere this past year that no one would have ever thought of as an incumbent. But now they are within a year, inclusive of the one that you mentioned, is also OpenAI, right? if you notice, Developer Day, not more than a week ago, we can half ago, what you saw was just a number of startups and startup ideas being completely eviscerated –

[0:38:00] JBH: Just digested.

[0:38:04] GM: – by just the capabilities that were introduced around GPTs and capabilities that were introduced around just context windows and all the additional features that OpenAI themselves announced.

[0:38:13] JBH: That’s right. For our audience, if haven't Googled, just go into YouTube. And just Google like, “Did open AI just kill all startups?” There's some really good rundowns on that. I don't believe that to be true, but it was a huge set of announcements.

[0:38:25] GM: Yes, and again, can OpenAI do all those things that kill all the startups? Probably not, because there's so much surface area to cover. It's impossible for one company to do all that, and that's why a great startup ecosystem continues to thrive, particularly in the Generative AI space. I think what you're going to have to think about as a founder, though, is you can't be working towards a feature that can be disrupted quickly. You have to be working towards, at the very least, a strong, unique product that maintains its value proposition, that a broader platform that certainly has resistance and resilience in an overall market.

I think that's harder to do. It takes longer. It takes more investment to build products and platforms over time versus features. I think the features are going to get really folded into the incumbents that are building already in the markets that they're in. But I think there's ways that you can build great products and platforms. I mean, just like what Jasper has done, they started as a prosumer offering in content creation for content marketers around Generative AI. It turns out, OpenAI came in and really sort of aided into that original core assumption in the market. But that enabled Jasper to take all everything that they learned and go into the enterprise and build great workflows and great applications and great embedded experiences for how enterprise marketers work today.

I think that's a lesson to be taken for all founders going through this journey, that even when you feel like you have called a product market fit a success, you will have to make some changes so that you can find long-term durability. My view is that as a founder right now, you can't build for features, and you can't be thinking just about the product alone. You have to be thinking both about product and go to market simultaneously. It's interesting, because if you talk to any second-time founder, that's really how their mindset is. First-time founders always have the sensibility that it's about product, and yes, it is, of course, starting with product. But second-time founders are like, yes, it's about product, but then how do I go bring it to market?

[0:40:41] JBH: So, George, what does that mean? I hear people talking about GTM all the time. But I would assume what you're saying is like, “Hey, I've got the next great widget. This is a cool product.” Is going-to-market like, “Hey, I have a letter of intent from a big customer and we're going to co-design this.” What are those? Practically speaking, what would those be –

[0:40:56] GM: In the early days, it's good design partners, it's good focus for folks who are going to co-collaborate in building what you're bringing to market. But I think soon after that, like when you're closer to the first iteration, or your MVP, call it your first iteration of the product, then you got to quickly move your mind towards commercialization and how you get paid for what you're building. I think in previous incarnations, we've had more time and we've had more ability to keep building and not have to worry as much about commercial value and selling, being good at finding the things that you're building have some ability to –

[0:41:40] JBH: Yes. It’s just being ruthlessly focused on what's going to sell not just what's going to sell, not just what’s like close to your heart.

[0:41:45] GM: Yes. Focus earlier on that you’d want.

[0:41:47] JBH: Interesting.

[0:41:48] GM: I think that's something that in this cycle of iteration and at the clock speeds that we're moving at right now, you're going to have to really find your commercial value faster, and even after you find your commercial value, you might have moments where you still have to pivot.

[0:42:07] JBH: Yes. The penalty is high. It's so easy to say and hard to do. When you're in the situation, until you fall in love with your product. You've got certain things and even though it's a simple statement, I actually can't underline this enough for our listeners, how important that is, that kind of ruthless focus on what people are going to buy.

[0:42:25] GM: I mean, for great founders as product people are tortured artists, I get it. I was certainly, one of those folks myself. But I think the best founders in time, not only are the tortured artist, but are the world-class salespeople. So, if you could take the combination of tortured artist and world-class salesperson and bring that together, those are the founders that will go the distance over time.

[0:42:47] JBH: Well, speaking of founders that have go the distance, we talked a little bit about the overall markets and things you're looking at in 2024, some like signal. You've got your own point of view, as an investor. You've invested in some, you've got your current portfolio companies, do you want to share some stories of how some of your current portfolio companies have checked off the boxes you think are important in developing in this area?

[0:43:10] GM: Yes. I mean, I'll give two examples of companies that have some tremendous scale. I mentioned Weights & Biases. But I think it's important to understand and we had a great blog post about Lucas and his journey around Weights & Biases more recently. The reason why Weights & Biases has worked so well in the market, is that Lucas had his first startup, which is CrowdFlower, and he had all of the challenges of building CrowdFlower in his mind, as a second-time founder, that went to market and not only built a phenomenal product, but figure out a way that they commercialize this, and scales. Insight was lucky enough to catch Weights & Biases, even before the commercialization really started. Now, we're starting, two to three years later, we're starting to see the tremendous scale of how big a business like Weights & Biases particularly as a leading tool in the MLOps, LLMOps world can get.

I think outside of even just the AI space where software businesses are becoming AI businesses with Christine Yen is doing with Honeycomb is fascinating, right? Because it's not just the fact that you're building a next-generation observability product that is fully differentiated from all the existing incumbents in the space, but they're doing innovative things like introducing generative capabilities into a traditional system observability market. It turns out that in the case of Honeycomb, what the unique value proposition of introducing Generative AI was that every observability product has somewhat of an esoteric language to be able to work in the context of finding that observable event.

If you introduce natural language as the front end, how much easier and how much more democratized could an observability product be? It's exactly the thesis that the team went in at Honeycomb, to go build a natural language interface using a fine-tuned open AI experience into a query builder for Honeycomb. It was the fastest adopted feature in Honeycomb’s history where 40% of all of Honeycomb users were on that product within the first 24 hours of it launching, to give you an idea of how fast adoption curve occurs.

So, what I would generally give advice to founders as they're going through this, is really continue to find those great moments of product differentiation early, as well as later in your product development cycle. And then be willing to continue to experiment and iterate in the first commercialization successes that you have, don't rest on those laurels, like find different ways that you can continue to commercialize and scale your business, because the market continues to evolve very quickly, and you want to be resilient for all those changes that are occurring.

[0:45:52] JBH: Very interesting. We’ll kind of wrap up here at the end of our discussion, and I really appreciate you taking the time. Just one kind of last idea here, you have a point of view as an investor. In the last few years, a lot of investors have different ways of approaching the market. Some of them don't have a point of view, some of them kind of like, I'll take your point of view, and I'll use that too. What do you think they're doing well in this space? And what can they be doing better?

[0:46:18] GM: So, I think most VCs are just very good at sensing when there's a shift in the market and really embracing that shift. If you think about where momentum is occurring, VCs have some of the best sense of momentum that could be, and that now, no surprise, has played out in how the Generative AI market has really evolved. Because VCs have put a lot of energy, a lot of investment directly into that market.

Now, I think where we as VCs can do a lot better over time, is just really, truly understand how hard the founder journeys are, right? And be more empathetic to how difficult these journeys are. Because as much as, you're like, “Hey, you should move into this portion of the market”, that takes work. That takes a lot of calories. And I think the best VCs, at some point, have figured out a way to have that empathy, in terms of really understanding what the founder’s journey is, and being able to be a copilot to their journeys.

I think, for me personally, that empathy came from just having to have been one, as in a founder and a builder in the past, and that kind of gave me a little bit of that empathy as I went into venture capital. For the venture capitalists that haven't necessarily done that, that's okay. Because I think you have enough experiences just being on board, being with founders, that you should be able to translate that to providing proper empathy for them. I think in most cases, the more that we are empathetic as VCs, and the more that we understand the momentum that's occurring in the market, the two of those things combined together, generally, you're going to end up with a reasonable set of investments made, as you’re kind of progressing.

[0:48:07] JBH: I love that. I love the idea too, that you don't have to have been a founder, but they just be with them. I like that phrase, that simple phrase is great. Just be with them for a while, sit with them, and experience their point of view. I love that.

All right, last question. My inbox and every bookshelf is just topped up with information in this space and reading material that I often don't get through. For our listeners, what are one or two things that you've read, or newsletters, or articles that – obviously, Insight has some great ones. But if there were one or two things that somebody really wanted to read, what would you recommend right now?

[0:48:42] GM: I mean, as far as like a lot of the thinking on responsible AI, certainly, the work that we've seen in market by observability leaders like Fiddler has been great. I would also add the constitutional AI frameworks that are emerging. Anthropic is probably one of the folks that have been leading that work and some of the work that's surrounding constitutional AI and RLAIF, the use of reinforcement learning and AI systems to almost guardrail how AI is in the wild, is a very emerging and topical field. I would sort of dig into that.

I think kind of more broadly, if you think about responsible AI, and where it's going to play out in the next few years, like we just have to build better-aligned systems to human values. And I think one of the original thoughts on that was just what Brian Christiansen had wrote around, a book called The Alignment Problem. So, I would highly recommend taking a look at that book, as a way of really thinking about how we can build better aligned systems. If you want to just have a discourse on just how we think about AI systems and the future of AI systems, sitting alongside of humanity, one of my favorite recent works on this topic, was a series of essays that were compiled into a book written by Meghan O'Gieblyn called God, Human, Animal, Machine. It was a wonderful amalgamation of her thoughts on what it took to build this current generation of AI systems, and what are the types of people that really are finding meaning around the AI systems that could be part of what I almost think of as copilots to our lives. So, those are probably the three books and three sort of sources that I've been inspired by.

[0:50:29] JBH: I almost didn't ask that question. Now, I learned at least two new things from you. So, thank you. That's great. Well, thank you so much for taking the time. Software Engineering Daily thanks you. I know our audience is super excited to hear from you. Hope you have a great rest of your day. Thanks a lot.

[0:50:44] GM: Great. Thank you, again. Appreciate it.

[END]