EPISODE 1949

[INTRODUCTION]

[0:00:00] ANNOUNCER: News feeds and recommendation systems have long relied on deep learning architectures that score each candidate item independently. As LLMs have matured, they have opened up a fundamentally different approach where a system can reason about content the way it reasons about language. However, that power comes with a fresh set of engineering challenges around cost, scale, and evaluation. 

LinkedIn recently rebuilt its newsfeed to treat content recommendation as a sequence modeling problem. The general approach is to predict what a user will want next, much like an LLM predicts the next token in a sentence. Tim Jurka has worked at LinkedIn for 13 years and is currently a VP of engineering. 

In this episode, Tim joins Matt Merrill to discuss how LinkedIn re-engineered its feed, how the team combines LLMs with traditional signals, managing inference costs at massive scale, steering content quality using natural language policies, and more. 

Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in back-end development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at DEPTŪ Agency. You can learn more about his work at code.theothermattm.com.

[INTERVIEW]

[0:01:40] MM: Hey, everybody. I'm Matt Merrill, and I am here with Tim Jurka, a VP of Engineering from LinkedIn. Today we're going to talk a little bit about what LinkedIn's doing with revamping their social feed with AI. But before we start, Tim, can you tell us a little bit about yourself and your current role at LinkedIn? 

[0:01:55] TJ: Thanks for having me, Matt. As you mentioned, my name is Tim Jurka. I am VP of Engineering at LinkedIn and I work on our consumer products. What that would be is things like the LinkedIn feed, search experience, profile, pretty much anything you will engage with when you open up the LinkedIn app. 

And I actually started here 13 years ago as an engineer on the feed. I was an AI engineer, one of the first AI engineers building the LinkedIn feed. It's been pretty amazing to go from actually building the thing to now see the thing operating at just massive scale with 1.3 billion members. It's been quite a journey. 

[0:02:27] MM: Recently you guys have rolled out some pretty major updates to how you're making the LinkedIn feed relevant to users. And I know that's welcome news to me, and it's probably welcome news to a lot of people who may not have otherwise engaged with LinkedIn because of content that was irrelevant or they thought was annoying. I'm really excited to hear about what you did, and also how it intersects with AI tooling which I think this audience will really like. Let's start with just what was the problem, right? What was the overall problem, and why was it challenging? 

[0:02:59] TJ: I mean, I can maybe start by rooting it in a problem of scale. Since just 2020, we've gone from about like 690 million members to 1.3 billion members. We've scaled considerably. And we have about seven new members joining LinkedIn every single second. And so with that scale comes a lot of challenges. Number one, you have tons of new industries coming on the platform, people in new roles, new opportunities. They're looking for something different from our platform. 

And there's a lot of people coming with evolving expectations of what they need from their professional social network. Starting from 2020, I mean, the world of work has changed a lot. We went through a global pandemic. We then went through the entire AI LLM revolution, which we're probably ostensibly still in the middle of. And people are just turning to LinkedIn more than ever to get this information about like, "Yeah, how do I navigate this? How do I reskill? How do I learn about what's happening? Keeping pace with things that are happening in my industry?" 

And so the problems we had to solve were twofold. One is make sure that we could kind of scale to this very kind of granular topical matching and representation of the entire LinkedIn feed ecosystem and make sure we're giving people the right content at the right time with timely insights. The other is with seven new members joining every second, you have to do a pretty good job in the first few seconds of a person's session narrowing down what they actually care about. And so we almost call this like the zero-shot experience. And both of these things together are kind of the challenges we were solving. 

Number one is like you need to get that signal very quickly to help the people get the insights that they need within a few seconds of onboarding. And they're not necessarily endlessly scrolling on LinkedIn. They're trying to get a job done. They're joining to find a job to learn something, and you have to really quickly get them that information to make sure you're maximizing the utility of LinkedIn. 

For people who are diehard users, I mean, there's a lot of things that even you alluded to. We started seeing the emergent of AI slop engagement bait. We needed fine-grained control to make sure that we are focusing on the highest value conversations, people who are authentic verified, and making sure that we have the capabilities to keep the ecosystem steered in that direction. 

And so this new foundation not only kind of up-leveled the fundamental infrastructure modeling capabilities on retrieval ranking, it also gave us these fine-grained controls to solve a lot of these problems from like faster time to signal world awareness, being able to do the zero-shot session kind of stuff, but also being able to steer the system to make sure that we're giving the highest value content to our members. 

[0:05:28] MM: Was the primary driver new user stickiness and a nice side benefit was existing user engagement or was it all of it? 

[0:05:37] TJ: It was across the board. Because one thing that you'll see is there's kind of two classes of content that we surface. LinkedIn has historically been this very network-oriented product where I connected with a bunch of people. But the people you connected with aren't always going to be the people that are giving you the cutting-edge insights for where you're trying to learn. And so you'll actually see in your feed a lot more what we call unconnected content, suggested content that is supposed to be really topically oriented to like, "Hey, I'm trying to learn about what supply chain risk means in the context of Anthropic and Claude models. How do I go deeper on that? What does it mean for my company? Or an electrical engineer trying to learn about modular reactors, try and go deeper there. You need to get deeper into where the subject matter experts are, even if they are beyond your network. And so it's not just solving the short-term zero-shot signal. It's also be able to map the interesting content based on where you're trying to learn much more quickly. 

[0:06:29] MM: Yeah. And does that also extend to your outreach? I know I get emails with content that I might be interested in from LinkedIn. Does it also include that or is that outside the scope of what you're doing? 

[0:06:40] TJ: It's in scope, but right now we're starting with rolling this out in the feed and we're iterating a lot there. This stuff kind of - I mean, it cascades across our products. This is stuff that we build out and make sure that we have this feature-rich capability everywhere you might encounter content on LinkedIn. 

[0:06:54] MM: Cool. Let's dive right into the interesting stuff. One of the things I was reading about, there's a really interesting blog article that you guys posted about this. Goes into a lot of detail, which is awesome. You're using sequential time-based interactions to drive relevancy, and this is not something I personally have seen much about in the AI space. Can you talk a little bit more about that? 

[0:07:18] TJ: Yeah. I mean, the underlying premise is actually quite similar to how you think about LLMs, which is LLMs are really just sequence models for language. They're trying to predict, given a token, what are the next tokens that come after? And then stringing together a comprehensive sentence, paragraph, whatever the context window is. You can do the same thing with content in the feed or any kind of item in a recommender system, where you're trying to predict sequences of items versus sequences of words. 

And so there's a lot of benefit of this compared to the old style MLP architectures where you might have heard of these feature crossing, DCN V2 deep and wide feature crossing, where you dump tons of features into the model. You try to find all these feature interactions. That's then your deep learning model. And every time you give it a new item, it weights all the features in that model and it scores the item, and it's like, "Here's the score of this one and this one and this one," and then you sort. 

Sequence-based models are actually saying, "Contextually, we're going to learn that once somebody has viewed a piece of content about modular reactors, then they're probably going to go deeper in direction X or Y." It starts modeling out a sequence of where you would go next based on all the data points that we have of every single expert on LinkedIn reading about any number of topics. 

And so it has a lot of benefits then of starting to go extrapolate. You might be starting at like an introduction to PyTorch, but you go 15 levels down the sequence, you might actually be trying to build a reimplementation of an LLM locally on your machine. And so can you reconstruct that sequence of how people would go down a content path and help kind of understand the engagement patterns from that vantage point versus de novo, like rescoring every single item just using all the features in the model?

[0:09:02] MM: That is super interesting and also just - we'll get into this a little bit. Just mind-blowing at that scale. If you can't answer this, I totally understand, but did you build this on any sort of open-source or widely available model, or is this something you guys have completely come up with R&D from your own perspective? 

[0:09:19] TJ: As with anything, we pull from the research that we see around us. There's a lot that has to be uniquely built for LinkedIn. And so, it's not anything off-the-shelf works for us. It requires us to piece together something unique. To my knowledge, this is fairly unique at this scale across this kind of the system of these sequence-based models and ranking and then the LLM-based retrieval. It pulls from concepts from a lot larger, including some of stuff that we've built internally. 

In terms of open source, we do leverage open source components in what we build. And it's honestly through experimentation that we figure out where we have to build things in-house versus open source. You try lots of different frontier models. You try lots of different approaches, and you kind of see what converges to solve the specific problems that we have in our scale, our kind of professional context, the kind of fine-grained controls that we need. 

[0:10:06] MM: It's tough for me to visualize what this architecture might look like, but I'm assuming you're using multiple models in different places. 

[0:10:14] TJ: I mean, we're continuously - if you think about the kind of life cycle of we have offline environments where we will train models using some interesting - an engineer has an idea of adding a certain feature to the model or trying a different model architecture. Offline, we can run thousands of these tests offline and replay them on historical data, and say, "Did we do a better job ranking items in terms of engagement?" Then we have to actually bring these online to see do the members actually - not in simulation, but in the real world. And so there we will split up our traffic and we'll run hundreds of experiments while there. 

At any given time, we're running tons of tests to try all sorts of stuff. And you may have seen some of this already publicly, making sure that we're trying to focus on authentic voice conversations, making sure that we're not rewarding AI swap. Making sure that our suggested content is really well-targeted to interests and being able to better model member affinity to different interests. All these things are variations of tests that we're running at any given point in time. 

[0:11:14] MM: And what's the engagement signal? Is it clicks? Is it time? Is it all of it? 

[0:11:20] TJ: Yeah. As a standard in industry, it's what we call a multi-head model or multiple objective optimization. And so, it's kind of a little bit of everything. You want to make sure that you have predictors for different ways that people will engage with things. That means positive predictors of like are you going to be commenting on this? Are you going to be re-sharing it? Are you going to be spending time on it? 

Also, negative predictors. Are you going to skip it? Is it going to be in the viewport for a second? Just pass over it. That's a negative prediction that we try to do as well. And you combine all these together into one multi-objective optimization to actually then ultimately optimize for - I mean, for us, it's really legitimately trying to get the most interesting conversations in front of people that we feel it's going to be time well spent. But it's a combination of all these factors that then ladders up to that. 

[0:12:08] MM: That makes a lot of sense that what defines engagement is a very complicated thing in and of itself. Yeah, using these models to help track that makes sense. 

[0:12:18] TJ: And maybe to illustrate it even more, the weights of all these different actions are personalized on a per user basis. If you're somebody on LinkedIn who's in marketing, you're trying to get follower reach and comments on - you're trying to achieve something different than somebody who is prepping for their job interview, "I just want the content that's going to show me how to nail the job interview at LinkedIn." Right? You're not going to be looking to post and to get comments and reach. And so you have to adjust the model and how it works depending on the actual intent of the person using the model. 

[0:12:50] MM: And it's literally by user. We're not talking about roles, or personas, or anything like that. It's very unique to each user. 

[0:12:56] TJ: It's fully personalized. 

[0:12:58] MM: Yeah. Wow. That is awesome. Let's double-click a little bit into that evaluation. That's something that I have personally been - I am not working at nearly the scale you are, but I have been doing a lot in my job is just how you define what is good in this space, right? And we just touched on that a little bit. I think the thing is, over time, how does this happen? And who is doing this? 

I can't imagine - well, I don't know. Maybe I'm wrong. But I can't imagine they've got armies of you, PhD trained in AI doing this. Are these engineers? Are they product people? And how often are you retraining at full scale versus just incremental improvement over time? I'd love to hear more about that. 

[0:13:44] TJ: In terms of training, we incrementally retrain constantly. We're pulling in the latest data as soon as we get it, retraining, updating the model. The more interesting part is the evaluation of how you are actually sure that this stuff is good. I mean, there's a number of ways you do it. There's the more traditional ways, cb(eval), which is like AB tests. And you look at your kind of metrics, and you're like, "Okay, what happened to A versus B?" 

But for something retrieval where you're using LLM to actually retrieve content that is relevant to a person, you actually want to be looking at some sort of - we call them policies, right? You define a policy for what consists of high quality content. This is something that's defined by our product managers. And these can be evaluated using people. We call them taste makers, people who are really well-attuned to the product strategy and where we want to take things and make sure that content is timely and insightful, etc. You'll have a timeliness policy and type policy. 

But also, you can start encoding this into like auto eval. You can have GitHub Copilot, CLI essentially doing revs of like, "Hey, I looked at a bunch of content that was surface and retrieval. It missed the policy in these places. Do you agree with me? Or do I need some feedback to iterate, make sure the policy is reflecting what you intend?" And so you get into that kind of like active learning loop. And so we do both. 

And it's been pretty incredible to just be able to have that capability. It gives you just much more sense and fine-grained control in terms of are we actually doing what we aspire to do. Is this actually timely content is actually interesting for this particular audience. In the old days of ML, you kind of hoped that that was the emergent property of your system. You put out a model and it's like we optimize for this thing, and we hope that it surfaces interesting stuff. Now you can actually kind of validate it. You can say for people in nuclear engineering, are you actually surfacing stuff that is relevant to their industry, or is this something that is either general-purpose humor or something that's not as fine-tuned? And so you can get into that level of granularity. 

[0:15:44] MM: And it's sounding like it's kind of happening in parallel almost. 

[0:15:48] TJ: Yeah. 

[0:15:49] MM: Which is wild. It's almost like an organism. It's very interesting. And about the who, you said it's product managers, taste makers. How are they inputting that signal in to correct that? What does that look like from a user experience? 

[0:16:02] TJ: I mean, there's a number of ways. They will directly - I mean, at the end of the day, it's a text file that's checked into GitHub. And they've written a policy. Almost you can think of as a prompt. We've built out our agentic tooling around that. So they can iterate on that prompt by getting some examples from GitHub Copilot CLI. And it's like, "Hey, here's a few examples. Here's what I think I would score them based on what you told me." And they can be like, "No, you missed the point on this one." And then the Copilot might then update some of the words of the policy to make sure that then it's reflecting kind of the intent of what we're trying to do. 

[0:16:40] MM: This is a very engineering-centric CLI-based process. 

[0:16:45] TJ: 100%. Yeah. 

[0:16:45] MM: Your product owners are very technical. 

[0:16:49] TJ: Everything is version checked. Yeah. 

[0:16:51] MM: Wow. Oh man, that sounds great. 

[0:16:53] TJ: We can see live over the course of a week, places where we've evolved our opinion on what constitutes inauthentic content. People giving kind of AI slop style replies. And we can fine-tune to make sure that we address that. 

[0:17:07] MM: And I'm very curious about the day-to-day for an engineer in here, right? I mean, even your product people are working in a code-based workflow. What is an average engineer doing in this environment? Because I think that's where a lot of people are wondering of like, "Okay, how do I fit in with my existing knowledge? I might not be a PhD in AI. How do I fit in and start to prepare myself for what this world might look like?" 

[0:17:35] TJ: There's no average engineer. 

[0:17:37] MM: That's fair. 

[0:17:38] TJ: Let me start with the people who really are moving a thousand miles a minute on this stuff are people who are generalists who can traverse more parts of the stack. They not only understand how the AI models work and how to fine-tune an LLM, do distillation, they also understand how to integrate that into the backends and the infrastructure. Maybe they even know how prompt engineering works. And all that kind of is a super powerful skill set for people to kind of cross over the boundary of decision-making into what does this product actually look like? 

However, that hasn't negated the role of having some specialized skills. To get an ML model to production in front of this many people, you need to make sure that you have nearline pipelines that are taking all the tracking events from the client and propagating them in nearline to all these feature stores to make sure I know this particular post by Jalen Brunson was the 25th thousand time in the New York metro area and these metro areas, which is all in real time being updated to then know where we target this stuff. All these nearline signals need to get ingested into the system. 

You then need to run this massive model in inference on a GPU fleet. And so you need inference engineers that are optimizing like the QPS throughput of the stuff because we don't want to overspend and buy every NVIDIA GPU out there. You can do a lot of things to increase the throughput of these systems. Sometimes you can offload some of the stuff to CPU workloads, keep it in the GPU workload. You have infrastructure engineers looking at that. You have AI engineers that are training and deploying. And a lot of engineers that like I said are generalists that are then making sure that these things all kind of work seamlessly together as we ramp to our members. 

And so the average engineer is somewhat of a misnomer because you still need a multifaceted team. And we almost think of it as kind of these pods of teams where you might have like a product manager that's helping shape some of the taste, and you have a designer that's designed the feed card, an infrastructure engineer and a backend engineer. And so one of these pods might be like working on a particular model end-to-end and bringing it to like the members. 

[0:19:39] MM: That's fascinating. At a system of this scale, how many pods and how many people roughly on a pod? 

[0:19:44] TJ: I mean, it was a couple hundred engineers to bring this to life. But not everything's conducive to these pod style. You have some deep infrastructure optimization work. But where you're moving fast on a new product idea and like some way to like improve the feed. That's where these pods can move really fast between like an idea, iterating on it, deploying it and having that closed kind of loop ecosystem where the team is making their decisions and running with it and using agentic tools to develop. 

[0:20:09] MM: That makes sense to me. You need the generalists to be able to react quickly, put those puzzle pieces together fast and get it presentable to the user, so to speak, or even out to the user. And then you need those really deep specialties to be able to tweak the fine details to make sure it's not too slow, that it's not completely irrelevant, that it's not costing too much. Yeah, that's really interesting. 

It sounds like that's a lot of people to be able to get this done as opposed to, "No, we actually did it with less people," which maybe it's just me wanting to see this through rose-colored glasses and save our industry. But I'm curious what your take is there. 

[0:20:46] TJ: I mean, these are systemic shifts that we've been working on for 12 to 18 months and like iterating our way into this. And you're shifting from one model architecture and infrastructure to this GPU fleet. That does take build out. There is upfront buildout. Our infrastructure team is phenomenal. They work on lots of things beyond this. They've been working on feed in like heads down mode for the last couple of years. 

But that's not the only thing they do. Once we got the buildout going, we are running and operating this with a much smaller team and focused on using the platform to build interesting new features and getting interesting content in front of members. Our infrastructure teams are then moving on to other interesting building out of agentic platforms, and they're kind of reading the next wave. And so it's not like each of these is a permanent large investment. You got to do the buildout to then leverage this platform to do interesting things and build compelling experiences. 

[0:21:36] MM: Yeah, the investment's got to come in there. 

[0:21:38] TJ: Yeah. 

[0:21:39] MM: Let's go back to scale, because the scale is just absolutely incredible. And I think that there's kind of two things that I'm thinking about is one is just how is this architecture inherently set up to scale just from a conceptual perspective. And then there's also the hardware or infrastructure part of it, too, which you've kind of hinted at. And I'm curious if you can describe a little bit of like how the interplay and how you thought about that to be able to create something of this scale with this cutting-edge technology. 

[0:22:09] TJ: When we started this, it's kind of like try to do everything on GPUs and see the best experience we can create for our members at the end of the day. You start there, and you quickly realize, "Well, this isn't going to scale." And so that's where you find something that's compelling at small scale test to 1%, 2%, 5% of members and say, "Okay, there's something here." 

And then you do a lot of iteration. You start with a GPU native architecture, but then you're like, "Hey, there's lots of these things that we can probably offload to CPU." Not everything needs to be like a GPU workload. You figure out how to make your embedding generation pipelines GPU accelerated, but then making sure that you maybe do some sort of batching so like you're not like just streaming everything directly into the system, but maybe you're aggregating on like 20-minute windows to try to batch things together and reduce the kind of continuous compute and updating the model. 

There's tons of things that we've done. The easiest thing to do is change your model size. I think everybody talks, like use a smaller model. But I think we focused a lot more on the system design. So you can do things like - you can build shared context batching. There's techniques like custom attention kernels that we were using to reduce the per request compute costs. 

[0:23:20] MM: That's not a concept I'm that familiar with. You said it's a custom attention kernel?

[0:23:25] TJ: Yeah. So you are now probably pushing me to start tapping in some of my infrastructure partners that honestly work more directly at the GPU level. Literally going down to the CUDA kernel level and optimizing the CUDA kernel for the specific kind of computations we're doing at inference time. They speed up to that level to make sure that we get the throughput that we expect. 

And so you're going really deep into the stack to make this happen. And I'm humble enough to say I don't know what I don't know. You should talk to some of our amazing infrastructure engineers that can probably go give you a whole podcast just on the GPU acceleration that they did, because folks like Ali Naqvi on the team have just done a phenomenal job there. 

[0:24:06] MM: I would love to. Let's get them. No. I didn't even know what I was scratching the surface of there, but that is fascinating. And just to go back real quick too, you said that there were certain things that you offloaded to the CPU. What are some examples of that? Because that's something that makes sense to me. But having something to grab onto, I think, might be useful. 

[0:24:26] TJ: I mean, there's a lot of places where you can do pre-processing before you load everything into GPU memory. Once things are in GPU memory, you want that to be the thing that you're using to then score like a massive amount of items. But there are things, like you can filter on different facets. You can say like, "Hey, this member is coming from California. We know the vast majority of content is going to be in English for this member." You can do like language filtering. You don't need a GPU to do language filtering. You can do this in a simple CPU index and just do a faceted filter on that. And so that's where you're starting to cut down on some of the unnecessary compute by using cheaper hardware and CPUs before you load everything else into GPU memory and then do your large-scale inference. 

[0:25:09] MM: Am I oversimplifying by saying like that type of stuff can be broken down into more procedural code that's run on CPUs, or is it still somehow using models and embeddings and things like that to be able to do that? 

[0:25:22] TJ: I think a lot of the stuff that's easily handed off to CPU is filtering and that kind of like non-model based stuff. There are also models that just don't require GPU. You can have simpler modeling techniques that can give you like an approximation of like this item is going to be completely out of scope for us to score for this particular member. And you don't need to do the full 100-billion parameter model for every single item. And so there are ways to break up even like the modeling capability of it. 

[0:25:50] MM: Yeah. So it's kind of like simple logical passes to get to the things that need 

[0:25:56] TJ: Yeah, exactly. Yeah. 

[0:25:57] MM: Yeah. Okay, cool. And the batching piece is interesting, too. I'm having trouble wrapping my mind around how you would batch some of this stuff. How does that look in like a data flow? You said something about batching embeddings. 

[0:26:11] TJ: Yeah. I mean, well, there's lots of places we use batching. I mean, you can like do batch scoring of items. You can also do just how you update the features. You can do that in batch as well. So, you don't do as many writes. If you think of the simplest thing in our nearline pipelines, you have how many people liking on LinkedIn every second. I actually don't know off the top of my head, but it's going to be some exorbitant number in terms of likes per second. 

When you're updating the feature counter, you don't really care if it's updated from 100 to 105 in the next millisecond. You can stream in the next 10 minutes of data just so you know the trajectory. And that way you're doing a write to database only once versus doing it with every - this is like more traditional software engineering and pipeline engineering. 

[0:26:53] MM: Yeah. Okay. But that makes a lot more sense. It's basically just using those models and those GPUs very intelligently because they're costly. 

[0:27:01] TJ: Exactly. 

[0:27:02] MM: How about infrastructure? Right. I can't even fathom. You're owned by Microsoft, so you've got hopefully that at your disposal. You got your own GPU farms. What does this look like in terms of hardware and infrastructure and cloud? 

[0:27:18] TJ: I mean, we really have anything at our disposal. But using frontier models without fine-tuning them at all is pretty inefficient because they're not task-aware of for what you're trying to do. And so we run this in our data centers, right? We have our GPU fleets in our data centers. We run models that we have fine-tuned on LinkedIn data to make sure that they're more context-aware. And so they can be more performant on the tasks that we care about. They don't have to be fully general-purpose. That would be wasteful for ranking a feed. And so that makes them more suitable to then run at the scale that we need them to and on our own GPU fleet. 

The reason I'm kind of pausing here a little bit is it's super context dependent in terms of what part of the life cycle we're in for this kind of stuff. Sometimes you might use a frontier model to do the kind of stuff we were talking about in terms of evaluation. You want to eval quality of - you actually probably want the best model that's as close to human intelligence as possible to be doing an eval against a policy. Because that's like a super nuanced thing of like I'm telling you these things about what makes it interesting to a member and what makes it timely and relevant. What isn't and isn't AI slop, right? 

And so that kind of eval, you're serving that to like a billion people. You're using it to score kind of a sample of your data and say like how are we doing here. Whereas for actually like running the inference, you want to fine-tune the model and tweak it such that it can run on the GPU fleet in a more effective way. 

[0:28:45] MM: It's very elastic is what I'm hearing. Yeah. 

[0:28:47] TJ: Yeah, exactly. Elastic in terms of both the techniques that we use depending on the life cycle of where we are iterating, eval versus inference, but also in terms of the kind of compute footprint that you have to employ. Whether it be kind of running frontier models in the cloud or you're running the actual feed ranking model on our GPUs in the data center. 

[0:29:05] MM: And in terms of just like I'm trying to imagine a scenario where LinkedIn gets hit particularly hard. I don't know. There's like a zero day or something, and everybody goes on and starts looking for LinkedIn articles. Is it just any other infrastructure scale problem you just need more when that happens? Is there something else that might be more nuanced? 

[0:29:24] TJ: I mean, what you're describing is like it goes into all your end-to-end software engineering and data center engineering, which you have edge, you have some systems at edge. If it truly is what you described an attack, we're going to be blocking it at the edge not by the time it gets to our inference containers. If we have a surge in traffic, yeah, we just provision more capacity elastically. We plan for that kind of stuff. We regularly run load tests to make sure we understand what kind of like our capacity is and what even unexpected peak capacity would be to make sure that we can serve it. 

[0:29:52] MM: That gets back into just good systems and software engineering.

[0:29:56] TJ: Good procurement, good redlinining of your systems. 

[0:29:59] MM: Caching. Yeah, prediction. Okay, cool. It's good. Some of us mere mortals can do that stuff. Just kidding. 

[0:30:06] TJ: It's almost like some of software engineering skills still apply in this new agentic era. 

[0:30:11] MM: Unbelievable. Unbelievable. Yeah, I thought Claude could do it all, but maybe I'm wrong. So, this is really kind of getting into the details, but I'm going to use this as kind of a way to kind of dive into some of these gotchas. So, one of the things that I read about in this article was that you were finding that numeric data like the count of an article would act like plain text and completely miss that it was a popularity signal. 

And I think a lot of engineers, and you mentioned this in the article, are just dumping structured data into prompts and assuming the model just gets it. And I think many times what I have seen is that it kind of fakes it pretty well. But when you start scratching beneath the surface, it's like, "What's going on?" I don't think it really gets that. Do you have a good story or a good example like that and what people can learn from that, and the mistakes that you made so you can help some other folks about those type of patterns? 

[0:31:06] TJ: I mean even that is like a pretty great example of - maybe two things jump to mind. One is even, yeah, the item popularity features. Even the early days of some of these frontier models, they were not doing a great job with math. You started with kind of RAG to like form out the task to a calculator. You would literally have Python eval, the value and get it back to you. With time, they fine-tuned the models to be pretty good at math. 

But when we say LLM, you can't run RAG at serve time for feed, right? You can't like call out to budget tools. That would be like way too slow. You're scoring a feed. And so this is where you kind of build out parallel capabilities. You have your LLM-based capabilities. And you can still use traditional kind of like counting systems that you would use in I'll say the olden days to track popularity and kind of like project out is this on the upswing in terms of virality. What is the pace in which people are engaging with it? 

And so we had to combine these systems together. When we say LLM, it's not like magical LLM solved everything. It's like LLM in combination with a lot of other features that are more quantitative and maybe aren't easily understandable by the LLM. So we use these as feature interactions with sequence-based model, etc. All this together forms kind of the model that we then end up using. 

The thing that we struggled with the most when it came with these LLM systems, like the new problems that we created for ourselves, is the scale of the system is really dependent on what you put in the context for the LLM. If you think about it, the LLM is looking at like the member context. Who is this person? What does their profile look like? Where do they work? What have they stated as their interest on their profile? What have they been reading in the feed in the last few sessions? And then the context of the item. Who wrote that particular post? What is the post about? If there's an image in it, describe the image, caption it. 

And if you just represent everything as just pure tokens and dump it in, you're just going to blow up the system. And so you have to be very judicious of like almost cutting down and summarizing and figuring out like what are the most important tokens that we factor in to like make the context as compact as possible but not too compact that it then starts losing precision and the whole purpose that we would be using the LLM. 

And that's actually turned into like an entire both for LLMs themselves. But also, we talked about the sequence-based models. How do you actually train the model against sequences? What sequences do you look at? Do you sample certain sequences more than others? Not everything in the sequence is equally important. And so really careful construction of the context window, the tokens you put into it, items that you put in the sequence, these are all things that are an entire pod is working on this to figure out how to maximize kind of the information gain here. 

[0:33:48] MM: What I'm hearing, and this lines up with the gut that I've been feeling, is that it's a big art of knowing when to employ LLM-based technologies and when to employ old-fashioned technologies, and how to tie them all together. And I think one of the big misconceptions outside of the engineering community, I hope most engineers understand this now, is that you can't just dump everything into an LLM and just give it a bunch of markdowns and expect it to do what it needs to do at scale. It might work for you at your desk using Claude. You can probably hack it. But when you want to actually make a system, no way. 

[0:34:26] TJ: Another way to put this is it takes a ton of refinement. The amount of iterations I mentioned on the policy that we have, etc., it requires a lot of guiding the system to get to the outcome that you want, especially at scale. You can make it work for a specific instance and probably get something pretty useful. My favorite quote is if you have an agentic system where each step of the agentic system works 99.5% of the time and it's like a 20-stage agentic workflow, then the end time workflow is useless. Because across 99.5% success across 20 steps, you're basically - the system is completely useless. 

And so we try to augment the system with ways to ground it in being a bit more performant by giving it popularity signals or making sure we have the right context in it. Making sure we're refining the policy such that it was representative of the types of content that we want to select. 

[0:35:13] MM: All right. So moving on to something a little bit more open-ended. I saw in one of your writeups about this project, you said that moving to LLM didn't really solve your old problems so much as to swap them for a new set, which makes total sense based on this conversation. And so I'm finding the exact same thing. That new set of problems, what does your team lose sleep over now that maybe they didn't even just one or two years ago? 

[0:35:38] TJ: I mean, I don't lose a lot of sleep. 

[0:35:41] MM: Good for you.

[0:35:41] TJ: Well, actually, I do. I do because I have a 2-year-old and a six-year-old, so I lose sleep from that. 

[0:35:45] MM: Makes sense. 

[0:35:45] TJ: But you're operating a lot more in this realm of trying to provide a lot of human-in-the-loop input to make sure that the system works as you expect. It's something that I think with previous ML systems you would really just train on large amounts of data, and it would be like the model is what it is, and you see the emergent behaviors of the model. Now it's much more about when you have exposed so many controls, you can use an LLM, you can use eval, these policies. That's a lot of degrees of freedom that you have in terms of developing your product. And it means that there's a lot higher pace of change in a lot of parts of the stack because now product managers are stepping in and they're stating an opinion about iterating on the quality of this subsystem, and then like an engineer's changing the model at the same time while somebody's iterating on the policy. And then all these things come together in like an integration environment. 

It's a lot more surface area that you have to make sure works together cohesively. And you have the right mechanisms, the right operational rigor cohesion to bring this together in a cohesive way to like an end experience. And so that's probably the thing that is both maybe a problem but also super exciting is that you expose all these controls in terms of what you can do to improve the product experience. But it also means a lot more coordination. And so we've built up agentic systems to help us coordinate across some of these different things in terms of how we deploy them and they interact. 

[0:37:08] MM: Yeah. And getting the balance right of being able to react quickly and not making that process too cumbersome so that you can't get changes out fast. Yeah, that's really interesting. 

[0:37:18] TJ: There's other things where it's like we had to build up a lot of bespoke tooling for understanding when somebody was like, "Hey, this recommendation didn't resonate with me. What happened in the system?" You'd look at all the coefficients of the model, and you'd be like, "I think these features were the reason that this was recommended this way." Some of the plus side of this is you can now actually ask the model to reason and say, "Why was this recommended? Explain it." Right? 

And that's somewhat magical in terms of the pace with which you can improve the product. You can understand where there's misunderstandings. And maybe we have not provided enough guidance for the model to be able to operate in the way that we expect it to.

[0:37:50] MM: Right. And this this problem set too, right? Not like it's a medical device. You're serving content to people. So, it's a such a great use case. I do get kind of scared with the medical stuff, but yeah. It's all about the right tool for the right problem. 

Speaking of that, one thing that also just not necessarily technical, but also very important to talk about is just ethical use of this. And I did see that you guys at LinkedIn have what's called a commitment to responsible AI. And that's really nice to hear. And can you talk about what that is and how you folded it into the engineering of this architecture? 

[0:38:27] TJ: Yeah, responsible AI, it's a top priority for LinkedIn. Fundamentally, it comes into like the kinds of signals that you're using in the model, right? We make sure to use professional signals and engagement behavior only, not sensitive demographic attributes in predicting things. 

And then with every single feed model that we ship, we run kind of regular audits to make sure that the kind of feed distribution, feed quality is equitable across different audiences and creators using a series of statistical checks. That's kind of how we incorporate into our workflow to make sure that, at the end of the day, the ranking that we're doing, etc., is equitable for every single member. 

[0:39:08] MM: And in terms of guardrails, just making sure that malicious content doesn't spread or anything like that. 

[0:39:15] TJ: Of course. 

[0:39:15] MM: Yeah. Okay. That too. 

[0:39:17] TJ: I mean, trust is a top - I think when I think about responsible AI, there's kind of like how systems take into account demographic variables. When we talk about trust, it's making sure that you have the right defenses to protect members against malicious content, scams, spam, all that kind of stuff. Both of these are separate and super critical technical things to get right in any ecosystem. 

[0:39:37] MM: Yeah, I'm sensing the pattern here is almost like in many cases in different ways defense in depth, whether it's your architecture doing a couple of things a different way or having a bunch of different guardrails. That's really interesting. 

[0:39:51] TJ: On that front, I think there's a lot of things that kind of organically come from LinkedIn's professional context in terms of some of these safeguards. For example, we have a 100 million members verified on LinkedIn. When you come to LinkedIn and you see a post from a verified member, you know this is a real person. And so you know you're getting credible verified professionals. That's ultimately the goal is that the system can find the most useful, credible information, and then surface to you. And you can trust that like, "Yeah, okay, this person is who they say they are, and they're credible in their field." 

And that's just kind of like an implicit thing in our ecosystem just by virtue of the fact that we are this professional platform where people verify their identity and say, "Here's what I've worked on. Here's who I am. Here's where I went to school." And I feel like that gives a certain level of depth to the experience in addition to the content itself in terms of your confidence as a member viewing that and saying like, "Yeah, okay. I don't have to necessarily double fact check this or worry about like if I'm getting something from somebody who's a malicious actor." We lean really hard into that from a trust standpoint. 

[0:41:00] MM: And I may be wrong here, but you all are pretty - this might not be the right word, but protective of your APIs, right? As far as I know, you do a lot to prevent bots from just putting out AI slop content. Is that correct? You are looking for humans entering content into your site, your platform, right? 

[0:41:20] TJ: That's right. We use a variety of different tools to make sure that we detect what we call like low-quality and unoriginal content, and then we take action on those to make sure we avoid broadly promoting them on our platform. And so a lot of the kind of bots AI slop, they tend to post exactly this kind of unoriginal stuff, and you can kind of sense it. And that's not what we're about. We want to make sure that, obviously, members should use these tools, but we want them to use the tools to bring the kind of authentic, original perspectives to LinkedIn, not super repetitive stuff through automation. That's a no-go for us. 

[0:41:56] MM: Yeah. My analogy to the old world is you're using LLM technology as input sanitization to not have to deal with it later in the pipe. 

[0:42:04] TJ: Exactly. People using browser extensions, or scripts, or third-party tools to leave automated comments is just - and first of all, it's officially not allowed in our terms of service, even though people do it anyway. And we make sure to remove those from the most relevant view of comments, so they won't show up in most relevant. We make sure we limit the reach of those. And for repeat offenders, we might limit access to LinkedIn altogether. 

[0:42:31] MM: It's refreshing to hear some of the basics here. Good system design. Knowing when to use the right tool for the right job, when to use the right size tool for the right job, input sanitization. Don't just throw everything in the same pile. It's nice to know that the basics still stand. 

All right, so the last question I like to end some of these interviews with, something for the audience to reflect on and how they may be able to relate this to their work. When you look at this whole project and as it still goes on, what's the skill you found your team or yourself wishing that you had more of? And I think that what I'm trying to get at is, for the folks listening, where should they try to point themselves next to make themselves more valuable or potentially try to fill that type of hole? Because this is probably a bellwether for what's going on. 

[0:43:22] TJ: It's a great question. I always start from the premise that whenever you're using these agentic coding tools, they don't know team boundaries. They don't know code repo boundaries. They know none of that, right? And that's part of what makes them so effective is like they can start taking a problem that you've formulated in your head, and it's like I can solve this by kind of navigating an end-to-end codebase or universe and make it happen. 

And I think a lot of software development teams have been built up with charters and understanding, "Here's my role, and here's my infrastructure partners' role, and here is my data science partners' role." And those walls, they're melting. The people who are most effective are those that can cut across these contexts and make sure that even if you're not a domain expert in an adjacent kind of capacity, that you know enough about it that you can kind of figure out how the broader picture stitches together. That's super, super critical. 

You still have infrastructure specialists. But more and more, they understand like the AI models that you're using and the architectures because you need to. You can't just wait for some AI engineer to mosey along and tell you kind of the performance characteristics of the next idea. 

And I think that extends even further beyond the technical realm, which is when decision-making happens this quickly, you're building so quickly and you're starting to build in adjacent places and you're able to get these ideas out in front of your members. You need to have internalized the context of the problem you're working in, but a bit more the product vision and like what you're actually trying to build. Making sure you understand the business context that you operate within. Because the bottlenecks then become are you able to just make those calls yourself. Is your batting average high enough that you're going to just make these calls? And more often than not, you're going to make the right call because you're going to immediately convert that decision into code. That is the most pure way to develop a product is if you can like have good intuition, be a good taste maker, convert that decision into code and then get that in front of members. That is kind of the life cycle. 

[0:45:18] MM: Yeah, that is really well said. What I just heard is curiosity and critical thinking are absolutely key. Being an order taker and expecting what to do to be given to you will not work in this new world.

[END]