EPISODE 1974   [INTRODUCTION]   [0:00:00] ANNOUNCER: The technical interview has long been used as a proxy for actual job performance. Brain teasers, data structure puzzles, or reversing a linked list became the price of admission to employment as a developer in many organizations, even though few engineers do that work day-to-day. It would seem AI has now broken that model entirely, since an LLM can one-shot any classic algorithm question. This is forcing tech companies to rethink what an interview should measure.   Vivek Ravisankar is the co-founder and CEO of HackerRank, which is one of the most well-known platforms for tech interview practice problems. In this episode, Vivek joins Josh Goldberg to discuss why old-style interviews stopped working, the company's apology tour for years of algorithm puzzles, how to keep AI evaluation fair and trustworthy, and where both models and the shape of hiring may be headed as software engineering becomes something every role does.   This episode is hosted by Josh Goldberg, front-end developer at Sentry and open-source maintainer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ESLint, a powerful static analysis toolset for JavaScript and TypeScript. Josh is also the author of the O 'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a co-founder of SquiggleConf, a conference for excellent web developer tooling. Find Josh on BlueSky, Fosstodon, and .com as Joshua K. Goldberg.   [INTERVIEW]   [0:01:52] JG: With me today is Vivek Ravisankar, CEO of HackerRank. Vivek, welcome to Software Engineering Daily.   [0:01:58] VR: Thank you for the invite. Super good to be here. How's it going?   [0:02:01] JG: Things are good. I'm always excited to talk about tech interviews, AI, and how it's all shaping together. But before we dive into all that, tell us, how did you get started with HackerRank?   [0:02:09] VR: Yeah, I started this company over a decade ago. I used to work at Amazon as a developer where I interviewed a lot of people and I realized that resumes didn't have a great correlation to their skills. I'd interviewed people who looked great on paper, completely bombed the interview and vice versa. And that led to the idea of maybe skills matter more than pedigree. Of course, we hadn't codified it that way, but that was the founding inspiration to get going. And we started the company with that ethos. It's still our ethos.   [0:02:40] JG: Lovely. And over the years, you've seen quite a few different styles of interview questions. What was the original premise for what kinds of things you were seeing companies ask and have folks' answer?   [0:02:49] VR: Yeah. I think if you were to just zoom out and think about all the evolution of questions, it always used to be one of those things of whichever was the most prominent software engineering company. And whatever their interview process was, it became the standard de facto across the board.   Microsoft was technically the first software company. And the questions that they used to ask early on were things like how many golf balls fit in a 747, how many traffic signals are there in Chicago, and things on those lines. Obviously, the idea here is not for you to get the right answer, but to test your thought process. How effectively did you think through? Computational thinking. And of course, the entire software engineer interviews used to be like that. Developers used to hate it. It has got nothing to do with the job.   And then Google came in to the picture and said like, "Hey, we're an amazing software engineering company. We are going to index the entire web, which means that we are going to ask some very hard data structures and algorithm style questions." And where do the best engineers work? They work at Google. And everybody started copying that style of interview questions.   And now the developers were in the perspective of, okay, it's better than asking how many golf balls fit in a 747 because at least you have to write code, but it still is not realistic. I don't write a breadth first search in my company. I don't reverse a linked list. I don't invert binary trees. Why are you asking me all of these questions? And there's always this undercurrent of interviews don't reflect your actual job.   And then over time, I don't think there was one particular company, just like AI as a factor, where it was able to one-shot all of these different algorithm questions and made them pretty much useless in an interview setting, which then forced every company to rethink their hiring, which forced HackerRank in itself to go ahead and rethink our interview process, what we want to hire. And now it's in the new era, the agentic era of interviewing, where whatever you're being interviewed on is going to mimic your actual work. So that's been the evolution so far.   [0:04:49] JG: It's interesting you bring up the perceived, perhaps actual mismatch between questions and day-to-day job responsibilities. Because I went through the CS hiring pipeline back in the day, myself around a decade ago. And there's this perception that my fellow students and I had back then, and many folks still do now, that there is a severe mismatch between what platforms like HackerRank and LeetCode and all the others do versus the actual questions. So how would you respond to that? Do you think HackerRank traditionally has been a representative interview for what folks do in their jobs?   [0:05:19] VR: I think maybe we can answer it in two parts. And maybe I'll start with an analogy and go and answer this in two parts, knowing that analogies are not perfect. So if you were to take the interview process of a pilot, a real pilot, and 90% of your flight is flown by AI, but that doesn't mean that you will get on a plane with a white pilot. You still want the person to know fundamentals of aeronautical engineering and others. And if you were to think about the interviewing process, the first step is to evaluate your fundamentals of aeronautical engineering. It's actually a multiple choice set of questions. No AI, obviously, to know whether you know the fundamentals.   The second round is a round on flight simulator, simulating different scenarios. The third is the actual flight exercises. And the fourth is number of hours that you've actually been trained. It's not exactly around, but like a credential that you need to have. And the fifth is your health checks. And then like you get certified as a pilot. I'm just compressing a lot of this stuff just for brevity, but like that's the high-level overview.   Now, the reason why I shared that is we have to ask ourselves what is the purpose of the interview and then determine what the rounds are. Now, obviously, when it comes to interview, you want to make sure that you are hiring the person who's going to perform the best inside your organization. And then there are like two different components of it. There is like a technical component and then there is like a cultural component, and both of them need to match.   When you think about the technical component, I do think there is a degree of knowledge of fundamentals that you need to have. How do you evaluate that is an interesting question. And there is an aspect around can I mimic real work? Where you're not going to actually deal with fundamentals on the real work on every minute basis. So I do think there are both of these components that are important.   I do think we went overboard on weighting. The weightage that we associated with the fundamentals was we went overboard relative to balancing it with what your interview process needs to be on real work mimicking. And I think that's starting to get rebalanced right now.   [0:07:26] JG: I think a lot of people would be very excited to hear you say that. There's a kind of mismatch and impact here, right, between the original drive of skills over pedigree versus what you actually see happen when you bias towards one specific area, right? Someone who has the ability to spend 20 hours a week working on HackerRank to get really good at these fundamentals that don't actually directly show up in the job. Do you see that as something that can be overcome? Is there an ability to make the measured statistic still relevant?   [0:07:54] VR: Yeah, I would say that is probably a little bit tangential to the skills over pedigree. And by the way, I'm not trying to defend myself here or anything. Because it's really funny, I don't know when this is going to air. We have bought a bunch of billboards in San Francisco where we're apologizing for asking people to reverse linked list, invert binary trees, grinding lead -   [0:08:15] JG: Wow. [0:08:16] VR: We're actually spending money. It's literally an apology tour. Whatever I'm trying to say it's not like me trying to defend and things. But I do think that aspect I would view, it slightly tangential towards skill over pedigree. I mean, you do need to prepare. You do need to put the reps in for you to get a job. It's part of the way that you work. And if somebody is putting in more effort, and as a result they are able to ace interviews or clear interviews, that's just good. That is still a skill.   The way that we think about skill over pedigree is are you subconsciously rejecting people who are not from MIT or Stanford? Or conversely, you're just like only hiring people from Stanford and MIT with a particular GPA. I mean, I face this personally. My co-founder, Hari, he's probably one of the best hackers, and hacker I mean in a good sense, which meant that he never used to attend classes. In college, he had a very low GPA, and companies would just automatically reject him. He didn't even get an interview call with Amazon or Google or many of the companies who came to our campus, university recruiting.   It's really funny. I used to vouch for him more than me. I used to tell the hiring manager, "Dude, you should probably hire him over me. But if you have two slots, you should hire both of us because I don't want to lose my job either. But you're just automatically rejecting because this guy doesn't have a GPA and he has a lot of things that he needs to work through."   When I think about skill over pedigree, it's less about the reps or the amount of prep that you make. It's more on subconsciously people just gravitate towards what's your GPA? What's your university? And that's the stigma that we're trying to eliminate.   [0:09:46] JG: Perhaps there's a third degree here, something like ability or privilege or the ability to spend a lot of time, right? If you're coding day to day, nine to five, you may or may not be able to spend additional time working on specific problems. If you were to completely redo the HackerRank from 2020, let's say, or even earlier, do you have an image in your mind of what you would be asking people?   [0:10:08] VR: By the way, we are actually doing that right now. We're shedding the past. We are reinventing the company kind of from scratch, really. I think there are three dimensions in which we are innovating. The first is the type of questions you ask. The second is what are you going to evaluate? And the third is your candidate experience. I think on the type of questions, we're moving from the standard algorithmic style questions to real world questions on code repositories in terms of what you want to evaluate. It's less on code correctness but more in terms of your critical thinking, judgment, and AI fluency.   And then your candidate experience moving from an IDE to an agentic development environment, or what we call as ADE, where the agents is at the front and center and the code is on the side. So if you were to like sum up, most hiring process ask you standard algorithmic style questions, evaluating code correctness on an IDE. And we're helping transform them into having real world questions on code repos, evaluating AI fluency on an ADE. That's the shift that we are making inside the company. And hopefully, we take the industry along with it as well.   [0:11:11] JG: That's really exciting to hear. I had planned to come in here and surprise you with a question, but you've already asked the question. So perhaps you'll nail it. Could you, before we move on to talking about the projects and other work in HackerRank, describe verbally how to reverse a singly linked list, please?   [0:11:24] VR: How do I reverse a singly linked list? Okay. So you take a pointer and you do a while loop and make sure that, at each step, you reverse the link and take care of edge cases and outlier corner cases.   [0:11:38] JG: Hired. Anyway, moving on. Let's talk about the innovations in HackerRank that you're -   [0:11:42] VR: - but I can write code for that. It's probably easier for me to write code than just explain the reverse link list. But yeah, let's go.   [0:11:51] JG: I also think that you should be given some grace as someone who's not quite in a strictly software developer role for quite some time now. But all right, let's talk about some of the features of HackerRank. I'd like to segment us a little bit. Let's first talk about the core interview questions, and then let's move on to the more AI-focused ones so we can really dive in later on.   For the last several years, HackerRank's been, as you said, reinventing itself. What do you see as the kind of pitch or evolution of interviews that you're working towards?   [0:12:16] VR: Like I mentioned, I think right now, majority of companies are still evaluating candidates on code correctness on your algorithmic questions, and that is increasingly starting to have lower and lower efficacy for multiple reasons. Maybe it's one of those things of that's not the kind of work that you do on a day-to-day job. But also, there is rising concerns around integrity, suspicious activity, "cheating". I don't like to use that word, but whatever, a lot of people actually use that.   And you're not even able to trust the signal that you get from this. I think, I believe it was Steve Jobs who said this, but although I've tried searching. I don't think it was attributed to him. But the quote is a nice quote. I don't know who said it. The best way to fight piracy is to make it official. And I thought like Steve Jobs said it when he launched Apple Music. This was like a dig against Napster, which is everybody wanted to listen to music online. But of course, Napster, it was a pirated version of it. It got shut down. And here's a way for you to launch it official.   And the way that we think about interviews and stuff is very similar. Hey, you want people to not "cheat" or use AI tools. Why don't you just give them Fable right there? Reduce incentive. Remove the incentive and you don't have to worry about it. Now that said, going back to the aviation stuff, I do think there is a place where you should evaluate their fundamentals or aspects of reviewing stuff without an AI. I do want to know Josh minus AI, what are you? Yeah, we are like mimicking or we are like taking a lot of inspiration from the aviation industry in terms of the interview process. And so far, it's resonating a lot with our customers.   [0:13:52] JG: That's really interesting. There's another quote I wanted to bring up with you, Goodhart's Law, which is that any observed statistical regularity will tend to collapse once pressure is placed on it for control purposes. Or when a measure becomes a target, it's no longer a good measure. So we saw this with the older school questions where people hyper-optimized to those. Are you worried about folks hyper-optimizing for whatever the next stage of interviewing is?   [0:14:14] VR: I almost feel like every time - this is not a thing on you, but we've had this discussion internally. Every time somebody brings up Goodhart's Law, you appear very intellectual. And it's like, "Okay, but what does it mean? Shouldn't we have any goals? Shouldn't we have any KPIs?" I mean, I get the idea that you're going to incentivize and you're going to overfit whatever you're trying to do. My guess is it may happen. But I don't have an alternate. What is the alternative to Goodhart's Law? Either no goals or make sure that you don't overfit. And both of those seem to be pretty hard to work through.   [0:14:47] JG: That's a good point. I think from my perspective, it's that, like you said, with the piracy, you have to make it legitimate. Or if people are going to abuse the statistic or the measurement, you want the result of their abuse to be beneficial. It's as you said earlier, if you make the interview more realistic, you're then more directly interviewing for what is actually relevant, right?   [0:15:08] VR: Yeah. I also think it's hard to do correlation/causation on this because it does take time for your performance to come into play within the company. And frankly, most companies aren't even good at tracking performance. So you're just going to use some proxy like tenure or promotions and things in those lines. It's also harder to do that, which we try to. By the way, when we think about correlation, we try to look at, okay, if a particular candidate has taken an assessment, taken an interview, is this person getting - this is public information. Is this person actually getting promoted, like tenure? What is the efficacy of the interview process and things in those lines? It's marginal or squared in terms of the standard algorithmic style questions.   I do hope the newer style questions that mimic real work, we'll start to find a better correlation. And then after some time, the Goodhart's Law might kick in and then we'll have to find another paradigm. We'll see how it goes.   [0:16:11] JG: It's almost a virtuous cycle there. You want it to kick in. You want to see how people adapt to your work.   [0:16:15] VR: Yeah, yeah, yeah.   [0:16:16] JG: Just out of curiosity, how do you measure something like that, the efficacy of your interview process?   [0:16:21] VR: I think there's different degrees of measurement. Maybe I'll start with the basic one. The first is we want to make sure that there is no adverse impact towards a particular race, ethnicity, gender. So before you actually launch a test, you'd want to conduct a sample across at least gender to begin with, if not across all different race, ethnicity, and others, to see if there's no adverse impact. That's one input.   The second one is your score distribution. The third one is actually the feedback that candidates give at the end. We call it DLI, Developer Love Index. How much do developers actually like their experience with it? So these are probably input metrics in terms of what happens. That's probably base zero. Then you go to base one, which is, okay, when somebody has actually cleared the screen or the initial take-home assessment, what percentage of them actually cleared the interview? And what is the correlation of people who actually showcase really good skills on assessment to interview? And you want that to be the next one.   The third one is probably likely the hardest, which is, okay, of the people who cleared the screen as well as an interview and joined the company, how many of them are like really, really good performers? That again, like I mentioned, it's hard to get performance data. So we try our best to look at proxies like tenure and promos, which are all publicly available information to try and see what the correlation looks like.   Certainly, base zero and base one, level zero, level one, you can do that. It's all within your control. Level two is hard. Level three might be how are you the next Zuckerberg? I don't know. So how do you actually figure that out? That might be interesting.   [0:17:51] JG: That would be interesting to see. But okay, let's move on to the AI then. It's going to be quite a lot of content here to discuss. So just to start off broad, you've alluded to this earlier, how do you see a good AI oriented or AI era interview going for, let's say, just a typical software developer position?   [0:18:06] VR: Yeah. So we've launched a new product called a Chakra. And Chakra stands for superpower. So the etymology behind that is we want to find the superpower of candidates. And it is an agentic interviewer. And the way that we've designed Chakra is it opens up a canvas. And the canvas can be anything. The canvas can be a development environment. It can be a whiteboard. In the future, it can be a Figma, Excel, whatever it is, and gives a real-world task or real-world work to the candidate, along with all the AI agents and assistants available. The candidate completes the work.   And after that, Chakra is able to follow up on questions. Why did you do A versus B? What would you have done differently if I had this particular constraint? Walk me through your thought process and all of that. So at the end, what you get as a hiring manager is a pretty comprehensive report of, first, was this person able to accomplish the task with AI? Second, how effectively did they use AI to accomplish the task? And third, did they understand what they're doing? Did they have fundamentals, critical thinking, judgment to make sure that it happens?   And those are very important signals for you to determine, okay, do I want to proceed with this candidate or not? And that's the first step on the screening. And then when it comes to interview, you want to make sure that you can continue the process in an interview where you're going to have the same repository that you gave in the screening but with different kinds of tasks or different kinds of features so that you're retaining the context of the candidate.   And now, the interviewer and the candidate are now pair programming on our agentic developer environment, which is effectively Cursor or Claude Code on web. That's what we have actually built in a collaborative environment for both of them to get a sense of what is it like to work together with the person as well as on a real world task. We call it the AI-led screen and the human-led interview. So that is the combination for the next-gen process.   [0:20:01] JG: As soon as you mention an AI interviewer, a lot of people's pearls get clutched. And for good reason, right? There are a lot of flaws with how AIs model and reason. Recently, Claude has said the word load-bearing in at least every prompt response that I've given to it. There are a lot of common biases or repeat points that don't work well.   Go given the very sensitive nature of an interview and the mixed non-deterministic results you get from AI, how do you make sure that this thing is actually interviewing properly and not just kind of going down rote original paths?   [0:20:32] VR: Yeah, yeah, yeah. No, it's a very good question. I think there are a lot of thoughts on this. I think there's one visceral reaction of why are you sending AI to interview me? That is one. And there's always like a debate of is sending a take home better than an AI? I actually think it's better to have an AI. Because in a take home, you're just boxed in. What if you have a question to clarify? Who do you ask? There's nobody that you can even ask a question. And if you go out of the tab and want to search on Google, you get flagged for integrity. A lots of things actually come up.   And go back to the developer love index, right, we've crossed over half a million AI interviewers on Chakra. The developer love index which is a rating that a candidate can give at the end of the session on a scale of five is now hovering at like 4.6 to 4.7, which is pretty amazing. And we were surprised, just so you know, in terms of it's actually higher than a static take-home assessment. That's at least like what the data is saying.   The second aspect of this is there are a lot of AI interviewers that are just like dorky video avatars and just like voice only and things in those lines, which is very, very frustrating when you actually see. We decided we had certain set of principles that we worked on. Firstly, we're not going to have video. You're not going to anthropomorphize Chakra. And we want to make sure that people know that it is AI.   And one of the reasons why we did that was I think when people speak to AI, it's usually in a voice form that people are used to. You've spoken to Alexa. You've spoken to Siri. You've spoken to ChatGPT Voice. It doesn't have a video avatar. So people are comfortable talking to AI in voice. So don't put a video on that. So the second one is the form factor of how it actually appears.   The third one, and by the way, it's all in no particular order, I think the difference between any AI product that is a slop and a magical experience is in the quality of your evals. That's what it comes down to. So we have a realy applied AI research team that actually works on evals where we run Chakra across many different scenarios, how much it can probe, how much it can pull back, how much it's able to ask follow-up questions so that it is meaningful and it's really good. And we continuously make sure that we can improve our evals.   And the fourth one is it needs to be like a combination of work and interaction. If it's just going to be a voice thing for 60 minutes, that's like super tiring. I even don't want to take that. Why am I speaking to an AI for 60 minutes? Most of the Chakra interviews are 45 minutes of work, where I'm going to give you a problem. You can go ahead and do it. I have all the agents. I'm here at the background. Chakra is in the background if you need any help. And then there's a 15-minute of follow-up where I'm actually asking you questions. That's a way better interaction pattern than doing it for 60 minutes on voice. Anyway, those are all the things that we have invested in that we've been very thoughtful in terms of the way that we built. Yeah, that's where we are.   [0:23:30] JG: Are you worried that If everyone uses something like Chakra, there won't be any strong differentiation between how different companies interview? Let's say that you build Chakra and then LeetCode builds Chakra. How do you differentiate the HackerRank version of this versus your competitors?   [0:23:44] VR: I think maybe I'll speak at the core aspect and then we can talk about like the specific features. I think there are two things that underpins success in this field. How strong is your evaluation and how strong is your integrity? So you need to be able to make sure that you can evaluate the skills that are required for the agent to get up. And when you evaluate, I need to be able to trust that this is the person who actually did it and you've evaluated that in a proper way.   So a company that cares about evaluation and integrity is going to be the company that's actually going to win at the core, because everything else is on top of it. So we actually build proprietary models on our own in terms of evaluation and integrity, run real strong evals. Know their company is actually even trying to invest in any of these things. Everything is just like a prompt engineered version of it or just like more like AI slop. And that's the foundation.   Then you start to add, okay, what are the type of canvases can you actually support? A development environment, a Jupyter Notebook, a whiteboard, and all of those things. What are the types of questions that you can actually do? What is the level of customization that you can make happen? Customization not just in the type of questions that you ask in terms of the rubric that customers want to bring into the system.   And then you have a whole set of things in terms of how do you make sure that you can pass the test with all the legal and IO teams within your customers who are going to say, "How is AI evaluating? Are you compliant to the NYC law? Are you compliant with the EU AI Act?" By the way, a lot of people don't realize this. Hiring is a regulated industry. When you say healthcare is a regulated one or banking is a regulated one, it seems like pretty obvious. Yeah, of course, it's regulated. Hiring is a regulated one. You have EEOC, Equal Employment Officer Commission, where it's going to fight for candidates who believe like it's an unfair practice. So you need to comply with laws. And even more so, if you're going to use AI in the interview process, you need to comply with laws.   If you think about the stack of things that you need to do, you need to first build your own proprietary models and evaluation integrity. You need to be able to support all the different canvases and customization and integration with the systems. And you need to be able to comply with all the laws and make sure that your AI evaluation is interpretable. You're not using any data to train the models. You're compliant with all of the laws. It's not biased towards a particular ethnicity or race or gender. You do all of that. Then you have competition. I don't think anybody is doing a job as good as ours. And I know it sounds very self-serving and braggadocious, but I don't think anybody is doing as good of a job across all of these dimensions. And it's going to be pretty hard for you to build this.   [0:26:18] JG: First of all, I'm excited to hear you. You seem genuinely excited about the product quality. But I do want to hone in on one little point you made there about bias, because there's been a lot of research showing that traditional AI models are in fact biased against, let's say, folks who sound or appear or present as women or people of color and so on. And perhaps you touched on this with making your own models, but how do you make sure that your interviewers aren't slightly or more biased against folks who aren't traditional cis white men?   [0:26:46] VR: There are two aspects in it. We've not yet gone in terms of evaluating their accent and things on those lines. At the core of it, everything translates into text because that's what we evaluate. Whatever you say, it's translated into text. Whatever you do is effectively text.   I don't evaluate how do you look or what's your accent? Do you have a thick accent or not? Now, there is a nuance there in terms of if what happens if you're not grammatically correct, if you're not working through it. Again, it goes back to the evals. We actually check what is the impact of things on those lines.   And there's a threshold of grammatical incorrectness where you're just not even coherent, where it does fail. But that I don't think is a function of your ethnicity or background and things. It's like you're not able to communicate properly. And that would be a problem in the real world if I'm going to hire you. So we are very conscious of not evaluating your voice or image and things in those lines for the purposes of evaluation. And everything gets transcribed to a text.   [0:27:45] JG: But there are some mannerisms or phrases that people in different areas use. I'm in New York, you're off on the West Coast. We do have subtle differences in how we speak when translated purely to text. Yeah. Like someone from Texas will say y'all. It's like kind of the canonical North versus South example. So is it something you can put into evals? How do you make sure that like someone who speaks the way, for example, an inner city Chicago black kid would speak versus like a Vermont upper-class white person would speak? How do you make sure that that doesn't sneak into the model bias?   [0:28:16] VR: And maybe it's because Chakra is heavily focused on right now tech and tech adjacent roles that the work that you do is - gain, if you consider a 60 minute session, you're doing 45 minutes of just work. And then the 15-minute interview is very technical, where it's actually probing. And there are only so many variations of kernel, operating system, threads, whatever computer science concept that you want to pick that is going to have an impact on your accent.   And certainly, grammatical correctness is a different one. We haven't actually seen that level of variation. But if you're going to use Chakra for a general voice interview for 60 minutes, I can certainly see variants happening. And even that, we try our best to make sure that our evals work in that way.   [0:29:00] JG: That's actually a great segue, because I'd like to talk about what comes next. Right now, using Chakra for technical roles. Of course, the next question I'm sure a lot of your accompanying customers have is, well, what else can we use this thing for? Could you use it for culture screen or non-technical roles overall? Where do you see the end line or the final goal of it? Is there a line that you won't cross for a tool or system like this?   [0:29:19] VR: Yeah, it's a really interesting question. It is one of those things I've thought about where we went from, I think, 2023, 2024, there was a lot of chatter on software engineering is dead. It still is there right now every time Anthropic or OpenAI launches a new model. But I think the narrative has shifted from software engineering is dead to software engineering is everywhere. Because what we see in our customers is, wow, this chakra really works well for software engineering job family. But my product managers are also building stuff. Maybe we should use Chakra for product managers. Or my designers are also building stuff. Maybe we should use that. Or my customer support person is no longer responding to tickets. They're building agents that can actually respond to tickets.   The hottest role in sales of go-to-market org is something called as a forward-deployed engineer. The hottest role in marketing is a go-to-market engineer. The hottest role in legal is a legal engineer. By the way, I'm not kidding. I just like posted today on LinkedIn. I think there's a company, Gong, the call recording company, that's actually put up a new role called talent engineer.   We're going from like software engineering is dead to, no, software engineering is everywhere. That is one dimension in terms of covering more surface area where we went from one job family across others, but still focused on engineering. And by the way, what is engineering? Engineering is your ability to translate your idea into actual working stuff. And you do that by decomposing the problem into a set of smaller problems, catching all the corner cases, building stuff that is maintainable and things. This construct, you can apply to any org that you want to go ahead and solve problems. And that's what people are evaluating and testing on. That's one dimension in terms of surface area. The other one that you mentioned is the culture screen, early recruiter screen. We have seen much more adoption of Chakra as an early recruiter screen to ask basic questions like, "Hey, are you willing to relocate? What's your comp expectations?" And things in those lines, more than the culture screen. I do think the element of culture screen will still remain within humans.   This goes back to the thing of at least what I believe. If AI is going to do a lot of the work that we were doing as humans, the other parts of the work that we should have been doing, that we weren't doing to the degree that we would like, we'll just get elevated a lot. Technically speaking, you should spend way more time in evaluating culture fit, but you just do a 30- minute interview to work through it right now. Maybe if you're able to offload the four-hour technical stuff to AI and you could spend two hours talking to the person versus just doing technical interviews or two hours evaluating culture fit, that might be better. And same thing with doctors. How much time are you spending in triaging the request versus spending time with the patient? So if AI can do a lot, then maybe you can spend more time with the patient and evaluate them. I do think the culture fit piece will stay with humans. And in fact, it may get even more elevated and add a higher weightage in the future. That's probably good for everyone.   [0:32:12] JG: It's funny you should mention that. My favorite story to tell about one of the recruiting situations I was in was there was a candidate who did very, very well technically, but failed the interview overall because he had found on LinkedIn, one of the recruiters, a woman had previously done modeling. And instead of asking questions about the company or role or comp, I just asked her about modeling. So I think you're very right that there are some things that you just cannot find when you're talking to robots that you have to have human to human interactions to discover.   [0:32:40] VR: Yeah, yeah, yeah.   [0:32:40] JG: Moving on, this is all really exciting stuff. Let's say that Chakra gets rolled out to a company. How do you evaluate long-term? And we kind of touched on this earlier, like the effectiveness of the system. Or are there ways that you have to determine this style of prompting or these eval areas are better or worse than others?   [0:32:57] VR: Yeah, I think we use in terms of how - the way Chakra evaluates is on three dimensions broadly. One is, were you able to do the work? The second is, how effectively did you use AI to do the work? And third is, do you understand what this is all about? So those are the three aspects. The first one in terms of like, "Hey, did you do the work?" That's much more heuristics based, very deterministic way of saying like, "Hey, were you able to solve the ticket or did you do the work?" and all of that.   The second one, how effectively did you use AI? The way that we evaluate is on framing, judging and steering. How well did you frame the problem to AI? How well did you judge the output, and how well did you steer the AI to get to the answer? We take the entire session, we break it into different episodes. We have like a logical way of breaking into different episodes. And for each episode, we give a grade in terms of how effectively did the person frame judge or steer depending upon what that episode is, like if it has one or more of these. And then this culminates in an overall grade.   And the third one, did you really understand? That's the 15-minute probing that we talked about earlier, where Chakra just tries to evaluate whether you really understood what you were doing. Whether you could extend the thought process, if added a different constraint, or if I gave you a new set of assumptions for you to work through. And all of these three things culminate in a unified score that's just easy for you to benchmark against other candidates. That's how Chakra evaluates.   [0:34:28] JG: I love that. There's a really common trope these days of software developers who just slop cannon something over their peers, meaning they throw something at the AI, it creates some output that vaguely looks good, and then they send it off for peer review. And then if they're lucky, a human will review it and see how bad it is. And if they're unlucky, an AI will review it poorly, and then it'll ship the product as a bug. What parts of that funnel do you think that would kind of fail on? The latter two?   [0:34:50] VR: The funny part is I was actually talking to an engineer yesterday. It's like, "Hey, we use a model." He was actually complaining, "Look, man, the amount of PRs that I just need to review is just crazy." And people aren't even looking at what the AI is building. It just makes changes in hundreds of files. I think Tobi Lütke from Shopify calls this slop grenade. I thought I really liked the term. And it's like, "there's too many PRs."   And he was telling - here's the funny part. And then I used the same model to just like, "Hey. go and look at this PRs and tell me what needs to change." And it comes with 50 other changes. Wait, so the person who built, who wanted to solve this problem with the same model, made all the changes. And now I'm using the same model to say go ahead and review it. And you're making all these changes.   At some level, there needs to be a human in the loop to answer this question. And I think, yeah, to your point, it will certainly fail in step two and step three. Where, if you don't know what AI is trying to do, or if you have not framed it well, if you have not like steered it well, if you're not judged the output, I think it will fail.   [0:35:58] JG: I want to ask you a few kind of tangential questions as we head closer to the end of the interview. Right now, we're seeing a sort of fantastic growth in model power, where these new ones, like you referenced Fable earlier, are just getting more and more powerful to the point where it's kind of unbelievable the work that they can oftentimes one shot. But this is still all online based. You're still paying Anthropic or some other company for these things. What do you think is the next step in models? Do you think we're going to have local offline models, significantly more powerful ones, coordination agents? What do you think is happening 5, 10 years down the road?   [0:36:35] VR: I don't know who said this. I think the difference between a monkey and a human is humans know how to use tools. I mean, iPhone is your second brain. You can write to-dos and things in those lines. So you're able to do more things when you started to use pencil, paper, phones, and all of those things. So I think one obvious dimension of models getting better is just the ability to use more tools.   And by the way, there is also an interesting thing there. We run our internal benchmarks for different model capabilities and stuff. Just because you use more tools doesn't mean that you can actually solve the problem in a better way. The correlation is not there. So that's like an internal benchmark. By the way, we're likely going to launch our own benchmark. I know there are a lot of benchmarks, we bench and all of those things. I keep saying, I don't know who said this. Maybe that's a good thing or a bad thing, but because I just like absolve information. I don't know who said this, whatever I'm going to say either. But somebody said this, which is, hey, it's easier to construct a hard benchmark. It is harder to build a realistic benchmark. We've actually taken in real world tasks, benchmarking these models. And we found that just because you use more tools doesn't mean that you're able to do a better job in the problem.   So anyway, coming back to your question, I think the first one is, yeah, I do think models are going to get much more effective in terms of tool calls. Models are certainly going to be much better orchestrators. I think we saw the launch of JEV. The etymology behind that is the Jevons paradox yesterday. JEV launched and it's making a lot of waves in terms of being an effective orchestrator of a model. I do think that's going to happen.   I do think there's going to be a combination of things that you do locally and things that you do on cloud, depending upon the task. I do think that's likely going to be a paradigm that will exist. I think it's just going to get more better on computer use. It's going to get better in terms of long horizon tasks. I do think it's going to get better in terms of when to ask feedback from you and when to do it on your own, when to be more proactive.   I'll give you an example. I use this new product that has come up, Instinct. I wanted to go to the Cars movie with my toddler. No tickets were available last Saturday. So I just left it. And then like an hour ago, I got a text from Instinct, "Hey. three tickets have opened. Do you want to work on it?" I didn't even prompt or anything, right? I do think like the models, models judgment in terms of when to ask, when to do on my own, when to come back to you, all of those things will get better. Yeah, so if I synthesize, better tool use, better orchestrator, which will help in efficiency, long horizon tasks, exercise better judgment, and a combination of local and cloud for different tasks. Those are all some of the dimensions I think we're going to see models get better.   [0:39:14] JG: It's kind of funny that you reference being easier to construct a hard benchmark than a good one. That's kind of what your company is trying to do, right?   [0:39:21] VR: Yeah. When you said that, it's like, wow, this is exactly what - and by the way, we're likely going to launch another business line. Because what's funny is like what we realized, whatever we are doing on the hiring front is very much transferable for all of these model companies. We are building an RL gym by creating all these real-world canvases and things. We are building verifiers, like what I mentioned to you in terms of how well the candidate did on all of these parts of it. We are building real-world questions. What does the benchmark consist of? RL gym, verifier, and real-world question.   And we have a big community of experts who can go ahead and build this for them, critique model outputs, like preference ranking. There's a lot of things that we can actually go ahead and do. This is not drawing boxes around like is this a red signal or a yellow signal. It's like actually building real world stuff to evaluate performance of models.   And interestingly, what we have also observed is it's not just restricted to the frontier labs. All of the customers that we work with, whether it's Salesforce or Amazon or Nvidia, they're building frontier models as well, but they're also building models for their own products, and RL gym and evals for their models to perform better as well. The opportunity of this can expand beyond just hiring and beyond just frontier lab because you can transpose the same infrastructure to a larger business in the AI era. And that's something that we're building from ground up. It's not yet launched, but will launch pretty soon. We'll see how that goes.   [0:40:44] JG: Good luck. I wonder if you could move or pivot or expand into other areas of developer life. Could you do code reviews or open source contribution validation just as a service based on the similar models or evals criteria?   [0:40:59] VR: We do have a lot of questions on code reviews, like, "Hey, review this code." I would say it has spiky usage. And what I mean by spiky is there are certain companies who totally love it, continue to use it. And certain companies are not okay with it. And the difference between these two companies - and by the way, developers are very opinionated. You probably know it, like Vim versus Emacs. I think the Silicon Valley episode, the guy breaks up with the girl because she uses tabs versus spaces or vice versa. By the way, this will sound super odd to any non-developer. Why did you break up with this girl? Because she uses tags versus spaces? But developers are very opinionated. Whatever I'm going to say is just like two different opinions on the code review piece of it.   I think that one category of people are like, look, in order for you to review code, you need to be able to understand the full context, the business context. And everything you should be in the repo. And so if I'm just going to give you a code review type and ask you to review stuff, it's just not going to work. You're not going to give me comments that are meaningful.   Then there's like a completely other category, which is like, no, I do think there are elements on how you are actually going to ask questions even if you don't know the business context, "Why are you doing this? What will you change?" that are just beyond syntactical correctness review comments that you're going to do. And I want to evaluate the quality of your questions that you're going to ask on this, even if you don't have the full context of the code repo and stuff.   I can see merits on both. But yeah, that is an interesting one in terms of evaluation. And that's what we're seeing in adoption in our customers. But if I just were to extrapolate in the long term, I do think we're going to have a different shape of how review thing happens. It's really interesting in terms of, I think we've changed the way that we build software, but I don't think we've changed the way that we review software. I mean, yes, there is Greptile and CodeRabbit and things in the science that you can actually do auto reviews, but you're still reviewing PRs, you're still reviewing code, but you've completely changed the way that you build software.   I have some ideas on it, which is maybe you should review the interaction with AI. What did the person actually talk to the AI? What did they actually say? That's probably, I wouldn't say better, but that's an important signal in conjunction with your PR that you need to think about. If I have to extrapolate, I think like the shape of code review in itself will change quite a bit. We'll need to see how it changes and then we'll obviously immediately adapt in the way that we think about hiring.   [0:43:22] JG: That's so interesting. And now that you've said it, it's such a HackerRank Vivek answer that you're evaluating not just the output, let's say, of an interview answer or the author of the pull request's actual pull request, but you're also evaluating the thing that went into that, how they interacted with the interviewer or how they interacted with their editor and agent while coding. There's a beautiful kind of symmetry or parallel there.   [0:43:45] VR: Yeah, yeah, I agree. Because, by the way, people do this all the time. I'm sure you've observed this, which is, if you're going to give me a PR slop of, okay, there's 50 files changed for a minor fix. Firstly, I'm going to probably reject that. And second, if at all I want to know, I'm going to say, "Okay, dude, share screen. Show me what you did with AI. Walk me through? Why is it generating so many diffs? Why do you need A, B and C?" And people are already starting to think about asking more stuff in terms of looking at what went in as opposed to just the final output. I am very sure Greptile, CodeRabbit, and a bunch of the AI code review tools are working to figure out a new shape in itself for code review, I don't think the reviewing PRs might work anymore.   [0:44:31] JG: Well, I look forward to interviewing you again in two years once it's all been figured out.   [0:44:35] VR: Yep. Yep.   [0:44:36] JG: At the very end of my interviews, I like to ask one or two fun questions just to palate cleanse and talk about non-work things. We briefly chatted before the interview about this, and I just want to give you an open-ended question. Tell us about filmmaking and your relationship with it.   [0:44:49] VR: Yeah, I've done some research. That's good. Yeah. So the two things that I did when I was in college was making movies and competitive programming. I found my then girlfriend and now wife during filmmaking. And I found my co-founder when I was doing competitive programming. So my ROI for going to college is through the roof.   And interestingly, there are a lot of parallels in both. The parallel is, firstly, in filmmaking, as well as in building software, it's almost like you have an image in your mind and you want to just like see it translated in a movie. You have some visuals going on and you want to see translated in software. Of course, you want to see, "Wow, what if this existed? Here's how it would look like." That is one. The second is the high is always when somebody is watching your movie and they're laughing at the point that you wanted them to laugh or they get shocked and awed for a twist that you had in your movie. And it's the same thing in the user. You know that when they're going to do this action, they're going to have a wall moment. And you feel the high when you're building software, when you see the user's reaction to how you've actually designed the product.   And it's very similar in other aspects as well. There is a lot of post-processing work that you need to do in terms of editing background score. It's almost like you got to like keep adding layers on layers and layers from the raw footage to the full production-grade movie, coloring, background scores, actual songs, which is much more popular in Indian movies, subtitles. You just need to keep adding more and more stuff until it actually becomes really, really good. And that's the same process, from your prototype to your finished production stuff.   And that's why you know it's always very interesting whenever people say AI is going to replace software engineers. I just did a prompt and I can create Salesforce. Dude, do you even know how much is there from prototype to production? It's a big, big process to work through.   The only difference though is once a movie is out, it's done. There is nothing that you can do about it. If people love it, love it. You get celebrated in the hall of fame. If people hate it, that sticker is going to remain there forever. But in software, maybe your first version sucked and you do have an option for you to continue to get better. And you need to continue to get better. And maybe people love your product right now. I think people were dunking on Meta for, oh, you guys suck in AI and stuff. And now everybody wants to use Muse. That's probably the difference. You can earn back your reputation within the same product category. And in a movie making, you probably have to earn back by making a new movie altogether. Yeah, there are probably some differences. More similarities than differences, though.   [0:47:28] JG: It's funny, you can have a company like Microsoft that creates both Windows ME or Vista and Windows XP or 7. And then you can have a filmmaker do both Avatar and then others. There's also, I think, an interesting reversal or difference here where software engineering is perhaps the most AI-filled career or area right now. And filmmaking is one of the less. Just curious, what do you think the impact of AI on filmmaking is or will continue to be?   [0:47:54] VR: I think there's a lot of things. And I interact with a lot of filmmakers just fueling my own curiosity, exchanging notes and all of them. I think in pretty much every aspect, there's a big shift. One aspect is when I think it's a very expensive process to do dubbing in multiple languages, but retain the lip sync. Because when you're going to dub in multiple languages - and by the way, this might be a little bit of a foreign concept. India has many languages. America, it's English. Everybody speaks it. That's why it's always funny when people say there's like different accents. You can't understand a Texas accent versus this. I say like, "Dude, you should go to India. It's a whole different language." You can't even understand.   I don't even fully understand Hindi, for example. And there are so many different languages, Kannada, Telugu, Tamil, Hindi, Bengali, and all of those things. And if you want to launch a movie that's going to cater to all of India, you need to be able to dub. But when you're going to dub, but if your lip sync is in the original language, you don't connect to it. And now you can use AI to change the lip sync in alignment with the new language, which by the way you can now do it for French and Spanish if you're in Hollywood to have much more mass appeal. That's one big shift.   The second one is whenever you're trying to create storyboards, it used to take a lot of time for people to hand draw things and storyboard and stuff. But now you can do a much better job in visualizations, which also fuels creativity. And the thing about this is when you're trying to make like a really big movie, you have to go to the studio to create this realistic landscape and things. How do you even imagine or visualize? Sometimes it takes a long time for you to even build and visualize. But today, you can visualize it very quickly, go iterate, and you can have a much better working relationship with the large studios in terms of the landscape of the set that you need to go ahead and create.   Music is a big deal. Editing patterns are a big deal. If you think about, okay, you made a movie. Now, how do you actually get people coming to the movie? Well, you need to have a trailer. And the trailer needs to be compelling. How do you know at what points you need to cut the movie? And what can you reveal without revealing the whole plot, but enough that's intriguing people to come and watch the movie? And now AI can actually help you cut the movie into interesting things in terms of the trailers that have actually worked. Yeah, it is starting to get embedded in all of the different life cycle of the filmmaking process. [0:50:21] JG: That's really cool. Thank you for that. Anyway, back to HackerRank. This is all the time we have, but Vivek, thank you again for spending this interview. It's been a fascinating conversation going from the beginnings up through evolutions and now AI era of you and HackerRank and interviewing. If folks wanted to learn more about you and your company and the products you're working on, where would you direct them on the internet?   [0:50:40] VR: You can go to HackerRank.com. And I'm available on X at rvivek. That's my X. I think that's my LinkedIn as well. But LinkedIn, you can search my name. Yeah, you can go to hackerank.com and sign up for a free trial.   [0:50:55] JG: Well, for Software Engineering Daily, this has been Vivek Ravisankar and Josh Goldberg. Thanks for listening, everyone. Cheers.   [END]