EPISODE 1747

[INTRODUCTION]

[0:00:00] ANNOUNCER: The Stack Overflow Developer Survey is an annual survey conducted by Stack Overflow that gathers comprehensive insights from developers around the world. It offers a valuable snapshot of the global developer community covering a wide range of topics such as preferred programming languages, tools, and technologies. Erin Yepis is a Senior Analyst and Ryan Polk is the Chief Product Officer at Stack Overflow. They join the show with Sean Falconer to talk about the results of the 2024 Developer Survey, which just released this summer.

This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

[INTERVIEW]

[0:00:48] SF: Ryan and Erin, welcome to the show.

[0:00:50] RP: Thanks, great to be here.

[0:00:51] EY: Hi, Sean.

[0:00:52] SF: Thanks for joining us. Both of you have been at Stack Overflow for a relatively short period of time. Erin, you've been there, I think, just over two years. Ryan, a little over a year. What actually brought you to Stack Overflow in the first place? Maybe, Erin, we can start with you.

[0:01:09] EY: Oh, I love this question. I've been in the analytics space for eight years now. But ever since I started going down this path, Stack Overflow has been right there with me. It's obviously something that I – Stack Overflow has been a part of my life, as far as work goes. When I saw that there was this opening for this position and that it involved the developer survey, I was so excited to join. It has met and exceeded my expectations since.

[0:01:41] SF: What exactly is your role? How do you describe your day-to-day?

[0:01:45] EY: I'm a senior analyst. I work with the product marketing team for research purposes, basically trying to synthesize insights from various different first-party, third-party data sources. We have so much internal data from our – obviously, our public platform, stackoverflow.com. We have lots of other data sources, too, from the other product lines that we have. Then we have external data sources just about the developer community in general. I'm just helping try to connect the dots about what we're seeing and why we think we're seeing that.

[0:02:17] SF: Okay, great. Then Ryan, what brought you to Stack Overflow?

[0:02:21] RP: Yeah. I joined about a year ago. I mean, first of all, it's an amazing team. Amazing group of people here. It's always fun to work with people so passionate about the community that they support, right? A lot of the reason I joined is for that community. We have 60 million visitors to our site every month. The ability to support those people and actually help build back up the community that we have is a great challenge and something that I was really excited to join and join in on.

Beyond that, being a part of the developer community is awesome, right? Most of my career has been focused around products that are focused on helping developers do their jobs in better ways. Really passionate about the space and passionate about working with developers to help make their lives easier.

[0:03:05] SF: Yeah, absolutely. Curiously, is there something particularly unique about working at Stack Overflow? I mean, given it's a platform that it's like, any engineer knows immediately what Stack Overflow is, but if you go and you tell, like, if I went and talked to my mom about Stack Overflow, she's, “Stack Overflow?” It's immensely popular and well-known within, essentially, this niche society that we all live in of technology, but outside of that is like, you might as well be talking about some random thing that very few people have ever witnessed.

[0:03:37] RP: Yeah. I typically say, it's like the other sites, but for developers, or something along those lines. Or, if you're searching for something technical, you're going to most likely, you don't realize it, but you most likely landed at that Stack Overflow for the non-technical crowd. Uniqueness though, I would say, very much so our integration with the people who have been a part of the site for 15 years, who have been highly dedicated to helping support their communities and help people answer questions across the board. That is a very unique element of what we do.

Also, the structure of what we do. We're not quite Wikipedia, but we're not quite Reddit, right? The structure of our site is really important. The fact that we focus on having accurate information, accurate answers, and we do that in a fast pace where people are constantly involved in that, really does drive towards an incredibly valuable set of community information and that's something that really drives us in our collaboration with our community members as well.

[0:04:35] SF: Yeah. I think that's interesting. It's really hard, I think, to figure out that balance between being something that's like a social-driven platform where people are contributing and not turn into the potential dumpster fire that someone might turn into, while being something that is high quality, but also is community-driven. I think one of the things that's interesting, and I'm curious to hear your thoughts on it, is like, why do you think so many people want to support and interact on this? You're talking about people who've been with you for 15 years. What do you think is their motivation for wanting to contribute their knowledge, essentially, for free on the Internet to be able to answer questions?

[0:05:18] RP: Well, I think it's a lot of the dynamics of the site itself from the beginning, which is that they are a part of a community. That community supports each other. There is truly the altruistic element of just wanting to help other people, no matter who they be. It's also, there's a lot less opportunities accepting the direct professional environment for mentoring people in the software development world. The ability to actually be a mentor to folks, especially later in your career, and we see a lot of our contributors on the site, our folks who are later in their careers who feel like they're giving back, so that's a really cool element of it as well.

We have mechanics on the site around job postings and other things as well, so it can actually be a benefit to your career to be engaged, and you can use your overall profile as an element of your resume to help show your experience as well. I think a lot of the helping and giving back elements is far more contributing to people's work than the direct mechanics.

[0:06:16] SF: Mm-hmm. You're right. I want to talk about the 2024 Developer Survey that was put out by Stack Overflow. Maybe before we get into specifics, Erin, could you provide a little bit of a background, a overview of what this Developer Survey is and what it's trying to accomplish?

[0:06:33] EY: Yeah, definitely. The Developer Survey is an opportunity to get a temperature check from we have data that can support certain statements about usage on the public platform. But there is a lot of things that we need the users to chime in on themselves to hear it directly from the source. It's an opportunity for developers to show themselves to the rest of the Stack Overflow community about what it is they like and don't like, who they are, where they're coming from. Stack Overflow has grown so much with the site itself and the Developer Survey.

It's the largest Developer Survey in the world, as we like to repeat. It's definitely an opportunity, not just for the community, but for people outside of Stack Overflow to learn about what are these latest trends, because there's so many people participating in the survey.

[0:07:29] SF: How many people actually participate?

[0:07:31] EY: This year, we had just over 68,000 responses worldwide.

[0:07:34] SF: Okay. Then the first thing I wanted to ask was around, was there anything surprising to you? You've been running these surveys for a long time. Was there something that really stood out in particular in the 2024 results?

[0:07:47] EY: I'll just say, we added a new question this year. We have our AI section. What I thought was surprising was the question about AI being a threat to your job. Most developers said, they did not feel it was a threat to their job. I thought that was surprising and good.

[0:08:02] RP: Yeah. I think there's some interesting elements of this as well that you see more usage. We saw the usage of AI tools go up from 70% to 77%. That's a decent upshift in a year. But the actual trust in those tools went down about 5% as well. Now, that might be more people using them, more skeptical people using them, but it's interesting to see that trust is dropping, not rising, right? Which means the technologies are – And once again, with the cost of training and so forth, the technology doesn't actually evolve that much in a year. You have more people using it, trusting it less.

We also see that, well, see, about 31% developers are still highly skeptical, but only 43% developers say they actually trust the accuracy of AI tools. When we say AI, most everyone's still pointing to LLMs embedded in code assistants and stuff like that. Still quite a bit of skepticism, concern out there around these tools, even though there's a lot of hype. Yeah, I would certainly say that, I think the developers who have been using these tools have been experimenting with them using a bunch of different code assistants would tell you that they're probably far less worried about losing their jobs to AI in the future. I think it's not necessarily meeting the hype in a lot of places.

[0:09:17] SF: Yeah. The more experience, the familiarity you get, the more you see the rough edges of it. You realize, okay, well, I get some job security for a while. I think it was a 12.1% of responders said that they think AI is a threat to their job. I still go back to, like if you've ever gone into a public restroom and you've tried to wash your hands in the sink, in the center, doesn't really work. We still haven't nailed that as the technology, just automatically being able to dispense water out of a sink when my hand waves over the center. Like, AI might be a little bit of a stretch before it takes our jobs and tremendously useful. I think it's some other things that were really interesting along this line, and you alluded to is that a lot of people highlighted trust being a challenge with AI, but they also still say that it's tremendously helpful from a productivity standpoint.

Even though they don't necessarily 100% trust the output, which is probably a good perspective to have right now, they still see tremendous value with what the, things like the co-co-pilots and evens things like ChatGPT are able to bring to them.

[0:10:22] RP: Yeah. I think there's an element there of, “Let me Google that for you” element, where it's the next step on code completion that helps you speed up your process when you're coding, but you don't necessarily – even on code completion, you go back in and refill in your variables and change things around and stuff. I think people are thinking of it as that ability to quickly add a method, or add a call, or something along those lines, not have to do as much work to craft that, but then have to go back and clean up the things that are wrong. That is a major gain, but it's not writing the code for them.

[0:10:55] SF: Yeah, and even some of the things that you see online about recently, I forget who exactly it was, but they made some headlines on being able to create a video game for scratch, just through prompts. I think the person who did that was also a very experienced engineer, so they already know the direction in the types of prompts that they need in order to do that. They're using their knowledge of engineering to facilitate that interaction. It's not like, just some random person would be able to actually get the same output.

[0:11:23] RP: There's a lot of really cool examples of people who spend a lot of time behind the scenes prepping something and then go through a couple of prompts and make it look amazing. But when it comes down to the day-to-day of a developer's job, it's much more nuts and bolts than that, right? It's much more actually having to dig in and use the tools, heavily use the tools, but maybe don't trust but verify.

[0:11:47] SF: Yeah. The same with trust. When I was looking at the results for most used programming languages over the past year across all the respondents, not too surprising that JavaScript remains one of the most widely used languages. It's been the top dog for a decade in these surveys. One thing I was surprised about was there was no mention of Rust in the professional respondents. There's legacy languages in there, like Fortran and COBOL holding on to dear life, and new stuff like Nim, but no mention of Rust. But when you go to the Learning to Code respondents, Rust is actually 18.5% of the respondents.

I'm curious, what are your thoughts on that? Why are we seeing – I feel like, if you're just paying attention to certain news outlets, you're expecting Rust is like, oh, it must be 80% or something. Maybe this is just on me for where I subscribe, but the fact that it wasn't anywhere in the professional responses was very surprising to me.

[0:12:39] EY: I think we introduced this new metric in last year's survey for most admired. Rust has been on the top of that list for last year and this year. That metric comes from of all the people that said they used a certain programming language or technology in the past year, did they also say, they want to continue using it in the next year? Rust score is very highly there. Then, yeah. I mean, I think to speak more to that outside of the survey, you can see, and as you were alluding to, Rust has a great community. There's a lot of evangelists there.

If you go to other forums where people can be a little bit more candid about why they choose the tools and technologies they do, Stack Overflow is all about write out the question, be very specific about what you're looking for, and then we're going to have helpful answers. If you want that color, you can see in other forums, such as Reddit, or LinkedIn in the comments. You can see people speaking directly to why it is they like tools like that. It's a great community. They're able to find answers. There's features that they love to use.

[0:13:44] SF: Ryan, did you have any additional thoughts on this?

[0:13:46] RP: No, I think this is a portion of the desire to use something versus the practicality, plus the implementation, right? It's definitely got a lot of hype and is plugged into a lot of our recent programming trends, including AI. But is not necessarily something that people in a professional role are directly experimenting with, or have a project that is allowing them to experiment with yet. I think that you see that desirability gap between what they're actually using and what they desire to use, it's pretty big on Rust. Everybody wants to use it, but I don't think they found an application directly for it in their environments, right? And so, there's a big gap. Then you see that in a lot of different technologies as well.

[0:14:30] SF: Yeah. Is that something that you've noticed, even from past surveys that is basically, the professional responses, is that lagging behind people who are in the learning phase? Maybe you're in the learning phase, you want to try the newest, hottest technology, but maybe they haven't found their way into professional programming at the same level. I'm curious if that's a trend that you've noticed with other surveys.

[0:14:53] EY: I think we noticed it with a couple of different things, but top of mind for me, just because it's something that is always top of mind, I learned coding through R. I know that every survey recently, it's just going down in popularity. But they’re still, R is still one of the top tagged posts for questions on Stack Overflow. It's not going down in usage, per se. I think it's definitely something that is easy for a new person to pick up. Then as they're getting used to it and asking questions and trying to find resources, they can find that on Stack Overflow. But then, as far as practicality goes, like Ryan was mentioning, you're at work, you're paid to do a job, you are not using R there.

[0:15:39] SF: Yeah. There's certain things that maybe are always going to be more dominant with people who are learning, but then as they build up those skills, they might move to something that is more widely used in a professional setting. When it comes to databases, PostgreSQL has taken over from MySQL in the last couple of years, as the most popular database. Why do you think there's been so much momentum behind the growth and interest in PostgreSQL?

[0:16:04] EY: I have an interesting tidbit for this. I'm glad to share it in front of Ryan, too, so he knows that I am working closely with our other analysts. We recently started looking at some of the trends in R, the answers associated with tags over the past couple of years and tags that go unanswered to see what is growing, what's not. Yeah, we can see there is a trend with MySQL and PostgreSQL that there are almost twice as many unanswered posts as PostgreSQL. There's a lot more answers per question with the PostgreSQL tagged posts for MySQL just recently.

I think what that is pointing to is that there are more – the subject matter experts, the people that come to engage with those posts that are, yeah, offering that mentorship, offering their expertise in their professional lives, or whatever they're learning to, they're moving to PostgreSQL. That's where the knowledge is transferring.

[0:17:05] SF: I think, sometimes these things end up becoming almost a richer situation, where if you're going and you're considering, like, let's say you're considering what database to choose and you see a really active community behind one particular database, then you're probably going to be more likely to choose that one. If the documentation looks great, there's an active community, I know I can get my questions answered, then the one that looks a little bit more stagnant, then that just ends up creating a flywheel, where more and more people end up adopting that technology in the first place.

[0:17:36] RP: I also wouldn't call out – wouldn't underestimate the impact of the capabilities of PostgreSQL over MySQL, the ability to scale, work across more complex environments as products become much more complex and less of a read-only style situation, where we're trying to gather as much analytics and data over our customers as possible. I think that the PostgreSQL capabilities align better with that more complex implementation market as well.

[0:18:05] SF: Yeah. I think from a future perspective, PostgreSQL has been faster to include advanced data type support, like JSON and arrays and XML. Now they have PG vector and JSON media. All these things also have, I think, really helped create this community that does become the flywheel of growth.

[0:18:23] RP: For sure. Yeah. Once again, that community is really powerful and how you support that and how you build up from there is definitely something that helps drive not only the underlying technology and the evolution of the technology, but its continued persistence in the market. I think a lot of software builders and software companies underestimate the impact of supporting their underlying communities and taking apart in them and driving contribution in multiple places, not just their own sites and their own information networks.

[0:18:54] SF: Yeah. I mean, I think a part of that always comes down to, it's difficult to sometimes measure the impact of those things, so they tend to get overlooked, especially in times when you're trying to run leaner. It's like, do you have a mathematical formula to attribute impact from this thing or not? Then essentially, those, a lot of times those things unfortunately get defunded, while other things that are more directly related to, attributed to revenue, or pipeline grow, or something like that get funded. On the database side, what database that's on the rise, do you think will continue to grow in popularity?

[0:19:29] EY: Just because they're new to the survey and new in general, the two that I pointed out would be Databricks and Snowflake. We have a lot of options on the developer survey. They're definitely not at the top of the list, but I think that's because they're new. Given that they have shown so much in their marketplaces, their AI integrations, they're just scaling, they're able to scale really fast in this new AI world that we're in.

Also, something that has been on the developer survey for a long time, again, not at the top of the list, but still, definitely staying strong as far as the top contenders go is Elasticsearch, which we know is also a big contender in that AI space.

[0:20:10] SF: Yeah. I mean, they're all fast. I mean, obviously, I think Databricks has been popular with data scientists for years, even doing traditional ML. Then both Snowflake and Databricks have made some key acquisitions, too, that has really helped with their AI efforts. One thing I was actually a bit surprised about looking at the database survey results was that BigQuery is responded to this, like owning more of the market from respondents than both Snowflake and Databricks combined, which I was a little bit surprised by, but I don't know. I guess, maybe there's just more BigQuery fans out there than I realize.

[0:20:47] RP: Yeah. I think when you look at the results on BigQuery, professional developers versus learning the code, it's surprising to see the percentage on professional developers is much higher than the learning the codes. There's certainly some element of a foothold in some industry there that they're maintaining, just driving up their results, for sure.

[0:21:07] SF: Yeah. How many of these 68,000 people work at Google, I guess, the – Another thing that I was, I guess I wasn't surprised by it, but was just made me think about it a little bit was, how much of the world runs on SQLite? Pretty much every mobile app has a SQLite database behind it. It's still, I think, essentially a project that's run by, solely by the creators and they handle all the updates and the roadmap. It's just fascinating to see this fairly seems like, niche project in, essentially, in terms of overall usage is up there with these gigantic companies that everybody knows and is dominating the industry.

[0:21:52] RP: Yeah, it's certainly, build a better config file, which is essentially, what the SQLite acts like. Lightweight data persistence and configuration file. You will still be at the top of the list 10 years later. Certainly, its flexibility, simplicity of implementation. As you said, the foundation of most mobile products pretty much puts it at the top of the list consistently.

[0:22:15] SF: Yeah. I mean, it's just like most databases, even going back to 30, 40 years ago, you're thinking about large scale in SQLite just really nailed that use case of hey, I just want a file that I can run, essentially, SQL statements again, against. Then along comes the world where we all have smartphones and we're carrying around computers in our pockets. Suddenly, it's really, really useful to have, essentially, this lightweight database attached to all these applications that we're running.

With the cloud platforms it's still not surprisingly the big three at the top, AWS, Azure and Google, but things get, I think a little bit interesting with the learning the code respondents. Google Cloud is actually at the top, then AWS, followed by Vercel and Firebase. Azure is actually sixth. Why do you think some of these platforms are so popular with those that are learning the code versus this professional setting?

[0:23:10] RP: I think there's a well-known business model that has been in this world for a long time, and that is appeal to the students and understand that if you can catch people earlier in their careers, you can keep the mind share throughout. There's a lot of products that peaked in college though, right? That is they don't really make it out of that environment. Because they were easy to use in college, but once you get to the business world, there's other products that already have a toehold. They're already established. I think Amazon has done a great job of establishing themselves across most business environments over the last 10 years, and it's hard to break in and get past that being the established tool of choice.

It's a constant battle. I think it's a good strategy for folks like Google to make their capabilities available. Of course, you have G-Suite as well that most students are using now and a lot of schools are even using Google tools as their primary element of how they work. I think we'll catch up from that strategy perspective. I think you're going to see more and more influence from kids coming out of college, and having the ability to then push that agenda through later on in their careers.

[0:24:17] SF: Yeah. Come subset of those, people are going to go on to found their own companies, or at least be in technology leadership positions where they can make infrastructure choices. If you've laid the groundwork, Google's laid the groundwork with the student population, then they're investing in the next generation of technology leaders.

[0:24:35] RP: For sure. I think it's a good strategy. It takes time. I expect that it pays out more in the next few years.

[0:24:42] SF: Yeah. I think the platform as a service type of companies like Vercel, they are great from a learning perspective as well. I know they're also, there's certainly companies that run their core infrastructure off of Vercel. A lot of companies starting out with some pass, eventually reaches a situation where they have the graduation problem, where if they are wildly successful, at some place in their growth, they typically want to start to move some of those workloads directly onto a native US, or Azure, or Google Cloud, so that they can really, really have control over the entire environment in the end, control costs, control resourcing, and so forth.

[0:25:20] RP: Transportability is also going to be key in the next few years as people look to move between cloud systems as they evolve and as they mature. I see a lot of startups choosing alternative cloud environments early on, but then shifting to the more traditional ones based on the incentives of scale, right?

[0:25:39] SF: Yeah, absolutely. I think under the tools that people are using, I think there, what was interesting was, it's probably the biggest delta between a product used in, or a tool used in professional environment, versus used in the learning and the code environment. I'm talking about, in this case, Kubernetes. Kubernetes was the fifth most popular tool in the professional category, but it's the 22nd most popular under learning the code. Whereas, Docker is about the same across both. I'm curious, what is your thoughts on that? Is it that we don't need a container orchestration where we're learning, maybe that's too complicated? Is it something that you only really see in a professional environment? What are your thoughts on that?

[0:26:21] RP: Yeah. I think it's, I mean, now you are getting more and more programs in colleges, where people are learning the elements of actual professional development, including continuous delivery, continuous integration and management of environments as you're building your products. For the most part, most college assignments, most college requirements don't really require the complexity of Kubernetes. A simple Docker container will do, or multiple Docker containers to prove out communications between systems. The orchestration elements aren't really needed, because you're not necessarily spinning up and managing these environments long-term. I think it's more, or less a case of good enough for now, unless you're in a specific class that is built around DevOps and site reliability engineering and the integrations of large environments.

[0:27:11] SF: Makes sense. I think that is something, even if you're relatively early in a company, depending on what you're doing, you might not even be using something like Kubernetes at that standpoint, or it might be abstracted away in some fashion through a managed service. They can manage Kubernetes environment. I'm sure a lot of the platforms as a service, essentially, are running Kubernetes behind the scene and you just don't have your hands on it.

[0:27:35] RP: Yeah. And, of course, the ratio of folks that depending on how large your organization is and whether or not you have dedicated, or distributed SREs, or DevOps team members, they might be handling a lot of this and you're just handling the basics of integrating in. It might be one of the cases that earlier stage developers early in their careers really only see this from a pass-by element of an environment that's already been handed to them. So, they don't think of it as a core technology either way.

[0:28:08] SF: Cause even if you go to a large company, like a Fang company, where they're probably using either Kubernetes or some container orchestration, that stuff's already been largely figured out. You're probably not building and architecting that from the ground up. You're hitting a button and you imagine, they can see this essentially because of the hard work that people put in previously.

[0:28:29] RP: For sure. Mostly obfuscated by the fact that it's just part of the everyday. They didn't have to actually build it and set it up and maintain it. There's value in that. And of course, there's also the original elements of the concepts of DevOps and SRE, which were more integration between the developers and the people actually delivering the software. So, maybe that's a bad thing from a perspective. We'd like to see more people actually engaged in understanding the underlying infrastructure. But the size of the environment, it always shifts people's perspective.

[0:29:02] SF: Yeah. On the respondents from about IDEs, one of the things that I was a little bit shocked about was that Vim is so high, it's fifth of all respondents. Still in the top 10 of those learning are still using Vim.

[0:29:18] RP: It feels like, the Monty Python sketch. I'm not dead yet. It's just, it's –

[0:29:23] SF: Just a flesh arm.

[0:29:24] RP: Yeah. I mean, there's still so many developers out there who love their environments. When you look at the actual spread on seniority and years in software development across the survey, there is a pretty dramatic spread from one year of software development, all the way to 50-plus years of software development. When you look at that dramatic spread, you're going to have products that have been around for a long time that people still love and still have their minds wrapped around how they're used and changing the underlying flexibility of Vim is pretty insane too, right?

You can change the underlying code base you're working on without changing the tool you're using to do it and you keep your mind in that right space. People love tools like that. And so, they persist forever.

[0:30:10] SF: Yeah. There’s certainly a certain beauty in the simplicity of Vim. I still use it a lot of times for editing, quick edits to config files and stuff like that, but certainly not living in it day-to-day as a my primary editor of choice anymore.

[0:30:23] EY: We have on our graphs for work with versus want to work with, the chord graphs. It was interesting. Visual Studio is obviously top of the list, but a lot – the number one IDE that those users want to work with, the size visual studio is Vim. Meanwhile, the Vim users only selected Vim as what they want to work with. They did not select any other. I think that's true. Their needs are being met and also, they are communicating that to other people out there. Hey, it's pretty cool over on this side of the eyelid. You might want to try it out sometime.

[0:30:56] SF: That's great. I love the fact that the Vim power users are like, “No, I'm good. I don't know. I'm not interested in anything else.” We touched on a little bit on some of the AI results earlier, but I want to spend a little bit more time diving into some of those. I think, one of the things I was pretty interested in was that 24.4% of the respondents said that they have no plans to use AI tools in their development process. Do you think that's a mistake to ignore the value that some of these tools can provide?

[0:31:27] EY: A deeper dive on some of the roles that were top of the list, as far as indicating they did not want to use AI, or not planning to, I think that has a lot to do with it. Context of what you are doing at work and where we are realistically with the AI tools that are available now. Yeah, we saw embedded developers and desktop/enterprise developers as those that mostly indicated they were not going to use AI tools. I think that is because of the nature of their work. Their expertise, they have a lot. They have a lot of knowledge about what it is that they do. It's very specific. There just isn’t a tool that they need to really spend time on. There is that, I know. There's like, this isn't going to solve any of my problems, so I don't really need to spend time doing it.

[0:32:15] RP: Yeah. I would say that I've spent a lot of years working in different environments, worked in the security, endpoint security space for a while. What I would tell you is a well-known factor there was the closer you are to the kernel, the harder you are to replace, right? The complexity of what they're doing and what they're working on is not something that is information that can be aggregated from the Internet, which is really what an LLM does. It's one of those things where probably these systems are answering very few of their questions when asked, so they're highly skeptical.

[0:32:48] SF: Right. That makes a lot of sense. I mean, if you just think about the training corpus for an LLM and you want to figure out, how do I center – vertically center a div? There's plenty of those responses that exist on the Internet to get you started there. Then you can generate 10 different ways to do that. But there's probably going to be a lot less about a machine instruction sets for some embedded device, or something like that.

[0:33:11] RP: Yeah, for sure. I think that's the nature of a lot of this. The more obscure the information, the less trustworthy it will be over time. There's mechanisms to solve that, but they're not built into the AI tools themselves.

[0:33:24] SF: One of the other things that people said, it was that basically, 45% of professional developers felt AI was bad at handling more complex tasks. What do you think it's going to take in order for AI to get to a place where it can actually help people with a more complicated, sophisticated programming task?

[0:33:42] RP: Well, I think that there's some elements here that are interesting. I think it's more than LLMs, right? I think it's more than just the – I think everyone has directly tied the word LLM and AI together at this point, which is not really the case, right? An LLM is a form of an AI. I think that it's more than just the technologies we've seen on large language models. There's multiple models interacting with each other, and some of the things that we're seeing, the experimentation that will eventually, I don't like the term AGI, or all these hype terms, but what I would say is, it's more types of models interacting with each other in a way that drives a more logical chain of decision making and allows them to interrogate and interact with each other to actually build a system.

We're on the cusp of that in a lot of different places. But if you can't trust one response, how do you trust millions of responses interacting with each other, right? It has a lot of propensity for things to go arrive very quickly. That's why you don't see a lot of automation around AI capabilities right now either. People are using them to fill in the blanks on things, but actually having the current AI technologies actually manage to automate back-end systems and maintenance and things like that is relatively minimal.

[0:35:04] SF: Yeah. There's so much contextual information that you need to take into account in order to do something like that, that's more sophisticated that you're going to run into challenges just from a context window perspective. Then I agree, I think that if you see where some of the successes that's happening in AI right now, and I mean, talking generative AI, it's really not generative AI as this monolithic model. It's more of a composable AI infrastructure, where you're bringing, stitching multiple things together.

There's the AlphaCode Project as just an example from Google, where they're using some forms of generative AI to solve programming competition problems. But what they're doing really is they're using the LLM to help generate a million potential solutions and then they're using additional AI systems to, essentially, evaluate and filter down to get to the best of those solutions. It's a lot more of these stitching together multiple techniques in order to get to a place where you can solve a more complex task. Then, when you do that, it's no longer a general model that you can just throw anything at it. It's for a very specific task that you want to solve for.

[0:36:09] RP: Yeah. When you get to the point where you have to write a specific set of models, model managers and a system to drive it, is it more efficient than just writing the code, or is it at that point, right? There will be a tipping point eventually where it is, but at this point, it's more experimental than it is useful.

[0:36:27] SF: Yeah, absolutely. Erin, in terms of how these surveys come together, what's the length of time that it takes to put this survey and what is the scope of putting something like this together?

[0:36:38] RP: Yeah, that's a great question. There are a lot of moving pieces. There's a lot of collaboration. I would say, there's the planning stage. I would say, as soon as the year begins, we're talking about the developer survey, because everyone's looking forward to it, right? It's a great time of year. We typically will launch the survey in the spring. At the beginning of the year, we're starting, talking about we have retrospectives. Obviously, we want to remind ourselves what could be better, what did work, what do we want to see more of? Then we're talking to our internal teams about, how are we using this information? How are we showing people this information? What did people like? What didn't seem to land? Can we tweak some things? What should we add?

We go through a couple of rounds of just making sure the questions, the survey instrument is finalized, and there's many people that will take a look at that and we're queuing it, testing it. Then we get it into, we get it programmed, we test that. Then we get it up. We get all of our creative stuff together. We're with our marketing team to put together all the branding assets, so we can start promoting it on our site and elsewhere. All of that, we're talking, yeah, about three months. Three months of work, of fun work. Then we launch it. That's a month, right? It's out there in the wild. Then we put it all together in a beautiful site. That's another month.

[0:38:01] SF: As we start to wrap up, is there anything else you like to share, or point people to?

[0:38:07] EY: Well, I would just say, all of our surveys, all of the results sites, you can go to surveys.stackoverflow.co. We have all of our surveys there. You can go look at all of the past surveys since we started doing this. Obviously, I'm more partial to the ones that I was a part of, which is 2023 to 2024. They're great. Go check those out. Then yeah, we have a great community, if you haven't heard of it, meta.stackoverflow.com. If you have any feedback, there is a survey, a developer survey tag, and you can post it up there. We love reading our feedback.

[0:38:44] EY: Yeah, this is a great resource. Go check it out. This is definitely something that's valuable to developers to understand the different trends in the marketplace. If you want to be a part of it next year, we run this every year and it would be exciting to have 100,000 respondents next year and continue to grow this, so that we can get a broad view of the market and everyone benefits from that view. It's useful, because we publish all of the information that we have.

[0:39:07] SF: Awesome. Yeah. I enjoyed digging into the details of the survey myself and I encourage anybody listening to go check it out. Ryan and Erin, thanks very much for joining the show today.

[0:39:17] RP: Thanks for having us.

[0:39:18] EY: Thank you.
[END]