EPISODE 1656

[EPISODE]

[0:00:00] ANNOUNCER: GitHub Copilot is an AI tool to assist software developers by auto-completing code. It's no understatement to say it's already transformed how developers write code. Adrián Mato Gondelle leads the design team for GitHub Copilot. In this episode, recorded in 2023, Adrián joins the podcast to talk about his work.

This episode of Software Engineering Daily is hosted by Jordi Mon Companys. Check the show notes for more information on Jodi's work and where to find him.

[INTERVIEW]

[0:00:41] JMC: Hi, Adrián. Welcome to Software Engineering Daily.

[0:00:44] AMG: Hey, thanks for having me.

[0:00:45] JMC: So, Adrián Mato is a designer in GitHub, specifically devoted right now to Copilot. But before that, so you yourself, would you define yourself as a designer, as a developer? Because you've had stints in the development area. So, how do you define yourself? And talk us about a bit your career.

[0:01:04] AMG: I like to talk about myself as a designer who likes to build things. I did work as a front-end developer before that, but mostly now spending most of my time as a full-time designer, actually managing a design team of designers, working in Copilots right now. Prior to that, I was mostly spending most of my time at GitHub, working in CI/CD, basically, building and testing deployments, and also putting together the first version of actions of packages and Codespaces, basically.

[0:01:37] JMC: Wow, nice. Now, you and your team are fully devoted to Copilot, right? Tell us a bit about Copilot? How did you get involved with it? When did the project get launched? Because, yes, everything's fairly new and I'd like to know more about the beginning.

[0:01:53] AMG: Yes. I'm very bad with dates. But basically, we have been working full time, me and my team for six months now. Copilot launch, like a year ago, I think, and was basically just reduced to code completion. Then, a few months ago, we were launching compiler chat, and then we made it publicly open, available for everyone. Obviously, we are also working more stuff that will be released at Universe in November, early November. I think it's the 8th of November.

The team is quite small. We are four designers working full time in a variety of different parts of Copilot, because it's not just a code completion, right? We are working in - we have announced before with Copilot texts like Copilot for PRs, [inaudible 0:02:39] sets, we need to maintain different surface areas like different IDEs. So, it's, I guess, the reach of Copilot is not just in the IDE. It covers cli.com and a bunch of other areas as well.

[0:02:51] JMC: Yes, exactly. Because from a designer perspective is a really challenging thing to design, right? An LLM is by definition, something that does not necessarily have an interface, right? It's not a front-end application, per se. It can be, I guess, if you just plug in a chat. But that is one way of interacting. So yes, you've mentioned Codespaces, which is GitHub's IDE, but it's probably not the most popular out there. So, where does Copilot get more engagement right now from? Is it from other IDEs? Is it used to CLI through the chat? What are the figures there, roughly? Not necessarily - we don't need to be precise.

[0:03:30] AMG: I guess, the popularity of IDEs is what you will expect. It's basically Visual Studio is the most popular one out there. Visual Studio, the older version of, I guess, VS code or the richer version. It's also quite popular. And then JetBrains, because Java developers are everywhere, are the three most popular ones. I guess, Neovim is also quite popular, but different orders of magnitude. Then, we have a lot of hacks out there, not what we like to say like, they're like official supported IDEs right now, and then there are like third parties with a lot of hacking to get it working in Xcode even, although it's very close as a platform. Yes, a bunch of other areas as well.

[0:04:14] JMC: Eclipse, Emax, who knows, Vim, and those are more editors. But the as you mentioned, of course, development happens mostly in the IDE, but also, especially with GitHub, but the same applies to GitLab or any other sort of like, collaboration mechanism. Pull requests are, in itself, a huge workspace, right? I know workspace means something in the IDE, right? The memory instantiation and so forth. But it is also a collaboration space, a pull request. So, you guys are working on Copilot there in what sense? I mean, I know you're saying that you want to announce things in just a few weeks' time. But what are you thinking of? Are you plugging Copilot there?

[0:04:59] AMG: Yes. If you just refresh what we were announcing for Copilot X a few months ago, you can expect that coding is not just happening in the IDE, right? There is a lot of collaboration in pull requests, daily tasks like checking documentation, it's always quite important and time-consuming. So, you can expect us trying to help speeding up and improving all those back and forth as an individual, and as a team collaborating in code changes, code bases, all of that.

So, pull requests in particular, are starting to look quite decent. There is a lot of manual labor implied in capturing changes that are going to be described or shipped. There's like a lot of code reviews that need like a lot of manual labor again, so we're trying to see how Copilot can assist you, as an author, and as a collaborator to just make it better. We're not like dramatically coming up with something novel. We are just making it easier and more efficient. The usage of your time will be more efficient.

For documents, there's a lot of [inaudible 0:06:05] documentation in general. People are always back and forth using different frameworks, different automated tools to generate documentation on top of code bases, API's documentation, stuff like that. So, we're trying to envision how do we speed up that process, and with advantage of like natural languages and semantic. I guess what I'm trying to say is like all the benefits of large models, in terms of how you interact with them will be beneficial to stop going to maybe, StackOverflow, as often or to manually maintain those documentation sites and just make it more efficient for you to retrieve what you're looking for.

[0:06:42] JMC: I tend to agree with you. So, OpenAI or ChatGPT felt a bit revolutionary, especially for those not tracking very closely for the last few years. The AI space in general, those that have been actually tracking, they are also surprised, but probably less so. But as you say, I think in the case of Copilot and the GitHub echo system, it's going to be an incremental benefit in the user experience across the board. Those minimum interactions that are complete that add up to huge waste of time or that require a lot of time reviewing code, doing manual tasks, being able to plug in Copilot there to provide automations, or some semantic, understanding the content of the code, the review, whatever is going to add up a lot.

So, my question to you is, code completion has been around for a long, long time. I mean, programming languages are structured and repetitive in a way. With a few statistical engines behind the scenes, you can actually hint where things are going and provide good code completions. What has Copilot added to that? Has it added anything in terms of interactions of how you request from Copilot's code completion? Has improved in the sense of is it more accurate? Are they better? In that sense, talking about code completion, what does Copilot add to the picture there?

[0:08:10] AMG: I think the best thing we did with code completion has been building within the existing frameworks and patterns. So, in terms of adoption, you are not like coming up with something so novel that people need to get used to. It was just basically, as you mentioned, like augmenting the code completion experience before you will get just completion for local functions, or methods, or maybe cross files. But it was pretty much just literally completing what you were trying to type.

Now, we're just building on top of those patterns and say, "Well, you can also reason within the file or within the project and start asking questions or wondering how things could be done." And the acceptance rates are not where we want it to be, although they are improving over time. I think the last time we were sharing some numbers, they were around 60-something. But the important part is more the rubber ducky in effect is you're just chatting with a Copilot, trying to make sense and drafting potential ideas or solutions for something while not needing to learn something new from scratch.

Even if you're selecting a piece of code, right-click, and you see like a bunch of options, explain these portions or like help me create unit tests for days, or help me refactor or improve these codes. Those things are already like included in actual editors. Sometimes you get an error of a dependency and you can right-click even without Copilot and VS, we'll find the right dependency path for you based on the packaged JSON. So, we have been doing that in the past. But now it's just like easier to reason with natural language. You just talk to your assistants or to your Copilot and be able to think about it. So, I don't think we are necessarily trying to replace the need to understand how to write code. We are just making it faster and easier, and sometimes more enjoyable when you're trying to find different ways of tackling a problem that you couldn't do before.

[0:10:14] JMC: I completely agree. So, I think that the most revolutionary part of all this new wave of LLMs the chat interface, right? The fact that, although there's a bit of a context break there, it's not the same as moving onto a browser, opening Google, opening Stack Overflow and looking for something that is almost a complete break. Without doing that, the need to do that, you still need to shift your mindset a bit from slash commands, from anything that requires an interaction with the machine, with the command line, to a potentially, and this is my question to you, explain how the chat interaction works to a chat window. Then, start this dialog, with the LLM in this case, and start discovering what the policy, what the code guidelines of your company, of your code base is, and so forth.

So, that is a bit revolutionary, because I think that in the past, you literally could not do that, right? With a certain degree of reliability.

[0:11:11] AMG: Yes. I think the compiler chat experience was just a natural next step for us. Because if you see how people were starting to use comments in the files when code completing, to kind of like start chatting or providing intention to AI, without actually writing the code was a clear next step for us. I think it's beneficial. I don't think everything needs to be a chat, obviously, and I know there is like a lot of people writing about, why are we always defaulting to the chat? I think it's just one step in the right direction. And I do think that there may be cases where, as we increase the accuracy of the suggestions, and just better tools coming up, like embeddedness, and other things, things are just moving very fast right now. They are getting faster. So, the cycle of feedback is getting faster, so you can also do other types of interactions, right?

Right now, if you chat and you get a retrieval, you may be waiting 15 seconds total to get the final answer. But we are streaming text in real-time to make it feel faster, stuff like that. As things are evolving, we will be playing with other patterns, potentially. But the chat is a very forgiving experience for interacting with models. Because, again, if you are getting the correct answer 60% of the time, or maybe 100% of the times, but it's not quite there. It's not perfectly what you're looking for, being able to refine the prompt, provide more clarity, context. Just refining the prompt until you get what you're looking for. Chat interfaces are just conversational interfaces, are a good way of doing that. Maybe it's not just a chat, but refining what you're looking for is important to get what you want.

[0:12:50] JMC: Exactly. The fact that it's a dialogue. That's why I was mentioning the mindset, rather, you are open to a conversation in which you, like with a conversation with a human being, you take for granted that someone can misinterpret your request, or your art, or your question. If you're requesting a code completion with a slash command, with a comment, whatever, the patient's there, or the mindset is completely different, right? It doesn't prompt you to refine it. You probably can do it, right? Or decline a suggestion or maybe suggest another option and so forth. But the mindset is probably not the same. That's why I love it. That's what I love.

[0:13:30] AMG: Yes. I think as tools are getting better, we're looking - if you analyze how Perplexity or even ChatGPT, everyone is working towards reducing the amount of explicit content the user needs to provide, because they are getting better at like guessing the right context for you. So, the prompting is just going to get more loose or forgiving or simpler, right? We're going to move from maybe giving a lot of details to get what you want, to something that based on the context, where you're using the AI, you can get very good, meaningful answers without spending 10 minutes thinking hard about the prompting. Because just staying in the editor, instead of like moving to another website to check on documentation is great for providing that focus and being in the zone. But if you're wrangling with a child for 10 minutes, that's also not beneficial for keeping, joining in the loop, and being focused on the task at hand.

[0:14:26] JMC: One of the other things that Copilot excels at is obviously the quality of the training data sets, right? Along with, for example, Stack Overflow, which has arguably the best data set of questions and answers out there in the world. Well, GitHub has the biggest repo of open source, the biggest set of repos and open source in the world. So, is that a challenge for you guys? Or is it an asset?

[0:14:52] AMG: I think this has been, historically, like a healthy tension to have working in GitHub. We have a very powerful API in general, not for Copilot, but just GitHub API. Everyone is building on top of GitHub as a platform. When we were releasing actions, there is nothing impeding you from bringing your own CI/CD, stuff like that. So, we kind of have this tension of people has access to all these open-source data that we also have. That's for SaaS, I guess, to think beyond what's possible and trying to innovate and promote, like what else we can do that no one else can do right?

It's a fair tension, I guess. We are trying to keep the game fair and being open, and letting people build on top of GitHub. But we are also trying to challenge ourselves researching, what else can we do that will be harder for others to achieve, not just because of the data we have available, but what else we can do, as a first-class experience that otherwise, you will need to rely on a third-party domain and things like that.

[0:15:52] JMC: So, the GitHub Copilot design team, the team that you lead, what personas do you guys have in mind? Probably many, but which ones are the ones that are top of mind your ideal personas, the ones that you work for? What are the drivers that you think are most relevant to these personas, and you strive to deliver a fantastic experience within copilot for those values, if you wish? What are the tasks that are they trying to solve for? What kind of features are you providing? What are the design decisions you're making to make this experience fabulous?

[0:16:24] AMG: I think the main persona we have been trying to appeal, at least at the beginning, was developers, obviously, because code completion was very close to the type of persona in the team. But when we were chatting about PRs, or documentation, or other features we are releasing, we're trying to have a good breadth of different types of personas, because you have PMs collaborating with developers. You have designers. You have dev roles. You have marketing. You have all these departments, so you can expect that apart from focusing on the task at hand of like coding, we are trying to, even with the chat, you can serve reasoning about other things beyond just the code itself.

So, we are expecting to appeal to a larger audience, and I think that's actually something we have done before in the past, like discussions or issues. They are not catered just for developers. It's just a form of collaboration and communication. PRs for viewers, it's not just tied to developers. So, we want to appeal to as many people collaborating in a team as possible.

Then, for the principles or guiding principles, we are trying to keep in mind that although this is very fresh and exciting, in the end, we may need, or not, AI to solve certain tasks. So, we still lead with needs, with scenarios, and with opportunities we want to tackle, and then we decide whether Copilot is the right thing to use or not. It's not an end goal, right? It's just another tool or, yes, more tools in the toolkit to help people get things done.

Obviously, it's the shiny thing. Everyone is willing to know more about what we're doing with Copilot. But sometimes Copilot will be pairing with existing features, like maybe in the CI/CD space. One thing that comes to mind, this doesn't mean that we're working on that. But you can expect like wrangling with Jamel files is a pain in the ass for CI/CD. The debugging is a pain in the ass. I could see like AI is helping us debugging, making sense of errors, or helping you even as a person working in analytics, maybe you can make sense of big data sets that will be beneficial not just for coding, but for other decisions.

So, I think there's like a long way for us to see how these models and these interactions will spread across other products, GitHub, extensibility, many, many things we can do which aren't just the coding part.

[0:18:43] JMC: Focusing on developers, anyway. As you say, in pull requests, in code revision. Code approvals, you will get the security engineer having a say, or even being themselves, say, an approver, or not a gateway, legal and so forth. That's fascinating and it's going to be really challenging for you guys to serve all those needs and those personas and the principles. But focusing on the developer, I've seen many Copilot competitors and Copilot itself, claiming to things that might or might not be true. And I'm asking you about your experience meeting clients and users about LLM and Copilot in this particular case. Helping new users, new developers, junior developers, in this case on board projects or even shift, but no, onboard projects easily because they have the companion. And Copilot for senior developers, allowing them to get rid of boilerplate code and like routine tasks, and allowing them to, in this case, not so much onboard new projects, because they've got the skill set to do that fast enough, but maybe familiarize themselves with the code base and specially discover new areas. So, use it as a soundboard for new ideas or just an exploratory tool to discover new things. Have you found any users or clients right now that use it in such a way for junior and senior developers?

[0:20:09] AMG: Yes, I think as I mentioned before, it's not replacing anyone, probably anytime sooner. I could see like a subtract in certain daily tasks that are not adding as much value as before. But I will say that you still need to have some notion of what coding is, and be fluent with coding. As a senior developer, you can expect that, if you don't remember the other - I'm just making an example up. But if you're trying to sort out in the most efficient way, there are like many ways of sorting. Write an algorithm for sorting something, like an array of data. You will expect that, maybe instead of just double checking, or refreshing yourself in the documentation, or somewhere else about it, maybe you can just ask Copilot, and it's going to throw at you a couple of different takes on the problem. And you can then decide, as a senior developer, what makes sense for you.

So, there's that. For junior developers, I could see them again, doing the drafting in different solutions for a given problem. I don't think it's going to prevent them from making mistakes. Because again, it's not like a deterministic system at this point. It's just like showing you suggestions based on the context available, and you still need to know what you're doing. So, that's why we keep saying that it's Copilot, and nothing else, is helping you solving problems and coding in this case, but you need to still know what you're doing, basically, and it's important.

I have seen like in the news, in the past, law firms relying on AI or models to scan a PDF, and summarize things, and sometimes you get right. Sometimes it's not you see the person responsible. So, we don't like to post on your behalf any code. You're getting suggestions, you are deciding whether you want to commit it or not, and push it to production. So, I do think it's speeding up the learning process. I was myself playing with an API the other day. I'm not super fluent with Go, for example. And the fact that I have a good notion of what I want to achieve as a developer, but not being fluent in a particular language, but you can read it. It's awesome to see that you can just put together like an API that otherwise you will be maybe spending a few days like getting used to the syntax and whatnot, and you can reason within those new languages or quicker.

Even if you haven't sold for something before, like, you want to do like a small animation or anything curve in CSS, more or less what you want to do, but you don't know exactly what you need to do. Even if you're familiar with CSS as a senior developer, those sorts of moments are magical when they happen, because it will take you hours. Again, this is nothing novel you're creating, but you're just making it more fun and efficient when it works. When it doesn't, we need to be forgiving and giving you the tools to get what you want faster. But yes, we're working on that, I guess.

[0:23:01] JMC: So far in your tenure, leading the design team for Copilot, what have been your biggest challenges so far?

[0:23:07] AMG: I think, every time we try to come up, we like to lead with principles to generate some sort of consistency and alignment on decision making, and how do we approach problems, and every time we come up, I mean, principles are kind of stable. But every time we want to generate some sense of order, in the experience to make it scalable, because we're not just designing one little corner. We are trying to - although the team is small for designers, officially, you can expect that collaborating with other designers in other functions of GitHub. We do need like those principles, those building blocks, and rules defined, so everyone can collaborate and help us spread Copilot and augment Copilot in GitHub. Every time we settle on something, things are so new, that they are changing quite rapidly. 

Even by the time we are comfortable with something, embeddings, dynamic embeddings come up, and it's like, crap, we are happy that we are improving the experience. But we need to change things, and it's not that you're starting over, right? You are always iterating. But the pace, and the need for iteration is maybe more demanding than when you were building on top of a RESTful API that you have known for years. So, it's quite exciting.

[0:24:21] JMC: How does it feel? Because has this been the most challenging product you've ever been part of? I can feel the Copilot team feeling like the GitHub, the cool GitHub team felt when GitHub was released, right? Because when GitHub was released, Git was there and Subversion and other, but GitHub was a one-of-a-kind and probably the first one in terms of providing all the features that we all now take for granted. Collaboration, web access to pull requests, web interface, and so forth, and being able to work remotely and stuff like that. It feels like Copilot is leading the way. There's plenty of competition, because this is a very dynamic market. But yes, is Copilot the most challenging product in which you have ever worked, not because of the company itself, but actually, because it's in a way one of the category creators and the pioneers in the LLM for software engineering space?

[0:25:16] AMG: I think it's one of those moments in the industry that you seem to think that it feels like we are - if I fast forward a few years, I think we are going to get less hyped, and we are going to be more calm. But right now, it's very shiny and very exciting. So, it's definitely one of the exciting moments in tech in a few years. I think the last one, I remember, maybe was the introduction of mobile devices, because it was like a - it's like a new paradigm we needed to design for, and you can argue is just a smaller screen. But it came with new interactions, new accessibility challenges, voice recognition became a theme. So, many new things, right?

I think it's one of those moments where there is so much going on that is new, and it's like a multiplier, where we don't know yet what's going to be possible in a couple of years from now. Not just on the technical side, in terms of like how much data we can fit to the context. But actually, everyone is kind of like doing chats, kind of like the same pattern, same solutions. But apply to a PDF, or to a CSV, or to coding. So, I'm very excited to see what we come up with and what new things are going to come out of things.

It's challenging, because how things are moving so fast, and sometimes very overwhelming how everyone is competing with each other, which is very exciting, but also demanding and consuming. So, I'm looking forward to what happens after the initial wave of hype, and when things get a little bit more settled, and then when we can do with a more calm approach. Because right now, I feel all the industry is rushing into being the first at doing something and there is like a lot of high stakes, I guess. But it's both stressing at times, challenging, but also exciting. I'm super happy to have a chance, I guess, to work on this, because even when we see a lot of competitors of Copilot. It's very refreshing to see even small startups, experimenting and giving us like things to think about. I love that sort of competition, and I think that's a value that startups can bring to the table.

In terms of tooling, the people building the solutions need better tooling. Right now, everyone is Q&A, doing quality assurance and measuring how things are going cost, quality. A lot of things are lacking in tools and everyone is building the same tools internally to validate some of that. So, you can expect and we are seeing startups trying to tackle that market as well.

Infrastructure and GPU costs are still an issue for everyone and you can see why big companies are kind of like associating with a particular company doing models and APIs for it. I rarely want to see how open source and smaller models, cheaper models may open opportunities to make it less expensive and demanding, and not just relying on big corps to offer these sorts of models as well. So, I think there is a lot coming up, I guess.

[0:28:22] JMC: What do you incline for more in this situation in which there's plenty of competitors, increasing number of them, and releasing new stuff. Do you hunker down and put your head down, because also you've got a lot of work? And as you said, GitHub universe, around the corner, you're probably working hard to announce - get things ready to be announced there. So, do you actually rather hunker down and put your hand on your ears and on top of your eyes? How do you filter out all the noise? How do you go about that?

I think I'll do the same as I did before. I've always been curious to know what people are coming up with. The challenge there is to not end up being a copycat of what you see. You want to look out for competitors, or even non-competitors, like just ideas that you need to be trying out like new products every other day. I'm not advocating for just being always up to the latest thing that is launching products. I'm just talking about like being curious to if you're interested in a particular space, even if it's not AI related. Go deep, research, what's available, what you like, what you don't like, taking notes of what is sticking to you, what's not, and slowly building that muscle and that eye for products and solutions that seem to make a good job at certain things and all that mashup of ideas, eventually we'll connect with ideas you're doing on a daily basis. So, we do like to share daily products we find useful, whether it's AI-related or not as a team, and we like to judge what we like and what we don't like about them, and try to exercise that objective critique, of how other products are doing things.

At some point, you incorporate some of those learnings or ideas into your process. But yes, it's challenging to not get distracted with - even not with competitors. Even if OpenAI is working on something that it will enable better retrievals, for example, in a few months from now, you still need to deliver something today and then work towards what's coming up next. So, I wouldn't say we get - we are just in a corner, not thinking about what's happening in the industry, but you also don't want to get distracted. It's a hard balance. But I think we are doing a healthy job by not becoming stressful and mostly just be curious as we as we did before. This has happened also with actions, right? CI/CD was very, very on fire a few years ago, and you have a lot of very specific solutions, although they are not maybe advertising as the single solution for your CI/CD pipelines like Vercel, or like, render, or other people are doing stack specific CI/CD, but it's refreshing, I guess. It's not stressful. It's refreshing for us. 

[0:31:14] JMC: Yes, in a way you just need to curate and filter out. But it's a tough balance to strike, but an achievable one. So, going back to what you mentioned before, which I thought would be challenging for you guys is the surface, right? Copilot can be consumed through many interfaces. So, how does your team keep consistency? How do you go about designing user interfaces that are still providing suggestions that are contextually relevant and helpful within that spectrum of IDEs, but that surface of consumption that is so broad?

[0:31:48] AMG: We do a lot of systemic design thinking. So basically, I mean, we have the design system, primer design system to guide the visual language and interactions and accessibility. But then, within the verticals of GitHub or within our space, in this case Copilot, you do want to start establishing, as I mentioned before, those principles, and building blocks that are going to be modular and extensible enough for people to build on top of, to iterate, to evolve, and sometimes even challenge to expand it or change them.

[0:32:18] JMC: Exactly. My question would be, perhaps Copilot required an expansion of such design system.

[0:32:25] AMG: For sure, because even if you're like a heavy user of VS as an IDE, maybe you do want consistency, and you don't want to be reinventing the wheel when we are providing solutions for the JetBrains extension, or if we implement something in dot com as well. So, there is like definitely internal documentation, discussions, and alignment to not just generate consistency for the sake of consistency, but having an opinion about like, is it worth exposing your references versus not? And what those references can give you?

So, it's not just like the UX of the interface. It's like we are trying to be consistent on how do we tackle, and offer those icebreakers, and offer those experiences in a consistent manner. So, whether you are in your IDE, or jumping back to CLI, or something else, you do have some familiarity and recurrence in how you're operating with Copilot. That obviously, requires a lot of like back and forth internally within the design team and the product people, because everyone has opinions, everything is fresh, new, and you don't have the same assurance as before. If you have tackled a problem before other projects or previous companies, you can be more opinionated about something. But when everything is so new, you don't want to be opinionated and developing an opinion, but you need to be more open-minded to challenging those assumptions more frequently.

I think, basically, leveraging DRIs, we do believe a lot in DRIs as a model to take decisions. Usually, we have what we call EPDs. Basically, engineering, product, and design tend to work together in projects. Usually, we have a DRI per area from engineering, product, and design to facilitate discussion. So, in this case, we do have people in the team leading the overall vision, can be still challenged, and is collaborative. But at the end of the day, we have some decisions to be made and we need like people having ownership for those decisions as well.

[0:34:28] JMC: I think there was a discussion in social media very recently about the cost. Some people claimed that no LLM technology, they were not talking about Copilot in particular, but it was not profitable. That the cost is operating in a deficit right now, and some important stakeholders, most importantly, Nate, the former CEO and founder of GitHub said that that's not the case and so forth. I'm not asking about the financials, but I guess, do you get requests from engineering to implement designing interactions, features, anything that the design team, in order to either reduce the compute power requested, the different requests from users can generate. Do you factor in those? Because it's constantly technology. It's running in very costly, very expensive hardware that right now is actually the axis of which is quite limited. So, is that also the back end of things? Is it something that you guys are considering when you're designing the GitHub Copilot experience?

[0:35:34] AMG: It's a new problem for us. But similar things, right? When you were designing for a RESTful API, maybe the cost is not an issue. But the latency and the retrieval, the error scenarios. So, you do take all this into account, not just edge cases, but how the information flows from the system to the end user.

In that regard, models are not so different. I will say there is like a clear correlation between the cost and the time to get a response, usually. The larger the context, usually, the more expensive and slower the responses are. So, maybe once we establish those building blocks, there is not so much creativity around the UX patterns, but a lot of creativity about did you prefetch? Did you get something in real-time? How did you go about caching? So, we do have more back and forth with engineering, and you do need to learn about that new terminology, and about like how things work. Maybe you're not an expert transpiring or transforming texts into vector databases. But you need to know what embeddings or types of embeddings why they became useful, and incorporate thinking in the solving of those patterns.

The more you understand the system, even if you're not the one causing it yourself, let's say, you do provide better caching opportunities or better strategies to make it feel faster, and also to reduce cost. In this case, it's not like a difficult thing to convince designers about. Because again, usually, something that is more costly tends to be also slower. So, if something takes 30 seconds, you don't contemplate, for example, waiting and getting all the retrieval in one go. But if something takes three seconds, maybe, you don't want to - I'm just making up an example. You may want to show a spinner because it's worth it, and then you get a response instantly, rather than streaming the text and the contents. Then, with that you can post format. You got to retrieval in three seconds. Maybe you can alter the contents and do a pass on the formatting, or how you're going to show up to the user. But if you are streaming something, you may not be able to do formatting of that response, because it's just text-based.

So, there is like a lot of stuff to think about. And it's challenging, but also refreshing. But yes, you need to be I guess, bringing more product designer, more product design than just design to the table. You need to be thinking about the actual product, not just producing yourself to UX, I guess.

[0:38:08] JMC: Yes. It's a broader scope than just UX. I agree.

[0:38:13] AMG: Yes. It can feel overwhelming at times. Because, again, you need to read a lot at the beginning, get familiarized with all these terminologies. Sometimes you may end up like, initially, people onboarding in the team, even engineers may come up with potential ideas, and you can tell who has just joined the team and who hasn't, because of the feasibility of the ideas. But it's still helpful, because sometimes you challenge the status quo, and then you realize that now you can actually achieve something we discarded a few months ago. It's still helpful.

[0:38:46] JMC: I think so, too. It's not ideal to whip the - or should I say smack the back of the head of a new - saying, "You twat. This is unfeasible, by all means." I mean, eventually, you'll need to bring them to the fold and say, "Hey, you know what, the underlying technology will not allow us to do what you proposed." And yet, I find it interesting.

[0:39:07] AMG: It just reminds of like, we also have a team called GitHub next. So, you were asking before, how do you balance the noise, versus delivering something to customers today? Having like R&D within the team, whether it's a separate team, or just fostering that experimentation, it's also important because, as you said, maybe something is not ready for production, but it's in the works. You can start experimenting with it, seeing what's feasible, what's going to be feasible in six months from now. Or even going further and say, "Well, we're doing a great job." But code completion for code bases and files, what else could we do here? Well, maybe we don't have time to ship something anytime soon, but we can start thinking beyond what's possible. So, you need to run in parallel those strings of work, I think.

[0:39:58] JMC: Another thing that I like that is probably beyond the scope of your work, specifically, of design at Copilot. But now that you mentioned research, one of the things that I like, big time, about what the Copilot at large organization is doing is putting our research there. It connects with another controversy. I've mentioned that cost controversy or the profitability controversy. But there's been another one not necessarily specifically related to Copilot. But to actually McKinsey, right? Saying, arguing, claiming that developer experience can be measured, right? Developer performance and so forth. And to a certain extent, I do agree with them. But it turns out that it's complicated, right? There were people chiming in and experts.

But one thing that I'd like about Copilot influencing positively developer experience that the researcher or the Copilot organization has been able to put out there through academic research, is that the qualitative aspects. So, they haven't focused so much on being able to measure quantify developer experience. There's some attempt there by saying that, I think your CEO said that 50% of the code is generated by Copilot and accepted by the user, which is obviously a massive amount of new code being generated really fast, probably faster than users, than human beings. But what I liked most about the research is the qualitative aspects and I can mention a few out there. I mean, there's plenty but I'll mention that most of the Copilot users felt that they were being reinforced to code better, right? That they felt encouraged to discover new technologies, new API documentation. I think you mentioned it before. It is helpful in the sense that it is a guide, right? Like a friend, a pair programmer out there that never gets bored of your questions, never runs out of patience, and will allow you to ask it as many questions and so forth.

There were more qualitative aspects. But I really liked the insights of the research team from Copilot discovered and surfaced. Anything else that caught your attention from them?

[0:42:07] AMG: No. I think the quality part of those insights is an interesting one for me. Quantifying the productivity of a developer can be misleading and reductionist, I feel like, and sometimes can get in these weird positions where developers don't feel safe anymore. So, we need to be very careful with what kind of data we want to disclose and even retain ourselves. Obviously, we are only retaining the data that helps provide better quality. We are actually storing almost nothing in that regard. But I think it's a difficult tension, because CEOs of companies may want to have actual five numbers or vanity metrics of quantifying how many lines of code are being produced. How many hours Copilot is running? Who is running it? But it can get very quickly into the surveillance and the bad parts of software, I will say.

Sometimes it's not even telling the whole story. Sometimes it's like the same people are trying to measure productivity based on lines, and then it happens to be that when you remove lines, it's usually better than when you are adding new lines to the codebase and whatnot. So, we try to - we're like actively thinking about the user space in a very healthy way, I will say, because you don't want to break the trust of developers. And you don't want to give the impression that productivity can be simply measured based on lines of code. Because again, we were chatting about this before, the nature of how collaborative is happier. How many people are involved in those decisions, even if you think about your metrics, when I was working in CI/CD.

In theory, you can measure some sort of productivity based on how many times to commit from development into production. You can try to get those kinds of metrics, but again, they are just numbers, and the numbers are just telling a portion of the story. So, we need to be very careful what we are offering and telling to the end customers. But obviously big enterprises do want to have those numbers to justify the cost and investment.

Anyway, we're trying to provide metrics that could be useful for both of us. You could say, well, acceptance rate could be interesting for us to know how much better we're doing, versus previous iterations, and can also influence or tell a story that people are getting value out of these without spying on users. If that makes sense. We need to be wary.

[0:44:29] JMC: Especially acceptance rate throughout time, right? If in time, the acceptance rate grows, by definition, I think that metric would say, "Hey, this thing is getting better. Or the user is understanding you better."

[0:44:40] AMG: Yes. You can cross that. There is an easy way in the API to know how many different languages are used in different code bases in your organization. So, you can expect that maybe you can cross information what languages are taking most advantage of Copilot, and maybe you're realizing that your company is heavily using Ruby. Maybe you want to also provide learning resources for those such languages, stuff like that. So, there is a lot of value in measuring and providing visibility to some of that data. But we need to find the right data and not just come up with vanity metrics that are making great articles in the news, but can be harmful to the organization or to the end developers.

[0:45:23] JMC: In general, I would argue that the research that GitHub next, on the GitHub research team have been putting out on. The qualitative side says that users of Copilot feel more fulfilled, that their cognitive load of jumping into a new project, or discovering a new code base, or the coding guidelines of any new project is a more feasible task, that it has actually provided more fun to boring tasks. So, in general, the feedback from these users that they feel more fulfilled, and it's brought joy again to the - again, that's the statistically representative, and yet, it is a fundamental part of work, right? This is, after all, a work tool, and I can't see a better outcome than making the life of the work of people more enjoyable, to be honest.

[0:46:14] AMG: We are happy so far with adoption, obviously. But if I think about like ways to improve it, probably you can see, there's like two clear groups using Copilot. There are people getting mad at not getting much value out of Copilot, and sometimes that's just because technical limitations, or sometimes they don't know how to take advantage of it. So, I could see as we know a better job, onboarding and helping them understand how they can extract value out of copilot. But you don't want to be disruptive and being too intrusive. So, we need to strike a good balance there and we're working on that, obviously. But to people that find ways to operate with Copilot on a daily basis, they do seem to be having a good time and taking value out of that, which is probably the best important metric, as you mentioned for us.

[0:47:01] JMC: Exactly. Okay, so without giving away any specific details, can you let us know what to expect? Manage our expectations, not as Copilot users in this case, but as Copilot news consumers. So, where should we expect news coming from in GitHub universe? Again, don't give us a scoop. But let us know where improvements changes, new releases go into effect in the Copilot or the software delivery and development world.

[0:47:30] AMG: I mean, obviously, the team is very excited about the 8th of November with Universe. Basically, the TLDR will be a lot of improvements and a lot of new features that will augment what Copilot can do for you. So, that's very exciting. I'm going to stop there. But you will expect a lot of new stuff to try out some of it at the very beginning, sometimes waitlists, but a lot of new stuff is coming up, and we are also going to be obviously iterating on what's available today to make it even better.

[0:48:01] JMC: Yes. For anyone wondering about this, specifically know that GitHub Universe is an event about all of the GitHub portfolio. Copilot is just one product out there. So, there's plenty of talks about any other things. Now, the good thing about GitHub Universe is that the events team has created a Copilot-specific track stream. I can't remember the actual name, but it allows you to see the agenda of only the Copilot-specific talks, and there's plenty, believe me. I just signed up for that one.

So, if you're interested in Copilot only, just go to GitHub Universe's website and look for the Copilot track, and you will be completely informed about what Adrián just mentioned that it's going to be announced. Although, he's been very secretive about it, and I get it. The legal team is probably is pointing at him with a pistol from behind.

Adrián, thanks so much. If anyone wants to know more about GitHub Copilot, they know the website, they can obviously, they should tune to Universe. But what about you? Where can they find you and ask you questions about potentially designing specific items of Copilot?

[0:49:07] AMG: Yes, basically, available on Twitter, @adrianmg is my handle, or you can just reach me out at adrianmato.com and just find me there.

[0:49:17] JMC: He's, by the way, the author of one of what I consider one of the most brilliant tools. It's called Pew Pew. Can you tell us about GitHub Pew Pew?

[0:49:24] AMG: Yes. It's one of those, I wouldn't say useless, because it's kind of useful for me on a daily basis.

[0:49:32] JMC: I think it's very useful.

[0:49:33] AMG: Yes. It's one of those - I was putting about this. It's quite tight. But I'm kind of like getting tired of a lot of subscriptions for every single tool you use on a daily basis. So, I've been trying to work on building these small tools for myself, and it happens to be useful at times for other people as well. So, it's a tiny CLI tool to manage all these repos you create, just when testing things, and then you end up with a lot of trash in your account. So, try to make it easier for you to delete and clean up your list of repos, I guess.

[0:50:05] JMC: Yes. Well, thanks so much for being with us. I wish you the best. You've got a lot of challenges ahead, because this is so new and so fast-moving, as you said. It must be extremely exciting as a designer, but also challenging. So, all the best. Thanks for being with us.

[0:50:21] AMG: Yes. Thank you so much for having me and a pleasure to talk to you. Thanks.

[END]