EPISODE 1671

[INTRODUCTION]

[00:00:00] ANNOUNCER: Today, you can access Netflix on virtually any device. For a Netflix user, this seamless experience can be easy to take for granted but it requires an enormous engineering effort. Jay Phelps is a Senior Software Engineer at Netflix where he works on Shared Client Foundations. He joins the show to talk about the start of his career, his work at Netflix, and much more. 

This episode is hosted by Josh Goldberg, an independent, full-time open-source developer. Josh works on projects in the TypeScript ecosystem. Most notably, Typescript ESLint, the tooling that enables ESLint and prettier to run on TypeScript code. Josh is also the author of the O'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a live code streamer on Twitch. Find Josh on Bluesky, Mastodon, Twitter, Twitch, YouTube, and .com as JoshuaKGoldberg. 

[INTERVIEW]

[00:01:08] JG: Jay Phelps, welcome to Software Engineering Daily. How's it going?

[00:01:12] JP: Good, Josh. Thanks for having me on. It's just a typical Wednesday for me. 

[00:01:16] JG: Would you like to tell us a little bit about yourself and how you got into coding? 

[00:01:20] JP: Sure. Yeah. My name is Jay Phelps. I'm on the Shared Client Foundations team at Netflix. We have client foundation teams for all the different individual client platforms like mobile. There's one for iOS. One for Android. There's one for web. There's one for TV. And then Shared Client Foundations kind of sits on the side and deals with foundational issues and efforts that apply to all the different client platforms. 

And so, in practice, what that really means is our team kind of does a lot of multi-tier migrations to different technologies. Right now, a big thing is migrating from Falcor which is this internal API library that we have to the more common GraphQL, which are similar. But major changes with that. Then kind of revamping of doing server-driven UI. Stuff that is fairly new to the world and to Netflix in particular. And those sorts of things. And then we deal with other stuff that apply to all the different platforms like quality of experience, logging, general logging initiatives as well. We kind of bridge the gap in a lot of cases between server people and client people. It's kind of what we do. 

As far as how I got into programming, I started programming I think - I mean, I don't know the exact date or the exact age I was. The first piece of software that I can definitely date was a QBASIC program. No. I'm sorry. It was a DarkBASIC program. That DarkBASIC was a language that was based on BASIC but it was for making games. And I made a kind of like a firework show sort of thing for the year 2000. Like that countdown, "10, 9, 8, 7." And then did like fireworks. 

And just to be clear, I was what? Maybe 10 or something? And so, it's like when I say I did it, I mean like I copy and pasted different scripts together to make it work. I have no idea how the particle generation stuff worked. No idea. It's just like copy this code, copy this countdown snippet and figure out how they work together and stuff like that. 

And then I did all sorts of different programming stuff growing up for the local - we had a volunteer movie theater that we had started where movies were $2 and candy was 25 cents and all that stuff. But because it was run by volunteers, every time someone was behind the counter was the first time they've ever been behind that counter. And a lot of times, the volunteers had never worked that sort of job ever. And so counting money - the "drawer" wasn't literally a drawer. But the "drawer" was always either short or tremendously long. And there were - because people didn't know how to count money, like count it back the right way and stuff. And it was a problem. And I was like, "That's a good point. I never really thought about that." But that's a skill that you have to kind of learn. 

And so, I wrote a piece of software that was basically a simple POS. A point-of-sale system to - they enter what they bought and then you click on what they bought. Then you type in what they paid you. You select what they pay you and then it tells you what to give them back. How many dollars, and nickels, and pennies, and et cetera. Made it foolproof. Those are some of my early exposures to doing programming. 

Admittedly, just like the story of a lot of people, I got into programming to make games because I wanted to impress my friends. And then I learned about hacking and I wanted to do that. And some of us got in trouble with - back in the day, Windows 95 used to store passwords unencrypted in a point text file that you could access by putting in a specially crafted floppy disc. You could put a floppy disc in, program would execute and you could copy the passwords to a floppy disc. And so, we got in trouble with that. Didn't get suspended because, basically, the punishment was we're not going to do anything as long as you tell us how to fix it. Got to prevent this from happening. 

I mean, I have so many good memories growing up doing programming stuff. We used to do LAN parties, for those of you who are on the older side of things. Remember, we used to bring our tower computers with our CRT or tube monitors even to another location and then connect them with network cables. Because this was before the internet and then before - or before. Then later, when the internet existed, it still was not a thing. Connecting computers across the internet sort of thing. Not in the early days for home use at least. Those sort of things.

[00:05:26] JG: Sure. It's funny. You were copy and pasting code. You were dealing with point-of-sales. You're basically a full modern developer before the age of 18. It's great. 

[00:05:35] JP: Right. And it was just for fun. I mean, it's interesting enough. I didn't - my dad - I love my dad. But he's a very traditional guy. And so, he actually is one of the reasons I got into programming. Because he came home from some seminar where they had tried to teach him some Visual Basic stuff. And I thought it was very cool and I thought, "That's what my dad does at work," at first. I quickly learned that that's not what he did at work. Not full-time at least. 

But I didn't really have a role model when it came to software engineering. I literally had no one who truly did it for a living. I had no idea you could do it without a degree. Because I don't have a degree. And so, I thought the only thing I had was you have to go to college to be successful. And so, I thought, "Okay. That means I got to go to MIT or something like that if I wanted to do software." And I was more interested in doing music at the time anyway. That's kind of my path right out of high school. 

I didn't want to be a singer or a performer because I saw people go down that route. Instead, I decided I thought I'd be smart. Instead of doing the performing thing, like whether it's art any kind of art sort of thing, you can usually make more money by teaching people or helping them. Helping the people who were attempting to make these things. I thought I'd be smart and go into recording engineering and work at recording studios. 

But, unfortunately, that was right about the time where Guitar Center basically took over and put all the recording studios out of business. And so, that's what happened to me, is I lost my job from the recording studio. And I was in California at the time. I grew up in Nebraska. But I moved to California and lost my job within three months of moving there. And, yeah, then kind of stumbled through. Ended up actually ironically working at Guitar Center, the people who put me out of business. Because I thought I could convince people to hire me contract-wise as a freelance engineer sort of thing and help them. And I did sometimes. But I just got really lucky because one of my customers was a Java programmer. And we just started talking about programming and stuff. And that I knew Java, and PHP, and JavaScript, and C++, and all sorts of stuff. And he was just like, "No offense." This is one of the weirdest things. I'm so glad he did. "He's just like, "No offense. But why are you working here? Why aren't you a programmer?" And I'm like, "Oh, I don't have a degree." And he's like, "Well, I don't have a degree." 

And that was like literally - it's kind of one of those things where you look back and you're like retrospectively like why didn't I ever really seek out the right people to confirm my innate assumptions? But that was kind of a pivotal moment in my career and in my life to be honest.

[00:07:57] JG: It's incredible how misconceptions, biases can seep in in the most unexpected of ways. That the idea that you need a software degree or degree in general to do programming, or software engineering, or some variant of those things is pervasive. But it's false. There are plenty of people, yourself included, that do not have that degree. Either didn't. Or when they started and then later got it. Or just never got it. And that's not a blocker at all for many.

[00:08:22] JP: Yeah. And it's so much easier even now. Today, with the boot camps and all that stuff, that's great. 

[00:08:28] JG: Yeah. I'm curious. If you were to try to learn to code all over again and doing it, say, on the time scale of a year instead of your entire adolescence and early adult years, is there a particular place or strategy you would go to or take? 

[00:08:41] JP: That is a good question and one I thankfully haven't had to think about. But I've been asked similar questions for people who want to get into software. I really think the boot camps are a really phenomenal way to very quickly - I think. I've never been to one. But I've helped at them. I have a little bit of exposure to them and then what other people have said as well. Boot camps seem like they're a really great way of being forced to reckon and decide whether you actually enjoy programming or not. Because I know a lot of people who wanted to get into "programming" and just kind of over the years kind of dabbled a little bit but really didn't go full into it to learn enough to know like whether this is something they really want to do. Whether it's something they enjoy. 

And I think boot camps can be a good way of really sussing that out in somewhat of a short period of time. There's obviously different lengths of boot camps and stuff. Because there's a lot of things that I think people don't realize before they start doing programming. In every industry, you have to continue to learn. You really do. But from what I've - this is going to be such a biased opinion because I'm a software engineer. But from what I've seen, of all the careers, software engineering is probably up there. I don't know if - I doubt it's the number one. But it's up there in the ones where you have to basically almost throw out most of your knowledge every X number of years depending on what your career goals are and where you work. 

I mean, certainly, there are people still writing Fortran today that have written that for 20-plus years. And that is a path you can take. But from my experience, most software engineers do end up relearning or learning brand new technologies, new frameworks, new libraries, new operating systems, new languages. All this new stuff. You really, in my opinion, have to be someone - if you want to advance in your career, I feel like you have to be someone who's okay with constantly learning. 

Realizing, of course, that not everyone can constantly learn for the rest of their life. You also have to think about that. Do you want to shift tracks into management? Or do you want to shift tracks into a type of software engineering where the churn is a lot less, like finance and banking? Things where - or the Fortran example where you can get away with not necessarily having to throw away your knowledge and start all over again to a certain extent. Yeah, I would do that. 

And then, ultimately, if you're a software engineer, you got to learn how to Google. Really, really learn how to Google the right way and know the resources that are best for your platform. Know your Stack Overflow or whatever it is that's good for your particular platform. And getting comfortable with figuring it out on your own. 

I'm not saying don't ask people for help. But at the end of the day, one of the best things you can do is teach yourself to fish, so to speak. To actually solve the problems and work through them. Instead of before you submit that issue ticket on that GitHub saying I have this error. What does it mean? When does it happen? How do I solve it? Getting used to spelunking into people. Searching in their codebase for that error code. Working backwards in the code and figuring out how does it actually work? Why would I ever get here? These are things that I think are very useful not necessarily - obviously, sometimes it's going to be faster just to make an issue ticket. It's a long dividend thing that this pays. Looking into other people's code. Learning about other programming languages that maybe you don't write day-to-day. Being able to debug things. Because, inevitably, you're going to have to debug things. What happens if you submit that issue ticket and no one ever responds? Which is going to happen. Or they don't respond for a very long time. Those are sort of skills that I would encourage people to learn. 

And the other thing is just to expose yourself to - me, early on, I have a friend. I'm still really good friends with him. But we don't live in the same location anymore. But I have a friend, Ben Walters, and we kind of grew up in our programming careers together. We kind of came up together. And having someone - it's not a requirement. But from my experience, having someone who can really kind of challenge you and push you, you see their progress, it encourages you to progress and to kind of challenge them back sort of thing. Kind of friendly competition. Not like I have to be better than them sort of thing. But it's like, "Wow. At the end of the day, we're talking about something." And I'm like, "That's really cool that you learned that. I want to learn that too." Or like sharing - I went down the compiler interest stuff and he went down more cybersecurity interesting stuff and things that I knew nothing about. And so, learning from him and what have you. That's something I would encourage as well.

[00:13:03] JG: Absolutely. That's a really good segue too to what you're doing now on the Shared Client Foundations team at Netflix. You mentioned that you do a lot of longer-term migrations. You work actively with people. Taking them from older used or previously preferred technology, to new or now preferred technology. 

While, yes, folks do have to learn new pieces of tech, new frameworks, whatnot, as a part of that, there are also probably a lot of applicable skills that remain transferable. How do you help people understand what it is they should or shouldn't learn? And then how do you teach them those new things when you're doing these larger migrations? 

[00:13:37] JP: That's a good question. It's the age-old software engineering adage. It depends. But aside from that, examples are one of the things - what I like to do, and some people have different methods of approaching it, but what I like to do is start off with these big migrations by getting some very well-versed people in the old technology. Bringing them on board as part of the initial journey of the new adoption and kind of white-glove helping them through to try and identify the challenges that might arise or the gaps. That's probably the more important thing. The gaps in the new system that we didn't account for. Trying to get that help as early as possible I feel like is very helpful. 

And then I'll go in with that person and kind of dog food. For example, at Netflix, we just finished a migration for TV search to this new paved-path API. Now if we did our jobs right, no one will have noticed anything changed, unfortunately. That's the nature of my teams a lot is that we're somewhat thankless outside the company because the things we do are pretty transparent. 

But, essentially, it is actually technically a complete rewrite of the search TV experience. It looks exactly the same but it is a rewrite and it uses actually a new API using GraphQL as well. And so, that was - I kind of helped - I worked with the search lead on TV and helped with that migration to try and prove out this new system. Because here in this year, we're rolling that new system out to other canvases, to other pages. And that means more people are going to get exposed to it. We kind of trial-run things with a select group of people and then broadly expand it to more use cases, more people. And along the way, you kind of pick up where people are struggling. And you work on documentation as one thing. This is what was before. This is what's today. 

Trying to make sure that things are as self-documenting as possible is one big piece to it as well. Making API is actually intuitive so that people don't even need to reach for the docs ideally. That's not always going to be possible, unfortunately. And, yeah, I think there's a high level for the current project that I'm working on, for example. 

At Netflix, we talk a lot about what's called paved paths. This is an example of that, where Netflix allows you in a lot of cases the freedom to choose which technologies and things you use. But the bigger the company you get - Netflix now is actually not nearly as small as it was when I joined originally. And it becomes a Wild West if everyone needs to is able to do exactly anything they want. You need some way of standardizing and encouraging standardization but not forcing it. And that's kind of what a paved path is for Netflix is it's saying this is the path that a team has paved for you to use. But if you need to take a side road for some reason, you can. You're the informed captain in that decision. You're the person with the best context to know does this paved path solves the problem I needed to solve? And is replacing it with something else going to be better for the company or worse for the company? 

And, of course, that means sometimes people make mistakes. Sometimes people choose not to use the paved path for incorrect reasons or misinformed reasons? It's very rarely a malicious thing. Or it's more of just a natural ignorant thing. Everyone has ignorance and not realize things. By and large, people do stick to the paved paths. And, yeah.

[00:17:16] JG: That's an interesting point, that it's not malicious. Very rarely do developers make malicious decisions. 

[00:17:23] JP: No. Yeah. It's so rare.

[00:17:25] JG: But there are things that are somewhere between malicious and just I didn't know, where folks are very opinionated oftentimes and will insist on their technology like a child with a favorite color. How do you make it so that people aren't, so to speak, convinced that the paved path is wrong because it doesn't go with their technology and then go down a thick bramble-ridden side path that ends up hurting them in the long run? 

[00:17:47] JP: Yeah. I mean, number one is obviously trying to get the paved path to actually work. Something that people want to use. And so, people have no desire to go outside the paved path. And a lot of that has to be batteries included sort of thing. It really depends on what type of paved path we're talking about. But making it just undesirable naturally. Making it very attractive to use that paved path so that not using it is not even really a consideration. And that's a really hard thing to do. It takes time. 

I would say the first couple years of most paved paths I've seen are not successful at that. It's a little bit of a pessimistic view that some people might disagree with. But from what I've seen, there's usually - I would say more than 50% fall in line. But then there's still a large segment of people whose use cases are not adequately - or who they feel are not adequately addressed by the paved path and continue outside of it. But that's a long tail of adoption that just requires iteration. And listening to our users, like the users of the engineers that is not the actual end users. 

[00:18:55] JG: Sure. I imagine you probably are also listening to the end users, for example, to verify that your new paved search is not crashing constantly on me.

[00:19:01] JP: Sure. Oh, yes. Yes. Yes. Yes. That too. Yes, that's a given. Yes. Absolutely.

[00:19:06] JG: I'd love to on this topic, one last little bit, take GraphQL as an example. It's no longer the flashy new thing that everyone's talking about. But it's still for many companies and people flashy, and new, and exciting. If you were trapped in an elevator with me for 20 seconds, 30 seconds and I just sprung the, "Why are you people going to GraphQL, you weirdos?" question, what would your pitch for that be? 

[00:19:26] JP: I'd press the alarm, the panic button, try and get the doors open as quickly as possible. GraphQL is something that's - first off, I don't think it's a good use case for every client-server interaction and application. In fact, a lot of times I would actually say it's not a good use case. It's contrary. Like you should stick to more RESTful sort of things. 

Really where it shines is where you need a unified API that is consumed by multiple different - I don't want to use the word device but it's the closest. Different use cases, I guess. The most obvious ones being, if you have an IOS app, and an Android app, and a TV app, and a web app, they're different use cases. And so, that means they're going to need different data. But they fundamentally still need the same data. I still need my list of movies, and my images for them, and my synopsis. 

But whether we show icons on top of it, that can vary on platform sort of things. That's just a stupid example. But that's really in my opinion where it shines is solving the over-fetching problem and not having these huge, bloated REST APIs. 

Where it's also really, really, really good is if you have for really big companies where you have that problem. Because then you don't have to deal with - if you have that problem and it's a really big company, you can very, very quickly just have these ginormous rest end-points that are basically solving nothing. I mean, solving everything and then solve nothing because they are just ridiculous and overburdened. 

But in my experience, when I've worked at companies that weren't Netflix, I actually think it's probably faster, easier, and more maintainable to just use a standard straight, boring REST API. There's a little bit of exception to that. And tooling has gotten a lot better on the GraphQL stuff. And if you're already very much familiar with GraphQL, already familiar with how to write like resolvers, like you've written Apollo server resolvers a million times, if you can spin something up very quickly, I'm not going to push back on you on that. 

But where I think people get into trouble is picking up too many new technologies, "Oh, that's actually a really good way to describe it." One of the biggest things I'd ask you is I think you should have a new technology budget essentially. When you're deciding a new project - and it doesn't have to be a concrete thing. I just mean that you need to be very, very cognizant of everything you take on that you've never used before is a huge risk that's going to take you more time than you think it's going to take you to adopt. 

And so, on the flip side, I can give you an example of the flip side which I did consulting at this - I co-founded This Dot with Tracey. And so, we did a lot of consulting. And she still does it today. And I can't tell you how many projects I saw that were simple crud apps that were adopting GraphQL, TypeScript, ImmutableJS, RxJS, Redux, you name it. All the latest coolest things with sagas and all this stuff and all because it's super cool and it's super fun. But at the end of the day, we were basically digging them out of the hole. Being paid to dig them out of the hole they dug themselves in technology debt, and over-adoption of new technologies, and keep it simple. 

That being said, there is something to be said about the fun it comes from using a new technology. And so, when I've been a manager, for example, I would never like veto someone saying, "We don't need GraphQL. But we want to use GraphQL because it's fun. And this is how much time it's going to take us more. And here some of the potential benefits that we might reap from it." Schema introspection tooling, and discoverability and ability to onboard new platforms if our company goes in that direction. They're not things that necessarily will pay dividends. But maybe they will. 

But if you acknowledge part of it is for the fun of it and for the challenge of it, I think that's a really important thing. I'm not speaking for all managers. But for me who's been both an engineer and a manager before, I think there's a lot to that.

[00:23:25] JG: Absolutely.

[00:23:26] JP: Keeping the engineer sharp. And like I said, things can pay dividends. Even if you don't necessarily need GraphQL, it can pay dividends.

[00:23:35] JG: Yeah. I think you touched on two really good aspects of the concept of new technology budget. One is that if a team is taking on too many new things at once, that becomes an unmaintainable nightmare. And oftentimes they have to be bailed out or bail themselves out later. But also, from a teaching perspective, the pedagogical, side human beings aren't made to learn many, many, many new things at once. It's much easier to learn one new thing at a time. 

If you have to introduce stuff to a team, it becomes more difficult when you're introducing several large new concepts for them at once, right? 

[00:24:05] JP: Right. 100%. I wholeheartedly agree. And because the other problem to it is the interaction between all of these systems. The more new systems you use, the more you have to kind of pave the way on how these systems are even supposed to interact. 

The ImmutableJS, and Redux, and RxJS sort of examples. There's weird TypeScript - oh, and that's the other thing. Adding TypesSript to the mix. There's weird TypeScript things that you run into and you search StackOverflow and no one has ever asked this question. Because you're blazing the trail for StackOverflow questions essentially. Just being careful about that I think is important. I'm not saying don't use these things. Just being careful.

[00:24:44] JG: You've previously said variations of the very interesting quote, "The problems you think are complicated often are not. While the problems you think should be simple can be impossibly complex when referring to big company problems." Could you dive into that a little bit and tell us what you mean by that? 

[00:24:59] JP: Yeah. I mean, unfortunately, the juiciest examples I actually can't talk about unfortunately just because that's big company juicy problems. But, I mean, very generally, I think people overestimate the amount of effort that goes into some things at these companies and then underestimate the things. 

As an example, how much internal tools and logging exist at Netflix is just crazy. And most really big companies. Really, really big companies. We have probably more software that runs internally than we do runs externally. Probably by a large margin to be honest. 

There's just so much that goes into making sure that people are productive, and things are efficient, and things stay up, and people are able to - yeah, be productive is honestly the number one. Most of the tooling is around people being productive. Whether it's debugging things, or being a call, or release software stuff. That stuff is just super complicated in my opinion. Needing to deal with - logging is a great example. It is really hard to log as much data as we log, and ingest it, and process it, and store it, and filter it in an efficient way. And the tradeoffs that come with all of this. Because you have to trade off flexibility to cost, and storage, and speed. And making those tradeoffs is pretty challenging.

[00:26:23] JG: One of the problems you've referred to being solved or being worked on by Netflix is server-driven UI. Being able to make the client or the shared clients rather flexible based on what the server needs. Can you dive into that this time now? Telling us what that term means and how it's used? 

[00:26:38] JP: Absolutely, for some people, it's hard to wrap your head around what server-driven UI is. I'll very generally explain it and then I'll get some concrete examples. The general idea is that most client applications, they ask the server what to display. Not how to display it. Not where to display it or what it should look like. And I mean that very generally. 

And so, what server-driven UI is it's moving those decisions to a certain extent to the server. The server can decide how something should be displayed. Not just what should be displayed. Not what content. And there's many levels of that. There's levels of granularity. There's really, really coarse-grained examples where you're just saying - the client saying, "Hey, if you want, I can put this type of row here and then I can put this type of row here, or here, or here. Tell me which road to put here." That's a very coarse-grained server-driven UI. 

Then there's like super, super fine-grained server-driven UI where you're saying, "I want this text to be on this button. I want it to be red. I want you to left align it and put it at the bottom of this box and then put another box on the left of that as another column," et cetera, et cetera. And there's levels between that of granularity. 

And if you're thinking about this, there's the - really, browsers are basically the OG server-driven UI system. If you ignore JavaScript for a second, the server is quite literally telling you, "Hey, here's HTML. Here's CSS with CSS. Here's not just what you should render but how it should look. Where it should be to a very granular extent. You have divs that can completely be all sorts of wild-styled. 

If you think of that, the goal at Netflix is never to build a browser like that. I mean, the performance of that is not what you're looking for. And you don't need that level of flexibility. But Netflix actually has two server-driven UI systems. We've got one called CLCS that's used for the non-member side of the system. And it is a much more granular system. It doesn't go as far as like a browser-level of granularity. But it is able to go like I've got a certain type of button. We've got predefined button styles. And it can choose between them and it can choose to place them in certain places on the page and so on. And it's a form-driven system. Because when you're on non-member when you're signing up and those sort of flows, they're very much forms anyway. It's a form-driven system. 

Then we have a second system called DEPP, discovery experience paved path, which is what I'm working on. And it is a server-driven UI system for the discovery experience, which is discovery at Netflix is a term that means discovering content of any form. Whether it's TVs, movies, games, articles, whatever it is. For the most part, it's not account settings and it's not playback. But even the playback, there are cases where we do - we have pre-play and post-play. And if you have ads, we show you stuff during, end. Those are discovery kind of canvases. And this server-driven system is being designed for those. 

We've kind of always done kind of a light version of server-driven UI for quite some time like in the sense of we choose what type of row to show you. Whether it's continue watching, or the most popular, or top 10. That's all algo-driven today and has been historically for many years. 

 We're kind of taking it a step further. We're building a system where we can be more flexible and choose putting toys next to a game or whatever. Not that we're actually doing that but that's an example that of something the system could do if we want to go do that someday. It really - and that's kind of the big key of the system is there are some immediate use cases. But it's really the system is being built to open up future use cases. Use cases that the company. Because the company has recently got into licensing toys, and making games, or licensing games, and doing live streaming. And we even have something called Tudum, which is kind of - it's kind of a gossip news website for Netflix content sort of thing is kind of how I would explain it. I'm probably not explaining it well. But that's how I look at it. And a hit. But, of course, not everyone knows about it. Not everyone uses it. 

But being able to expose more content in different types of content more flexibly and let our algorithms drive that is kind of the big goal. And just giving the business a system that we can then build on for the next 10-plus years and whichever direction we want to go in.

[00:31:14] JG: Just to be clear, the name of the system is Tadum, which refers to the sound you get when you open Netflix, the app.

[00:31:23] JP: Tudum. Yep. Exactly. That's exactly - people internally call it Tudum, usually. But, occasionally, people will call it Tudum. I feel like it's because - I don't know this. I don't have the history behind it but I assume it's because saying Tudum feels weird. 

For me, I'd rather say Tudoom. It feels more like a nice word. Whereas to Tudum - maybe my brain is because of the word dumb is in there. It just doesn't like saying that as part of - I don't know. I'm just trying to think subconsciously why people say Tudum internally. But Tudum, Tudoom. 

[00:31:55] JG: There's a certain gravitas with Tudoom that's quite nice. Out of curiosity, you've described how the two systems, member and non-member, differ. One is more granular. Is there a particular reason why you wouldn't be able to unify the two to make the level of, say, granularity configurable? 

[00:32:12] JP: Great question. And it's something that we definitely looked into and will continue to collaborate on. There are some systems that actually are reused between them. It's not like a top-to-the-bottom non-reused. But the number one thing is that, by and large, non-member and member experiences with common exception, but by and large, they're kind of focused on very different things. 

Once you're a member, we're kind of more focused on primarily just finding you the best possible content and best that you can watch, or view, or what have you. Whereas like non-member, obviously, it's to get you to sign up or what have you. And so, because of that, we have kind of conflicting requirements. We don't need the level of granularity required by the non-member like CLCS system. And so, that comes at a cost though if you have that kind of granularity. That means that the client has to support that granularity to a certain extent. That means the backends have to be aware of that granularity and like have recipes that are able to build higher-level components from those lower-level components. 

And also, just use cases. For example, I talked about that non-member is more form-driven. Whereas the member side is not. And so, there's assumptions baked into their system that are kind of incompatible with what we needed. And these are all solvable problems. You throw enough engineering time, and effort, and all this stuff. These are all solvable problems. 

But in true Netflix fashion, we kind of informed and decided that it was better for us at least in the short term to go a different way and build a different system for velocity's sake and not try and build this big monolithic system that burdens everything. It also has some additional benefits. There's redundancies. Meaning if there's a bug that's introduced in one system, it usually doesn't impact the other. 

And so, if signups go down because of a bug in that system, it's not going to impact your discovery experience if you're already a member and vice versa. That wasn't necessarily like the reason or something like that. But there's some niceties to having a segregated system.

[00:34:16] JG: Sure. There are two interesting different almost opposite decisions you've alluded to. Both of which sound very reasonable on their own. One is that, as you said, granularity comes at a cost. The two different systems are tailored to their two different use cases. But then you've also previously referred to how the member system is being built for future use cases to expand in scope even though it's not yet necessarily needed. Such as the games or toys is the example that you gave. Not necessarily what will happen. Just want to be very clear. 

[00:34:45] JP: Games definitely has happened. But, yeah. Whether we put a toy there. I don't know. Just threw it out as an example.

[00:34:53] JG: The AB experiments will determine that. But how do you make that decision then of knowing which tech debt to take on versus which future tech debt to avoid? 

[00:35:02] JP: I mean, it's hard to objectively say how you make the decision other than just we make it through discussions and an informed captain making the ultimate decision and the call on whether reuse the system, or we build something else, or do a kind of a partial of both, which is kind of what we went with is there's some systems that are reused and some things that are reused and some things aren't. 

At the end of the day, we designed this new system to hopefully be in a way where we can get more granular over time and kind of incrementally adopt more granular features only when we need it. For particular canvases and particular rows, or sections, or what have you, we can get more granular than we have in other use cases and not have to build some new system or something like that. That's kind of the idea behind it is we didn't have the budget both in time, and finances, and desire to do some really big dramatic re-architecture for the clients. Because anything you change on the server-driven system, the clients have to support that in some way. Or you have to do it in a backwards-compatible way. 

And so, there's always migrations, and rewrites, and all sorts of stuff. But we can't just say all discovery canvases have to be rewritten to this new granular system when have no use case for it. We don't need to be able to say put this button on this form in this place. And we go to the next form page, do this, and et cetera. We just don't have that use case today on the member side.

[00:36:31] JG: Sure. before we jump into the final fun parts of the interview, I want to apply some of these same lines of thought that we've been talking about large-scale architectures around to lower-level language primitives. You've previously been a member of the RxJS team, which is a very popular library in JavaScript. You're excited about things at the language level. How does that impact your work as a Shared Client Foundations engineer to enable and help make more productive your fellow developers? 

[00:36:58] JP: I think one of the biggest things I had to learn very quickly is restraint. And what I mean by that is I love RxJS. I love WebAssembly. I love all sorts of technologies that are very, very powerful but also very, very easily abused and very niche. 

And so, by restraint I mean - ironically, more people ask me at work, "Hey, should we be using RxJS. Or should we use WebAssembly?" And I'm usually the first person to say no. And it's not that they don't have uses. It's not that I don't love them or have desire. It's just that I've kind of learned the trade-offs so well and seen them used and abused so well that I have to be that voice of reason. And so, I think that's one piece of that that helps me inform and be a better foundational engineer. 

I mean, at the end of the day, when you work in foundations, and platform, and that sort of stuff, it's really also easy to get disconnected from reality of your consumer - your developers that are using the things you're building. You see it all the time. And I've experienced it even myself like both as a producer of foundation stuff and as a consumer of foundation stuff that people - it's really easy to lose touch with the latest technologies and the actual problems. And so, it's a good idea to plug yourself. 

That's why I think partially, A, dog fooding is really important. Never throwing things over the wall. Never saying, "Here's the solution. Go use it." Instead being like, "I think I have a solution. I'm going to go use it and confirm that it actually does do what I do." And not just prototyping real migrations or real adoption. And iterating. Just continuing to iterate and learn from those real-world experiences. 

Then, also, when you can, I try and plug myself into either projects or at the very minimum discussions on tests that are being run that I don't necessarily need to know about but I want to know about because I want to learn what's going on. What are the problems people are running into? This is kind of a nerdy, weird thing that other people were like, "What?" I really like logging-related challenges. And so, a lot of the stuff I kind of volunteer to work on is logging-related over the years. And so, making logging easy. Because logging is - it sounds like such a simple problem. And it is if like all your logging is like Google analytics. But for these big companies, logging is a huge pain in the butt for both clients, the ingestion, and the consumers, the data science folks. 

There's a lot of problems I could talk about at length about trying to solve. And some of this server-driven stuff is actually enabling us to investigate and experiment with solving those problems in novel ways. Ways I've never seen before. Server-driven logging. Not the server actually logging. But the server telling the client what to log and when to log it, which is kind of a weird kind of thing to wrap your head around. But has benefits and cons. And there's different levels of that again. But those are kind of things that I've been kind of investigating. 

[00:40:00] JG: That last bit is fascinating. I, in the past, have been on teams where we've investigated very, very primitive versions of that. Or let's say as soon as we see that the client has had an error or some uncaught or caught exception, we then start logging much more. But having the server control, that could really, really give you much more precise usefulness per byte sense for your logs. That's fascinating.

[00:40:24] JP: And these are mostly usage logs like in the sense of user interaction logs. Like user sees a thing. User clicks on a thing. Those sorts of things. And where they're on this page. What have you? And one of the biggest challenges we have - I don't want to get too boring. Because I know people don't like logging mostly. But one of the biggest challenges we have is just the scale that we operate at and how many clients there are. How many different pages there are and the evolution of those pages? 

We have a lot of inconsistency between logging of things. On one page you log it this way. And on another page, you log it a different way. On another client, on iOS you log it one way. On a TV, it logs at a different way. And that inconsistency is very costly to lots of different systems. Both in dollar amount but also in time, and bugs, and all sorts of things. 

And then just the introduction of new things. Data science very often is like, "Hey, we want to introduce some new field that needs to be logged when something is presented." And it seems like such a simple thing. But in reality, in a complex codebase, it can be really difficult to get that new piece of data from point A all the way to point Z to where it needs to be logged. Threading it through your system can be - especially on some of our pages, on some of our devices were written - initially written at least seven years ago. And they've just been iterated on since then. 

And a big company with how many tests we run, that means that the code is pretty gnarly. And there's exceptions to that, obviously. Trying to solve that. Or at least not solve it but mitigate it and make it a better experience is one of the things that I also work on.

[00:41:57] JG: Beautiful. Let's talk about a variety of topics towards the end. Can you define and explain what back pressure is? The concept of programming language design? 

[00:42:06] JP: Yes. I have a Medium article that elaborates on this and it also links to a talk if you'd rather watch that basically explains the same thing than a talk forum. But, essentially, back pressure is a term that's borrowed from fluid dynamics, which is a fancy word for me. Fluid, physics stuff. Plumbing people, they know all about back pressure. Car people, a lot of car people know about back pressure. But, essentially, it's an analogy that is, in software, it's the resistance of data flowing through your application in some way. And that means data that comes into your application and then goes out of your application. Somehow, it's coming in and then somehow that is resisting going out. 

Usually, that could be memory. Usually, that's memory or CPU-bound or disk-bound. Or it could be all sorts of different reasons. But the quintessential example that people give is reading and writing from the disk. If you were trying to - most file systems can be read faster than they can be written to. that's usually a disk limitation. 

And so, if you naively just read as fast as you can from the disk and then write - if you wanted to copy that file and you read as fast as you can and then tried to write as fast as you can, you'd actually have to buffer. You'd create these big, big deficits. And if the file is really large, the buffer could grow to the point of being larger than the available memory. 

And even if it's not, think about that in a server standpoint. Just because, "Oh, I have 16-gig of memory, I have plenty of memory, it only takes 500 megabyte -" I only have to buffer up to 500 megabytes before I can catch up and start accumulating. That's a killer. That cannot happen. 

Even go smaller. Let's say two or five megabytes. If you're talking thousands of requests per second trying to do these copies because this is a web server, you could very easily use a lot more memory than you have available. Or even if you don't, you have to scale things up and be paying more money for AWS resources, or Azure, or whatever. 

And so, there's basically three ideas to back pressure. Or three strategies for back pressure. The most ideal usually, but not always, is to control the back pressure when possible. And in the disk case, that's a perfect example. You can choose to only read from the disk as fast as you can write from the disk. That way you don't need to buffer anything at all. You read then you write to the disk. And then you have no buffering problem whatsoever. 
Unfortunately, that's not always possible in like - for the disk, it is. But for a flip side example is a user is - whatever you're dealing with user input. A keyboard. Try as we might, we can't tell and enforce that the user stops typing so fast. And I think if you're a UI person, you've probably at least once in your life written an application where you put an [inaudible 00:44:55] to an input box. And then based on that input, you then performed some operation directly. And you noticed that the UI was super sluggish and super slow when someone would type fast. And so, how do you solve that? That's a type of back pressure that's push-based where basically you cannot control the source. The source is going to push an event to you and you have to do something with it. 

The strategy in that case is either doing some sort of debouncing or throttling. Essentially, you sample in some way. Or you drop. Or you could buffer in theory. You could say I'm going to accumulate all these and then do them all once I have the result. That's another strategy. But for the keyboard solution, you wouldn't do that. You would do some sort of sampling. You would wait until they're done typing or you would sample every end number of milliseconds and then make the request or do whatever it is. 

And so, where you might buffer is an example of server-to-server communication. Usually, it's very important that you - when you have lots of microservices and you're making server-to-server communication, like GRPC or whatever you're using, it's usually important that you don't drop that request. It's usually pretty important. 

And so, usually, you have to lean on buffering in that sort of scenario when you have back pressure. And so, back pressure in that case might be you have server A that's making N-number of requests to server B. And then server B is supposed to make request to server C. But server B takes longer than what's coming. It takes longer for it to do its work than server A is sending it. Server A is sending more requests than B can handle. And, eventually, B is going to buffer, buffer, buffer, buffer, buffer and then fall over. 

And so, in that scenario, you would kind of combine strategies. You would say, "Ideally, I use some sort of pull-based back pressure system between the -" like you say, "Server A, hang on to your request and until I tell you that I'm ready for it." And then when you tell, you say, "I'm ready for the request." "Okay. I'm ready." It processes it and moves on. 

And the reason why you don't buffer on your own, you buffer at the source, is if the buffer falls over - if that server falls over from having too much of a buffer, you don't impact yourself. You impact the requester. And so, think about a vastly distributed system where B in that scenario is depended on by not just A but all sorts of other systems as well. Now it can stay resilient by basically only - instead of everyone going down that depends on B, only A now goes down. A is overwhelmed. And that sucks. But X, Y, Z that also depend on B continue to function. 

And it's hard to - when we start talking about A's, and B's, and graphs, and stuff over audio, it's obviously really hard to visualize that. But some primitive examples are in the Medium article. But the idea being is that back pressure is a fact of life. And almost every developer will encounter it even if they don't don't call it back pressure. And those three strategies, controlling the producer, buffering, or sampling are essentially how it boils down to how you handle it. Or a combination of all three.

[00:48:02] JG: That's a beautiful explanation. Thank you.

[00:48:04] JP: Oh, thanks. Yeah. 

[00:48:05] JG: Let's talk about typing on keys again. What is your favorite piano piece to play? 

[00:48:10] JP: That's a good question. My actual favorite is actually really simple. But for whatever reason, I really, really enjoy - it's really simple. It's called Vincent (Stary, Stary Night) by Don McLean. It's "Stary, stary night, paint your palette blue and gray." I don't know if - it's an older song. Many folks probably haven't heard of it before. But it's a really fun song. And so, I really enjoy playing it. It's one of the first songs that I typically will whip out just because I like how it plays. 

And then I like playing songs that people recognize and think are funny, like A Thousand Miles, Vanessa Carlton, "Making my way downtown, walking fast". Like [singing the song inaudible 00:48:47]. Yeah, exactly. It's a real crowd-pleaser. Everyone loves that one. 

[00:48:56] JG: How often do you get a chance to play the piano? 

[00:48:58] JP: I have a digital 88-weighted keyboard sitting next to me. but I don't really play it that often these days. It's something I usually do on vacations and holidays. And if I see a piano on the street, which is how you know I know how to play piano is because we ran into one in Romania, of all places. There was one right outside our hotel. And so, yeah. 

[00:49:18] JG: I want to give a shout-out to that conference, by the way. Shout out to RevoJS, the conference in Romania. Timisoara. Wonderful, wonderful time. Excellent conference. And the hotel had a piano that I walked by and saw Jay Phelps playing beautifully with Ben Lesh just sitting there on his phone rudely. 

[00:49:35] JP: Yep. That sounds like a typical conference for him and I. But, yeah. It was a great conference. I really enjoyed it. But I don't get to play. I mean, it's somewhat of a perishable skill. The biggest thing is remembering - I mean, I played trumpet, and played piano, and then played like a little bit of clarinet and stuff growing up. And trumpet and piano are in different scales. Reading the music, it's hard to like remember going back. Actually, reading music is actually difficult for me because I'm like, "Is that - which note is that? In which scale?" Yeah, I don't get to play as much as I used to. I'd like to. 

I mean, I would love to get - my goal is to have a nice, big piano someday and have more time to play and do more of those interesting things. But at this moment, that's - especially with my little dog, which you all can't see. But he's sitting on my lap right now. He's adorable. I have a lot of responsibilities because he's only 11 months old still. He's still a pain in the butt. 

[00:50:32] JG: He's very cute though, for our audio listeners. 

[00:50:36] JP: Yes. He is. He is. 

[00:50:38] JG: I personally got back into piano after learning it as a kid when I was a teenager. Because there was a beautiful advertising for Halo 3, the believe line, that used the Venetian Boat Song. I believe the piece is called To Great Effect. To underscore this very, very somber scene. And Halo actually is a great example of a universe that is not very well explored in the main series. But the extended universe of Halo is fascinating stories, and wonderful characters, and thoughts, and really much deeper, darker content than the main areas. Now this is relevant to you because you're a big Star Wars fan. And I'd like to you to comment on a couple of Star Wars things if you're okay with that. 

[00:51:14] JP: Sure. Yeah. By all means. For those who - I mean, you can't see it. But behind me is a row of Star Wars helmets on the wall. Not a row. I mean, many rows of Star Wars helmets. I don't remember how many I have. I don't know. 30 something. Or something. 25. 

[00:51:30] JG: There are several dozen visible. Darth Jar Jar? What is it, that theory? And why would it redeem the prequel trilogy? 

[00:51:36] JP: Oh, great question. For those who don't know, there's a YouTube theory. Several videos but there's a couple that are more famous than others that Jar Jar Binks is actually either Darth Plagueis or just some other Sith Lord that we don't know about. And he's actually the one who was the puppet master all along. And the videos are actually pretty convincing. If you watch the videos, they explain like - for example, he uses - in the battlefield scene, when he's fighting the - and winning against all these battle droids that are supposed to be amazing, he uses a technique called drunken boxing, which is a real, known, legitimate fighting tactic and fighting style. 

And if you watch the videos side-by-side of examples of drunken boxing and Jar Jar, he's like literally doing all the best - all the moves exactly how you're supposed to do them. And there are several scenes in which he gets what he wants in circumstances he should not at all get what he wants. And you can see him in the background waving his little hand using Jedi mind tricks to convince people to do things. 

And it redeems the prequel trilogy just because of how much I hate Jar Jar. I mean, as I've gotten older, I've been like, "Okay, whatever. I'm passed grief and I'm accepting now. But I feel like it would make - it would be a lot more of, "Oh, surprise," than like, "Oh, Palpatine somehow returned." That was a very like, "Oh, really? That's the best we can come up with? Palpatine has somehow returned? Okay." 

[00:53:14] JG: That quote is a meme now.

[00:53:15] JP: Is it? 

[00:53:16] JG: Yeah. Somehow X. Yeah.

[00:53:19] JP: Somehow. Yes.

[00:53:20] JG: All right. Second and final question for you on the Star Wars subject. Is there a particular piece of lore that is not well-known to the general public that you would like to make well-known? 

[00:53:31] JP: In Star Wars? Oh, that's a hard thing to know because I don't know what's really well-known in the lore. I mean, if we're talking like canon stuff, I would really encourage people to watch Andor on Disney+. It's a terrible, terrible name for the television show. Literally, it means nothing to anyone - almost anyone unless you're really, really obsessed with Star Wars. It means you're like, "Who's Andor? I don't care. Goodbye." 

Really, it should have been I don't know. I don't make up good titles. But should have - to describe the actual show, it could have been like the true cost of a rebellion. Or the darkness of a rebellion. Or something. Because the show is gritty for a Star Wars show. I mean, it's not like blood and gore and like World War II documentary crazy stuff. But it shows a side of the Star Wars universe that has never even been eluded towards. 

And all the other content, the rebellion is good, the empire is bad. And that's it. There's no if, ands, buts, middles, or anything like that. And Andor completely flips that on its head and shows you the dark side, pun intended, of a true rebellion. Rebellions are built on terrible, terrible things that people do, and live with, and have to live with. And I'm not saying I'm advocating for rebellions. But point being is that these people do things that are despicable. And that's the nature of rebellions. 

In the flip side, there are people who are so disconnected from the reality. I mean, look at Russia. The propaganda in Russia disconnected from the outside. I'm sure America - I mean, America, 100%, is propaganda too. I'm not saying that like we're immune to propaganda. I'm just saying that like from what I've seen, it seems to a very fairly extreme degree relatively speaking. I mean, I'm sure it's worse in other places. But it kind of shows that side of the empire as well that people don't necessarily - they see that their shoplifting crime has now been changed to a six-year sentence because of the rebellion did something. I hate the rebellion then because it affected me. And I have no idea why like the rebel - I'm off in some planet in the middle of nowhere. The empire is a different problem. Like a problem that doesn't really impact me. But now it's impacting me like in a negative way. Not me, obviously. I'm saying an example from the show. 

I feel like watch the first episode. The first episode really sets the tone. If you hate the first episode, you won't like the rest. But I highly recommend it. I also recommend it for people who aren't even Star Wars fans. I think you could really enjoy it and not be a Star Wars fan. I watched it with my girlfriend who was not a Star Wars fan and she was very skeptical at first and very quickly became enthralled in it and didn't obviously get the little like nuances and Easter eggs or what have you of it. But it's a very different show than the rest of the Star Wars shows. I love it.

[00:56:34] JG: Someone would yell at us in the internet if we ended this interview on a show on a platform other than Netflix. Just for crossing our t's, dotting our i's, is there a particular Netflix show that you would like to promote or just say that you are a fan of personally? 

[00:56:48] JP: There's so many shows. I will go a little raw or a little vulnerable and describe that I am somewhat of a trash TV buff. I really like crappy, reality television The Ultimatum. I love cringe. I love being like, "Oh, my God. I'm in so much pain because of how cringeful. I can't believe this person. I can't watch this. I have to watch this." Yeah, I'll binge-watch Ancient Aliens one week and then the next week I'll be watching Love is Blind or something. I feel like, definitely, there's something that prior to getting into it I would have been like, "Ha-ha-ha-ha-ha. Yeah, right." I was surprised at how much I got into it. 

[00:57:31] JG: The Great Werner Herzog, the very prolific, very highly-esteemed filmmaker, on a separate podcast, the Conan O'Brien Needs A Friend Podcast, admitted very happily that he is a big fan of Here Comes Honey Boo Boo. Highly recommend. I think that it is not at all shameful to watch these shows. It is its own art form. Its own form of entertainment.

[00:57:53] JP: Yes. 

[00:57:53] JG: Great. Thank you so much, Jay. This has been an absolute pleasure. Final question, is there any particular way that you'd like people to find you on the internet? 

[00:58:01] JP: Sure. Really, the almost only way. Because I'm not really on anything but Twitter. And you can find me at _jayphelps on Twitter. Or you just say hi to me if I run into you at a conference.

[00:58:17] JG: I hope that happens. Anyway, thank you again, Jay. This has been a pleasure. And this is Josh Goldberg, Software Engineering Daily. Cheers, y'all. 

[00:58:24] JP: Thank you, Josh.

[END]

SED 1671		Transcript

	(c) 2024 Software Engineering Daily	1