EPISODE 1770

[INTRO]

[0:00:00] ANNOUNCER: Gil Tayar is a principal software engineer at Microsoft, developer advocate, and conference speaker. Gil's contributions to the Node.js ecosystem include adding support for ECMAScript modules in Node.js, Tomoka, and testdouble. He joins the show to talk about his history in software engineering, MonoRepos vs. PolyRepos, the state of JavaScript, and more.

This episode is hosted by Josh Goldberg, an independent full-time open source developer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ESLint, the tooling that enables ESLint and Prettier to run on TypeScript code. Josh is also the author of the O'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a live code streamer on Twitch.
Find Josh on Bluesky, Mastodon, Twitter, Twitch, YouTube, and .com as Joshua K. Goldberg.

[EPISODE]

[0:01:06] JG: Gil, welcome to Software Engineering Daily. How's it going?

[0:01:07] GT: Good, thank you, Josh. Thank you for having me.

[0:01:10] JG: I'm very excited about this. You've been around in the industry for a little while. You've had your hands in quite a few areas like at Microsoft and Wix and Node.js. But could you tell us just to start, how did you get into the wonderful world of software development?

[0:01:23] GT: Ah, well, the age of 13, I'm Jewish, so Bar Mitzvah, and I asked my parents for a PC. At that time, it was the ultimatum and PCs was a ZX81 with 1K of RAM, which I upgraded to 16K and I learned basic that way. From there on to Turbo Pascal on an Apple II where I worked on the wonderful IDE by Anders Hejlsberg who is still one of the luminaries of the software industry. Now, we're both working in the same company, Microsoft.

Then the army. I joined the army as a software engineer. I did the software engineering course there, it's like a nine-month, very boot camp kind of thing, and spent six years as an instructor in the school of software engineering. So, I continued being an instructor. From there on, like easily got into the industry, companies like Magic, which nobody remembers today, but definitely Wix. I even started my own company in 2000, the height of the dot com era, about a year before everything crashed down. But we survived and the company was sold for bits. Then, on Wix culture and today my role as a software engineering at Microsoft. So, 35 years. Yes. More or less.

[0:02:43] JG: I believe I remember at a conference, someone mentioned that you were their instructor in the army. How does it feel that people you've taught to code are now also speaking at conferences and movers and shakers in the industry?

[0:02:53] GT: Yes, I taught in the army, Tali Barak, a wonderful speaker also. I taught a lot of people and mentored a lot of people, not only in the army, that's really, really old news. But also, I did a boot camp once, front end, so taught like 10 people and we still get together sometimes and then reminisce. I mentor a lot of juniors trying to figure out how they can enter because at least in Israel, juniors have a really, really hard time getting into the industry. So, I help them along. My nephew and my niece came to the software industry with a little bit of help from me. So yes, a lot of people and it's fun. I like mentoring and helping people along whether they're junior or senior. In a way, it's what I do. It's what I like.

[0:03:40] JG: How do you stay in the same mindset as them? How are you able to relate to these people who are coming in with such a different set of technologies and people in places around them?

[0:03:50] GT: I mean, I don't program in COBOL anymore. Well, I never did, but let's say. I keep abreast of all the trends and everything because this is what I love to do. I don't do that because I need to. I do that because for me, it's fascinating. I went through a lot of revolutions with the ZX81, so the PC revolution. I still remember in the army, I wanted to go work on PCs and not on mainframes and minis, which were all the rage back then, and they were like, "Come on, this is just for games. What's interesting about that?" I told them, "You just wait," and I liked the prophecy and it worked.

So yes, I keep a lot of revolutions, and it's fascinating. Computer industry and software industry, the OOP revolution, the functional programmer revolution, the web revolution, the mobile revolution, now the AI revolution, it's really, really amazing and really, really interesting. So, I'm keeping like even front end, like I started around 2000 in front end code where you had to have one big file full of JavaScript and you could do very little and then Knockout came out, and then Angular, JS, and then React and it's so fascinating to see all the different solutions coming up for the same problems. I just love it.

When people come to me, I know what I'm talking about and we talk in the same wavelength. They are younger though.

[0:05:14] JG: Yes. I want to rephrase the question a little bit. When you're a deep expert on something, it can be hard to maintain empathy with someone who's very new to it. They're facing very different challenges from you. For example, when I was working at Codecademy, we would have some deep experts on C++ who deeply understood memory management, trying to explain how to write a for loop to someone who had never programmed before. So, do you find it difficult or do you have any strategies around taking someone who's, say, a niece or nephew and teaching them despite you having a deep understanding of everything in the computer?

[0:05:43] GT: Yes. I don't think it's challenging. I like doing it. I always like, I know a lot of people when explaining they're like diving deep into the problem and you don't understand anything because you're like deep in the rabbit hole and everybody's around you and say, "Well, where are we? Where are we? I don't understand anything." But they're not saying it because they're polite or confused or ashamed that they don't understand anything. I always started, "Okay, let's start with the background, the context of things." And slowly explain each and every single step by single step in a very - I really, really try to make it orderly and logical and structured and a lot of people come to me and say, "Wow, this is the way it should be explained." Again, it's because I take the step back, I give the motivation, give the understanding of the context and then slowly and logically and understandingly. Once you do that, it's fine. I mean it works. No, I don't win - when I'm working on JavaScript, I don't explain memory management, obviously. So, you need to understand the scope of what you're explaining.

[0:06:43] JG: That's a great way of putting it. You must have a lot of patience. I understand a lot of folks don't have quite the emotional endurance to be able to sit through and re-explain the whole area to people over and over again.

[0:06:53] GT: One way to say it is patience. The other way is like, "Oh, just give me a stage or a junior sitting next to me where I can explain to them something." And I'm like, "Okay." It's like somebody comes with a question, I'm like, "Okay, let's start from the beginning. Let's explain. Promises. Okay, what you're doing here is blah, blah, blah, blah, blah, blah." Et cetera, et cetera. I love it. So, it's not patience. It's love.

[0:07:17] JG: It's not patience. It's love. Just as a -

[0:07:21] GT: I think -

[0:07:22] JG: Yes. Great advice to live by. Just humor me before we start diving into some of the more deep aspects of build systems and web tooling. Suppose I know JavaScript but I've been transported in time from the era when callbacks and callback hell were quite popular. How would you just briefly, 30 seconds up to a minute explain, say, Promises or Promises A+ to me as someone who wants to understand?

[0:07:47] GT: I have a talk about that. 2014, that was my first talk. I called it from callback hell to Async heaven and through the Promise purgatory. It's a wonderful talk where I slowly explain from callbacks. I go to Promises okay, and how did I do that? I don't remember anymore. Then, from Promises, slowly, I went to the yield thing where you yield the Promises and then Async heaven, and boom I'm guessing 20% of the audience really, really understood that back then, probably also now. But still, I think it was a very orderly way of explaining it.

You just go explain the problem with callbacks and then move in from callbacks to Promises. The idea is taking the callback and treating it as a value to back to Promise. So, it's a value. And then once you have Promises, when you extend the dot then, dot then, dot then, and then you do the yield, blah, blah, blah, and everything is great. It's tough, but it's doable. Well, at least somewhat.

[0:08:48] JG: Yes, there's a kind of beautiful parallel, I think, between the way you're explaining it and the way it's technically built up. Because for those who haven't gone deep into Promises, it really is built that you have the concept of generators, which then powers Async/Await, but in order to do so, you need to work with basic raw Promises underneath.

[0:09:05] GT: Exactly. What I'm seeing today is the exact opposite. People are using Async/Await and they have no idea what a Promise is. So they're not using dot then and dot catch, and once you get into the Promise world where you're not awaiting something but you're like taking the Promise and putting it aside and then running it, they get confused. What is this I'm doing? I asked somebody, you have an Async function and then you console log a call to that function, what is it console logging? A lot of people say, let's say it's just returning a five, they're saying five, and not a Promise to a five. A lot of people today, because there are so many levels of abstraction and because it's so ingrained today, don't really understand the connection between Await and Promises, not to mention callbacks.

[0:09:51] JG: Do you have any particular strategies or ways that you would educate folks if you were, say, working on a team and you wanted to help folks understand how that all is built up?

[0:09:59] GT: Slowly. Always slowly. You said patience. In the end, if you go too fast, people don't understand. So, you slowly and logically and methodically, there's none of the way.

[0:10:10] JG: It's kind of inevitable too in any area of tech that we are constantly building things on top of the older, that we're learning how to better do stuff, creating higher and higher abstractions. So, by nature, although we're more powerful as developers, dear God, I can't go back to Async or pre-Async to callback hell. We also have more we have to learn to really truly understand the full stack.

[0:10:29] GT: Yes, absolutely. And I programmed an assembler, so I go even deeper.

[0:10:35] JG: There is a certain satisfaction to knowing where all the bits and the bytes are headed. 

[0:10:39] GT: Yes, a lot.

[0:10:41] JG: Well, let's talk a little bit about the stack because you've done a ton of work on build systems and web tooling and Node.js over your career. What is your relationship to the build system or CI/CD for node projects these days?

[0:10:53] GT: It combines two of my big things that I love, which is build systems and Node.js. Even in 2000, when source control was just an idea that like 20% of developers used, and build systems were like non-existent, there was make, and that's about it. The company I built, we decided to build a build system, and it was this huge amount of Perl code, which took XML files or I don't think it was XML. It was lying and we need to parse everything and build a build out of it. It was extraordinarily beautiful. 

Obviously, unmanageable after a while, but we really, really tried because it was really important for me. It's not the build system. It's the idea of modularization that is important to me, not to look at all the code as one huge big monolith, but rather as a set of modules or packages, as we call them in JavaScript today.

So, that's one like love. The other love is Node.js. I've done tons of stuff like Assembler, Pascal, Basic, C, C++, Java, Python, and now Node.js and JavaScript. And the most I've ever been productive with is by far, Node js. I have this developer today. I gave him a plus one ticket to NodeTLV. There's a conference in June in Israel, NodeTLV. So, I have a ticket because I'm speaking, and I gave him my plus one ticket. He comes to me, he says, "I'm really excited about NodeTLV." I said, "Why?" He said, "Because I'm writing in C#," and I'm like, "I'm really, really missing the productivity and the ecosystem of Node.js."

He's right. It's unparalleled. I mean, it has its problems. But nobody has built anything remotely to the scale of npm in the world. It's an amazing success. Tons of problems, but an amazing success story. 

[0:12:50] JG: Yes. It's truly not comparable directly to any other system in existence. Or just the sheer ability of people to iterate, to publish, to learn from each other, and as we were saying, build abstractions in npm is never foreseen in human history, which is kind of a weird scale to be able to apply to something that's come out over the last decade or two.

[0:13:08] GT: Absolutely. I remember there was this holy grail in the early 2000s of building a component model. Everybody wanted to build a component model because the idea was, erroneously, if we have lots of components that are well-built, we can just glue them together and build an app. A bit naive, but npm is the closest we got to this goal, because I can truly - I need to do mutex today, an async-mutex that locks parts of the code. It was like, "Okay, npm, search for it. There's the Snyk Advisor, which tells you if it's a healthy package or not, like based on lots of different metrics." And boom, I chose a package. Npm installed it well and used it. No problems whatsoever. This would never have happened in any other language.

Well, modern languages are better. Go, Rust, in some ways Python, but there's still a horror show there. But npm tops them all.

[0:14:05] JG: I look forward to people angrily writing in about your description using the phrase "horror show." That's not my problem.

[0:14:13] GT: Once you installed some kind of package manager in Python, maybe it works, but there are so many and they're so complex.

[0:14:20] JG: Sure. I feel not just an intellectual need, but for the sake of humor, a need to start complaining about the node and the npm ecosystem with you. Because as you mentioned, there are some flaws, some drawbacks, which is, of course, natural when you're evolving, when you're moving quickly. You've said a great quote that I want to build on. You said previously, modularization is the number one problem in software development. What would you say is the number one issue or difficulty that you personally feel or experience with writing Node.js?

[0:14:51] GT: Writing Node.js? Well, it's been a while. I'm now a front-end guy. I've become used to TypeScript and I do not want to transpile. So, I have to use these shenanigans like JSDoc typings to make it work and it works like 95% of the time, but that's a difficulty configuration. Like it's not out of the box. If you take Rust or Go, everything is out of the box. You get your linter, you get your format, or et cetera, et cetera. If I have a new project for Node.js or React or front end, configuring it is, you need a wizard to do that. Now, I've done it so many times that I can do it like with my eyes closed, but every time I do it, like ESLint now has the flat config, so okay, let's try that. And this doesn't work with that, et cetera, et cetera. Combining all these into one thing is difficult and you need to understand each and every one of the tools.

Once you get into composing them, for example, as you well know, TypeScript and ESLint, or TypeScript and React, and bundling in TypeScript, and bundling in this and that, then it really becomes really, really, really difficult. You either accept the defaults today or you become a wizard at configuration. Those are the two options today.

[0:16:09] JG: We actually just released a beta for V8 or TypeScript ESLint that changes how you configure type to linting. So, I look forward to you and everyone else wizarding through that.

[0:16:17] GT: We will look into it.

[0:16:18] JG: Oh, it's very exciting. But yes. So, let's say that you are leading a team and you needed to, let's say, help your developers set up all these tools, whether you're going with the basics or becoming a wizard or luminary yourself. What would you kind of take as your algorithm for helping folks understand what to do in that space?

[0:16:37] GT: Well, there's understanding what to do and there's helping them set it up. So, what I did at Roundforest was I was the wizard, I built a configuration and for each package, and I built this tool where you could take a package and then it's a template and you just create another package from it. That works really well. That's one thing.

The second thing is do as little as possible. I mean, the more configuration you have, the more tooling you have, the worse you are because you need to configure and you need to make them become, combine one another. So, I really, really try to cut down on configuration and tooling. So yes, ESLint,
Prettier in the back end, and TypeScript for type checking, JSDoc typing. It's not for transpiling, because I hate transpiling. Transpiling, the reason I dislike it, it's not like I dislike it, I use it in the front end. But the reason I prefer not to have transpilation is because it makes everything more complex. It becomes a pipeline. So, from my point of view, less is more. That's one thing.

Another thing that I do not do, and this goes against everybody, what everybody is doing, if I have a monorepo and I have lots of packages, okay, every monorepo has lots of packages, everybody is trying to make one configuration for all the packages. It's not easy. Not only that, once you have that, changing that configuration becomes like, there's this grand wizard that understands everything, and they're the only one that can change it, because there are so many packages dependent on it, and there are so many configurations, and nobody understands anything.

What's the other way? And it's arguable, but I work like that in round sorts, and I really liked it. Every package has its own configuration. There is no sharing. I mean, you could share, but don't. This brings two things. One, you're not afraid of changing the configuration because it's only that package becomes affected. Two, it's a small package. The configuration is very simple, very easy to understand, very easy to change, very easy to reason about. That for me is the most important thing. That is why modularization is the number one most important thing.

Once you have lots of packages and they're independent of one another, not like in modern monorepos. Once they're independent of one another, they're easy to reason about, they're easy to understand, they're easy to build separately, et cetera, et cetera, et cetera. And the most important thing is, as I said, they're easy to understand. And we had a monorepo of about 400, 500 packages and it worked like a charm, because everything was independent.

[0:19:09] JG: One drawback, though, of every package having its own file, you're nodding and smiling at this. One drawback is that if you need to update something, then you have to update it in those 400 to 500 packages. What do you do in that case?

[0:19:21] GT: There are two answers to that. One is I don't. Why do I need to upgrade everything in all those 400 to 500 packages? That's one. The other is if I do need to, for example, I want to add a rule for packages, what I say is this. Whenever a developer goes into a package to modify it, first of all, date all the dependencies, that's very important. Second, they run a script that knows how to update those packages like a code mod for configuration. And that code mod knows how to check if it's possible to change the configuration or not, et cetera, et cetera, et cetera. Does it work 100% of the time? No, it works 95% of the time.

I don't have - the people that say shared configuration aren't wrong. They're right. But I'm also right. The question is a balance. What is the better right or what is the worse wrong? My point of view is I would prefer the burden of sometimes updating the configuration, because it's only sometimes. When you have four, remember, if you have 400 and 500 packages, you can't do a configuration anyway. It's just not possible. Okay. So, I prefer the burden of once in a while getting problems with updating the configuration rather than the burden of shared configuration where you can't really change anything, because you can't really make it work on 400 packages.

The fact that for me, a package has to be independent. It has to be built independently, tested independently, et cetera, et cetera. Otherwise, it's just a monolith in another form. So yes, there is a problem with non-shared configurations. I understand it. I felt it at Roundforest every day. But I still think it's better than a shared configuration monorepo.

[0:21:04] JG: So, what are the advantages then for you of having a monorepo versus a whole bunch of individual single repos?

[0:21:11] GT: One only, very easy one. I can search across all the packages. That's it for me. By the way, at Roundforest, we didn't have one project. It was not a project monorepo, it was a company monorepo, just like at Google. So, we had lots and lots of projects residing in, and each one used different packages from the monorepo, and the way we dealt with it was we had Visual Studio Code workspaces, where each workspace had only the packages that belong to the project.

So, when I searched in VS Code, then that searched only the packages that belong to the project, and then I could do whatever, search, reference, et cetera, et cetera, search for references, et cetera, and it just worked. That is the main thing. Managing 400 Git repos is basically impossible. Managing one monorepo where you can search thing, that is, for me, the only advantage of a monorepo like that, because I prefer not using shared config monorepo.

[0:22:13] JG: How do you deal with the build systems issue then where you have 500 different packages, you don't want to rebuild the other 499 on one change?

[0:22:20] GT: Now, people will get really, really angry. Okay, we build locally. I know, I go, "Oh, no, but you don't know what's on the computer." They were chilling at my time, like in the early 2000s, you never knew what was on the computer. But today, look, this is the JavaScript ecosystem. It's npm, everything is local in the package JSON, all you need is JavaScript of the correct version. And if you have an [NGMR/NDMR inaudible 0:22:45] that deals that automatically.

So, why not build locally? And we're talking packages. I don't build everything. So, when I'm working on one package, I'm working on it, building it and running the test takes not more than a minute, maybe two minutes. So, why push it to CI? Have this whole process that takes like whatever, 10 minutes, because there's a lot of overhead, and then you get back, "Oh, there's a failure." We just build locally. We had a script to do that. The script basically did an npm install, npm build, npm test, and npm publish to publish the package.

Now, people say, "Well, what if I want to work on two packages at the same time?" Well, that's the problem. The need to work on two packages at the same problem is not a good thing. It's a problem, because now, once you work on two packages on the same time, they start becoming dependent on one another in terms of thinking, in terms of testing, you're not testing into two packages, you're testing only one package because you're anyway building them all together. So, you suddenly have hundreds of packages with no tests whatsoever, and you start really, really getting afraid of changing them because there are no tests and they could like affect all the other hundreds of packages. So, you start becoming afraid because the packages are not independent.

You're basically back into a monolithic codebase. The codebase we had at Roundforest and before that at Applitools, each package was really independent. Really, really. So, it had its own tests and you could only build that package and publish it, so you had to work packages by package by package. And a lot of people say, "Oh, that's a really bad developer experience." I said, "Yes, not perfect." But it creates packages that are really, really independent. They have their own tests, and the tests are rock solid because they're built independently and tested independently.

So, it's weird. I'm taking away some freedom, which is you build packages by package, okay? But what I'm gaining psychologically is an amazing ecosystem where you can have 400, 500 packages, and they're all independent, but now there's just no problem. If you think of it, the npm ecosystem works exactly like that. Each package is independent of one another. So, what I built was like a mini npm inside the company. Now, people say, "No, it doesn't work. It doesn't work." Yes, I know. But go to Roundforest, work there, and see how it works.

[0:25:18] JG: I love it. It's solipsism from the package or monorepo level.

[0:25:23] GT: Absolutely. I was surprised how well it worked, actually.

[0:25:27] JG: I wouldn't think it would work unless someone were talking to me on a podcast who's had experience working at a company making it work.

[0:25:35] GT: Me too. I'm still like, "Oh, my god, this is a great idea." I was like, it was bits and pieces. It just happened and then suddenly, "Oh, this isn't happening. This isn't just happening. It's a methodology." So, it works.

[0:25:48] JG: Let me try to poke a hole in it though. You mentioned you're working more in front end these days. Suppose you're working on a design system which is then consumed by a dozen different variants of a front-end or app or some such, and you want to make an API change that also and/or changes some colors, so you need to test TypeScript, type compatibility, logical compatibility, accessibility in case the color change messes stuff up. How would you do that if you have the one package that then is consumed by others?

[0:26:13] GT: What you don't want to do is if you have a design system, check all the other 10 apps because there's no way you can do that. There's just no way. So, you built it in a backward compatible way. If it's not backward compatible, you do a simple merger. Now, once you change that, the apps consume that package, and let's say they find a bug there. Okay, you introduced a bug, because we're people. So, the package chooses not to upgrade to the new version of the design system, stays there, goes to that package, fixes the bug, adds a test, obviously, publishes, and now they can consume.

I know it's a roundabout way, but look, a project monorepo where you share config and everybody works on all the packages, it works for 40, 50 packages. Once you get to the scale of 400, 500, you just cannot build everything, you just cannot change everything. If you have a package, you cannot say, "Okay, let's test it in all the other packages." It works because people have a project monorepo. One app with lots of packages around it, like lots, as in 40, 50. That's it. But that's not scale. That's a monolith with folders, what we call them packages because we're cool.

[0:27:32] JG: A monolith with folders. Yes. That's fascinating. If I remember right, that's how the Microsoft design systems were working. When I was in an office team, every time we wanted a new version of say, Fluence, we would have to pull in and check, okay, does this work? We would have our own pull request with our own validations, checking does this new system work?

[0:27:49] GT: Exactly. You have tests. I don't care. At Roundforest, we pulled in automatically. Whenever you started working on something in a package, we just upgraded all the dependencies, some for minor, and sometimes some for major. Just run the tests. Like 99% of the time, the test pass, and boom, we're good. The 1%, okay, we fix a little bit here, fix a little bit there, or we say, "No, no, this package I don't want to upgrade." Which side?

[0:28:15] JG: Before we change topic to talking about Node.js, was there any other hot takes or hard opinions you wanted to put forward in the concept of monorepos and packaging?

[0:28:24] GT: Yes, I think I put enough.

[0:28:26] JG: Okay. You've incurred enough Internet hatred for yourself to deal with for the next year in this one conversation. Yes. Let's talk about Node.js. How did you first get involved with that lovely project?

[0:28:37] GT: I was at Wix, and there was this project that needed transpilation. Babel didn't exist, so I used the Google Transpiler. I didn't know like a lot of - I knew only front-end JavaScript and I understood that I needed something in the back-end that transpiled the user's code. Remember Wix is like a place where you can build your own website. Our project was not only build your own website but allow people to code in their website like back-end and front-end, et cetera, et cetera. So, we knew we had to transpile and we had to do some lots of build stuff et cetera, and the obvious pick was Node which was in version 0.12, 0.14, 0.11, 0.10, I don't remember exactly and I picked it up and I wrote a POC in two days. That was callback. That was the days of callback. Not a promise in sight and I was like, "Oh, my god, this is amazing. I want more of that." Obviously, we continued working with Node.js and it's still my pet. I love working it and I love the ecosystem and I love project itself. That was the
first love and I started talking about stuff in Node.js and one of the things I talked about was ESM because modularization and packages and module systems obviously very much aligned. I got really interested in ESM and how Node.js worked with ESM, which was back in the days not working with ESM. It used only common JS.

I gave a talk in Japan, and Myles Borins was there, and we started talking in the cab, and everywhere, and he suggested I become a contributor and an observer in the Node.js ESM working group, and that was the start of my involvement with Node.js. Mostly ESM, mostly as an observer, mostly as a speaker about ESM or whatever it is in Node that is current, and not so much a contributor.

[0:30:36] JG: Not all of our listeners might have a deep love and understanding. What is ESM and how does that compare to common JS or whatever Node was doing before ESM?

[0:30:44] GT: Right. So, we're talking back in Node.js, not front end. Okay, so let's talk Node.js. Node.js was back in the days when there was no module system in JavaScript. There just wasn't. Import from and export were just thoughts in the minds of Dave Herman and others. So, they invented one. It was called CommonJS. It's not the event that they took somebody else's module system, if I remember correctly, and used it. It was a required thing. Const, X, load-equals require load dash. And that was what we used and to great success, npm is built on CommonJS, and that was great.

But then ES6 came along and ES6 said, "Okay, but we need a common module system, and they invented or standardized the module system, which is called ESM, ECMAScript modules, with the syntax where we now import from and export to whatever, export default and export this and export that. So, now we have two module systems, CommonJS and ESM. ESM, it took a long while for it to enter Node.js and to enter browsers, by the way. It was 2015 was when it was standardized, but I think browsers got it only in 2018, something like that. Node.js really got it only in 2020, I think. I don't remember the dates, but somewhere around that area.

So, a long, long while to make it standardized. Today, we have to compete in a way. Node works with two systems, CommonJS and ESM. So, that's ESM.

[0:32:21] JG: I can see what you were talking about 20 minutes ago in the interview with how you layer your descriptions, where by the time you actually said, the technicals of what ESM where it made total sense in the context of how we came from an original or original four-node system and then built
up to the language specifier. Because what a lot of people miss is that A, as you said, it's old. It's from almost a decade ago that it was standardized. And B, it's an addition to the language. It's not like CommonJS, which is just a few identifiers and function calls that happen to be magic.

[0:32:50] GT: Exactly. This is why I'm a huge proponent of ESM. Today, npm is a hodgepodge of ESM and CommonJS and front-end developers who are today using ESM only needing to import packages that are CommonJS and it's a big mess. As we all know, bundlers are trying to hide that mess from us and succeeding 99% of the time. It's that 1% that is like hair pulling out. So, that is why I want something to succeed in Node.js. I think it's incredibly important for the ecosystem.

[0:33:24] JG: What do you see as the steps that are being taken now that are particularly positively impactful towards making that success happen in our lifetime? I want to thank Sindre Sorhus. I hope I'm saying the name correctly. Sorry, if not. He's this wizard who has about a thousand packages to their name. Great packages. just use the mutex one. Thank you, Sindre. And they decided that they're moving all their packages to ESM only. And all the next versions of the packages will be ESM only and not CommonJS. That's difficult. Why is that? Because in Node.js, ESM can import CommonJS. That's not a problem. But CommonJS cannot import ESM.

So, if you want to is the latest versions of Sindre's packages, you need to convert your app to ESM.
So that's one flag in the mud, or how do you say that? Flag in the mud.

[0:34:21] JG: Sure.

[0:34:22] GT: Whatever. And more and more packages started doing that, either being ESM only or being dual mode. I think of today it's about 15% of the top packages that are ESM, either ESM only or dual mode. So, that is great. Hopefully, it will continue. I'm very ambivalent, but I now just introduce the ability to require ESM, still experimental. I'm like, "No, please don't do that. Don't, don't,
don't, because I've been preaching not to do that."

A lot of people are trying to convince me that it is a good thing, but it has its problems. But maybe, in the end, like code wins our wins. PR wins. So, it's going to be in the ecosystem and maybe it will help, maybe it won't. But I think that in the end, we're going to be ESM full like in a few years. That's the goal. So, those are the two things I think, the move of packages and maybe require ESM.

[0:35:17] JG: From a naive or first-timer perspective, being able to import a CommonJS package from ESM and an ESM package from CommonJS seems like being able to go both ways would be very lovely. Why might one of those ways not be ideal for the ecosystem in your mind?

[0:35:32] GT: Right. Two reasons. One is ESM in essence is Async. How do I know that? There's a feature called top level await, where a package in the top-level code can just await something. And that means that importing it is an Async process. That is why you have to weight import, something dynamically, and you can't just import it. So, it is an Async module system. Requiring an ESM module is weird. What if there's a top-level elevator? And the answer today is if you require a package or a module that has top-level await, the require fails. It's that simple. That generates a split. I can't use top-level await because maybe somebody will require me. That's one reason, not a big reason.

The other reason is once more and more packages move to ESM, this will force the hands of other packages and other packages, et cetera, et cetera, et cetera, to move to ESM. Once you have the ability to require ESM, people can stay in CommonJS forever. As we all know, I want demobilization. So, those are the two reasons I think it's problematic. I'm not sure I'm right. Absolutely not sure. 

[0:36:46] JG: Sure. There is pain. I know Sindre got a lot of flak online for moving to ESM only, but as you just said, there is benefit. Pushing the young bird out of the nest once it's able to fly.

[0:36:55] GT: Exactly. 

[0:36:57] JG: It doesn't hurt by the way, that Sindre, for those who haven't seen on GitHub, has an incredibly cute dog in the profile photo, which is probably very helpful to the cause. Before we start wrapping up, another interesting thing about ESM, a lot of folks especially in front-end will import stuff from odd extensions like images from .pngs or webpy or .css from .css files. As someone who's worked deeply in the import and loader areas of node, how do you feel about non-standard imports and their place in the ecosystem?

[0:37:25] GT: I had a lot of problems with that. It's only the front-end world. I mean, back-end, we don't do that. The maximum is import JSON. We don't do that in the back end because we don't have - I don't think we need a reason for that. But in the front end, it's very, very common. It started with importing CSS and CSS modules and now we're importing everything. SVGs, PNGs, whatever. It's problematic. Because one, it's non-standard. So, the definition of what exactly this is changes. And two, the bundler needs to deal with that. Each bundler deals with it a little differently. So, if I try to move from webpack to parcel and from parcel to VIT and from VIT to whatever is ES built, then I always invariably, I have to deal with the problems in these parts. And it's very, very difficult to move between bundlers for that reason. So, that is why it is like it. It's non-standard. It creates lots of problems, and people say, "Okay, just don't move bundlers," but a project doesn't live for two years. It lives for 5 years, 7 years, 10 years, and you have to change technology. And if you're non-standard, it makes it very difficult.

[0:38:33] JG: But standards are informed by common usage, and the fact that so many people are doing things like import styles from .css means that there is a real need here. So, how do we get to satisfy that need without this kind of bizarre, painful, real-world experimentation?

[0:38:47] GT: I don't know. Seriously. I mean, I've just started front-end. I know front-end, but I've just started really developing front-end and the project I have, we're importing CSS and PNGs and SVGs and whatever. So, I haven't really wrapped my head around, okay, what if I don't do that? Is there a problem? How do I solve it? My guess is that if people say, "Okay, I don't want to do it via imports, but some other mechanism," they'll figure it out. And it's, I think not that difficult.

But I seriously don't know. Maybe the developer experience is so good that it's the only way out. I don't think so, but I don't know. I really and seriously don't know.

[0:39:29] JG: It's a hard question, but there's also built-in support node now, right? And is intercepting the right word and saying that when you import a particular file, there's some kind of importer or loader that's able to come into play?

[0:39:41] GT: In Node.js, yes. There's loaders. I have a talk on loaders. You can search for them in YouTube. Yes, you can intercept just like we did in CommonJS. But in CommonJS, we also intercept it. That's why you can do a TS node and run TypeScript because what TS node does is it intercepts all the requires and transpiles it on the fly. But in CommonJS, it was problematic because the interception was like, we just reverse engineered everything. But now, Node.js is tied to the way everybody does that so they can't change anything around it. In ESM, they decided to do it the correct way and describe the standard way to implement loaders and it's really, really nice. It took a while, but now it's here and it's really, really nice.

[0:40:29] JG: Well, in an example use case, other, of course, than CSS, be for loader or in import.

[0:40:35] GT: TS node, like that's what they use, loaders. Babel transpilation or any kind of transpilation, basically. The second is mocking modules. You want to say, if somebody imports this module instead of this, they get that. We use it a lot in tests. So, the loaders are the only way to do that and that's great. I actually wrote the first loader for mocking modules at test double and it was a really, really nice experience. What else? Oh, HTTP loading or loading from zip. You can do that too, if you want, instead of just from the file system, like Node.js does.

So transpilation, mocking, and other forms of protocols, those are the three off the top of my head that I can think of. Oh, APM, application monitoring. So, testing performance, et cetera, et cetera. Yes, that's the four.

[0:41:27] JG: I haven't thought of that one, being able to put the application monitoring directly into the code with import calls.

[0:41:33] GT: Yes.

[0:41:34] JG: That's really nifty.

[0:41:35] GT: It is.

[0:41:35] JG: The testing though is so interesting because that's been such a pain in the tush for such a long time for node folks using whatever it is, Jest or VTest or whatever, Mocha, people are using, where there's just this set of sometimes semi-standardized with each other ways of, if I import something, I want it to be mocked out, and people have such a hard time properly setting up, jest.mock, v.mock and so on. Do you think that that's going to get friendlier over time now that node loaders are a little more standard or actually standard?

[0:42:07] GT: Good question. Jest has their own way of doing module loading. They're basically replacing CommonJS for some kind of such a thing. VTest is a bit more standard in that it transpiles the code on the fly, but it's still using Node.js ESM. But I don't think they're using loaders. So, they're just transpiling everything, including the imports, so that they work in an expected way. I think they could use loaders, not sure. But VIT itself is, oh no, VIT itself is a front-end. VTest uses VIT, and VIT is a front-end bundler. It can't use Node.js loaders, so that's out.

[0:42:46] JG: A lot of backend projects use VTest. That's really funny. That it's built on a front-end bundler system. 

[0:42:52] GT: I'm not sure why. It just makes them slower and it does weird things to your code. So, I'm not a fan. I use it. I love using it in the front end, but in back end, Mocha, NodeTest itself, Ava, all the regular ones that just import the test files and just run them, I think they're the best. They're the simplest and easiest to understand.

[0:43:15] JG: I would love to dive into that with you, but we're almost at a time here, and I wanted to save time at the very end for a fun personal question. You are a prolific reader and consumer of books and movies. What's your latest reading kick in the last few months?

[0:43:32] GT: Well, mainstream, very mainstream, but Three-Body Problem. I really liked it. I read it like two years ago, I think. And then the Netflix series came out and I said, "Okay, let's read it again." Wonderful. It's weird. It's different. It really is Chinese. It's different than American or British sci-fi. So, there's that. I'm doing a marathon on my favorite ever author, Samuel R. Delany. He's a sixties writer. He's still alive, still prolific, a great writer, mostly doing mainstream, very weird literature. But he's amazing, like every 10 or 20 years, I'm just like, "Okay, let's read it all again." And it's still amazing. 

[0:44:18] JG: Wow, he is prolific. He's won quite a few awards.

[0:44:22] GT: Oh, Grand Master, everything. Yes, he deserves it. He really is one of the more complex and interesting sci-fi authors, like ever.

[0:44:31] JG: What do you look for in a sci-fi author or their books?

[0:44:34] GT: Well, it depends on the mood. I mean, I love my space opera, so I'm fine with that. But in the end, I tend to more, it's the more psychological, complex, very literate sci-fi. So yes, again, it depends on the mood.

[0:44:49] JG: Personally, and this will result in a question. Personally, I really like books that make me question and kind of increase my ability to understand either myself and/or the world around me. Asimov's Foundation series made me really think about, "Well, what is the behavior at mass of people and systems? What are we moving towards?" Is there a particular book or series that you would recommend folks use to get into thinking more about themselves in the world around them?

[0:45:13] GT: Oh, that's a question. Well, Delany, sorry, I have to go back to him because he's the first author that introduced me to the idea of postmodernism or post doctrinism, where you understand that text and language can be read in different ways and can be understood by different people in different ways. Culture is a very - it's not ingrained into us, but it changes and develops and everybody has their own culture. He does it in such a beautiful way.

But if maybe the author that has changed me most or influenced me the most is one of the big three. There's Asimov, you mentioned, and there's Arthur C. Clarke, which is wonderful. And Rendezvous with Rama has this amazing quote in the end, like it's a huge deal, it's what is the meaning of life, and it's loving and learning, and it's the best quote for me ever.

But the most influential author is Robert Heinlein, by far. I know he's very liberal and not libertarian, very, very much so. So, that's not me. But his ideas around family, sex, friendship, love, they've touched me very, very deeply and influenced how I think of the world and people.

[0:46:30] JG: It's fascinating how books that on the surface may have characteristics one disagrees with can still be founts of wisdom and beauty. When I was a teenager, I read the Terry Goodkind Sword of Truth series, near-religiously. I loved that series. Then later, later, later in my life, learned that it was strongly objectivist. One would even say objectivist propaganda. I had no pickup on that when I was reading it. But I did take a lot of philosophies out of it, like standing up for yourself, standing up for what's right. And at the time, I think it really linked to the Jewish concept of tikkun olam, which is healing the world around you. In retrospect, I have no idea how I got there. But at the time, it really helps me understand those things.

[0:47:09] GT: Yes, the same for Heinlein. Looking back, you understand how his philosophy of zero government is deeply flawed, but all the other concepts are amazing.

[0:47:24] JG: Well, that's a lovely way to end an interview. Gil, I want to really thank you for chatting with me this last hour. I think this has been a wonderful conversation on node and the ecosystem and build tooling and love and teaching. So again, thank you. This has been great. Is there anything else you want to plug or any particular ways folks can find you online?

[0:47:40] GT: Oh, Gil Tayar on Twitter, which is the best way. Feel free to contact me, talk to me, ask questions. Don't be shy. Really, I'm there, always. Thank you.

[0:47:48] JG: Well, we all appreciate it. For Software Engineering Daily, this is Josh Goldberg. Thanks, everyone. Cheers.

[END]