EPISODE 1850

[INTRO]

[0:00:00] ANNOUNCER: Traditional package management systems for JavaScript have faced several inefficiencies related to dependency storage, resolution, and project performance. pnpm is a fast, disk-efficient package manager for JavaScript and TypeScript projects, serving as an alternative to npm and Yarn. Due to its efficiency and reliability, pnpm is increasingly popular for managing monorepos and large-scale applications.

Zoltan Kochan is a full-stack web developer and the creator of pnpm. He joins the show with Josh Goldberg to talk about his background and package management in the web ecosystem.

This episode is hosted by Josh Goldberg, an independent full-time open source developer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ESLint, the tooling that enables ESLint and Prettier to run on TypeScript code. Josh is also the author of the O'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a live code streamer on Twitch. Find Josh on Bluesky, Mastodon, Twitter, Twitch, YouTube, and dot com as Joshua K. Goldberg.

[EPISODE]

[0:01:24] JG: Zoltan, welcome to Software Engineering Daily.

[0:01:27] ZK: Hi.

[0:01:28] JG: Hi.

[0:01:29] ZK: I'm happy to be here.

[0:01:30] JG: As a consistent and now longtime user of pnpm, I'm excited to talk to you. We've got a lot of great stuff to talk about, like architectures and modern repo management. Before we get into all that though, how did you get into coding?

[0:01:42] ZK: I started coding at school, like at 2004. I didn't have a computer back then, or maybe it was a little bit earlier. So, I kind of had the whole journey from the invention of Internet to the invention of AI. I mean, the first couple of years, I was coding without Internet. I got Internet only in 2008. So, I was using just old school books and pirate content, I guess, in Ukraine.

Actually, early on, I was planning to be a painter. I was doing a lot of drawing. I didn't want to join the university or institute. It was for painters, but I missed like entry days. I mixed up the dates so accidentally I failed that gym and later on I entered a math faculty to become an engineer. That's I guess most of it. I started with Pascal at school and then I used C# and I used C# at my first job.

[0:02:58] JG: So, you're saying there's an alternate universe where instead of creating pnpm you became an artist, a painter?

[0:03:04] ZK: Yes, but I don't think I would be very successful at that. I was always painting like real stuff, not some imaginative abstract stuff, which I think is more exciting for a painter.

[0:03:20] JG: So, you got your first job in C#, what was that?

[0:03:23] ZK: So, the first job was at an outsourced company, it's called Svan, and they have a little office in my city, Uzhhorod, which is in Ukraine, and they had different projects for American companies. I started as a junior programmer there, and I was only doing like unit tests, which is interesting because people didn't use like test development, people were just coding, and I was writing the tests for them, which now seems weird, but that's how it worked. I was so happy when, after a few months, I started doing real tasks that are not related to tests. 

So, at that company I worked almost two years. And then I joined JustAnswer, which is a Q&A website where verified experts answer questions. And I worked as a full stack developer on a .NET stack. After four years, maybe it's JustAnswer, I started really to look into Node.js and frontend. And that's when I found pnpm.

[0:04:38] JG: I have to ask, as someone who was in .NET before JavaScript, how did you find that transition going from the strongly typed, strong foundation of .NET and C# to the, at the time, even more so, wild and wacky and loose world of Node.js?

[0:04:52] ZK: So, there are pros and cons. What I really loved about JavaScript, that's before I used TypeScript is that it's so faster to prototype and to get the working like little CLI or tool like for scripting it's so much easier to use JavaScript because you don't need to deal with all that boilerplate with C# like even to write a hello world CLI app with C#, you need to declare a class, declare a function, and stuff like that. Maybe it changed since then, I don't know, because now there is more than 10 years have passed since then, and I know that C# has improved a lot. Now, you can even use C # on all platforms, stuff like that.

But that was my experience back then. Also, you don't need all that IDE, which is actually really expensive to buy a Visual Studio license. You can just use it as an editor for JavaScript. But maybe with C#, it's also possible, but it's not so common.

[0:06:03] JG: Yes, these days, you can use VS Code for C#, but it is not quite the primary path. Yes, it is kind of amazing to think of the trajectory that the .NET area and ethos has gone through. TypeScript recently announced that they are rewriting in Go, which at that time, 10 years ago, would have been completely unimaginable. Microsoft using Google-oriented or sponsored language for TypeScript development for anything other than .NET. It's an amazing change.

But okay, moving on, Node.js. So, Node.js typically ships with npm, which is their package manager, the Node Package Manager, as it occasionally is called. Why would you want to create something like pnpm, a different package manager than the default?

[0:06:49] ZK: So, how it happened? Actually, just to answer, we had a huge monorepo. It's about, okay, it's not a huge monorepo by today's standards, but at that time, it was like 140 components. And the way it worked, there was a simple script which in a for loop ran npm install in each directory. Maybe they were linked together through symlinks or not, I don't remember. And we had like a Sinopia, which is the old name of Verdaccio, running in San Francisco and we were using it from Ukraine. This installation was terribly slow.

In San Francisco, it was faster, maybe 10, 15 minutes. I don't remember, but in Ukraine, it was 30 minutes just to run and install in this monorepo. Also, it used a lot of disk space. So, like dozens of gigabytes because of all these duplications in node modules, folders like 100 times. I'm not sure how I found pnpm, but I found it and I tried it instead of npm in this monorepo and it immediately improved speed drastically. So, back then npm was a lot slower, not everyone remembers it, because it was like in 2015, '16, it was npm3, it was before Yarn came out and even on small like projects npm was like, it was slower than the current npm probably 10 times. The current npm is super-fast in comparison but pnpm was really revolutionary at that time and I decided to look into it. At that time, pnpm was not really maintained. It was created by Rico Sta. Cruz. I think he's from South Asia. Now, he lives in Australia, I think. And it was like a fork or like a clone, I don't know, of another package manager IED, created by Alexander Gugel. The reason these package managers were created was because in npm3 the structure of node modules has changed, if you remember, I don't know.

In npm2, node modules used like a nested structure so every dependency had its own dependencies in a sub directory inside the node modules in the sub directory so you could have really deep tree structure in a node module and on Windows it was causing constant issues, because on Windows there's a limit to the file path. Also, the other problem with this structure was that dependencies were duplicated many times inside node modules so they wanted to fix these issues, and in npm3 they changed the node modules layout to a flat layout where now dependencies are hoisted to the root. And as an alternative to this approach, this IED package manager was developed by Alexander Gugel and the idea was to use symlinks to avoid both nesting real directories and to avoid hoisting everything to the root. So, that's the main reason why pnpm was created.

[0:10:23] JG: You used a few terms there. There's an old joke among developers, a famous cartoon on the Internet about how in increasing order of size. There is the planets, the stars, a black hole, and then node modules. So, it's nice to see so much work has gone into reducing that size and complexity. What exactly does hoisting mean in this context and how does that change or not change how Node.js resolves packages?

[0:10:46] ZK: So, by hoisting, we mean that every dependency in the dependency graph is moved to the top of your node modules' directory. So, even if you only have like webpack in your dependencies, webpack itself has like hundreds maybe of dependencies of its own. If you install webpack with npm and you open your node modules directory, you'll see hundreds of directories there. Even though in your packages, you see only one dependency, which is Webpack. So, this is a simple explanation of hoisting.

If in the dependency graph, there are multiple versions of a package, then this hoisting algorithm will pick one of these versions and put it to the root of node modules. And the other ones will be nested in one of the dependency's sub-node modules directory.

[0:11:42] JG: Okay. Then you also mentioned symlinks. Now, as a user of pnpm, what we experience is that you can install pretty darn quickly, and oftentimes if it's the same installation you've done on a previous run on your device, you can just do it offline or even on an airplane. But how does symlinking play into all of this? And what even is symlinking?

[0:12:03] ZK: I'm not sure the symlink have a role in the speed. It just allows us to use an alternative layout for structuring node modules. Actually, you also asked about Node.js resolution in the previous question, but I did not answer, so I can mention it now. So, the way Node.js resolves dependencies is that it searches for node modules directories in the current directory of the JS file that has the required statement and then also looks in every parent directory as well, and when it finds this dependency, it resolves it.

An interesting important point here is that for this resolution, it uses the real location of the module, not the symlink location. So, it means that we can have a symlink at one place, but we can have all the dependencies at a different place, where the symlink points. So, with pnpm, if you run pnpm install, with webpack for instance, you'll see only webpack in the root of the node modules directory, and it will be actually a symlink. And you'll also see a hidden directory, .pnpm. And if you open the .pnpm directory, you'll see lots of directories there, like in the form of a package name, at, and version.

So, this is a flat directory structure, even flatter than the npm one with the hoisted node modules. Because it has a version of the package. So, the webpack in the root of node modules points to a directory in this .pnpm directory. If you open any directory inside this .pnpm, you'll see another node module there, and you'll see all the direct dependencies of each dependency. So, if you open webpack in .pnpm, you'll see symlinks to the direct dependencies of webpack. Because when you require webpack from your project, Node.js will resolve the symlink to that hidden directory inside .pnpm. And it will actually look for dependencies of webpack in that like directed directory inside .pnpm.

[0:14:38] JG: That's fascinating. Yes, it's really, I think a great example of doing less work to achieve just a really clean solution to a common problem. As a result, to users, it just feels seamless. You run pnpm install and it feels the same as an npm installation just faster. But under the hood, it is kind of shockingly different.

[0:14:56] ZK: Yes, the speed difference is actually achieved by something else. It was also a part of the initial pnpm version when I found it. So, if you open the pnpm website and go to, I think, the motivation page, you'll see there a nice illustration of what makes pnpm so performant. So, instead of having like separate installation stages where in the first stage, all the dependencies are resolved, then a separate stage when all dependencies are fetched, and the third stage when all dependencies are written to the node modules directory. In pnpm, there are like separate independent pipelines for each packages. So, one of the package will be resolved, fetched and written to node modules at the same time when others are still being resolved and independently at different stages of the installation. Actually, even Yarn has these different installation stages, but still it's fast enough, I think.

[0:16:03] JG: Before we move on to the higher-level features like workspaces, it would be great to get your opinions on the current state of package managers. As you noted, node and npm themselves have changed a lot and also Yarn has had quite a few versions. Do you see any particular pros, cons, feature comparisons between the current versions of those three package managers?

[0:16:24] ZK: So, I believe currently, if you check, pnpm is most popular for using in monorepos.
I don't know actually how good are the other package managers at monorepos. I really stopped comparing at some point, but people say that it currently has the best feature set from monorepos. You can see also our website on the workspace page. At the end of the page, we have usage examples, I think, and there are some big names there. Github projects with thousands of stars like Nuxt or ViT, some well-known projects use pnpm for their monorepos.

There are some features that are available only in one of the three package managers. Like Yarn has the plug-and-play feature, which we kind of have also, but it's not really well-maintained. So, we use the Yarn's libraries for that. So, if you need plug and play I would recommend using Yarn. pnpm has some features that others don't have. Recently we have added configurational dependencies which is basically a new type of dependencies that run before everything else and you can load settings from there or use it for plugins. It's really new so I think it will be very powerful, but it needs more work to be done.

But I think Yarn also has some plug-in system. I don't know how it works, but I think they have like core plugins, which are installed by default and they have maybe some community plugins. In pnpm, we have hooks like in pnpm file CGS, you can hook into different parts of pnpm.

[0:18:20] JG: It's always lovely talking to open-source maintainers, very rarely does anyone who's not an open-source maintainer in one answer mention things like we use one of our direct equivalent packages or projects as a dependency, such as you do with Yarn, or for this particular feature, although we have it, you should go use our other equivalent project for certain scenarios. It's really lovely to see that kind of cooperation in the ecosystem.

[0:18:44] ZK: Yes, well, I've worked on pnpm for like nine years maybe, so many things have changed since then. But Yarn's core contributor was Mile. he has also worked on Yarn for six years maybe, or more. So, we have talked a lot and we had - when there is some hate in the community, you can empathize with that. I wouldn't talk about Yarn. I know how much work goes there.

[0:19:18] JG: Yes. There is an unfortunate amount of community virtual, but you're all working towards a better future and it's really lovely to see. Speaking of that future, let's talk about some of the features of pnpm that you mentioned. So, just to take what you said about hooks or the configurational dependencies as an example, why would I want, let's say, a configurational dependency if I'm managing my team's package or projects?

[0:19:43] ZK: Yes, I think I was a little Bit inspired with bit for configurational dependencies. I work for Bit for like three years, I think, or more, three, four years. If you haven't heard about Bit, it's like a GitHub for component-based development. It has its own version control system, package manager, like hosting CI, everything, to make component-based development easier. What makes Bit interesting is that you don't have to choose between a monorepo and a single package repo, because you have dynamic workspaces where you can fetch only those components which you need for a particular feature. So, it solves the scaling issue of monorepo, but has all the pros of the monorepo. And one of the ways that Bit makes it possible is that it has these environments for components and the environment has all the necessary configuration, like not only the dev dependencies, but also any rules for the linter, for the bundler, think about it like all the dot files that you commit to your repo.

Then you have - so this is one of the disadvantages of a multi-repo approach that you need somehow to synchronize all these dot files. But with Bit you have like your environment component and this way you can share all your conflicts by using the environment. I wanted to make it easier with pnpm also, so to be able to publish like a package that will contain all your settings for instance, maybe you use some patches to fix dependencies. If you haven't heard about it, you can use patch files with pnpm. It's also supported in Yarn and in NPM you can use a package for that, patch packages I think. So, if you have some fix for maybe an unmaintained dependency in your node modules, and it's very deep in your dependency graph, so you can't just fork it. I mean, actually, you can fork it and use an override. But with the patch it's even easier, because you don't need to publish your fork to the registry. So, you generate the patch file, and you commit to your repository and you tell pnpm to use this patch file for the dependency. But if you have many repositories and each repository uses this dependency, you'd have to copy the patch to each repository and make the configuration change in each repository.

With configurational dependencies, you can add this patch file to the configuration of dependency and then just use the configurational dependency in each of your repositories and it makes it easier. Also, you can, for instance, have some standard overrides in configurational dependencies, or you can have the list of package names that you want to hoist to the root of node modules. Because sometimes you need that with pnpm.

We also have a feature which other package managers don't have currently, which is called catalogs. It's a way to centralize, to synchronize versions of dependencies in a monorepo. So, you have these catalogs declared in the pnpm workspace YAM file at the root of the monorepo. Then inside your projects, you can just put catalog column in the version field, instead of the version. This way it's easier to control each project in the workspace will use the same versions, which is otherwise possible only using some linting. I think Yarn has constraints, it's called, which allows to create linting rules for this, and there is a third-party package for this, I should think, called link sync.

But these catalogs, it's a bit nicer because it's centralized in one place and it's nicer to see on the change in one file, not in many packages on files throughout the repository. For the future that we could move these catalogs to this configuration dependencies, and maybe companies could have like catalogs of recommended versions throughout their repositories, stuff like that, or even like an allow list of dependencies. Maybe I know that some companies have limits to what packages they can use for their codebase. So, it would be easier to ship this.

[0:24:58] JG: Yes, if there's one developer behavior that's well understood in the web ecosystem, it's that people prefer and enjoy configuration files and avoid and hate linting. So, it's nice to see that being moved to a first-class supported feature. A bit back brought up your company and we should talk a little bit about it because it's fascinating and also very relevant to this area of repo and component management. So, you work at Bit.dev and Bit.dev does use things use things like pnpm. But let's say I have a monorepo with a design system and a bunch of apps and some frontends. Why would I want to use something like Bit.dev or a [inaudible 0:25:35] platform to be able to manage all those components and frontends and so on?

[0:25:39] ZK: From my point of view, the main advantage is actually this possibility to create
your own workspace. So, your company won't have to decide whether to use multi-package repositories or to use single-package repositories or have multiple monitoring repositories for different things and then somehow link them together. With Bit, every component is independent. It's shipped. Actually, you can think about it as a single-package repository. So, each component has its own version control system, but unlike with the Git hosted approach, you don't have to, like Bit creates a much better user experience, even like in the UI, the website, BitCloud. So, there is BitCloud and Bit.dev. Bit.dev is, I think, mostly about the open-source aspects of Bit, because you can actually a Bit for free and host it on servers because a Bit is an open-source project, but then it has BitCloud, which has additional proprietary code and paid features.

But like with Bit, like the code you use, like when you make changes in multiple components in the PR, you will see all the changes to all the components in one place so that's actually not possible currently with GitHub, maybe some other companies allow it, I don't know. It's very well optimized currently for UI components too, so it's like it's like has the how it's this project called for UI library site, I don't remember, where you can visually see the components and even like click on them. Storybook, yes. So, it's kind of like a mixture of Storybook and GitHub, I think, visually.

So, the problem is monorepos is that you can't scale it. No matter what you come up with at some point it will break. You can't scale it to that amount of components that you will need in today's world. I know there are like Turbo and NX, which improve things a lot for monorepos, but still at some point, it can't handle that amount. So, I think it is a really unique approach because nobody else tries to solve it this way, that nobody wants to create a new version control system and a new Git. So yes, just check it out, I guess.

[0:28:22] JG: Yes, creating a new Git does sound rather daunting. There have been some things built on top of Git, as a service, that have sprung up, but it's nice to see the innovation here. We don't have too much time left for questions, but real quick towards the end, I do want to ask you, you've talked about monorepos versus multi-repos or single. You've talked about different forms of package management, why and how newer innovations in pnpm and Yarn have happened. For the next, say, five years, are there any more upcoming trends or new areas you're excited about being invaded in?

[0:28:57] ZK: Well, the CEO of Entropic said that developers will not be needed in one year, so I don't know.

[0:29:01] JG: Things that are actually going to happen, I should clarify.

[0:29:05] ZK: To be honest, the problem with a popular project is that you have so much stuff to do that you can't really make a long term - like you can't think about big stuff. We had many actually, like you know, Bun came out, right? There is also Orogene, which is created by Kat Marchan, I hope I pronounce her name correctly, but she was a core contributor to npm-CLI. There is also Vault, which is created by ex-npm maintainers and founders. So, sometimes we look at these competitors and think about what could we do differently.

For instance, we spent a lot of time on thinking about rewriting pnpm in Rust or some other language, because Bun came out and it claimed that it's 10 times faster than pnpm. So, to be honest, for a long time, I thought that maybe the long-term plan would be to rewrite pnpm in Rust. But then we created a proof of concept and it turned out that the Rust rewrite didn't make it faster. Maybe we just failed, I don't know. But it looked like most of the time we spent on network requests, which are not influenced by the text that we use.

So, I think now my big idea for the next few years would be to create some registry integration, because we have, I think, optimized almost everything we could on the CLI part. But if we would have our own registry for instance, I don't know, if we would do it or we would use some existing registry. But if we could like for instance move resolution to the registry side, it could make installation faster or use some different archiving algorithms for the files that are faster, or maybe we could fetch not the whole package, but just the files that are required from code.

So, I think it would be interesting to look at the registry side. It could be also a way to get some additional funds if you would have like our registry. But competition is really big currently, because I know there is JSR from Dino and Vault is coming out and something else, maybe Bun will have its own registry. I don't know. They are very wide in their plans.

[0:31:57] JG: It would be unsurprising. We have at Software Engineering Daily an episode from a few months back into 2024, an interview with Luca from Dino that touches on how, yes, once you push a lot of these concerns into the registry, as you said, you can do a lot of optimizations for them. A big focus is TypeScript supports and documentation generation. That'd be fascinating.

Zoltan, I have one last question for you before we wrap up. What is your take on spicy food and where have you had your favorite pieces of hot food?

[0:32:27] ZK: Thanks. Yes, I really like spicy food. I'm Hungarian and in Hungary there is a famous soup goulash, which should be consumed spicy. And there are some sauces, a Hungarian sauce called Eros Pista. But then I discovered Asian food, which is even more spicy and Mexican food. And I really, really love Indian food. I have many friends in India from work, from my previous work. For one week, I visited Bengaluru and there it was heaven. A heaven, food heaven for me.

[0:33:07] JG: Yes, a lot of folks from North America, from the West might not know that there are quite a lot of fantastic, very flavorful Eastern European foods, such as goulash, and it's lovely to hear that you had such a good time. Was this for a conference that you were in India?

[0:33:20] ZK: No, it was a work trip. JustAnswer has an office in India. So, I was visiting colleagues. 

[0:33:27] JG: Well, great. That's all the time we have. One actual final question for you. If people wanted to find out more about you, your projects, anything you're working on, are there any particular locations you'd suggest they go on the Internet?

[0:33:37] ZK: They can follow me on Bluesky, on GitHub, on X. Most of my activities in the GitHub repository in pnpm. We also have a Discord for pnpm, which is quite active.

[0:33:51] JG: Well, thank you so much, Zoltan for coming on. It was fantastic talking to you about the history and features of pnpm, and package management, and workspace, and repo, and component management. Looking forward to seeing all the great stuff you're doing with the new features like the dependencies and catalogs. For Software Engineering Daily, this is Josh Goldberg and Zoltan Kochan. Cheers all, thanks for listening.

[END]
SED 1850		Transcript





	(c) 2025 Software Engineering Daily	- 13 -