EPISODE 1964


[INTRODUCTION]


[0:00:00] ANNOUNCER: Cory Doctorow is a science fiction author, journalist, and technology activist, and a special advisor to the Electronic Frontier Foundation. He is the author of dozens of books, including the nonfiction Enshittification: Why Everything Suddenly Got Worse and What to Do About It, and the novel Picks and Shovels. His work spans tech policy books like The Internet Con and Chokepoint Capitalism, the solar punk novels Walkaway and The Lost Cause, and the best-selling YA Little Brother series. He holds honorary doctorates from York University and Open University, writes the daily blog Pluralistic.net, and lives in Los Angeles and London.

In this episode, Cory joins Josh Goldberg to talk about his latest book, The Reverse Centaur's Guide to Life After AI. They discuss the difference between a centaur and a reverse centaur. Why the same AI tools leave some workers empowered and others feeling exploited, why companies impose worse-performing workflows anyway, and what workers can do about it.

This episode is hosted by Josh Goldberg, front-end developer at Sentry and open-source maintainer. Josh works on projects in the TypeScript ecosystem, most notably TypeScript ESLint, a powerful static analysis toolset for JavaScript and TypeScript. Josh is also the author of the O'Reilly Learning TypeScript book, a Microsoft MVP for developer technologies, and a co- founder of SquiggleConf, a conference for excellent web developer tooling. Find Josh on Blue Sky, Fosstodon, and .com as Joshua K. Goldberg.

[INTERVIEW]


[0:01:56] JG: With me today is Cory Doctorow. Cory is a writer, activist, journalist who has been with the Electronic Frontier Foundation, or EFF, for nearly 25 years. Cory, welcome to Software Engineering Daily.

[0:02:06] CD: Well, thank you for having me on. It's a pleasure.

[0:02:08] JG: Well, we're so excited. You've got a lot of stuff under your belt. Some people may know you as a sci-fi author or author of very many books on society and tech and culture. Other folks may know you as the Enshittification guy. Would you like to start us off by saying how you got into tech and writing?

[0:02:24] CD: Oh, my gosh. Well, I got into tech because my dad was a computer scientist. So our first computer wasn't even electric. It was mechanical. We had a computer called a CARDIAC, a very low-powered CARDboard Turing-complete device that was invented at Bell Labs by engineers who were worried that you had an incoming generation post-transistor who wouldn't understand how actual logic gates worked, and who built a CARDboard computer that you printed out while they printed it, and you assembled. And then you move tokens around.
You would write out a program and assembler, and then it would give you lookup tables that would show you how to move the tokens into different registers and the adder and so on to get your output.

That was in 1976. I was five years old. 1977, we got a teletype terminal connected to the University of Toronto via an acoustic coupler. They had a VAX system there. My mom was teaching kindergarten. So teletype terminal didn't have a screen. So you can only compute if you had paper. So she would bring home paper towel from the bathroom at kindergarten. We get a thousand feet of computing up one side and a thousand feet down the other side and then roll it back up and send it back to school for the kids to try their hands on.

'79, Apple gave all the heads of computer science at all the computer science departments in Ontario Apple II Plus to try for the summer. So my dad was at that point head of computer science at a school in Scarborough outside of Toronto. And so we had an Apple II Plus in '79. 1980, we got a modem. We only had enough slots in the machine for either the modem or the 80-column card that gave us lowercase characters. And I don't think I saw a lowercase character for like the next four years because that was where I knew I needed to be.

I also, at the same time, was writing science fiction stories. I started selling those as a teenager for a story I saw when I was 17. I ended up dropping out of four undergraduate programs, never got a degree, and finally out of the University of Waterloo, where I was working on what we would today call machine learning. Back then it was cellular automata. And took a job

programming CD-ROMs for the Voyager company in New York, which was like the coolest multimedia company in the world at the time.

When the company kind of imploded, I ended up as a freelance Gopher developer. And when I finally got my first Gopher contract from an ad agency, it was like, "We have a client that wants a Gopher site." I wrecked it by coming in and going, "You have to see this thing called a web browser." And so that became a web contract.

I ended up being kind of a freelance CIO, worked for a lot of different companies, helping them get on the web and figure out how to wire up their offices and get ISDN modems. And did a lot of stuff around the periphery; worked for an executive MBA program on high-tech stuff.
Eventually founded a company with one of my co-founders and a neighbor. We raised money in the capital markets. During the dot-com bubble, I moved to San Francisco and then ended up working for the Electronic Frontier Foundation, where I worked for the next 25 years. And here I am.

[0:05:15] JG: And here you are. It's interesting that we're here to talk about your book about reverse centaurs, AI, machine learning. And you had machine learning on your resume in your life before it was even called machine learning. How does it feel to have witnessed the rise of it from cellular automata to AIs and agentic workflows?

[0:05:31] CD: Oh, well, this isn't the first time that's happened. My dad's master's degree in applied mathematics, which is what they called computer science before they called it that at the University of Waterloo, was also cellular automata stuff, except his was on punch cards. AI has been through a lot of winters. And because we have this phenomenon of buzzword compliance and a highly financialized tech sector, it is normal that whatever is hot gets called that thing.

So I have watched the tech sector describe every technology as push, as metaverse, as Web 2.0, as Web 3.0, you name it. There has been a kind of mania for hastily changing the pitch deck to include the new thing. And to the extent that the company has shipped a product that people use to bolt this entirely superfluous add-on to a product that people use often, to its great detriment and sometimes to the fatal detriment of the company.

As a fake computer scientist, I never got a CS degree, although I hold an honorary doctorate in computer science from the Open University in the UK. And I'm a visiting professor of CS at the OU. So as a pretend computer scientist, I look at AI, and I say, "This is some pretty gnarly and interesting computer science stuff." Fundamentally, what happened was 10 years ago, some scientists said, "We have these machine learning algorithms. We have an idea for applying a minor variation to the way we operate them." They expected to get a minor variation in the outputs, and instead they got a significant improvement. They found a piece of low-hanging fruit. It turned out to be an orchard full of low-hanging fruit. They could repeat the trick and increase that variation and get a nonlinear increase in the outputs. But Stein's Law of Finance says anything that can't go on forever eventually stops. They reach diminishing returns.

And rather than discovering that if you just teach words to the word-guessing machine long enough, you eventually create a superintelligence that, if you're unlucky, turns us all into paperclips. They discovered that, as with every other computer science breakthrough, that you are able to achieve some impressive things, and then you've got to think of something else.

[0:07:30] JG: Which brings us to the something else. And you've already actually used a few phrases that I like here. Buzzword compliance is a great one. And in the book, Reverse Centaur's Guide to Life After AI, I kind of want to walk through it in terms of some of the fun phrases you brought up. So let's start with reverse centaur. What is a reverse centaur compared to a centaur? And how does that apply to AI?

[0:07:51] CD: Well, this is not the first time that labor and capital have come into conflict over automation. There's actually a pretty rich literature. One of the truisms of that literature is that when labor drives uptake of automation, it is typically in service to improving the quality of the outputs. Workers just think they can do cool and amazing things. If you go back to the web bubble and you read the business press, it's full of articles going, "What are CEOs going to do with all these young workers who've used the web at university who expect to find it in the workplace?" They're not equipped. It's the opposite of the articles we have now, which brings me to what happens when you have capital-driven automation. Right?

When you have capital-driven automation, it's typically in service to increasing throughput, often at the expense of quality and worker well-being. And so this is particularly valuable when you

have a cartelized or monopolized market where you can produce inferior outputs without seeing a loss in sales. This is more or less what the thesis of Enshittification is: that in a world in which competition has been sidelined, regulators have been captured, workers have been tamed, and new market entry through interoperability has been made illegal by the expansion of IP laws, companies can make worse things and still keep their customers. They are the only game in town, and no one's going to punish them through non-market means. And so they can just, like, Enshittify stuff, which I think we've all felt very sharply.

And so in the world of capital-driven automation, you see increase in throughput, often at the expense of quality, often at the expense of worker well-being. And that's why today, when you pick up the same business press, it's full of articles about how are CEOs going to convince these workers who think AI is bullshit that they should be using AI on the job- really the opposite of the web era.

So this brings us to centaurs and reverse centaurs. In automation theory, a centaur is someone assisted by a machine, whether that's you using an IDE or a spell checker or even riding a bicycle, an act that makes you look like an actual centaur. And the metaphor here is you have the human head and the robot body, the horse's body, the strong, fast body with more endurance than you have, which applies no discernment, applies no judgment, and does not direct the action. The humans on top; the body is taking orders.

The reverse centaur, the corollary here, is the machine driving the human. The human conscripted to be an unfortunate peripheral to the machine, being worked at the pace and to the endurance of the machine, which is faster, stronger, and has more endurance than you, not just being used by a machine then, but being used up by the machine. 99 of your colleagues are fired. You have to mark the homework of the AI and replace all of them, and you take the blame when things go wrong. You have to work long hours under dreadful conditions, doing the worst part of your job under conditions that make it impossible not to make mistakes because you acquire this automation blindness where you can't spot the errors. Because most of it's right except when it's really wrong in ways that will kill people. And that's reverse centaurism.

And the reason that matters is because this distinction, centaur reverse centaur, resolves an otherwise seemingly intractable paradox about AI in the workplace particularly for programmers,

which is that you find people who are skilled practitioners of their art, who are reliable narrators of their experience, who say, "I've taken up AI in my practice, and I produce better outputs than I ever thought possible. I'm so excited to have these wonderful new tools."

And then you have other workers, equally skilled, equally accomplished, equally reliable, narrating their experience, who say, "My God, you would not believe how much tech debt we are producing and the scale that we're producing it at. I work in civilian aviation, never get on a fucking airplane again because we've just filled the sky with intractable tech debt."

How do you resolve this? Is one side lying, the other side telling truth? Was one side trained to use AI well and the other side not using AI well at all? No, one side is a centaur. They're deciding where in their workflow AI makes sense. The other side is a reverse centaur. They are producing inferior outputs at a faster pace under worse conditions. And that is why this tool can produce these very contradictory accounts from the field. And it validates, I think, the most important aphorism about science fiction, which is that the important thing is not what gadgets do, it's who they do it for and who they do it to.

[0:12:07] JG: That brings us to one of the other core theses of the book. You would think that given this understanding of one form is much more productive and long-lived and effective than the other, and the other form is a reverse centaur, which is a horrifying thing to look at, why is it that so many companies, you bring up in the book examples such as Amazon, are putting us into these slot factories? What's the perceived efficiency or growth potential there?

[0:12:30] CD: Again, if you believe that your customers will accept an inferior output, if you can lower your wage bill and improve throughput, you kind of have a - I don't want to say a fiduciary duty, but certainly a moral hazard inducing you to do this.

In particular, I think the AI bubble has to be seen in light of the string of tech bubbles we've lived through and the kind of material basis for those tech bubbles. Because over and over again, these companies that have achieved incredible growth, to the point where really they've ran out a runway to grow. If you're Google and you have a 90% search market share, you don't grow. I mean, you raise a billion humans to maturity and make them Google customers. Sure, but

Google Classroom is going to take 15 years to produce results. And so, you've got to find some other source of growth.

And these firms have had these crazy stories about growth. They've said, "We're going to grow by becoming each other." Right? Google was going to, like, do Google Plus and become Facebook, or Facebook was going to do the pivot to video and become YouTube. Or they say, "We're going to grow by conquering markets that don't exist: NFTs, blockchain, metaverse, Web3, and so on." And now you have AI and superintelligence and so on.

The reason for these growth stories it's not what progressives like me sometimes say, which is endless growth is the ideology of a tumor. It has nothing to do with ideology, or at least ideology isn't the important piece of it. It's entirely a material phenomenon. Because if you're a company that is growing, you have a really high share price, not because of irrational mania, but because a share is a claim on the future earnings of a company. Companies that are growing have lots of future earnings.

The corollary is when your company stops growing, it's overvalued because its future earnings are the same or lower than they are this year, which means that you see these panic sell-offs of stock in listed companies that used to grow and have saturated their growth. And the only thing to forestall that is either growth, or if you can't find growth, a story about growth.

And the reason you don't want to sell off - well, first of all, if you're like the individual who runs the company or one of the executives who's been compensated largely in stock, a mass sell-off that sees like a 50%, 70%, 80% drop in the value of a company, well, that's a 50%, 70%, 80% drop in your net worth. I mean, no one is as exposed to fluctuations in Meta share price as Mark Zuckerberg, right? He doesn't want Meta to grow forever for ideological reasons. He just doesn't want to become 80% poorer tomorrow. It's like a completely rational, material, non-ideological phenomenon.

So it's not just that the individuals involved in managing the firm are exposed to this, but also the firm itself. If you care about its well-being, you have to worry that when you stop growing and your stock becomes as liquid, that a lot of the growth tactics are going to go away. If you're a

growing company and your stock is very liquid, you can buy other firms using stock. You can hire key employees using stock.

Remember when Meta announced its AI super intelligence play, and they announced that they were paying a programmer $100 million? They did not pay him 1 million $100 bills. They gave him $100,000,000 worth of stock. $100,000,000 or a million $100 bills- they're hard to come by. You have to get them from a customer or a creditor or maybe an investor. Stock you make on the premises: you just type zeros into a spreadsheet, and you've got $100,000,000 worth of stock. If you try to make your own hundred-dollar bills on the premises, they will take you away in handcuffs.

And so companies don't want to reach the situation where their growth stalls. All the people who know how to fly the plane grab parachutes and jump out, and then everything they could use to pull out of the tailspin is not available to them. That's the reason they want to keep growing. And so AI is this latest story about growth, and it's a kind of meta story about growth, not meta in the Facebook sense, meta in the above sense, in the sense that they're also telling other companies how they can grow without doing anything innovative, without coming up with a better product or coming up with a better process or finding a way to innovate. What they're saying is, "We're just going to let you fire all your workers and replace them with chatbots. And you're going to split the wage savings between you and Sam Altman." And you will continue to grow, not by selling more units necessarily, but just by extracting more through every sale, by improving your unit economics. And so I think that makes the bubble very powerful, because you don't just have the storytellers of the bubble who are excited about this. You have the audience who's really excited about this. You have bosses really excited about this.

[0:17:10] JG: You bring up another good example in the book: grocery store chains who replace cashiers with automated registers. And there's kind of the fork in the behavior. What some grocery stores could do is see that the cashiers are more efficient and then give them other tasks, such as interacting and helping with the customers. And I actually wanted to ask you about that in this context then, because if every grocery store in a town is getting cheaper prices for their apples, a cent less per apple, that gives them a little bit of a competitive edge. So it feels like the situation is almost forcing people to at least try out the AI chatbots that are being

phrased to them as more efficient. Is there some way to counter that? To give labor a little more power in this market?

[0:17:49] CD: Well, I mean, the obvious way is a union, right? But let me complicate that story just a little because yet they are getting a penny more per apple and they're losing three cents per apple more in shoplifting because they've replaced all the cashiers with machines. And then they ask us, the public, to foot the bill for co-ops. And then they ask us, the customers, to pay in time by putting everything behind plexiglass and making us wait for one of the two customers left in the business to show up and open the door so that we can buy the things that we're looking for.

And so it's not as simple a story as that. It's a thesis about rational markets that often breaks down at the margins. You know, there's this joke about economists that if an economist sees a
$20 bill on the ground, they won't pick it up because if it was really there, someone would have already picked it up. Because, you know, as we say in economics, there's no such thing as a free lunch. So I think that, like, there are management fads that suppose that there will be improvements in the cash basis of the operations of a firm, and that whose perpetrators or whose captives are then improbably surprised to discover that the incredibly obvious foreseeable outcome of eliminating all the people from the store who stop shoplifters has happened, which is to say you have a lot of shoplifting.

And one of the things about being serious about this stuff, and particularly working in it for a long time, as I have, and maybe just being old sometimes, is that you acquire a kind of object permanence that can make the rest of the world a little baffling. It just sometimes feels like our policymakers and our business decision-makers have so little object permanence that they would lose a game of peekaboo. Like, how is it that we can just make the same mistake over and over again and be surprised when we get the same outcome? It is very frustrating and sometimes a little demoralizing.

[0:19:47] JG: It is a little frustrating and a little demoralizing. But I'd like to move on in the flow to talk a little bit more about the AI bubble and especially how it pertains to software developers.
Since this is a software podcast, we don't have the ability to make some sort of large cross- sector union the way the music folks do. A lot of us are in roles where managers are breathing

down our necks, telling us you have to use the AI. We're going to judge you on lines of code, a metric that, as we all know, is not correct. What advice would you have for software developers facing this hellscape?

[0:20:18] CD: Well, like all the best Americans, I'm Canadian. And there is a joke from Eastern Canada whose punchline is, "If you wanted to get there, I wouldn't start from here." There was a time before Google fired 12,000 workers after doing a stock buyback that would have paid their wages for the next 27 years. There was a time when Silicon Valley engineers were in such high demand and were so valuable to their employers that they commanded a lot of power. And this is why you got the free kombucha and the massages and, like a surgeon who'd freeze your eggs so you could work through your fertile years. It wasn't because your bosses liked you.
They didn't let you go to those all-hands meetings where you could, like impertinently criticize their technology and business strategy and tell them they dress badly because they thought you were awesome. They did it because Silicon Valley engineers added an average of a million dollars each to their firm's bottom line, and there weren't enough of them. And if you walked out the door, a million dollars walked out with you. And there were 10 bosses at the gates who would offer you a job if you quit.

And so now we are in this circumstance where supplies caught up with demand, where firms are willing to produce lower-quality outputs and endure more downtime and defects in their code because they think that their customers won't mind as much. So this is Enshiftification at the code level and not just at the service delivery level. You see this in system-wide outages in AWS and lots of other platforms that are theoretically too big to fail and supposed to be run like utilities and instead are being run like startups in terms of the SRE commitments.

And so this means that now, with 10 other workers at the factory gates will take your job if you quit, after half a million layoffs in the sector. It's tough. And you say tech workers are not in a position to unionize like musicians. Musicians didn't unionize because they were the princes of labor who their bosses had no choice but to take what they were offered from their workforce. Musicians unionized because they were brutally exploited, and they fought for a union.

And I tell you what. If you look at how your boss treats the workers who aren't in high demand, not an Amazon programmer, but an Amazon warehouse worker who's injured at three times the

rate of any other warehouse worker, not an Apple programmer, but someone assembling an iPhone in China who works in a factory that has a suicide net around it, not someone who programs AI, but someone in Kenya who does the AI data cleaning and who ends up with lifelong psychological trauma because of the tagging they're supposed to be doing of extreme and gore content. You can see what your bosses would do to you if they could. And so if you wanted to get there, I wouldn't start from here, but here's where you are.

And there is the tech workers coalition. There's Tech Solidarity. At EFF, we're unionized to the Communications Workers of America. There are tech shops that are unionizing and tech shops that are trying to unionize. And the bad news is, as you say, it's not a good position in terms of your bargaining strength. The good news is. More workers in America want to join a union than at any time in living memory. And more Americans support the unionization of their neighbors in their workplaces than at any time in generations.

And the unions have got larger cash reserves than they have ever had in the history of the labor movement. And right now, they're not spending it on organizing and they should be. And one of the ways we get there is by demanding that we as workers need to have consideration from our comrades who are unionized already and that they should not treat those cash reserves as existing solely to benefit the workers who've already been organized but to add to the pool of unionized workers.

And so go to Tech Solidarity, go to the Tech Workers Coalition, find out how to unionize your shop. Start having union conversations. Because wishing that there was a way to have a better workplace without unionizing, it's like wishing that we had cryptography that only worked when good guys used it, but that stopped working when bad guys used it to hide child sex abuse material or terrorist plots. Wanting it badly is not enough. There is one reliable, durable mechanism for workers acquiring power in history, and it's solidarity. That's it.

[0:24:25] JG: Solidarity. It's interesting you bring up the crypto aspect. You do reference this in the book as well, that that's actually what the Clinton administration had tried to do, to position it as a musician and that only the good guys would use it. And that doesn't happen. That's not how that works.

[0:24:40] CD: Yeah. And it's not just them. I mean, right now in the UK and in the EU, there is a giant movement to force firms to decrypt user communications in the name of national security, fighting child sex abuse material, and so on. Some of those goals I think are legitimate. I think child sex abuse material. Obviously, fighting that is extremely legitimate. I don't think you get there by banning working encryption. And I think the collateral damage from trying is catastrophic. As we say at the Electronic Frontier Foundation, wanting it badly is not enough.

[0:25:13] JG: So similar question for this area then. You've given us advice on solidarity, perhaps even unionization. How would you advise folks trying to work on what is the right and wrong way to do encryption and data protection?

[0:25:24] CD: Oh my goodness. Well, I would fall back to my work on Enshittification here and say that while individuals and startups, they can employ best practices. I mean, we look at, say, Signal, right, which are who are doing data protection very well. They have practiced extreme data minimization. They have forward looking programs to do quantum resistant encryption.
They are involved in policy fights and completely uncompromising in their willingness to backdoor the system. They are also working very rigorously to highlight the fact that prohibitions on jailbreaking phones makes them vulnerable. Because if apple and google say that they won't carry them unless they backdoor their encryption, then the majority of their users are going to lose it. So they're really fighting the whole policy fight as well as making best-of-breed, open, inspectable technology.

And of course, your security technology has to be open and inspectable because there's no security in obscurity. And as Bruce Schneier says, anyone can devise a security system that works so well that they themselves can't think of a way of breaking it. It doesn't mean that it works. It just means it works on people stupider than you. So all of that being said, none of that stuff is rocket surgery. We have companies that have been doing this forever. We know what best practices look like. This raises this question, why don't we have more best practices? And the answer is that we have an Enshittogenic policy environment. We have an environment where if two companies decide no one should have end-to-end encryption, no one gets it.

And so far, that's been good in the West. But in China, Apple turned off all the working encryption tools for its iOS platform. And because that platform's locked down and it's illegal to

reverse engineer, that's what the Chinese get. All their cloud storage has got a backdoor. Working VPNs are prohibited in the app store. The encryption at rest, the full disk encryption is defective, has deliberately introduced defects. The encryption on the wire also has deliberately introduced defects. So you have this place where one person gets to decide whether or not billions of people have privacy and security, whether their governments can find out who the dissidents are, round them up and put them in a camp, which is a thing that Americans need to worry about now as well.

And so in an environment where you have regulatory capture and monopolization, very little market power for the labor side, and also this copyright-based prohibition on reverse engineering that stops us from jailbreaking platforms to install software on them that the manufacturer disprefers, you end up in a world in which not only might you be tempted to do the wrong thing.

And I'm sure some of the people listening to this either work for startups or might dream of having a startup someday. And I have to tell you, when you convince 150 of your best friends to quit their jobs and risk their kids' college funds and their mortgage to come work for you, and when you make a product that makes millions of people happy, and your investors show up and they say, "I require you to make 5% of those users' lives a living hell, or I'm pulling the plug on the whole thing," that it's very easy for you to tell yourself, "I'm going to be a hero and destroy the life of 5% of my users to preserve the income of the 150 people who work for me and the benefit that 95% of my users get." And the reason that I'm a hero for doing it is everyone's going to think I'm a bastard. And only I am going to know that I'm the one who fell on my sword and I have to live with the moral injury of having done it. And so it's so easy to rationalize your way into that as an individual.

And then as a firm, once you're in a large company, it doesn't matter how much moral injury you experience when your boss orders you to Enshittify your product if it's clearly going to be more profitable to do it. We saw this with Google in 2019-2020. This is stuff that came out with the DOJ's successful antitrust prosecution of Google, the so-called search case, where they decided the way they would increase search revenue after hitting 90% market share was by making search worse. So that you would have to search more than once and see ads more than once.



And you see this factional dispute within Google between the revenue side and the tech side. And the revenue side's argument is, "We'll make more money." And no one has anywhere else to go. We pay Apple $20 billion a year not to enter the search market. Every browser, every manufacturer, every carrier, doesn't matter what you're using. It's a Google search box that is wired into our servers. Why would we care if our search was worse? What are people going to do? Use Bing? And so the guy whose counter argument is, "I didn't miss my mother's funeral to make search worse," that guy loses the argument.

And so if you want to do data protection right, it's not enough that you have to know what the right thing to do is. You also have to exist in an environment where doing the wrong thing makes you less money than doing the right thing. And you can build that environment somewhat yourself. There's a thing called the Ulysses Pact that everyone who's now seen the Odyssey knows what the Ulysses Pact is. It's when you tie yourself to the mast because you know that in the future you'll be weak and right now you're strong. So you tie yourself to the mast so that you can't be tempted. So this is things like irrevocable open source licenses. No matter what your investor says, you can't make that proprietary code again.

It's things like structuring your business as a B Corp or an S Corp or a worker co-op. It's any number of API guarantees that make it easy for your users to leave. It's making federated platforms, so that when users go, they can remain in contact with all the users that they're enjoying on your platform. All of that stuff makes it far more expensive for you to betray your users, which means that it's harder for other people to coerce you into doing it. And it's harder for you to rationalize your way into doing it.

[0:31:03] JG: As much as I love AT Proto and open source and open APIs and everything you just said, is that not a competitive disadvantage, though, in the capitalist landscape? You're not able to be as monetarily profitable and hyper-growth oriented, no?

[0:31:17] CD: Only because of the Enshittogenic policy environment, right? In 1998, Bill Clinton signed Section 1201 of the Digital Millennium Copyright Act into law, and it's a law that establishes a $500,000 fine in a five-year prison sentence for jailbreaking. When Facebook started, they had this problem that everyone who could have used Facebook was already using

MySpace. It was actually when they opened up to the general public instead of just college kids 2006.

So they gave those users on MySpace a bot, so that when you quit MySpace and went to Facebook, you didn't resign your account. You gave your login credentials to the bot. It impersonated you to MySpace, scraped all the waiting messages and stuck them in your Facebook inbox. And then when you applied them and pushed them back out again, that margin was your opportunity, right? This is what Jeff Bezos says all the time, "Your margin is my opportunity." The margin that MySpace was extracting through ads and manipulative behavior was Facebook's opportunity, and it made them into a successful company. The problem isn't that we have federation. The problem is that Facebook was able to pull up the ladder behind it because they have since used Section 1201 of the DMCA to block anyone from adversarially interoperating with them.

So you're right, it's a disadvantage. Unless what you're doing is hacking Facebook so that the people who currently stay on Facebook because they love their friends more than they hate Mark Zuckerberg can eat their cake and have it too. Come use your platform. Stay in touch with their dopey friends who are stuck on Facebook because their dopey friends are stuck on Facebook because their dopey friends are stuck on Facebook and they're paralyzed with this collective action problem that has us taking our friends hostage. They can come and be your customers and that margin can be your opportunity.

This is why tech was so exciting 25 years ago, because every time a company got big, sclerotic, and stupid, they were disrupted. People moved fast and broke their things. Right now, you're only allowed to move fast and break poor and weak people's things. I think we should move fast and break kings. I think we should move fast and break tech giants' things. They are a danger to themselves and others.

When they get hundreds of billions of dollars on their balance sheet, they do absurd things like stock buybacks and AI data center buildouts that see them spending a trillion dollars a year to make $50 billion a year. They need to have that money taken away from them by someone with better ideas before they hurt themselves and others.

[0:33:35] JG: Move fast and break kings is a great slogan, great rallying cry.


[0:33:38] CD: I agree.


[0:33:39] JG: Before we dive into kind of the later side of the bubble and perhaps what comes next, I do have one more advice question. I'm a former independent open source maintainer. And one of the things that pushed me out was the horrendous slew of AI slop span. You also mentioned in the book a certain literary magazine, Clarkesworld, that received just an obscene amount of slop. What advice would you have for people trying to field and protect versus slop while still giving actual humans or perhaps AI-assisted humans the right of way?

[0:34:08] CD: So this is a culture problem, right? I think that people who slop their PRs lack the discernment to distinguish useful code from useless code. And we have seen the free software ethic become an open source ethic, become a kind of open source as a career calling card ethic. And I think a lot of people are like, "Well, if I have 25,000 contributions to significant projects on GitHub, I can get a great job. Or I have bragging rights or something." I think the problem is they don't know that their code sucks. And I guess it's because they understand a lot about how code works and nothing about how code fails. They need to do something.

I run into this all the time because I'm not a software developer and I haven't written any code in a quintillion years. And yet I'm an Ubuntu user and I get into trouble on my own by like copying and pasting from Quora or whatever, when I'm like, "I want this menu to work differently. I'm going to go recompile this piece of software or whatever." And I'm just sitting there going, "Make install, please." And that's fine. Because if I vibe coded a patch that fixed something for me and then it broke, I could just download the official release and it'd be fine again. But if I vibe code a patch that breaks for 5% of users, of which I'm not one, because it breaks when you're using non-Roman script, or it breaks when you use just one emoji that you never use, or what have you, then you create a lot of headaches for the maintainers that you yourself can't perceive. And I don't think we can fix that just by scolding people.

I haven't really thought this through, so I'm thinking this through as I speak. So I think maybe what's happened is, normally, if you're a bit of a dope, and you've got an idea for a patch, that won't work. You write that patch, you submit it, the maintainer either gently or not so gently

goes, "Look, dope, here's why your patch doesn't work." And maybe you work with them and actually you learn how to write good patches from them. In the same way that writing good bug reports or anything else, like it's not a thing you're born knowing how to do.

And what we've done is we've taken dopes and we've given the ability not to make one bad patch, but to make a thousand bad patches. And maybe what we do is we delete 999 of them. And we say, "Look, dope, pick the one that you think is important, right?" You want to fix something because you've got an issue on a scratch. Something doesn't work that you think needs fixing. You want to be a part of this collective project. Why don't I, the maintainer, and you work together with your code assistant, if need be, to try and make the patch work the way we would with any other patch from someone who is just starting out? And we give them at least that much grace.

And if they're not interested, if they are like, "No, I just want to make a thousand patches and put it on my resume," then we show them the door. But like we've had automated code gen tools forever, right? When I was teaching myself Apple basic, my Apple II Plus did not have a debugger. And I read about it in like probably Bite Magazine. I read about debuggers. And I wrote a debugger for the Apple II. That was not very good. And I had a new automated tool that let me work with my code in a way that I couldn't before. And I was able to produce more code. And sometimes that meant that I could get it over my skis, but it also meant that I was able to fix my code faster. So I could write more code, but I could fix it faster. I could get it to run faster.
And then because it was running faster, I could find the defects faster and I could fix them.


And I think that used well, code gen tools that use the form of statistical inference we call AI can do the same thing. It's a matter of degree, but they have to be in the hands of someone who has discernment, particularly if they're working on a project that has other users. I think the place where vibe coding belongs is like you've got a HomeKit gadget and you've got a Raspberry Pi and you want to glue the two of them together and you vibe code an app. And it works until a dozen and you vibe code another one. That's fine. It's like the people who write Visual Basic or HyperCard programs that solve some tiny business need around their shop rather than hiring a programmer in it and trying to convey to the programmer what they're looking for. They just make the thing that they want. And it doesn't fail very gracefully, but it works perfectly. It exactly scratches their particular edge. That's what vibe coding is for. It's not for submitting PRs to a big

open source project that has a lot of people who rely on it and who are going to get right up the maintainer's butt when it stops working for the good reason that they rely on.

[0:39:00] JG: I think that's also very good advice for folks in a corporate landscape. You know, the sitting down with someone one-on-one, helping them, working with them, and developing personal rapport is particularly useful for co-workers. You know, especially if you have a senior engineer helping a junior who's been much more AI trained and focused than them.

[0:39:16] CD: Yeah. I mean, people who actually work in software engineering know that the major part of their job is not writing code, right? It's thinking and figuring out how the problem works and understanding how it works as part of a system. The difference between software engineering and coding is the difference between thinking about a system and thinking about some lines of code.

And again, coding is fun. I had the enormous privilege and pleasure of working with EFF's first ever staff technologist, who was a very playful coder named Seth Schoen. And Seth, he worked on a lot of big open source projects where he was thinking things through and so on. But he also would write obfuscated C. And he rewrote DeCSS, the program that was illegal. It was a felony to distribute because it would decrypt DVDs. He rewrote it as functional Turing complete haikus.

[0:40:10] JG: That's incredible.


[0:40:13] CD: Go look up the DeCSS haiku. It's online. And I'm not allowed to tell you he wrote it because he did not disclose, he wrote it because it was technically a felony to write this poetry, which was his whole point, which is that the government had made it a felony to create literature and that this was a facial violation of the First Amendment. Go look it up. It's amazing.

Writing code is incredible, but it's not software engineering. Software engineering, there's actually a really good piece in CACM, the communication CACM that just came out that looks at the proportion of a software engineer's job that's writing code. It's just not the major important part. And so LLMs, they let you do part of the job faster, but they don't let you do the rest of the job faster. And they create the danger that if you do that part faster and then try to debug it or

QA it, do the code review for it, that because there's so much of it, it's going to be really hard to keep up with.

There's this well-understood problem called automation blindness, which is that when a machine usually runs right and you're supposed to be vigilant for the very rare instances in which it runs wrong when you're supposed to be the human in the loop, you can't maintain vigilance. Just something with the human neurological apparatus seems incapable of maintaining vigilance for things that only occur occasionally which is how it is that we've created in the TSA the water bottle spotting as motherfuckers the human race has ever produced who are nevertheless 95% of the time incapable of spotting the fake guns that red teams bring through the x-rays, right? Because they're not they can't remain vigilant for these things that never happen.

If you're a programmer using AI to write code in a way that does not require superhuman vigilance, an example might be in radiology, the difference between a radiologist who examines x-rays and gets a second opinion from the AI when the AI disagrees and says check that x-ray again, which is a thing where you're just doing your job, but at a higher degree of fidelity that stops people from dying of cancer, versus the radiologist who sees nine tenths of their colleagues fired and who then has to review a hundred times as many x-rays that the AI has vetted and says are cancer free. And they have to make sure that the AI is right. And if the AI is wrong 3% of the time, they're probably going to miss all of it. And then people are going to die. And that's not what the machine does. That's who it does it for and who it does it to. That's the social arrangement of the machine, not the technical capabilities of it.

[0:42:29] JG: To me, that was probably the most impactful example of a reverse centaur versus a centaur itself. And I do want to talk to you, Ashley, about what you quote as "dogshit unit economics". But first, I have to ask, for the sake of the technical users, there are a lot of people who are trying to protest models or AIs as created today in terms of free speech. And you're a notable free speech advocate, and you mentioned this. Can you walk us through how is it that free speech laws protect things like the DeCSS haiku and model generation for AIs?

[0:42:59] CD: Well, so the DeCSS haiku arguably is still a felony, unfortunately, because Section 12 of the DMCA is very bad. And a judge, a very dumb judge who has done a bunch of

terrible things in his career, including letting the Sacklers off, decided that the people who published 2600 magazine were not allowed to publish DeCSS because it was not a literary work.

In the 90s, the NSA had classed, as you said, classed working encryption as a munition. And EFF and our fellow fighters for working encryption, privacy, and security raised a lot of arguments about why this was bad. The NSA said, all of this parade of horribles you proposed that will happen if we don't have working encryption, they're just hysterics. We will let you use a cipher called DES50, a defense encryption standard of 50 bits, that is so secure that no bad guy will ever break it, but not so secure that we can't break it. And so we'll catch all the bad guys.

And we made a lot of arguments about the technical insufficiency of DES50. No one really cared. The NSA had hired the largest plurality of top mathematicians from all the Ivies and big ten schools for 50 years. People were like, "Who are you to say that this doesn't work?" So then we proved it. John Gilmore, who was EFF's co-founder, who helped write Solaris, design the SPARC chip, write GCC, and founded the first ISP as well as EFF, and also the Usenet alt hierarchy. He's quite a storied technologist. John built a computer. He made some custom ASICs. The computer was called Deep Crack. It cost a quarter million dollars and it could brute force all of DES50 in two and a half hours. We brought that into Quora. We brought that into Congress and we said, "Look, if the mafia is willing to spend a quarter of a million dollars, they're going to be able to decrypt all the financial transactions in America, all the corporate secrets, all the individual information. Someday when we're pushing firmware updates to anti- lock breaking systems or people's pacemakers, that will be an open book to griefers and foreign spies and what have you." And again, they were like, "Yeah, that sounds stupid. Go away."

So then we brought the lawyers in. And Cindy Cohen, my former boss, has just retired from EFF and has written a wonderful memoir of this period, Cindy Cohn - called Privacy's Defender, I should say. Cindy Cohn formulated a radical argument. She said the First Amendment protects expressive speech, and code is a form of expressive speech. And a lot of people at the time were like, "I don't get it. How can code be expressive speech?"

But we represented a programmer called Daniel J. Bernstein. He was a graduate student at UC Berkeley, and he was publishing the source for a cipher that was stronger than DES-50, and

therefore a munition, on Usenet, this early messaging system that predated the web. And we went to court on his behalf, and the judge agreed with us that this math that he was producing, this code he was producing was a form of expressive speech and that the First Amendment protected it.

And then we went to the appellate division, and the appellate division agreed. And they NSA did not want to try their chances at the Supreme Court. And so since about 1993, it has been legal to have encryption that works. And this is how we protect everything. I mean, everything. And it's because of this free speech tradition.

And so when people say that we should ban certain code with a kind of scrutiny - so there's some speech that's unlawful, obviously, but the constitutionality of a law that bans certain speech is very fraught and requires that the ban be very narrowly construed and fall into a very small number of purposes, and that the speech itself be of a low degree of democratic importance.

So restrictions on political speech, for example, are very hard to craft in a way that comports with the Constitution, because one of the purposes of the First Amendment is to guarantee political discourse. And so this is one of the reasons that most so-called political disinformation and misinformation is probably legal. Not all of it. I mean, stuff like vote next Wednesday when the vote is on Tuesday is illegal. Fraud, like deceiving people is illegal. But having a thing labeled as a deepfake, right? Where you're like, "We're about to show you a deepfake of politician X saying some things that are a paraphrase of what we think he would do is legal." Because the First Amendment protects it, because the First Amendment is very, very delicate about restricting political speech.

And one of the things that we're seeing right now is what happens when a government decides that it is not interested in protecting dissident political speech, and how dangerous and frightening that is to the democratic project. And so there are outputs of AIs that are unlawful speech. But I would be reluctant to say that we should lower the degree of scrutiny we apply to restrictions on code itself, including code that embodies a model. Not least because it's very hard to make that restriction stick. What are you going to do? Make every Git server in the world comply with orders to take down the code that embodies the model? How is that going to work?



[0:48:23] JG: It doesn't seem practical. I really wish we could spend more time talking about what goes into this. You also have content in the book about scraping and how that can be very, very good. But we do have to move towards the end of the interview. So let's talk about what comes next. What is dogshit unit economics in AI? And what do you think comes after a bubble burst? Or perhaps even why is the bubble going to burst?

[0:48:42] CD: So AI has bad economics and it has bad unit economics too. So unit economics, that's how a business performs when it moves another unit, right? When it makes a sale or acquires a customer. So like what happens at the margin? So the web lost money early on. And AI is losing money. And some people who say that AI has got a path to profitability use the fact that the web lost money as proof that AI will make money someday. But for that to be like a valid logical construct, it would have to be the case that losing money was itself a predictor of making money. And broadly, losing money is a predictor of losing more money.

And the exceptions to that have to have a set of criteria, have to satisfy a set of criteria that are themselves not a guarantee, but at least increase the likelihood that you're going to go from a money losing cash basis to a money winning profitable basis. And first among those characteristics is good unit economics. So every new web user, even in its money losing his days, made the web more profitable as a sector.

Every visit to the web by a web user made the web as a sector more profitable. And every generation of web technology was more profitable than the previous generation. This is very different from AI. Every AI user that an AI company acquires knocks another hole in their balance sheet. Every time that user comes back, the balance sheet bleeds more red ink. Every generation of AI loses more money than the previous generation of AI. It's the money losingest thing we've ever done, and we're losing a lot of money to it.

Seven companies make up 35% of the S&P 500, the so-called Magnificent 7, and six of them are losing hundreds of billions of dollars. The company that is profitable is NVIDIA. That's the company that's getting the hundreds of billions of dollars and then loaning it to those six other companies so they can buy more of NVIDIA's products from NVIDIA. This is not good. The

sector is not making anywhere near what it's spending and it's spending more, but its earnings are not coming up.

You may have seen Anthropic argue that it was profitable. But if you look closely at that announcement, what you'll find is that anthropic is only profitable if you ignore the generally accepted accounting practices, GAAP, the gold standard for figuring out whether a business is profitable.

Fundamentally, what they've said is we're profitable in such a cool way you can't express it with normal math. And they can't tell us which math they're profitable by just that there is a math under which they're profitable. It's a little like their announcement about Mythos, which is like we're about to have an IPO. And we've invented a security auditing tool that's so powerful that you're not allowed to see it. Did we mention we're about to have an IPO?

And I just think like we have a thing that's really valuable that we'd like you to buy but we're not going to show it to you because it's so cool. It's not an argument that should persuade people. That's a monumentally unpersuasive argument. And it may be that it does good things. It needn't be useless to not be true. You could invent a security auditing tool that will perform a
$20,000 security audit using $50,000 worth of tokens. And that would be an impressive computer science feat and a way to go broke. And so without being able to look at it, we can't make any claims about it and we should apply a skeptical lens.

The economics get worse when you look at the claims about inference. If they can do what, say, the railroad companies do, which is like you put a lot of money into capital, you put down the rails and you buy the rail cars, and then your operating expenses are really low. So even though you had a giant CapEx, you can make it back. Or if you go bankrupt, then the people who buy your hard assets at a bankruptcy can make it back operating those assets and amortize them out over a long duty cycle. They keep claiming that their inference costs are coming down.
When we look at their actual internal figures, that's not in evidence.


So Ed Zitron is very good AI finance and tech critic, got a hold of a balance sheet from OpenAI, their finances from OpenAI. And what he saw was that it's true that they are on the line item for

inference, it's really low. But if you keep reading down the balance sheet to the line item for marketing, what you find is they claim they're spending as much on marketing as Coca-Cola.

Now, Coca-Cola has three global ad agencies that mostly just service them. They have billboards everywhere. If you want to find out where Coca-Cola's marketing budget has gone, you can just count noses in the ad agencies and count billboards by the side of the highway, right? And you can figure out that they're spending that money on marketing. That's not an evidence for OpenAI. Ed's theory, which I think is true, is that they have taken a bunch of $100 bills and sold them for $1 each by giving people really cheap inference. And rather than calling that an inference expense, they're calling it a marketing expense.

But even if we take them at their word and say, "Okay, you figured out how to make inference cheap, you've got the hard assets, you've done the capital expenditure involved in making the model, you can rest on your laurels," we can then look at the low switching costs as evidenced by the huge exodus from ChatGPT to Claude when the new Claude model came out that was demonstrably better than ChatGPT, which means that OpenAI doesn't get to just make a model and stop. They don't have to make another model or they're out of business. And the day they do, Anthropic has to make a successor to Claude.

How do you get these balance sheets right side up? How do you take a trillion dollars in expenditure and $50 million in revenue and turn it into a profit? How do you do it when your assets need to be replaced every three to five years? How do you do it when your operating expenses are not coming down? How do you do it when your unit economics suck? I don't think you do. I don't think that business has a future, at least not the way they're running it.

[0:54:34] JG: I look forward to the sequel, the reverse centaur, describing post-bubble all the open source models. I would really love to keep talking with you about this, Cory. This is fascinating stuff, but unfortunately, we've reached roughly the end of the book and the end of time. If you wanted people to find out more about you, your writing, the EFF, where would you direct them on the internet?

[0:54:52] CD: Start at EFF.org. Join, be a member. We're a member-supported nonprofit, but also sign up for our mailing list, find out what's going on, get involved in your local tech and

politics questions, and be a part of the oldest, most important, and most effective digital rights group in the world.

To find my stuff, I'm published by Macmillan. I've written more than 30 books, science fiction, science fiction for kids, science fiction for teens, picture books, graphic novels, tech policy books, collections of essays, even a little art book of my weird collages. And all of them, except for the weird collage book, which was a very limited edition, are for sale anywhere books are sold. And you can get my eBooks and my audio books anywhere books are sold, except Amazon won't carry my audio books because they're DRM free. So you can get them places that aren't Amazon, including from me directly at craphound.com/shop. DRM free, no user license. You bought them, you own them. Don't violate copyright law. But sell them, give them away, loan them out, it's fine.

And then pluralistic.net is my newsletter. I write four to six essays a week there and I syndicate it according to the grand tradition of POSSE, post own site, share everything. I turn every one of those essays into a thread on Mastodon, another one on Bluesky. I paste it into Medium and Tumblr. I'm the last Tumblr user on Earth. I have a old-fashioned mailman list. I use mailman because it's open and I use no analytics. I don't even know how many subscribers I have, let alone what my open rate is. My website has no tracking. My privacy policy is I don't collect any of your information, never will, and never use it in any way. And it's Creative Commons attribution. So if you see something there that you like, you can share it, you can give it away, you can sell it, you can translate it, make a video out of it, set it to music, whatever you want to do.

[0:56:41] JG: It's refreshing to hear. Well. All that being said, thank you again, Cory, for coming on to Software Engineering Daily. It's been lovely to talk to you. For SED, this is Josh Goldberg.

[0:56:50] CD: Oh, it's been my pleasure. Thank you, Josh. Thanks, everyone.


[0:56:53] JG: Thanks for listening.


[END]