EPISODE 1813 [INTRODUCTION] [00:00:00] ANNOUNCER: Jeffrey Ullman is a renowned computer scientist and Professor Emeritus at Stanford University, celebrated for his groundbreaking contributions to database systems, compilers, and algorithms. He co-authored influential texts like Principles of Database Systems and Compilers: Principles, Techniques, and Tools, both of which have profoundly shaped the field of computer science. Jeffrey received the 2020 Turing Award together with Alfred Aho "for fundamental algorithms and theory" underlying programming language implementation and for synthesizing these results and those of others in their highly influential books, which educated generations of computer scientists. In this episode, he joins Kevin Ball to talk about his life and career. Kevin Ball or KBall is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI in action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn. Or visit his website, kball.llc. [INTERVIEW] [0:01:18] KB: Hello, Software Engineering Daily listeners. I'm KBall, and it is my privilege and honor today to welcome Turing Award winner, Jeffrey Ullman, to the show. [0:01:27] JU: Hello, everyone, and thank you for inviting me to your esteemed podcast. [0:01:32] KB: Yes. I'm excited to talk with you. I know you have such a rich and varied background. How do you introduce yourself these days? I know you mentioned a lot of the stuff you did was old hat. So what do you describe yourself as today? [0:01:47] JU: I'm basically retired over the hill supernumerary. Yes. Let's face it, folks. Computer science is a young person's game, and I ain't that anymore. [0:02:01] KB: That is fair. Though, I think, some of the things that you did have been shockingly long-lived in this field, which moves so quickly. I think when I mentioned to someone, "Oh, I'm interviewing Jeffrey Ullman," they immediately were like, "Oh, the Dragon Book, the Dragon Book." I'd love to just spend a few minutes. I know that, once again, older, but what do you think made it so successful? It feels like this is something that first edition was decades and decades ago. I talked to new computer science grads, and they're still talking about the Dragon Book. [0:02:31] JU: Yes. Well, I think the answer is in the name. It was the cool cover. Apparently, people were proud to be seen walking around campus with a book that had a picture of a dragon on it. Therefore, they enrolled in computer science. That's probably the biggest contribution I've ever made in my life is getting some really bright kids who, otherwise, might have wasted their life in physics or something to go to the CS. [0:03:04] KB: If we look at compilers, I know when you started, the types of languages we were compiling for were perhaps simpler than some of the things that we've had today. There have been a lot of developments. Are there any changes that stand out to you over the years in terms of language design and compilers that you found particularly interesting? [0:03:23] JU: Well, I think the biggest thing is that we're now compiling for parallel machines in some sense, whether it's an 8 core processor or some massive supercomputer. There are issues which, I guess, the last version of the Dragon Book began to address a little bit primarily because Monica Lam, this is sort of her field, is parallel compiling. But that's something that when we wrote the first of the three editions, we didn't even really think was worth thinking about, and probably it wasn't at that time. That's '77, I guess. Yes, 1977. [0:04:07] KB: Yes. Well, and I think one of the threads that I've seen as I started researching your career and your work that very much shows in that book is thinking about and connecting from fundamental theoretical abstractions, and thinking about the different layers of abstraction and then how you apply them to practical problems, and move from this very abstract theoretical approach into higher and higher levels of context or implementation detail. I'm kind of curious, are there ways in which the abstractions we need have changed with that move towards more parallel computing? [0:04:46] JU: Well, let's see. Certainly, if you were compiling for a serial machine, you didn't need any of the abstractions that deal with concurrency control, for example. The MapReduces, I guess, an interesting abstraction of parallel computation, things that wouldn't have made any sense back in the 1970s. It certainly wouldn't have been useful. [0:05:09] KB: Yes. MapReduce is a good segue into - I saw some of your more recent work really was tied into these large-scale data mining and dealing with web scale. You had a book in that space as well. MapReduce is one example. What are the other key pieces that you think have gone into as we've scaled up and up and up into the current day? [0:05:33] JU: Well, I mean, it's pretty obvious that what has happened is getting all the data in the world together has given us this amazing power that we haven't. I'm talking about these large language models. I'm sure we know that there are - well, there's still some of the rough edges to be worked out. Let's put it that way. But, obviously, it gives people power to do all sorts of marvelous things. This same thing was true, what, 5 to 10 years ago when the massive neural nets came along. Again, what made them work was not only the availability of very cheap, massive computing devices but the availability of a massive amount of data. This is, of course, something that people often dismiss is the importance of the data. It's not just the algorithm. It's the data as well. For example, I've heard people are now worrying that LLMs have reached their limit because we've already taken everything that's ever been written, and there ain't no more. That's an interesting - well, we'll see, I guess, in a couple of years whether that turns out to be true or whether, for example, there are ways to manufacture data that never really existed in the way, oh, let's say if you're trying to do machine learning to recognize tumors. You can take an image of a tumor, and you can enlarge it 10% or rotate it 10% and create additional useful data without actually having any more data. Anyway, that's an interesting question. [0:07:21] KB: Yes. I think one of the key questions there is like what makes it actually useful, right? In the image example, that's great because it allows you to extrapolate things that look different to the computers, but that we as humans can validate are the same or equally valid. What does that look like in some of these other spaces? [0:07:40] JU: Yes, I have no idea. [0:07:44] KB: Yes, fair enough. [0:07:45] JU: As I said, it's an interesting question. I'm sure people are looking in this direction right now. [0:07:52] KB: Yes. While we're talking about this, I think I saw something that that you wrote about the importance of dimensionality reduction when you start to deal with these very large-scale data situations. I was pondering if there was a connection there to what people are doing with distilling models in terms of trying to pull out what are the relevant dimensions. [0:08:11] JU: Actually, I don't remember writing anything myself on that. I mean, obviously, the - [0:08:15] KB: It may have been in the books that you - [0:08:17] JU: Data sets book does talk about dimensionality reduction. It's often a useful tool in understanding data because there are various techniques for focusing on what's really important in some very complex high-dimensional data. [0:08:34] KB: Yes. Thinking about, once again, that abstraction lens that I saw in this world of LLMs, and you highlighted there are some rough edges. There are some challenges. What are the key abstractions for us as software engineers in working with these things? Or are there abstractions that we are missing and maybe need to figure out? [0:08:53] JU: It's a good question. I don't think we have really said the last word about how you use them. One of the interesting things is Stanford, for example, never taught a course on how to Google stuff, how to create search queries. But we are now teaching a course in what's called prompt engineering. Probably many schools are doing it or will be. There seems to be the idea of a prompt let's say. Maybe it needs to be abstracted or understood in some way. I have yet to see just a good set of principles being written down for how you use it. I've been playing around with a little bit myself. One of the things I do in my old age is I do a lot of editorial work. There's this journal that I work for where they have 300 editors, and I'm the only one in computer science, and so a lot of stuff gets dumped on me. I understand nothing about what I'm reading, and there's nobody to hand it off to. So what I've been starting to use various LLMs. Ask it things like first defining what a good referee is and then saying, "And here's an abstract of the paper. Suggest some people who would be good referees." Sometimes, it actually gets it just right and does introduce me to people I've never met but who turn out to be good referees. Other times, it just - sometimes, it'll say, "Well, this is too hard for me, but here's how you pick a referee. You read the literature and you -" Basically a completely useless comment. Frankly, I don't know what makes - is it the difference in the detail on the abstract or just how I phrase the definition of what I want? I don't know. There's a lot of very mysterious stuff, I think, that is still to be learned. [0:10:53] KB: There is, yes. Well, and I think you're right. There's an opportunity here. I saw a tool that was essentially trying to compile prompts more or less like a very simple compiler would, where you had different layers, and you could see it combining and lowering things down to different styles for different LLMs. I guess I could say it didn't have any optimization, right? It was entirely focused on how do I go from a high level set of components to something that will work for Anthropic versus ChatGPT. But I think there is something there of the equivalent of a compiler that understands what are the right ways to prompt these different models and maps from intent to something useful. [0:11:37] JU: Well, yes. Good research topic. What can I tell you? [0:11:40] KB: It would be a good research topic. Speaking of research then, you mentioned recently you've been actively working on a different project. Do you want to tell us a little bit more about that? [0:11:51] JU: Well, let's see. Again, I'm trying to just sort of keep busy in my old age. I guess it was really about the turn of the millennium that I and a number of friends started a little company called Gradiance. It was designed to automate home works, and the architecture that we hit on was this. To the student, it looks like a series of short multiple choice questions, where they're asked to solve a problem and then pick a correct answer from, say, four choices. But the difference between this and a routine multiple choice question was we felt that we wanted to use the home work to not only test but to teach. If you got it wrong, we'd give you some sort of a hint and ask it to do this whole thing again. The trouble is with multiple choice, well, if you guess A and it's wrong and guess B the next time and eventually you'll get the right answer without having to do anything. The way we worked it out was we developed what we call root questions, which are questions that have more than one correct answer. Now, how does that work? For example, if the question is solve the equation 2x plus 5 equals 10. Solve for x. Now, that has only one correct answer. I think it's two and a half. It looks like there's only one correct answer. But, in fact, what we're asking the student to do is solve the problem. Okay, you're going to write it down. If you've got it right, it's x equals two and a half. Now, the choices are not what is the value of x, but tell us something true about x. For example, you can have a correct answer, x is not an integer, or x is less than 3, or twice x is a prime. There are any number of correct answers that are easy to recognize if you know what x is. If you don't know what x is, you're just guessing. We thought we would make a lot of money selling this service. We didn't actually. Basically, people were willing to use it and still do use it. Our largest enrollment is from the University of Cairo, for example. People still use it, but they wouldn't pay for it, basically. I guess now almost two years ago, I gave it a talk on this technology in Bangalore, and someone in the audience was one of the original Infosys guys got all excited about this and said, "We could use this to teach mathematics," and you can. If you look at it, if you go to the Gradiance site, the materials that we have are really, well, computer science from Java programming, obviously, compilers, operating systems, data mining. But I said we can do this in mathematics. So he actually raised some money and got a team together. They're putting together - we've been working so far on ninth grade mathematics. By the way, ninth grade mathematics in India is mostly stuff I learned in 10th and 11th grade. But that's another story, I guess. At any rate, we're hoping to field these materials that anybody in India or around the world, I guess, can use for free. Again, the idea is not only make sure that students have learned the material, but whether having trouble and give wrong answers is an associated hint or explanation of some sort that we hope eventually gets them to get all the questions right. [0:15:53] KB: Yes. I feel like this is a broad topic area that people are very excited about right now, which is like how do we allow people to essentially scale education and go at their own pace and not depend so much on individual teachers? [0:16:07] JU: Okay. Now, that's an interesting point. Again, roughly the same time that we started Gradiance, these MOOCs were all the rage, and it was somehow assumed that these were going to replace universities. [0:16:22] KB: Yes. And where did that go? [0:16:25] JU: Yes. Where did they go? Okay, well, it's a good question, and I think it's an important one to answer. Part of the problem is this. When, say, General Motors decides to replace assembly line workers by robots, nobody asks the assembly line workers whether they think the robots can do a good job or not, okay? When you offer academia alternative ways of, essentially, introduce technology into the education process, teachers have traditionally had the ability to say, "No, this is not acceptable. I don't want to use this. Keep it out of the classroom, basically." This is - I can't think of too many other fields where that is true. It turns out, I think it was a little bit worse than that. Not only was there obvious resistance on the part of people whose jobs are threatened. But it turns out that the MOOC doesn't really give you enough for 95% of the students, that people need help. Again, you can automate a little bit of that. I think Gradiance does a little bit, but you're always going to need somebody to go to when you're stuck. The MOOC cannot eliminate the need for teaching professionals. It can perhaps take some of the burden off of them and thereby make them more productive, which by the way means we need fewer of them. That's life. It's true in every field, right? I mean, I think we could do more to - if instructors were willing to, let's say, play a MOOC and then much the way they used to prescribe a textbook, right? I mean, the textbook did some of the job of teaching. The MOOC can do a little more of the job of teaching, but you still need somebody to conduct the class and in particular to handle all of the special cases that keep coming up. Then, of course, there's also the problem of validation. You can take a MOOC and have your smarter older sister actually do the work for you, and nobody really knows. Just because you've registered for something and then seemingly done the MOOC requires, it doesn't mean it was you. Okay, there's technology for - you go to a special center, and you get fingerprinted and everything. But that's not really going to work. That's not a long-term solution. You need people, instructors to not only make sure that everybody gets it. The people who are having trouble in a million different ways, they are helped through whatever problems they're having. But, also, just to validate that they have, in fact, learned the material, I think, is - that's not going to go away. I know there are technological solutions that do it, but I can understand why schools don't want to automate the validation of their own students. [0:19:50] KB: I'd love to take this conversation about learning in a slightly different direction. You talked a little bit about Stanford now offering a course in prompt definition or prompt engineering. [0:20:00] JU: Prompt engineering is apparently the term. [0:20:02] KB: It's the term, yes. We can argue about whether it's engineering or what it is, but you talked about that. We've talked a little bit about applying technology to help scale learning, both what you're doing at Gradiance, a little bit what these massive online courses were doing. I feel like we are in a time period where there's a lot of resistance to technology change. There's a lot of resistance to LLMs in different parts of the world. So I'm kind of curious how you think about how do we bring people along with the changes that are happening. [0:20:37] JU: Boy, it ain't easy. I think you just have to get new people. [0:20:42] KB: I mean, that is one solution, right? You wait for one generation to die off but - [0:20:47] JU: Look, having aged, I see it in myself that it just becomes harder and harder to adapt to new technology. By the way, the field of HCI has been around for forever, really, human-computer interaction. I haven't seen anybody addressing the problem of dealing with the old folks and the fact that - I can't explain it, but you just - [0:21:19] KB: I've seen this as well. In my parents' aging, they - my dad is in his 80s now. My mom has passed away. But the rate of change is just hard. I feel like in the tech industry, we often change just to change and neglect the cost of that. [0:21:36] JU: But it's not just change for the sake of change. I mean, the fact is, for example, I had to learn to use and depend upon a cell phone. [0:21:48] KB: Yes. [0:21:49] JU: For example, I have a 25-year-old grandson who was out visiting the escapades. He likes to borrow a car and just drive around, and so he did that. I have a NAV system, and I know how to use the NAV system. But he says, "Well, I just want to use Google Maps." Okay. And he says, "Well, you needed the right adapter, and you can just plug your cell phone into your car, into my car." I've never known that. But apparently, you can. These are the kinds of things. I guess young people, they all know that, that that's how you navigate these days. I was visiting my son. I have a five-year-old granddaughter, and I brought along a tablet. They're not allowed to play with these normally. But when grandma and grandpa show up, they - [0:22:45] KB: Yes. You get grandparent privileges. [0:22:47] JU: Yes, right. She wants to download an app to do drawing. But the app has ads, and that gets you to - basically, there's a strange ecosystem because it looks to me like all of these apps, all their ads are for other apps. That's basically an economy based on everyone taking in each other's laundry. I'm just not sure exactly how that works, but this is what happens. She's wandering from app to app. I said, "Look, you got to get around the ads." Fudging around, I sort of found it. Fifteen minutes later, I noticed she's found her own way, and it's better than my way. That's what five-year-olds do. [0:23:34] KB: Yes. It's funny. It reminds me of when my son was that age. We would joke that somebody would hand him a device of their own for whatever to entertain him. Within minutes, they'd be saying, "Wait, how did you do that? What is that?" [0:23:48] JU: Yes, exactly. I think a big question is how do you make it possible to teach old folks who have - I guess we have our own skills and our own ways of doing things that don't always work. For example, how do you recognize what it is your customer, say me, is capable of doing or feels normal doing, I guess, or has the skill to do, and use that to get them to navigate the device? [0:24:27] KB: I think this is really important, and it's a big blind spot. Because as you said at the beginning of our conversation, tech is generally a young person's industry. The people making these decisions and thinking these things may not themselves be interacting with older folks very often at all. Even middle-aged folks, there's a big difference. I'm in my 40s. I naturally interact with most technology via text. My kids go to voice and video for everything. It blows my mind. I want to look up how to do something. I'm writing in text. I'm looking for an article on it. They're looking for YouTube. We as an industry, especially looking at our society aging, we do need to figure out how do we incorporate the preferences, needs, and differences of older generations. [0:25:19] JU: I'd love to see that. [0:25:20] KB: Have you seen anyone doing it well? [0:25:23] JU: Doing research in the area or? [0:25:24] KB: Either research or even like some company managing to do a good job with their products on it. [0:25:29] JU: Well, I have not seen really good examples of the systems that adapt to, let's say, a variant mode of thinking about it. But I get the feeling it should be feasible. [0:25:44] KB: It should be feasible. Well, and I feel like I do wonder if this is a place where modern machine learning, LLM technology can help because it allows for what I've been describing to some people as more intent-based user interaction paradigms, where it's able to do some amount of interpretation. If we can use that in a way so that we can try to infer intent even if your natural mode of trying to interact is different than my natural mode is different than your grandchild's natural mode, to be able to figure out what you're trying to do and help you with it. [0:26:19] JU: Yes. I think the new LLM-based technology, as a simple case, it should be able to summarize a YouTube video. [0:26:27] KB: Yes. [0:26:30] JU: I think an interesting case that doesn't seem to yield to LLM technology would be. I mean, I have a map on my cell phone. I want to connect it to my car's screen. I don't know. Okay, it's in the manual, right? There's a 500-page manual in my glove compartment. Somewhere it must tell you how to do that, but I don't even know that it's there, that it's an option. Another example, my son had to show me that my trunk has a button that lets me open the trunk from the outside without pressing the button that's in the car itself. I think all cars have this these days. Nobody told me that it was there. [0:27:22] KB: Yes. There's sort of a dissemination of information problem, right? If you knew to ask the question, you could have found the answer. But how do we surface the questions? [0:27:33] JU: That's exactly right. Now, the car should say, "I notice you've been pressing the trunk release button quite a bit. Did you know that you can?" [0:27:43] KB: Yes. [0:27:44] JU: That's not unreasonable, and I'm not sure it's an LLM problem. [0:27:50] KB: But there is something there, and it actually reminds me of what you're doing with Gradiance of like there are many correct answers here if you have the underlying model. Can we build our technology to notice that there's a class of them that you are just systematically ignoring or unaware of and surface them to you? [0:28:11] JU: Anyway, that's what HCI people ought to be doing, I think. [0:28:15] KB: Yes. One of the things I love about talking with folks who are older and who have seen so much about this is you often have a much wider perspective than someone who's only seen 10 or 20 years in the industry. What else do you see as gaps in our current tech ecosystem and environment? Where should people be doing work that they're not? [0:28:36] JU: That's always the hard question to ask. If you look at the great changes, nobody realizes that we need it until somebody provides it. I'll give you an example. The date is very important. I think it was like March of 1992. I was invited to DC for a discussion about what was then called the Information Superhighway. It was some academics like me and the heads of telecom companies and other industry types. We wrote a report. It came out of this. The interesting thing is nobody mentioned the World Wide Web. I mean, that was some discussion of gopher, I think, and a big question was would the Information Superhighway run along cables or telephone lines. Here in Switzerland, the World Wide Web existed, and none of these 300 experts, I take the blame for it as well, was aware that this was even an option. I mean, I remember several months later asking one of my colleagues. Why do you always keep referring to things as colon, slash, slash?" It just - nobody was even aware of the need for something like the World Wide Web. I think you can probably say the same thing about cell phones, personal computers. There's this famous quote from, I guess, the head of DEC at the time, "Why would anybody want a computer on their desk?" The answer is I cannot really tell you what the next or even guess what the next great thing is going to be. [0:30:42] KB: Certainly not, but you have already pointed to something that's missing right now, which is HCI research focused on older generations. [0:30:51] JU: I think it's a good research topic. I don't think it's going to change the world. I mean, if I had to guess, I would certainly have to think about quantum computing. I've always been sort of a skeptic about quantum computing. Apparently, there are some applications. That is you can simulate some things better using quantum phenomena. By the way, I should say it does look like quantum communication is real, this spooky action of distance story. [0:31:27] KB: Yes, instantaneous communication, mind-boggling. [0:31:30] JU: Yes. That people are actually possibly able to do things of that nature, and that may become real. But what people have been expecting is that there are going to be these quantum computers with the right kind of qubits that will enable you to implement Shor's algorithm and break RSA codes and elliptical codes and things like that. I mean, that would certainly be a transition in our way of thinking if that were the case. And sort of dubious. I mean, things like the D-Wave machine, apparently, use a kind of qubit that cannot implement Shor's algorithm and the scale-up that seems to be going on, which, by the way, it may not be following Moore's Law. We don't know. Are we going to double the number of qubits available every two years? In particular, qubits that are adequate to really run Shor's algorithm. I just don't know whether that's going to happen. It might. You might be able to predict some big change coming along the pike. That's something. It's a candidate. Let's put it that way. Whether that's going to happen or not, I don't know. We talk about artificial general intelligence maybe. [0:32:58] KB: Yes. You mentioned a few technology shifts you saw that you did not expect in the past. If we were to look back at software engineering in particular as a field rather than tech in general, since most of our audience, I think, are software engineers, what have been to you some of the biggest shifts or breakthroughs in the way that we think about software over the years? [0:33:23] JU: Going way back. The first programs I wrote were in kind of a machine language, and I graduated to FAP. That's the IBM 790. Then I learned Fortran, and that was a big power booster. I think people are all so excited about LLMs are going to write your code for you. Well, I think it will probably make life a little bit easier, make you a little bit more productive. There seems to be some good evidence that that's happening compared with moving from FAP to Fortran. I don't think it even compares. As I said, the fact that it makes sense to think parallel has certainly been a big, big change. Again, I think it's more subtle and gradual but the change from thinking of software as algorithms to thinking about it as algorithms plus data. Again, slowly that has become - it just changes the way you think about that you can do with a computer. [0:34:37] KB: Absolutely. Well, one could argue that a lot of the advances in machine learning are just more and more you're programming with data, and algorithms are, if not a forgotten element, at least taking more of a backseat. [0:34:51] JU: Because you need - the data just sits there until you apply an algorithm to it. [0:34:56] KB: Fair enough, yes. [0:34:58] JU: For example, we talk about prompt engineering. Well, I mean, ultimately, I think that subject is going to become algorithms for turning desires into sequences of text, which at some point may not even be text. It could be videos or whatnot. [0:35:23] KB: Yes. You're guiding it with a human language algorithm, essentially. [0:35:27] JU: Yes. Okay, in a sense, it would be an algorithm. Well, who knows? There may, in fact, be tools. Your computer will help you to formulate a prompt. There could be computer-implemented algorithms as opposed to just recipes that people follow. [0:35:45] KB: Absolutely. Well, and I think a lot of the development in that space has been applications building prompts generation around these tools, so they can do things for you. Awesome. Well, thank you so much for your time. Deeply appreciate it. [0:35:58] JU: Thank you for inviting me. [END]