EPISODE 1958

[INTRODUCTION]

[0:00:00] ANNOUNCER: In recent years, the main impact of AI has been on the digital world and most notably, software engineering. However, its longer-term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move, or a car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos and sits at the frontier of physical AI and robotics. Amit Goel has been at NVIDIA for 15 years, and he is currently the Head of Robotics, Ecosystem, and Edge AI product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two-tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space.

Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in back-end development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at DEPT Agency. You can learn more about his work at code.theothermattm.com.

[INTERVIEW]

[0:01:49] MM: Hello, everybody. My name is Matt Merrill, and I am here today with Amit Goel from NVIDIA, who is Head of Robotics and Edge AI. I'm really excited to be here to talk to you about physical AI. Why don't I turn it over to you, Amit, to introduce yourself?

[0:02:06] AG: Hey, Matt. Great to be here. I'm Amit Goel. I head up our robotics ecosystem at NVIDIA, which means my team gets to work with all the developers who are doing some amazing work with AI and robotics. I also lead our product team that builds the computers for running the AI on the device on the edge. It's called the Jetson family of products. I've been in NVIDIA for 15 years. I'm really excited to talk about physical AI today.

[0:02:36] MM: All right. I am super excited to talk about this, because I think that a lot of folks listening here are probably, at this point, pretty familiar with the software side of Gen AI and LLMs. What I want to do today is just start from zero, right? That's actually, I'm not playing a role here. I purposely didn't dive into this deep, so that I could be fresh. When NVIDIA, or anybody, says physical AI, what do you mean? What makes it different from LLMs and coding agents that most of our listeners are using every day?

[0:03:10] AG: Yeah. The big difference between LLMs, or in general, digital AI and physical AI is that you're working with atoms. What that means is you are dealing with sensors; you're looking at spatial information, 3D information, you're looking at videos, you're understanding dynamics of the world. Physical AI, essentially, is a model that can take as inputs the information of the world around you and predict what's going to happen in that world. Everybody is probably familiar right now with self-driving cars. If you live in the San Francisco Bay Area, you'll see the Waymos driving around. That's probably the closest way to think about physical AI in the real world. You have a machine that is taking the information from the cameras, the Lidars and the actuators in the car and making a decision on what it should do next based on what the driver wants out of the car.

It's taking the data from the observations from the physical world and determining actions that have to be executed in the physical world. It's been around for a while. The simplest form is probably your Nest Thermostat that's controlling the temperature of things in your house, based on your activity, and even a small traffic intersection that's observing what's the activity that's happening and controlling that particular traffic light. Physically, it has been around for a while, but it's been very narrowly defined for one particular use case. Now, what's happening with the really large models and general models is that you can have a single model that can understand all of this and similar to what happens with LLM. You don't have a sentiment that now this is modeled. You don't have a translation model. You don't have a summarization model. You have a one big model that understands everything in the textured world, and you can program it to do what you want it to do. We are getting to the point where we can build this general physical intelligence models that you can use for a lot of application in the physical world.

[0:05:27] MM: Got it. Yeah. That was one of my big questions. I never really thought about that, like 

a self-driving car, or even a Nest Thermostat is technically physical AI. But when I was prepping for this, seeing some of the stuff that is currently happening is so much more advanced than that. It makes sense that in the past year, the explosion and what these models can do enables a lot of this. Let's dive in and try to give folks a mental model of how these things break down. One thing that I was reading about prepping for this was a concept of a world model. Can you tell folks about that and how that fits in here and how it might drive one of these devices?

[0:06:06] AG: Yeah. The way to think about the world model is in the context of what are the inputs and what are the outputs. For a world model, the inputs are observations of the current state of the world. It could be a sensor data from a motor of a car of a robot. It could be a signal that is audio command that has been given as an input. It could be a text-filled prompt. Of course, it has to have the cameras and the visual representation of the world that's coming into it.

The next question is: what does the world model predict, right? Inherently, the world model, because it's dealing with the physics of the world, is a physical AI model; it is multi-modal because the real world has multiple modalities. We are hearing, we are listening, we are processing information. It's multi-modal: many inputs and also many outputs. The typical output of a world model is a video, which given a prompt given a current state, if the world model understands what happens in the world, it can predict, based on what you told me, this is what's likely to happen. In the case of a car, you can say, "I'm going at this particular speed. This is what I see is happening around me. Tell me what is likely going to be the state of the world in the next five seconds." Based on that, I can then control the car through what action it should take.

Same analogy applies to a robot, right? You asked me to pick this thing and put it over there. This is what I see is my current state of everything around me. This is the object that I see. How should I move? Should I go left? Should I go right? Should I go backwards? Do I need to bend? It can predict what is the next few things it needs to do. The world models can predict the world around you. It can do a 3D representation of the world. It can also understand, "Hey, I want to be there. This is my objective. How do I get there?" What to think about in our head it's called planning.

The world models can predict the plan to go from the current state to the desired state. This is essentially the planning function of the world model. The other thing is you can say, "Hey, this is my current state. I'm planning to take this action. Can you tell me what is going to happen if I do this?" That is super important for robotics. If you can say, "I'm going to do this and tell me what's going to happen," you can actually decide whether that's the right thing to do or not. Rendering the entire world around you, given an action, predicting what the world is going to look like after you take that action. Or, given the current action and the desired state, predicting the action I should take to get to that desired state, these are all of the outputs of a world model today.

[0:09:00] MM: That sounds like it could be, especially for something like a self-driving car, an absolutely enormous model, or set of models. It just seems like a lot - video, potentially audio. I don't know. Just a lot. I mean, I'm almost having a hard time wrapping my mind around how much stuff is going into that.

[0:09:19] AG: No, you're right. I mean, the amount of data that gets fed into a world model is orders of magnitude more than what you would feed into a text LLM. Because just a video of 15 minutes is several gigabytes of data. Of course, the amount of data that gets processed in a world model is a lot. At the end of the day, I think the size of the model we see varies depending on what you want the model to do. You were asking about physically, the difference between LLMs, and one of the things is that you have to run some of these models on the device itself, on that particular robot, or on that particular car. You do not have the luxury to say, it's okay if it takes one minute to get back to with an answer. I'm going to throw all the compute available at it. You need to think about how you're going to make distilled version of these world models that can run on device.

For something like evaluation, which can happen offline, right? If you're training a model, you collected some amount of data, but you want to train what it is going to do if it was in a particular street of New York. You can just ask the world model to give it a particular image and say, "Can you create a world around me? And I would drive this thing in New York." That part is super heavy, but that part can also run offline. You can throw a lot of compute at it.

At NVIDIA, we've been working on the Cosmos models, and a variety of those exist. We have many different sizes of the model, starting from things that can run on to the edge, all the way to, really, we have nanos and super models. They can be used for creating data. They can be used for evaluating things offline. You're 100% right. These models are processing a lot of data.

[0:11:12] MM: You anticipated my next thought comment, which is just the speed. These things can't wait. It's not like me waiting for Fable to analyze my next presentation, or something.

[0:11:20] AG: That's right.

[0:11:22] MM: Yeah. Yeah.

[0:11:23] AG: You cannot just leave it around and come back and see what happened, especially if you're deploying it on the device itself.

[0:11:30] MM: Yeah. That's amazing. I'd love to walk through - you get these interview questions of, I remember back in the day, used to be like, what happens when you press enter on Google, right? I want to give you this for one of these situations. We can pick a type of robot, or whatever. What happens in 100 milliseconds between that thing capturing, let's say, an image, or some sort of multiple sensor inputs, and then a motor command going out of that thing to perform an action?

[0:11:59] AG: Yeah. Typically, if you think about, let's say, a robot, the moment you are capturing the information from the robot's camera, typically, you need more information than just the camera. You need to also understand what is the state of each of its actuators. You need to understand some of the other sensors, like IMU, which is an inertial measurement unit that tells you the pose of the different parts of the robot. Once you capture the information, first you need to synchronize all of this. What that means is my camera is observing at this particular time. At the same time, what was the state of the motors? What are the state of the IMUs? You need to fuse them together.

The next thing you have to do is, if you're given a task, let's assume we are running a version language action model. You need to tokenize this data- all of this data: the data from the current frame, but also the data from the past frames, because your next action is dependent on not just what you're currently seeing, but also a little bit of history. You have to take the data, you have to tokenize it, and then you feed it into this model, which is now going to do two things. First thing it has to do is it has to do a planning, which is the task of taking my current observation, my past observations, and my objective function. What should I do, right? If I was asked to pick up that particular cup, do I need to move left? Do I need to move right? Do I need to go up or down? How far do I need to reach? This is the planning task. And today, this is something that is done by an order-regressive model that will predict the sequence of tasks that you need to do, based on the objective function and current input.

Now that's your reasoning-in-planning brain that kicks in. The planner tells you, okay, based on the tasks that's given to you, this is the next step you need to do. Now comes in the second piece of the model, which is generating all the actions. That part is very complex, because depending on the robot type, you could be controlling 30 to 40 motors, and you need to generate actions of all of that synchronously. You cannot see really generate that, right? You need to control the output. One thing that's very unique about physical world is that a lot of things are continuous. You have to control the current, and you cannot send it for this much current; it has to be continuously generated. The models that you need is different than what you need to predict a discrete output. A sequence of text versus the current for 50 motors that you have to send it at every millisecond. The frequency at which the output must be generated is very different for a robotic and a physical AI model.

You have to make these two models work together. A reasoning model that is planning can exist at a low frequency. The inputs of that goes to another model that actually generates the motor control commands, and that one is running really, really fast, because it has to control all of that at high frequency. Once you have that output, typically that's happening in the brain, but typically at the end of that, it goes to a small computer that is attached to the motors, which does your PID control based on the input that it got from the model.

[0:15:33] MM: I'm not familiar with PID. What's that?

[0:15:36] AG: PID is the closed-loop control, that low-frequency control loop; that is, you have a target position, you get the current up position, and you control the current. That is running right onto the motors. It does not need to know what's happening on the other 30 motors, because its objective is to control that one motor, and that runs on a very small computer, or a microcontroller.

[0:15:58] MM: I'm thinking of this in terms of a BIOS, or something like that, just very simple.

[0:16:02] AG: Yeah, yeah, yeah. It's a very simple phone where that's running on the -

[0:16:06] MM: Just taking commands. I'm listening to this, and I just can't help but think, like, this is the world's most insane state machine ever. That's just so much to keep track of.

[0:16:17] AG: Yeah.

[0:16:19] MM: You talked about this in a way that almost seems like there's two different models happening. There's a reasoning and a planning. But yet, they're all sharing the same context, right?

[0:16:31] AG: That's right.

[0:16:32] MM: How does that work? How does it share that?

[0:16:35] AG: Yeah. That's an interesting part. There are many ways these models are being architected. One of the ways that we are looking at this is having a shared latent space, where the attention from one model and there's a cross attention between one model and the other model; the embeddings are shared, so that you have a really high-bandwidth communication between these two brains. If you look at the latest Cosmos 3 family of models that NVIDIA released, we have these two heads, right? A reasoning head. On top of that is the - we have the reasoning tower and the action tower on the output side, and they talk to each other. They have shared context in which they exchange information in the latent space.

If you were running them distributed, you may want to take the output of one, decode it, and send it to the other one and again, encode it. But that slows down the bandwidth. As I was telling you, for a robotic system, they need to operate really in a closed loop. That's why we build Cosmos with that shared context, so that both of these can coexist and exchange information with very high bandwidth.

[0:17:46] MM: I'm almost picturing this just like, I don't know, I'm trying to think of a CPU analogy, but it's logical. It's almost like a logical division inside the same hardware. Is that correct? Or do I have that wrong?

[0:17:58] AG: You're absolutely right. Think about in a physical space, we have a shared memory where you have two different processes; you don't want to send the data over because it raises your bandwidth. The same thing we are doing at the model level now, where, in the latent space, we can share the information and the context between these two models, so they can operate; although they're operating in different frequencies, they can share the same context.

[0:18:23] MM: Got it. That's really interesting. I was going to ask about that two-tower design, because that was something I came across and it didn't make sense to me. But as you explained that reasoning step, yeah, that does make more sense.

[0:18:35] AG: Yeah. Another way to think of it is like, it's an architect, which is planning everything that you need to build. Then there's an executioner, which is given the plan is doing the execution. They both need access to the same information, but what they do with that information is slightly different.

[0:18:55] MM: One of the things here that's like, I mean, we talk in textual models about using GPUs, because they're really good at this task, but I can't help but think this requires just so much more horsepower, so to speak, than what those need. If you can simplify it, what is the difference in the hardware that's powering these things?

[0:19:16] MM: There are a few things that is different. As I mentioned, when you're thinking about the world model, the number of modalities that you're dealing with is quite different. What that does to your architecture is when you're training a model, when you're dealing with the text inputs, the bandwidth that you're processing for data itself is not that hot. When you're working with the world models, and you're training a world model, you need to retrieve the images; you need to retrieve the actions, or the modal control commands. You need to retrieve the text annotation for all of that in parallel and bring it into your model. That creates pretty unique challenges from the hardware perspective as to what do you keep in the storage. How do you manage the data? We, in fact, had to go ahead and build out a completely different stack in order to process this data, because GPU resources are super scarce and precious.

A lot of this pre-processing typically was done on a CPU. You would have your model waiting for the data to do the training, but it's just not possible to do that with the existing pipelines that were built for text-based model training. We created something called a curator, which was the ability to now take large amounts of videos, process it, store it in a way that now you can keep the training data coming in. That's on the data side of things.

On the hardware side of things, when it comes to training these models, as we know with LLMs, also, there are multiple phases. You have this pre-training, there's mid-training, and then there's post-training. You see a lot of post-training happens when using reinforcement learning; you're using different tools. The same is true for world models. But when you're training a world model, let's say it's a robotics world model, how do you verify it? How do you do the reinforcement learning? For a coding model, reinforcement learning as you go run it, compile the code, verify it works or not. For a world model, this job is to simulate the world; you actually need to go to a physics simulator as a tool. That physics simulator is best run on another kind of GPU, which has the ray-tracing cores that can do the sensor simulation.

It becomes a heterogeneous computer architecture. On the one side, you need something that can process all of this data. Then, of course, you have your transformer model that needs to get trained. Then after that, you have to continuously call a simulator where you go and do the reinforcement learning and testing, whether if it's working or not. It becomes a pretty complex architecture. Inherently, it has become the heterogeneous computing problem. That's one other thing that developers who are used to LLMs get access to a whole bunch of computers, all latest Blackwell clusters, that they can go and just train their models, test the models, evaluate it, serve it all on that cluster.

Physically, it requires actually three computers. You need a computer for training the model. You need a computer for where you do simulation, either to create data or to do reinforcement learning post-training. Then last piece of it: deployment. You cannot deploy it on the same infrastructure that you trained it on. You need to deploy it on a small computer that's sitting inside the brain of a robot. You need to go do some more work on the model quantization, distillation, compression to go deploy it. It requires a very different way to think about the compute hardware that you need if you're building a physical AI model.

[0:23:10] MM: I'm joking here, but I'm imagining you being at a cocktail party talking with a software developer about textual LLMs and going, "That's cute."

[0:23:19] AG: Yeah.

[0:23:20] MM: Just kidding.

[0:23:22] AG: I think Jim Fan, who leads our research team for a physical AI in humanoid robotics, he often jokes like, "I'm always envious of my friends in the LLM world. Their problem is I don't have more GPUs. My problem is like, okay, first of all, the data that I got: half of the sensor data is missing, and the synchronization is not there. Then I'd have to - I get the GPUs, I train it, but next I have to find a simulation environment to go test it, deploy it on the rear robot. The cables are breaking." It's a very different word. Yeah.

[0:23:56] MM: That is something, the simulation piece of it, never even - I mean, yeah, the horsepower even just to do that simulation is mind-boggling. Switching topics a little bit. When these things are deployed, they need to react to somewhat unpredictable things, right? They effectively need to learn. I read a little bit about imitation learning and watching humans do tasks. Can you say a little bit more about how that happens in the - at least in current research and in the real world and how that operates?

[0:24:29] AG: Yeah. two Things that's being done right now to make these models more generalizable. First is to collect data in diverse environments. If you are supposed to assemble, let's pack a bag of shoes, right? Often, the size of the bag, the shoes is going to be different. The size of the box is going to be different. One way to do this is you have a lot of people collecting data in different environments, in different factories, bringing the data off, doing the tasks in different environments, and training the model so that it now has seen enough distribution, so that the real world falls somewhere within that distribution. That's why when you see imitation learning, that's what is happening. We are collecting the data, increasing the distribution in which it's been collected.

As it turns out, however much data you collect in the real world, the real world has an infinite distribution. It's continuous. You can collect it in 10 different lights, but there's going to be something that's always going to be out of distribution. That's where the second part comes in, where you need to think about having reasoning model paired with a model that has seen in our distribution. They go together, right? You cannot just have a reasoning model, but not have enough real-world data, because the reasoning model can tell you this is closest to this particular real-world distribution and then you can go act on that, right? The reasoning model is learning from data in the wild. It's using all the text data that's available. It's using the video data that's collected across different domains and learning from that.

Combining this pre-training with data in the wild, textual video data, along with a real data, action data, which is people doing the task, robots doing the task, combining these two, the world models are now trying to solve this problem of how do I know enough that I'm physically capable of doing the task? I know how to pick up an object. I know how to slide something. I know how to bend over and perform a particular task. How do I do it in different environment in a different context? Both of these things are being trained together. The first one, how do I do a certain particular task is being done in combination of either imitation learning and reinforcement learning. Then, how do I do this in any environment? This is where the data in the wild is playing a role.

[0:27:14] MM: Okay. The reasoning models are these various tasks specific?

[0:27:18] AG: The reasoning model, this is where the generalization is coming from, right? You're trying to collect diversity of information. Your robot might be designed to work in a factory, but you're putting in all the data from homes, from outdoors, from indoors, because that's when you build a reasoning capability that can generalize and you can drop out of those pricing.

[0:27:40] MM: Yeah. That makes sense to me that you need to give it tons of different inputs to be able to react to different things. Okay.

[0:27:49] AG: Yeah. Because if you were just looking at factory, you would probably not see a cat, or cow coming. When you're doing the reasoning model, you expose it to all elements.

[0:28:00] MM: This is a little random digression. I was on Reddit the other day and somebody posted - I lived near Boston, so it was on the Boston subreddit. Somebody posted a picture of this guy with his wacky looking backpack on. It was a crazy backpack. People were making jokes like, "Oh, he's the rocketeer," or whatever. Somebody was like, "No, I think that they're gathering world data for AI models." I see you nodding your head that that's probably what that guy was doing.

[0:28:26] AG: Yeah. One of the things that is happening is simulation is important, as I said, for physical AI, because that's where you can safely test your models. That is where you can do reinforcement learning faster than real time speeds. The challenge with simulation models is the cost of creating that simulated world is quite high. What you probably, likely saw was somebody carrying a backpack with some cameras and Lidars. What they are doing is capturing the real world and passing it into a model that can create a 3D representation of that world, what we like to call Real-to-Sim. You take the real data, run it through a model and creates a simulated version of it. Now, once you have that simulated version of it, you can drive in that simulated version and test how it's going to - how your robot, or your car is going to behave. You can use that and do augmentations. What if these buildings colors were different? What if the lighting was different? What is the road was narrower?

You are closer to the actual distribution. This data is often used for post-training. This is where I want my car to operate really well in this particular block. Okay, I'm going to collect the data and let my general model post-train in this world. Now it becomes really good at that environment.

[0:29:56] MM: That's really interesting. That's actually the opposite of what I might have thought. Thank you for clarifying. Cool. One thing that just came to mind here is hallucinations and text models, or you just chuckle at them. But hallucinations in a physical space sound a little scary. I would imagine that there's better guardrails for things like that, and I'm curious how you handle that in the physical AI space.

[0:30:22] AG: One of the things that is important for physical AI is, as you mentioned, that if you are out of distribution, if you're hallucinating, the impact is pretty significant. You have to do two things for that. Either you can make your model really conservative and says, "Okay, this is out of distribution. I see something that I've not seen before, or this is not part of my learning curriculum. I'm not going to act on it."

The other thing you can do is, well, I'm going to act on it. In fact, the way I'm going to make sure that I know how to handle these out of distribution scenarios is I'm going to actually give those and simulate in that environment. I'm going to evaluate my policies, or my models in a lot of different environments, which some of them may not even be real, and then score them on their performance. One of the ways to make sure that your robot policies are not hallucinating, or if they are out of distribution, they're still safe, is to expose them to those adverse environments in simulation and see how they perform. That's one of the ways to make sure your policies and the models that you're training are within the guardrail.

The second thing also is, and remember we talked about a planning brain and an action brain, right? The planning brain is like a VLM, or an LMM, where you can do the guardrails into that so that it does not plan something that is not acceptable. If it does not plan, then the action brain cannot - is operating on the plans that are coming out of it. How do you put the guardrails around your planning? It's very similar to how do you put guardrails around an LLM to not give you an answer, or an information that is not suitable, right? You can restrain things at the planning stage. On the action stage, again it is mostly learning in domain, you can go to the simulation and test it.

The third thing that we are also thinking about is when it comes to safety and actions is you have to have some level of redundancy, some level of external element that is observing and verifying. Because that separates the concerns. The person that is acting and the person that is monitoring, if there's another AI that can observe what is happening and then have a way to control the robot, or the car, that's another way to manage safety. I think it's a combination of all of these things that are coming together to make sure that as we are building these models, as they go into the real world, we can make sure that we're not being extremely conservative and not being able to do half of the tasks. We don't want that, but we want them to be able to do that safely, so we are bringing this outside in element of safety where you're monitoring, you're looking at it and making a decision, you make sure that your reasoning brain, planning brain has the guardrails and then extensively test in simulation.

[0:33:42] MM: That's one thing I wanted to dig in when you say - when you say it evaluates those plans, it makes sense in training that it does that, but what I want to clarify, is it also doing this in the real world, too? If it is unsure about its action, is it actually going to run simulations in real time to evaluate which one may be the best?

[0:34:02] AG: That's the biggest value of the world models is. They can do counterfactuals. If you look at the work that we published recently with our self-driving car team called Alpamayo, it's a reasoning model. And within that, it can reason, it can predict 10 different futures and choose one of them. Because it is a world model, it is not limited to only the right thing that - it starts this imitation learning on this is the right way to do it. It can explore 10 different directions, 10 different outcomes, score them and act on the safest and the one that gets you the most through your objective function.

[0:34:47] MM: I see.

[0:34:48] AG: That's the beauty of the world models. It is doing that at evaluation time.

[0:34:53] MM: Wow. Yeah. Yeah.

[0:34:55] AG: It's important to have these models open for that reason as well, right? Because what is acceptable in one part of the world versus what's acceptable in another part of the world, thinking about driving rules, right? What is an acceptable driving behavior in Boston is likely different from the acceptable driving behavior in the Bay Area. And if you go to a different country, of course, it changes. Taking these world models that can predict all the different counterfactual scenarios and then post-training them to choose what is acceptable is important.

[0:35:29] MM: When I heard you talk about the security, I can't help but think like, this is security in layers, security in the planning stage, security in the, yeah, in the execution stage, evaluating options. That's really good. And that is not something that, well, I don't know, I was about to say, that's not something you hear too much about in the LLMs, but that's probably not true. We're starting to do things like that as well.

[0:35:51] AG: That's right.

[0:35:53] MM: You mentioned the open models, which is really exciting and also a little bit scary, because one of the things where my catastrophizing brain goes is like, what happens if this tech gets into the wrong hands? I know, you're not a soothsayer. We all, at least our listeners have the best of intentions, but is there anything being done in the research community to help get ahead of this in terms of ethics, legalities and just human safety in general?

[0:36:23] AG: Yeah. I think, fundamentally, to think about it, if these models are open, as long as we have more people who have access to them with the right intention, they will outnumber the people who will have access to this with the bad intention. Hopefully, that's the word we live in. In that case, I think being open and accessible actually gives us more tools for everybody to build systems that can monitor them, as I said. The thing with safety is you have to assume that what if it happens? How am I going to counteract? I think we saw a recent, what happened in a couple of weeks ago with the agent getting it back in the Huggingface. They were able to use an open model to actually go and triage that and fix it, right?

I think that's why we believe that having a pretty strong, open ecosystem is vital to make sure that there are checks and balances. It's vital for people to create models that are better aligned to their philosophy. It's especially true for physical world. I was off kidding with somebody that me and my wife can't agree on how we load a dishwasher. That isn't the right way. So, we need an open model where I can, she can reinforce, train it to do it the way she likes it, or I can do it. If somebody decides for us how the dishwasher is going to be loaded, there's no way that's going to work.

[0:38:04] MM: I mean, do you have a dishwasher loading robot? Are you doing that at home? Because I'm totally jealous. Is that a perk of working in NVIDIA?

[0:38:11] AG: No.

[0:38:13] MM: No, okay. All right.

[0:38:13] AG: Not yet, but I'm looking forward to it. I think it'll happen soon.

[0:38:18] MM: Give me a prototype. That's really interesting. Tell me if I'm completely off here, but I almost think of this in terms of human morals, right? The deeper the moral is set in someone, the harder it is to compromise it. I think what you're saying is that these models are open and they are deeply ingrained in ethical things. It's very hard to unprogram that. Is that kind of what you're saying?

[0:38:43] AG: I think if these models are open, you can introspect them and you can steer them. They're not. Then you can put monitors. You can put all kinds of things, but you cannot - You're limited in what you can do. Then you also introduce a single point of failure as well, right? I think diversity and choice and one model checking the other one is the best way for making sure that you have a better answer to the security problem.

[0:39:17] MM: I can see what you mean by the transparency being huge, because if there's a bad guy, you don't know what he's thinking, but if you can see what he's thinking, that's a little bit of a different story. It's really interesting. I want to talk a little bit about, so you released Cosmos 3 Edge. You mentioned this Cosmos model. For somebody who's not familiar with this, right? Like, I don't really know what that - I think you said it's a model, but I don't really know what that means, Edge. Can you talk a little bit about what that is?

[0:39:45] AG: As I was telling you earlier, the world models can do many different functions. They can create, predict random worlds in which you can evaluate your policies. They can take real-world data and augment it to change the environments. Some of these things are pretty compute intensive. One of the functions of a world model is also, what we were talking about earlier, is that they can be the planning brain for your robot, because they can predict a lot of different scenarios and pick one that is most appropriate, that gets you to their objective function. That model needs to run on the robot; that's part of the world model.

This model is not really generating pretty videos, 10-second-long videos. It doesn't have to do that. This model does not have to - you can still down to certain particular domains that you care about, but it should be able to do the task of predicting the future and controlling the robot. We took our Cosmos 3 model, which comes in different sizes, and we created a 4 billion parameter version of it. The main purpose of the Cosmos 3 Edge is that it's a world model that can take the inputs as text, or actions, or videos and generate the next action that the robot needs to do. By making it a 4 billion parameter model, we can now deploy it onto the robot. It can be used as the policy of the robot itself. We optimize it. It can run real time on a Jetson Thor, which is a computer, which is the size of a passport that has 2,000 teraflops of compute.

[0:41:35] MM: Did you say the size of a passport?

[0:41:37] AG: Yeah, it's the size of a passport. Passport size. Yeah.

[0:41:40] MM: Holy moly. Wow. Okay.

[0:41:42] AG: That's what is inside a lot of robots. If you look at the 1X robot, the Boston Dynamics Humanoid, they are all powered by this small computer, but it has 2000 teraflops of compute. It can run 10 billion, 4 billion parameter model on it. That way, it can do the reasoning. You were asking about manufacturers can do all of that on device.

[0:42:04] MM: That is wild. That is really cool. I can see why that's a big deal. Some mini model that can run on robot.

[0:42:13] AG: On the robot.

[0:42:13] MM: It is so funny, because when I think Edge, I think of you know, like, "Oh, CDN distributed geographically." But when you guys talk about Edge, it's like, "Nah, it's in a factory, or it's, yeah."

[0:42:23] AG: Yeah. Yeah. It's this battery-powered, running on device, plus running all the time. Yeah.

[0:42:29] MM: Yeah. The power consumption of those things has to be reasonable, too. Not only is it small, but the power consumption. Wow.

[0:42:36] AG: Yes. Yeah.

[0:42:37] MM: That's crazy. Okay. I'm a normal developer. I want to start playing around with this next weekend. How do I get my hands on one of these passport size computers? They're only like 20 bucks, right? I'm just kidding. How accessible is this? I would love to play with something like this. How does a normal dev get into this?

[0:42:59] AG: Yeah. I think it has become much easier in the last, I would say, 12 to 18 months. In the past, the hardware was very expensive. It was because it was designed for roboticists, right? One of the things we started working with the team at Huggingface was to target and enable all the software developers who are used to writing programs to start programming robots. To do that, you need to have a robot, right? You have building physical AI. Of course, you can do a lot in simulation, right? That's the first step. You have to get started with this. You use your existing computer, you download a simulator, and you train, you download an existing model, connect it together and see how it works, right?

[0:43:45] MM: This is all just running on your laptop, or computer, or whatever.

[0:43:48] AG: Yeah. You can run on a decent laptop today, which has a 4090 GPU. You can get a DGX Spark, run it on that. You can run the simulator, you can train the models, test it. But the interesting thing that is about physical AI is that experience of touching a real thing and dealing with the connection and the motors and the variations there. Now, it's like, you can, for a couple of hundred dollars, you can get a robotic arm that you can train, you can tele-operate it, collect data, format it, publish it to Huggingface, train a policy, convert it, deploy it onto the small Jetson and let the robot do its thing.

[0:44:34] MM: Oh, wow.

[0:44:35] AG: All of that, I would say, for you to get started, if you have a good computer, a $,1000 budget, a couple of days, and you can have your first physical AI model doing things.

[0:44:48] MM: Wow. What are the arms? Is that something NVIDIA produces, or is that something that other, like third parties produce?

[0:44:56] AG: It's just third party. There are lots of projects. If you're into 3D printing, you can just order the material, 3D print these things, get up to that.

[0:45:03] MM: Oh, that's really cool. Okay.

[0:45:05] AG: Yeah. Pretty big open-source ecosystem. If you look at SO-101 arm, that's the most popular with the makers right now. You can 3D print it. Lots of vendors do the kitting of all together and they ship it to you, but you still get the joy of connecting it, building it all together. Yeah. That's there. Then you can start from there and go to a Humanoid and get one of these humanoids for $15,000, $10,000 these days that you can make it do whatever you want; dance, or clean your home.

[0:45:37] MM: Wow. When somebody's looking into the those, I don't know what we call them, classes of product, what would I Google for, or what would I ask Claude for in terms of trying to find that? That's something that to me, if I were to step away, I'd be like, okay, I guess, I could search open-source robotic arm, but I feel there's probably something more specific that people could search for.

[0:46:00] AG: Yeah. I think if you're interested in using this new age of robotics, which is end-to-end models, that is taking pixels in, controls out, you want to look for models that are supported by the support NVIDIAs, like crude, or physical intelligence, or these popular AI models, right? Those robot hardware that has been tested with these models. The way to search for is look for different robot policies, AI models that exist and say, "I'm looking for a hardware that works with this particular model." You don't want to have to customize the model for a different hardware.

[0:46:47] MM: Yeah. That's a bridge too far for somebody trying to get into it.

[0:46:50] AG: Exactly. That's why the Huggingface checkpoints are really good, because they already have checkpoints for this as a one-on-one arm. If you're looking at action-chunking transformer model, you are looking for a crude model, you're looking for any of the popular AI models for robotics, you can find a checkpoint for specific hardware and that's one way to go find it. Because then, you can just cobble it together and have your first roll out. Then you can say, "Okay, I want to go customize it."

[0:47:22] MM: Yeah. Oh, that sounds fun. Okay. Cool. I hope that inspires some people. All right, let's close it out. I got a couple of open-ended what if questions. What sounds like science fiction right now that you actually think might come true in five years with this?

[0:47:37] AG: If you think about robots today, what comes to your mind?

[0:47:42] MM: Probably that Boston dynamics thing that can jump upstairs and jump up on platforms and things like that. Yeah.

[0:47:48] AG: Yeah. I think for most people, when they think about robots, it's the embodiment that comes to mind, right? As you said, that Humanoid, or a big arm that they saw in an automotive plant that was carrying the entire car and turning it upside down. I think in five years, we will see definitely robots that are more around us, more ubiquitous, that are multifunctional and can do many things. The robots today are purpose-built. You think about robots. You think about either that humanoid, or you think about a Roomba, or you think about that industrial robot, you don't think about a brain that can do 10 things for you. You think about the body, not the brain.

I think in five years from now, I expect that we will have robotics embodiments that are going to be capable of doing multiple things. When you think about robot, you will not think about the body, but you will think about, "Oh, that thing that can do all of these tasks for me, the embodiment is just an artifact." That would change. You would not think about a robot for A, but you would think about a robot that can do a class of things for you.

The other thing that's going to change is for the robotics people is the task of programming was very intimidating. It's like, you need to be an expert. You need to be either a technician, or a PhD to be able to program the robots. That is going to look very different. With these foundation models, you would be able to likely show a task, do it one time, do it with the robot two times, and teaching the robot with data is what is going to be magical. It seems impossible. If you tell a robotics person today that, "Hey programming a robot is just like, go show it. Hey, this is the task I want you to do. Here is a video. Can you go do it?" That seems like magic today, but that's going to be how we're going to tell robots how to do a task. You're going to either demonstrate it, you're going to operate it just one time, two times, and the robot will go and do it.

[0:50:02] MM: I mean, if we didn't have this inflection point with software-based LLMs in the past year, I probably would not believe you. But seeing what happens in the past year, I believe it. Yeah. It's crazy.

[0:50:17] AG: I think those are the two things that I want, think would happen. Multifunction robots that can be just talked by telling them what to do, showing them what to do.

[0:50:26] MM: Right. You make such a good point about - I mean, for the history of robots, it's always been the thing with the arms and the face and it talks. Doesn't have to be like that, right? So, yeah. I can't wait until I have a disembodied arm robot that can do whatever I show it to do in my house, like load dishwashers.

AI is reshaping a lot of software engineering jobs, right? I mean, it's a little scary and also enthralling at the same time. Some people are wondering where their careers are going to go. Maybe you've inspired some people to look into this. If I'm a senior engineer today, I have no robotics background, but I have a heck of a lot of software background, what would you start learning besides the basics that we just talked about playing around? What's the best way to really get into this?

[0:51:16] AG: The best way, if you want to get into robotics, it's just starting, I believe, with simulation. Because that is where it still feels like you're working with software, but you get to experience the physical challenges, where you don't think about - You get exposed to gravity, you get exposed to friction, you get exposed to all the real-world challenges that you will encounter when you get to the hardware. I think that's the first thing, is get out, start working with the simulator, get exposed to those terminologies and the data that you're not used to when you're working with code and software.

Then once you get comfortable with the simulation, then deploying it on hardware. As I mentioned, there is a lot of different hardware modalities that are available now, where you can deploy the models that you've trained. You can get one of those and start deploying them. You have to get physical when you're working with the physical AI. Don't be intimidated by it. Embrace it. I think the more and more software developers and hardware developers work together, I think we'll have better systems. Today, still, I feel like that a lot of times the hardware developers who are building - the mechanical team is a little bit operating differently, decoupled from the software team. I think when they start understanding each of those languages, which is what would happen if as software developers get into physical AI, they need to understand all the mechanical constraints. That would be pretty exciting. I think we'll have much more capable systems.

[0:53:12] MM: Before we talked, I was wondering if you're going to say like, "Oh, you need to go get a PhD in math or something." I'm joking. I think what I'm hearing is like, no, that's starting to change. Just like in software, when you wanted to mess around with, let's not even talk about LLMs, but machine learning, it used to be, you got to know a lot of math, you got to really get in there with the concepts, and now it's becoming a lot more user friendly. Sounds the same thing.

[0:53:36] AG: Yeah, I think the open-source ecosystem around robotics is pretty strong. You have everything that you need from tools to assets, to models. It's all available. You can really go heads in and get started.

[0:53:51] MM: All right. Well, Amit, this was fantastic. Super, super interesting. I really thank you for being here and explaining this to us.

[0:53:59] AG: This was great, and I really enjoyed being here. And hope you have a good rest of your week.

[0:54:05] MM: You, too. Send me that dishwashing robot when you get it.

[0:54:08] AG: Yeah, I will.

[END]