Tesla Simulates to Train AI

Tesla took the opportunity to show off some new techniques for training their Full Self Driving software in addition to their other big AI projects.

With all the talk of AI cross-use in the other presentations, it was safe to assume something new was cooking for Tesla’s FSD, and we were not disappointed. Tesla engineers were extremely excited to show off a new method of simulating real-world environments for their AI to learn from - and it’s very impressive.

Tesla’s FSD operates on a mostly visual system. 8 cameras take in everything around the vehicle, and the AI decides what to do from that input. But if that’s all it took to be a perfect driver, humans wouldn’t get into so many accidents. Like any other new driver, FSD needs to practice.

Tesla currently uses GPU server farms to build out environments to train their FSD intelligence. Out of the 14 thousand GPUs Tesla has, it takes 10 thousand to train the system, and 4 thousand to auto label - an AI system Tesla uses to pick out cars, pedestrians, and other objects from the background. On top of that, it takes over 160 billion frames of video to train an AI system for a specific stretch of road.

Now, obviously they don’t train these systems on a real road, as mistakes there have consequences. So Tesla have previously used video and mapping software, with the help of artists, to build out simulated versions of roadways to test the FSD software on.

It generally takes an artist around two weeks to build out a single scene - just one intersection or stretch of road, with all its obstacles. But with new software tools, Tesla can now generate most of San Francisco in one week, with only one person directing the tools.

And better yet, all of the data they build these models with comes from real-world data. So it will include all the little quirks that real drivers have to deal with.

Now obviously, this will take way more computing power than Tesla currently has. But with the first Dojo cabinet being constructed, you can see how one breakthrough leads to another, and another. Engineers build a dense computing chipset, which helps make Dojo a reality, which lets Tesla’s FSD learn much faster in a digital simulation, which finally ends up informing the logic and navigation software of Optimus.

Everything Tesla’s AI team does, fits together like a puzzle, and it’s all snowballing right in front of us.