Tesla’s Autonomy Stack: Five Architectural Bets

Tesla has been unusually public about its architecture — AI Day 2021 and 2022, a series of CVPR workshop talks, and a steady stream of release notes — which makes it one of the few production autonomy stacks whose evolution can be traced from the outside.

What follows is my reading of that public record. Everything here is from published talks and releases; I have no inside knowledge, and anything about how well it works is a separate argument from how it is built. I am interested in the architectural decisions, because most of them are choices any camera-centric stack eventually has to make.

A caveat worth stating up front: this is a fast-moving, heavily-marketed subject, and some claims below come from company presentations rather than peer-reviewed work. Dates and numbers are as publicly reported. Treat the technical substance as more reliable than the performance claims.

Bet 1: Own the compute

Autopilot 1 (2014–2016) ran on Mobileye’s EyeQ3 — a supplier part with a supplier’s software. After that relationship ended, Hardware 2 moved to NVIDIA’s Drive PX2 with Tesla’s own perception software on top, and in 2019 Hardware 3 moved to an in-house dual-SoC “FSD Computer.”

The interesting part is not the silicon, it is what vertical integration removes: the constraint that your architecture has to fit a part designed for someone else’s architecture. If your roadmap requires fusing eight camera streams through attention layers at frame rate, you can design for that rather than negotiate for it. The dual-SoC layout also builds the redundancy story into the compute rather than bolting it on.

The corresponding risk is that you now own a hardware roadmap, and the fleet you already sold is fixed. Every subsequent architecture change has to fit compute specified years earlier — which is a real constraint on a stack whose models keep growing, and the reason “HW3 versus HW4 capability” has been a recurring topic.

Bet 2: Fuse in vector space, not image space

This is the change with the clearest technical lesson, presented at AI Day 2021.

The original design ran detection per camera and fused the results afterward, in a hand-written C++ layer that stitched per-image outputs into a single world model. It degrades badly in an obvious case: a vehicle spanning two cameras is two partial detections, each individually poor, and no amount of post-hoc geometry recovers the object cleanly. The fusion code accumulates special cases for every such situation.

The replacement lifts all eight camera streams into a shared bird’s-eye-view representation before any object reasoning, using attention to let each BEV query pull features from whichever cameras can see that location. Outputs come out already in a common metric frame — what the talks call “vector space.”

The general principle: fuse at the representation level, not the output level. Fusing outputs discards exactly the information needed to fuse well — the uncertainty and the partial evidence. It is the same reason early-fusion beats late-fusion across most of multi-sensor perception, and it is not specific to Tesla.

The same architecture added an explicit temporal dimension — a queue of past features indexed by both time and ego-motion, so the model can remember what it saw before an occlusion, and answer “is that lane still there” for something last visible eight seconds ago. Static structure indexes naturally by distance travelled; dynamic objects by time.

Bet 3: Predict occupancy, not just objects

Object detection has a category problem: it can only report things it has a class for. A mattress in the road, an unusual trailer, debris — the honest answer is “something is there, it is this shape, do not drive into it,” and a detector with a fixed label set has no way to say that.

Occupancy networks (AI Day 2022) predict volumetric occupancy — plus flow — over the space around the vehicle, independent of class. The planner asks “is this volume free?” rather than “what is this object?”, which is the question it actually needed answered.

Alongside it, road structure was reframed as a sequence-prediction problem: lanes as tokens emitted autoregressively, with the connectivity of an intersection generated the way a language model generates text. Framing map topology as generation rather than segmentation is a genuinely elegant fit — intersection connectivity is a graph with grammar-like constraints, and segmentation was never a good representation for it.

Bet 4: End-to-end

FSD v12 was the largest architectural change: the hand-written planning and control stack replaced by a network trained to map sensor input to driving actions. Musk described it as removing on the order of 300,000 lines of C++.

The argument for it is that the interfaces between modules are lossy. Perception hands the planner a sanitized abstraction, and every hand-designed boundary throws away information the downstream stage might have used. It also moves the improvement loop from “an engineer writes a rule” to “add data covering the case,” which scales differently — and if you have a large fleet, it scales in the direction of your main asset.

The arguments against are real and unresolved:

• Validation. You can unit-test a rule. What is the test for a policy? Evidence becomes statistical — miles between interventions — which needs a great many miles to move, and cannot easily prove a specific failure has been fixed rather than made rarer.
• Interpretability. When it does something wrong, the answer to “why” is not in a stack trace.
• Guarantees. Hard constraints — never exceed this deceleration, never cross this boundary — are properties you assert about a rule-based system and hope for from a learned one. In practice this pushes toward a residual safety layer outside the network, which reintroduces a hand-written module at exactly the place where its limitations matter most.

Where the industry has landed, broadly, is that end-to-end training with intermediate supervision — keeping perception outputs as auxiliary tasks so the representations stay inspectable — captures much of the benefit while preserving something to debug. Whether that is a stepping stone or the destination is, I think, genuinely open.

Bet 5: Learn a world model, then train against it

Bet 4 creates a problem that Bet 5 exists to solve.

An end-to-end policy trained by imitation learns to copy expert actions on the distribution of states the expert visited. The moment it makes a small error it is slightly off that distribution, where its next action is a little worse, which takes it further off — and the errors compound. Ross and Bagnell formalized this in 2010: for a policy with per-step error $\varepsilon$ over a horizon $T$, behaviour cloning has worst-case regret

while a method that lets the learner see corrections from its own state distribution gets $O(\varepsilon T)$. The quadratic term is the entire practical difficulty of imitation learning: at highway horizons, $T$ is large and $T^2$ is brutal.

The classic fix (DAgger) needs an expert to label the states the policy actually reaches. On a fleet, you get a weak version of this for free — every disengagement is a state the policy reached and a human correcting it. But interventions are rare, unevenly sampled, and you cannot ask for more of a specific kind.

Unless you can generate them.

The world model

A world model is a learned, action-conditioned simulator: a generative model of what the sensors will show next, given what they showed so far and what the vehicle does.

Tesla demonstrated exactly this at CVPR 2023 — a generative video model producing multi-camera-consistent driving footage, conditioned on actions, so you can roll the same scene forward under different commands. The same idea shows up across the field: Wayve’s GAIA line, NVIDIA’s Cosmos, and a growing pile of academic work.

Three things it buys, in increasing order of ambition:

1. Evaluation. Replay a logged scene with a modified policy and see what would have happened, without needing to have driven it.
2. Rare-event synthesis. Condition on a description or an initial state and generate variations of the case you have eleven examples of.
3. Policy improvement. If the simulator is good enough to roll forward under the policy’s own actions, it supplies precisely the on-distribution states that behaviour cloning cannot get — which is the $O(\varepsilon T^2)$ problem, addressed.

The third is the real prize and the least proven. A learned simulator has learned artifacts, and an optimizer pointed at it will find them: the policy discovers a behaviour that scores brilliantly against the model and is nonsense on the road. The failure mode is not “the sim is a bit inaccurate,” it is “the policy is actively searching for the places where it is wrong.”

Where RL comes in

With a simulator you can optimize a return rather than copy a demonstration:

which lets you state what you want directly — progress, comfort, clearance, rule compliance — instead of hoping it is implied by the demonstrations.

def rollout(policy, world_model, obs_history, horizon, gamma=0.99):
    """Score a policy inside the learned simulator."""
    obs, total, discount = obs_history, 0.0, 1.0
    for t in range(horizon):
        action = policy(obs)                       # differentiable through the rollout
        obs = world_model.step(obs, action)        # p(o_{t+1} | o_<=t, a_<=t)
        total = total + discount * reward(obs, action)
        discount = discount * gamma
        # a rollout is only worth as much as the model's calibration at this depth
        if world_model.uncertainty(obs) > UNCERTAINTY_LIMIT:
            break
    return total

That break is the part that matters. Truncating a rollout when the model leaves the region it was trained on is the standard defence against optimizing into simulator artifacts, and it is the difference between model-based RL working and producing confident nonsense.

Reward design is the other hard half, and it is hard in a specific way: driving rewards are mostly constraints, and constraints expressed as penalties get traded off. A policy that can earn back a collision penalty with enough progress reward will, eventually, find the trade. Hard constraints belong outside the objective.

Preference-based methods sidestep some of this. Rather than writing $r$, learn it from comparisons — this trajectory versus that one — which is the same machinery that made RLHF work for language models, and fleet intervention data is a natural, if biased, source of pairs.

Foundation models and reasoning

The other thread is scale-driven: vision-language models carry common-sense priors that no driving-specific dataset contains. A hand-written detour sign, a police officer waving you through a red light, a ball bouncing into the road ahead of a child you cannot see yet — these are semantics, not geometry, and they are unbounded in a way that defeats enumeration.

The obstacle is latency. Control wants 10–100 Hz; a model that reasons in tokens does not. The shape the field seems to be converging on is two-tier: a fast reactive policy in the control loop, and a slower deliberative model that supplies context, goals, and constraints to it — the same split as System 1 and System 2, arrived at for engineering reasons rather than cognitive ones.

How much of this is in Tesla’s shipping stack is genuinely unclear from public material. The BEV and occupancy work was presented in technical detail; the world model was demonstrated; the reasoning direction has mostly been discussed rather than documented. I would not read the last part of this section as a description of a deployed system — it is where the field is pointing, Tesla included.

The sensor question

Tesla removed radar in 2021 and ultrasonics in 2022, going camera-only. Nearly every other serious program fuses lidar and radar.

The case for vision-only: cameras are cheap and information-rich, sensor disagreement is its own hard problem, and one modality means one pipeline to improve. The case against is the one from sensor physics — cameras infer range rather than measuring it, and they degrade in precisely the conditions where you would most like an independent measurement. Radar’s direct Doppler velocity and weather penetration are not things a camera can be trained into having.

This one will be settled by fleet data neither of us has. What is fair to say is that it is a bet on inference quality being able to substitute for direct measurement, and that it is being made against the industry consensus rather than alongside it.

What generalizes

Strip away the company and five things survive as ordinary good architecture:

1. Fuse sensors at the feature level, before the semantics.
2. Put the output in the metric frame the consumer needs, not the sensor frame it came from.
3. Represent free space directly, so you are not limited to categories you enumerated.
4. Be suspicious of hand-designed interfaces between learned modules — they are where information goes to die.
5. If you train a policy by imitation, you eventually need a way to generate the states that policy actually reaches. A world model is one answer; it is not the only one, but the $O(\varepsilon T^2)$ problem does not go away on its own.

The end-to-end and vision-only bets are the contested ones, and reasonable people are still running the experiment in both directions.