Camera, Lidar, Radar: Localizing Without an HD Map

For a decade the standard answer to “where is the vehicle?” was: build a centimetre-accurate prior map offline, then match live sensor data against it. It works well and it is how most deployed systems reached their first driverless miles.

It also has an operations problem that scales badly. The map has to be surveyed, validated, stored, distributed, and — the expensive part — kept current. Roads are repaved, lanes shift for construction, a new intersection opens. Coverage becomes a business constraint on where you are allowed to drive at all.

So the interesting question is how much of that you can do online, from the sensors on the vehicle. That question is really about what each sensor measures and where each one fails.

What the three sensors actually measure

Camera is passive. It integrates incoming light onto a grid, giving dense appearance and excellent angular resolution — a few tenths of a milliradian per pixel — for very little money and power. What it does not give you is range. A single image determines the direction to a point, never the distance; depth comes from motion, stereo baseline, or a learned prior about how big things usually are. Failure modes follow from being passive: low light, direct sun in the lens, and low-texture scenes where there is nothing to match.

Lidar is active time-of-flight. It measures range directly and precisely — a couple of centimetres, largely independent of distance — which makes it the only one of the three that hands you metric 3D structure with no inference step. The costs are angular resolution that thins out with range (a 0.1° beam spacing is 35 cm between points at 200 m), sensitivity to airborne water and dust, and price. Rain, fog, and the spray behind a truck all return energy early and produce phantom points.

Radar is active RF. Two properties matter and neither is shared by the others. It measures radial velocity directly via Doppler, in a single scan, with no differentiation between frames. And centimetre-wavelength RF propagates through the fog and rain that attenuate both light and near-infrared. The price is angular resolution — a conventional automotive radar’s beam is degrees, not tenths of degrees — plus multipath: RF bounces off the road surface and under vehicles, so you get returns from places nothing is. Modern 4D imaging radars improve angular and elevation resolution substantially, but they do not close the gap to lidar.

The engineering point is that these fail independently. Fog defeats camera and lidar and barely touches radar. A featureless tunnel wall defeats lidar geometry while camera texture and radar returns still work. A dark, unlit road defeats camera while lidar is unaffected. Fusion is not about averaging three noisy estimates; it is about making sure that no single environmental condition takes out every input at once.

Estimating ego-motion

Without a prior map, pose comes from integrating relative motion and then correcting the accumulated drift.

Visual-inertial odometry tracks features across frames and fuses them with an IMU. The IMU supplies metric scale — which monocular vision cannot recover on its own — and carries the estimate through the brief intervals where vision fails. It is cheap and accurate over short horizons, and it drifts, particularly in yaw.

Lidar odometry registers consecutive scans, usually with a point-to-plane ICP variant. Modern systems like KISS-ICP and LIO-SAM are accurate and fast. The critical caveat is geometric degeneracy: ICP can only constrain directions in which the scene has structure. In a long tunnel or a straight highway cut with uniform walls, the geometry is nearly invariant to translation along the corridor, and the solution slides. A well-built system detects this — the smallest eigenvalue of the registration Hessian collapses — and downweights the degenerate direction rather than trusting the fit.

Radar odometry is the underrated one. Because each detection carries radial velocity, you can solve for the sensor’s instantaneous ego-velocity from a single scan, by fitting the expected Doppler pattern across azimuth over the static returns:

Least-squares over the detections, with RANSAC to reject the moving ones, gives $(v_x, v_y)$:

def radar_ego_velocity(azimuth, radial_velocity, iters=100, tol=0.1):
    """Instantaneous sensor-frame velocity from ONE radar scan.

    azimuth, radial_velocity: (N,) arrays over detections.
    Static world points satisfy  v_r = -(cos(a) vx + sin(a) vy).
    """
    A = -np.stack([np.cos(azimuth), np.sin(azimuth)], axis=1)   # (N, 2)
    best_inliers = None

    for _ in range(iters):                       # RANSAC: reject moving targets
        idx = np.random.choice(len(azimuth), 2, replace=False)
        if np.linalg.matrix_rank(A[idx]) < 2:    # near-collinear bearings
            continue
        v = np.linalg.solve(A[idx], radial_velocity[idx])
        inliers = np.abs(A @ v - radial_velocity) < tol
        if best_inliers is None or inliers.sum() > best_inliers.sum():
            best_inliers = inliers

    # refit on all inliers
    return np.linalg.lstsq(A[best_inliers], radial_velocity[best_inliers], rcond=None)[0]

No data association between frames, no feature tracking, and it works in weather that has taken the other two sensors offline. Note the rank check: if every detection sits at a similar azimuth — a narrow field of view staring down an empty road — the system is ill-conditioned and the lateral component is not observable. Integrated alone it drifts, but as a velocity constraint in a factor graph it is a strong, cheap, and unusually robust input.

Wheel odometry and steering stay useful and are nearly free. They are also biased in exactly the situations that matter — slip under braking, tyre radius changing with pressure and load — so treat them as a constraint with an estimated scale factor, not as truth.

Correcting the drift

All of the above drift. Two mechanisms bound the error.

Loop closure. Recognize a previously visited place, add a constraint between the two poses, and redistribute the accumulated error over the whole trajectory. This is what makes SLAM a map rather than a long dead-reckoned path. It matters less for highway driving — you rarely revisit — and enormously for depots, yards, and urban routes.

Global priors. GNSS, especially RTK, bounds absolute error directly, but degrades in exactly the places localization is hardest: urban canyons, tunnels, under bridges. Multipath here is insidious because the receiver reports a confident, wrong fix rather than no fix, so a solution that trusts the reported covariance will be dragged off by metres. Standard- definition maps — OpenStreetMap-grade road topology — are a weaker but far cheaper prior, enough to snap to the right road and disambiguate which of two parallel carriageways you are on.

The usual home for all of this is a factor graph: odometry factors between consecutive poses, loop-closure factors between distant ones, GNSS and radar-velocity as unary and velocity constraints, all solved with a robust kernel so that one bad association does not bend the trajectory. Incremental solvers (iSAM2 and relatives) keep it real-time by re-linearizing only the affected part of the graph.

Building the map structure online

Pose is only half the problem. Driving also needs the semantic content an HD map used to supply: lane geometry, road boundaries, stop lines, connectivity through the intersection.

The mapless approach predicts that structure directly from live sensors, usually in a bird’s-eye-view representation: lift multi-camera features into a common BEV grid, fuse lidar and radar into the same grid, and decode vectorized map elements — polylines with semantics — rather than rasterized segmentation. Producing polylines directly is what makes the output consumable by a planner without a brittle post-processing step.

Two honest limitations. Range is bounded by what the sensors can see, so online maps are short-horizon; you get a hundred metres or so, not the kilometre of look-ahead a prior map gives for free. And occlusion is unrecoverable in a way it is not for a prior map — a truck beside you hides the lane you need to merge into, and no amount of model capacity makes that measurement exist. Temporal accumulation across frames helps a great deal with both, which is why the strong systems maintain a persistent local BEV memory rather than predicting from the current frame alone.

Where this actually stands

The practical position, as I read it: online pose estimation with lidar-inertial or visual-inertial odometry plus loop closure is mature and works. Online map structure prediction is the part still improving quickly, and it is where the interesting failures are — long-range geometry, unusual intersection topology, heavy occlusion.

Which is a reasonable place to be. It means “mapless” is less a binary than a question of how much prior you need and how stale you can tolerate it being. An SD map plus strong online perception covers a lot of ground, and it does not carry the operational burden that made HD maps a constraint on where the vehicle was allowed to go.