Deploying AV Inference Models Under Constrained Onboard Compute

Real-World Constraints of Onboard Inference

Autonomous driving developers face a contradiction: inference models need larger parameter counts to improve decision quality, but onboard systems have extremely limited compute and latency budgets. NVIDIA points out this pain point directly in its developer blog—AV developers need inference models that can run within onboard compute and latency constraints, not models that look good on benchmarks in the data center.

This is not simply a model compression problem. In-vehicle scenarios tolerate latency in milliseconds, and any single inference timeout can affect the stability of the control loop. Therefore, model selection and optimization must be derived backward from deployment constraints, rather than training a large model first and then figuring out how to stuff it into the vehicle.

The Structural Conflict Between Large Models and Onboard Deployment

The original article clearly states that larger models can provide stronger reasoning capabilities, but this directly conflicts with the onboard compute ceiling. Reasoning models typically rely on longer chains of thought and more intermediate computation steps, which is inherently at odds with the deterministic low latency pursued on the vehicle side.

Common industry approaches are distillation or quantization, but both come at a cost: distillation loses some reasoning depth, and quantization at low bit widths may undermine numerical stability. NVIDIA's article frames the problem as "running within limits" rather than "bypassing limits," implying that co-design at the model architecture and inference scheduling levels is needed, not just post-hoc compression.

Reasoning Models refer to models that perform explicit multi-step reasoning before generating a final answer, typically characterized by longer chain-of-thought (CoT) outputs or multiple internal iterations, such as decomposing a problem step by step, self-verifying, or calling tools before answering. Compared with conventional models that produce results in a single forward pass, these models spend more computation on "thinking," and thus perform better on complex decision tasks, but their inference latency and compute overhead are significantly higher and non-deterministic—variable output length means worst-case latency is hard to predict. For the vehicle side, this non-determinism is a core risk: the control loop requires a predictable upper bound on response time, while chain-of-thought length fluctuates with input difficulty, precisely breaking this premise. Therefore, when using reasoning models on the vehicle side, it is often necessary to limit the maximum number of reasoning steps or set different budgets for different scenarios.

Distillation and quantization are the two most common model compression paths, but their mechanisms and side effects differ. Distillation (Knowledge Distillation) lets a small model imitate the output distribution or intermediate representations of a large model, significantly reducing parameter count, but the student model has limited capacity, and what is often lost first is exactly the "depth" required for long-chain reasoning, manifesting as reduced accuracy on multi-step problems. Quantization reduces weights and activations from FP16/FP32 to INT8 or even lower bit widths; the main benefit is reduced bandwidth and memory usage, but lower bit widths compress the numerical dynamic range and introduce errors in sensitive stages such as attention softmax and layer normalization, potentially causing numerical instability or accuracy collapse at extremely low bit widths. The two can usually be combined, but both require re-validation of accuracy and latency on the target hardware rather than directly reusing cloud-side conclusions.

Design Approach Starting From Constraints

Optimizing reasoning models for onboard deployment involves at least three levels:

• Model level: control the number of reasoning steps, avoid unconditionally expanding long chains of thought, and take short paths in simple scenarios.
• Runtime level: dynamically adjust the inference budget according to the current compute load, ensuring an upper bound on worst-case latency.
• Hardware level: fully utilize the heterogeneous compute units of the onboard SoC and distribute inference workloads to the appropriate execution units.

The original article does not elaborate on specific technical details, but by presenting "compute and latency limits" as a first-class constraint, it itself shows that onboard inference cannot follow the cloud-style approach of "throwing more compute at the problem to improve results." For developers, evaluating whether a reasoning model is suitable for mass production should not rely solely on accuracy, but also on its latency distribution and tail latency on the target hardware.

Heterogeneous compute units refer to the multiple compute resources coexisting on an onboard SoC: the CPU handles control logic and scheduling, the GPU handles large-scale parallel matrix operations, dedicated accelerators such as NPU/DLA are hardened and optimized for fixed-point convolution and Transformer operators, and there are also specialized modules such as ISP and DSP. Different units vary greatly in compute power, precision support (FP16/INT8/INT4), and energy efficiency, and the same model layer may differ in latency by several times when placed on different units. So-called workload distribution means partitioning and mapping the computation graph to the most suitable units according to operator type, and allowing multiple inference pipelines to run in parallel without blocking each other. This requires the compiler to have operator fusion and scheduling capabilities, and the runtime to be aware of the real-time utilization of each unit; otherwise, one unit may become a bottleneck while the others sit idle.

Practical Implications for Developers

The demand for this kind of constraint-aware inference is expanding from AV to broader edge scenarios. Whoever can maintain inference quality under limited compute will be better positioned to actually ship models into products. NVIDIA's public framing of this problem also means that toolchain and hardware support will continue to lean toward "running within constraints."

When selecting models, developers are advised to treat the onboard latency budget as a hard screening condition, conduct measurements on the target hardware first, and then compare model capabilities.

Tail latency refers to high-percentile values of the latency distribution, such as P95 and P99, rather than the average. In real-time systems, the average has almost no reference value: even if 99% of inferences complete within 20ms, if the remaining 1% reaches 200ms, it may directly cause control commands to miss their cycle, trigger degradation, or even safety intervention. Sources of tail latency include input complexity fluctuations, memory bandwidth contention, thermal throttling, and scheduling queues. Therefore, onboard evaluation should focus on the shape and upper bound of the latency distribution, and determine budgets in combination with worst-case execution time (WCET) analysis; for reasoning models, a hard upper limit should also be set for the maximum chain-of-thought length to ensure that the worst case remains within the acceptable range of the control loop.