Global Performance Tuner

Online inference requires low latency and high concurrency, and vendors must reserve GPU resources to handle burst traffic, resulting in a higher unit cost; batch inference allows tasks to be queued and executed during off-peak periods, resulting in a lower unit price.

Global search is inherently more time-consuming than single-path building, and teams need to trade off tuning budget against release cadence, for example relaxing the budget for core models while keeping the default strategy for long-tail models.

The traditional TensorRT workflow typically follows only one build path and relies on manual tuning; the Global Performance Tuner upgrades single-path compilation to multi-path search.

The core idea of the Global Performance Tuner is to make the build process not stop at the first acceptable result, but instead compare globally across multiple candidate paths to find a better solution.

By tuning separately for different target hardware, the same model can produce engines customized for each deployment environment.

The larger the search space of automatic tuning, the longer the build takes.

In a development phase with frequent iterations, developers may still need a fast build path to validate functionality, and then enable global tuning before final delivery to obtain the optimal engine.

Precision-aware tuning typically requires developers to provide calibration data (for INT8/FP8 quantization) or explicit precision tolerance metrics (such as a relative error threshold).

The Global Performance Tuner automatically explores multiple configuration combinations during the build phase and selects the better-performing option based on actual measurement results.

The Global Performance Tuner provides automatic, precision-aware performance tuning, converging the exploration process previously spread across multiple build attempts into a single automated workflow.

NVIDIA introduced the Global Performance Tuner in TensorRT 11.1.