ONNX Runtime
ONNX Runtime is an open-source cross-platform high-performance machine learning inference engine developed by Microsoft that supports running models in the ONNX (Open Neural Network Exchange) format. It is compatible with multiple operating systems and hardware accelerators, including CPUs, GPUs, and NPUs, and optimizes and accelerates inference for models exported from mainstream frameworks such as PyTorch and TensorFlow, with broad application in production deep learning model deployment.
Core Facts
TensorRT unleashes GPU hardware potential through operator fusion, kernel auto-tuning, and memory optimization.
Triton Inference Server is an open-source model serving framework from NVIDIA that supports dynamic batching and concurrent model execution, and is compatible with inference backends such as TensorRT, PyTorch, TensorFlow, and ONNX Runtime.
ONNX Runtime is an open-source machine learning inference and training acceleration engine primarily developed under Microsoft's leadership.
Timeline (last 90 days)
In real projects, inference frameworks such as ONNX Runtime, TensorRT, TFLite Micro, and OpenVINO are used far more frequently than PyTorch training code.
Running AI models on-device typically uses techniques such as quantization and pruning to compress the model and then executes inference on the CPU, GPU or NPU of a personal computer; common frameworks include llama.cpp and ONNX Runtime.
Inference acceleration tools include TensorRT, ONNX Runtime, OpenVINO, and others, used for graph optimization and hardware adaptation of models.
An inference engine is a software component specifically optimized for model inference performance, common examples being ONNX Runtime, TensorRT, and llama.cpp.
Common tools for edge deployment include TensorFlow Lite, ONNX Runtime, and TensorRT.
BitLens supports generating quantize_static with nodes_to_exclude for ONNX Runtime.
Mainstream inference runtimes such as llama.cpp, vLLM, and ONNX Runtime mostly rely on Python for model conversion, graph construction, and scheduling.
To integrate an inference engine in C++, a common approach is to use backends such as ONNX Runtime, whisper.cpp, or TensorFlow Lite, converting the model to ONNX or GGUF format before loading it.
EdgeWise chooses the open-source ONNX Runtime inference engine, letting partners compile the selector themselves on platforms such as Linux, RT-OS, iOS, and custom ASICs.
Local models require the operating system, drivers, and runtimes (such as DirectML and ONNX Runtime) to jointly perform graph partitioning and operator mapping in order to find the most suitable execution unit.
29 more timeline events
All Facts (20)
TensorRT unleashes GPU hardware potential through operator fusion, kernel auto-tuning, and memory optimization.
80%VerifiedTriton Inference Server is an open-source model serving framework from NVIDIA that supports dynamic batching and concurrent model execution, and is compatible with inference backends such as TensorRT, PyTorch, TensorFlow, and ONNX Runtime.
80%VerifiedONNX Runtime is an open-source machine learning inference and training acceleration engine primarily developed under Microsoft's leadership.
75%VerifiedONNX Runtime is a high-performance inference engine open-sourced by Microsoft that supports multiple hardware backends including CPU, GPU, and NPU, and has built-in acceleration methods such as graph optimization and operator fusion.
65%VerifiedTensorRT is an inference engine developed by NVIDIA, and ONNX Runtime is an inference engine developed by Microsoft.
65%UnverifiedFastEmbed采用ONNX Runtime作为推理引擎，不需要依赖PyTorch或TensorFlow
95%UnverifiedONNX Runtime employs a layered architecture of graph optimization plus Execution Providers to support multiple hardware backends
85%UnverifiedFastEmbed-rs relies on ONNX Runtime under the hood to load and run quantized models.
60%UnverifiedONNX Runtime's graph optimization is divided into three levels: basic optimization, extended optimization, and full optimization.
60%UnverifiedThe core architecture of ONNX Runtime is a pluggable Execution Provider (EP) system, including the CPU EP, CUDA EP, TensorRT EP, DirectML EP, and CoreML EP.
60%UnverifiedIn real projects, inference frameworks such as ONNX Runtime, TensorRT, TFLite Micro, and OpenVINO are used far more frequently than PyTorch training code.
50%UnverifiedRunning AI models on-device typically uses techniques such as quantization and pruning to compress the model and then executes inference on the CPU, GPU or NPU of a personal computer; common frameworks include llama.cpp and ONNX Runtime.
50%UnverifiedInference acceleration tools include TensorRT, ONNX Runtime, OpenVINO, and others, used for graph optimization and hardware adaptation of models.
50%UnverifiedAn inference engine is a software component specifically optimized for model inference performance, common examples being ONNX Runtime, TensorRT, and llama.cpp.
50%UnverifiedCommon tools for edge deployment include TensorFlow Lite, ONNX Runtime, and TensorRT.
50%UnverifiedBitLens supports generating quantize_static with nodes_to_exclude for ONNX Runtime.
50%UnverifiedMainstream inference runtimes such as llama.cpp, vLLM, and ONNX Runtime mostly rely on Python for model conversion, graph construction, and scheduling.
50%UnverifiedTo integrate an inference engine in C++, a common approach is to use backends such as ONNX Runtime, whisper.cpp, or TensorFlow Lite, converting the model to ONNX or GGUF format before loading it.
50%UnverifiedEdgeWise chooses the open-source ONNX Runtime inference engine, letting partners compile the selector themselves on platforms such as Linux, RT-OS, iOS, and custom ASICs.
50%UnverifiedLocal models require the operating system, drivers, and runtimes (such as DirectML and ONNX Runtime) to jointly perform graph partitioning and operator mapping in order to find the most suitable execution unit.
50%