MLC-LLM
An open-source framework designed for cross-platform deployment of machine learning models, with a focus on large language models. It supports deployment to browsers, mobile devices, and other edge environments, and is compatible with hardware backends including WebGPU, enabling on-device inference across diverse platforms.
Timeline (last 90 days)
In recent years, with the AI compute improvements of mobile chips such as Qualcomm Snapdragon and Apple Neural Engine, and the maturation of inference frameworks such as Llama.cpp and MLC-LLM, models under 7B parameters can already achieve usable generation speeds on flagship phones.
Open-source frameworks such as MLC-LLM and Transformers.js are specifically optimized for browser-side inference scenarios, capable of loading model weights in ONNX, GGUF, and other formats into browser memory and executing inference.
The WebLLM project typically uses the MLC-LLM framework for model quantization and compilation optimization, with Apache TVM serving as the underlying infrastructure for MLC-LLM.
The underlying infrastructure of MLC-LLM is Apache TVM, an open-source deep learning compiler stack capable of automatically optimizing and compiling models into efficient code for specific hardware platforms.
Mainstream edge inference frameworks include Google's TensorFlow Lite, Microsoft-led ONNX Runtime, and MLC-LLM developed by Tianqi Chen's team.
MLC-LLM, developed by Tianqi Chen's team, can compile large language models into native code highly optimized for specific hardware.
All Facts (6)
In recent years, with the AI compute improvements of mobile chips such as Qualcomm Snapdragon and Apple Neural Engine, and the maturation of inference frameworks such as Llama.cpp and MLC-LLM, models under 7B parameters can already achieve usable generation speeds on flagship phones.
50%UnverifiedOpen-source frameworks such as MLC-LLM and Transformers.js are specifically optimized for browser-side inference scenarios, capable of loading model weights in ONNX, GGUF, and other formats into browser memory and executing inference.
50%UnverifiedThe WebLLM project typically uses the MLC-LLM framework for model quantization and compilation optimization, with Apache TVM serving as the underlying infrastructure for MLC-LLM.
50%UnverifiedThe underlying infrastructure of MLC-LLM is Apache TVM, an open-source deep learning compiler stack capable of automatically optimizing and compiling models into efficient code for specific hardware platforms.
50%UnverifiedMainstream edge inference frameworks include Google's TensorFlow Lite, Microsoft-led ONNX Runtime, and MLC-LLM developed by Tianqi Chen's team.
50%UnverifiedMLC-LLM, developed by Tianqi Chen's team, can compile large language models into native code highly optimized for specific hardware.
50%