[KongchangAI]
· 2 min read· 1,420 words

Huawei Open-Sources MACaQ: Quantizing 100B-Parameter Models in 64 Minutes; llama.cpp 1.0 Released

Huawei Open-Sources MACaQ: Quantizing 100B-Parameter Models in 64 Minutes; llama.cpp 1.0 Released

Huawei's quantization tool, llama.cpp 1.0, and other developments jointly drive efficient operation of large models on ordinary hardware.

This round of AI news centers on "getting large models to run on ordinary hardware": Huawei's open-source quantization tool MACaQ compresses 4-bit quantization of a 100B-parameter model to 64 minutes with peak VRAM of 32.5GB and over 2x inference speedup; llama.cpp releases version 1.0, promising interface stability and supporting over 150 architectures, with speculative decoding nearly doubling generation speed on a single RTX 5090; MiniMax's open-source 4K image generation model needs only 6.4GB of VRAM after community quantization, runnable on 8GB GPUs; the coding agent leaderboard introduces a cost-per-problem dimension, with DeepSeek V4.1 Flash standing out at about $0.30 per problem; Google's Gemini 4.2 agent helps mathematicians formalize century-old problems, automatically generating 1.8 million lines of code. These advances jointly level the VRAM and speed barriers, making economics the core competitive advantage for using AI at scale.

The two biggest barriers to running large models locally—VRAM and speed—are being tackled at the same time. Huawei has open-sourced a quantization tool that compresses 4-bit quantization of 100B-parameter models down to the 64-minute range; meanwhile, llama.cpp has officially entered its 1.0 stage with a promise of interface stability. Add to that MiniMax's image generation model running on 8GB GPUs, and a coding agent leaderboard that puts "cost per problem" front and center—this round of AI news almost all points in the same direction: getting large models to actually run on ordinary hardware.

Huawei MACaQ: Quantizing 100B Models in 64 Minutes

According to a share from Bilibili's "Will叔每日AI动态" (Uncle Will's Daily AI Updates), Huawei's open-source quantization tool MACaQ completed a rather telling real-world test: fully quantizing a 100B-parameter-scale model to 4 bits took about 64 minutes, with peak VRAM usage of 32.5GB.

Huawei's open-source quantization tool MACaQ

What's more convincing is the post-quantization inference performance—the prefill stage was about 2.3x faster, and the decode stage about 2.6x faster. In practical use, prefill speed determines time-to-first-token for long prompts, while decode speed determines how smoothly subsequent tokens stream out. Both roughly doubling means quantization isn't just a "save VRAM" compromise—it's a directly perceptible experience improvement in production scenarios.

The significance of this result is that it turns quantization from "an offline task that takes a whole day" into "a routine operation you can finish over lunch." Teams that iterate models frequently no longer need to reserve a long quantization window every time they switch models.

Quantization is the process of mapping model weights from high-precision floating point (e.g., 16-bit) to lower bit-width (e.g., 4-bit). It significantly reduces VRAM usage and compute, but introduces precision loss. Common post-training quantization (PTQ) requires no retraining and determines scaling factors via calibration data; quantization-aware training (QAT) simulates quantization error during training, typically achieving higher accuracy at greater cost. The "4-bit quantization" mentioned here is extremely low bit-width, placing high demands on calibration and algorithm design. MACaQ's ability to quantize a 100B model in 64 minutes shows that its calibration and parallelization strategies are notably efficient.

llama.cpp 1.0: Local Inference Interfaces Are Finally Stable

llama.cpp has officially released version 1.0, with the team promising that the interface has entered a stable state and supporting over 150 model architectures.

Supporting over 150 model architectures

For developers who build applications and wrap toolchains on top of it, "interface stability" matters more than any single-point performance gain—it means upgrades no longer easily break existing integrations, and the ecosystem growing around it can invest with confidence.

Separately, real-world testing shows that a certain model in the Qwen series, with speculative decoding enabled, increased generation speed on a single RTX 5090 from 74 tokens per second to 134 tokens per second. Nearly doubled throughput shows that the speculative decoding approach—"accelerating with a draft model"—is already practical enough on consumer-grade GPUs, and the performance ceiling of local inference keeps getting pushed higher.

Speculative decoding is an inference acceleration technique: a lightweight draft model quickly generates multiple candidate tokens, which the target large model then verifies in parallel, keeping the accepted tokens. Since verification can be batched, overall throughput improves significantly, and the output distribution matches the target model (theoretically lossless). The smaller the draft model and the higher the acceptance rate, the greater the speedup. The article mentions that a Qwen-series model on an RTX 5090 with speculative decoding enabled went from 74 token/s to 134 token/s—a typical gain from this mechanism on consumer-grade GPUs.

MiniMax's Open-Source Image Generation Model: Even 8GB GPUs Can Run It

The original version of MiniMax's open-source 4K image generation model H3 Image requires 68GB of VRAM—a number that shuts out the vast majority of individual users. But according to the share, the community produced over 20 quantized versions in a very short time, with downloads exceeding 110,000. The smallest quantized version needs only 6.4GB of VRAM, meaning an 8GB GPU can run it.

MiniMax's open-source 4K image generation model H3 Image

This is a pattern worth noting: after an open-source model is released, how quickly the quantization community responds often determines its actual reach. No matter how impressive the original specs are, if it can't run, it has no users; a round of rapid quantization can pull the hardware threshold from the data center down to consumer-grade GPUs within days. The influence of an open-source model increasingly depends on whether it can be "redistributed" to ordinary people's machines.

Coding Agent Leaderboard: Cost-Effectiveness Becomes a New Dimension

Artificial Analysis has updated its coding agent leaderboard. DeepSeekAd V4.1 Flash isn't first in score, but its cost per problem is only about $0.30—nearly 10x cheaper than Kimi—making its cost-effectiveness stand out.

Artificial Analysis updates its coding agent leaderboard

This dimension is growing in importance. As the absolute capability gaps between models gradually narrow, cost per problem and the economics of per-task execution become key variables in model selection. This is especially true for teams that need to run large-scale evaluations and batch-execute coding tasks—cost differences get directly amplified by task volume, and a model that's nearly 10x cheaper may be more economical in total output even with slightly lower scores.

Artificial Analysis is an independent AI model benchmarking and analysis organization whose leaderboards are often used to compare model performance on coding, reasoning, and other tasks. Unlike simply looking at accuracy, the leaderboard has introduced economic indicators such as "cost per problem" in recent years, reflecting cost-effectiveness in real-world deployment. For teams that need to call APIs in bulk, cost per problem multiplied by task volume is the total expense—so even with slightly lower scores, a low-cost model may be more economical.

Google's Agents: From Problem-Solving to Formal Proofs

On Google's side, the Gemini 4.2 agent system helped mathematicians formalize multiple century-old problems, automatically writing 1.8 million lines of code in just one direction. The value of this kind of work isn't "AI replacing mathematicians," but translating human mathematicians' proof ideas into machine-verifiable formalized code—work that is tedious, time-consuming, and error-prone, precisely the domain where agents excel. Additionally, according to the share, Gemini 4.5 is already in gray-scale testing on Arena.

Formal proof means writing a mathematical proof completely in a machine-verifiable formal language (such as Lean, Coq, or Isabelle) so that a computer can check the correctness of each logical step. This process is extremely tedious, requiring the translation of human intuitive proofs into rigorous symbolic derivations, and typically takes years. Agent systems can automatically generate large amounts of formalized code to assist mathematicians with this tedious step, but final correctness is still guaranteed by the proof assistant kernel, so untrustworthy "AI hallucination" conclusions are not introduced.

Three Threads Converge on the Same Trend

Looking at this round of news together, the thread is quite clear. Quantization tools are compressing VRAM and time costs, llama.cpp is stabilizing the infrastructure for local inference, and open-source image generation models are rapidly lowering hardware thresholds through community quantization—together they're leveling the same barrier: making models run and run fast on ordinary devices. And the coding agent leaderboard's emphasis on cost gives the same answer from another angle: when using AI at scale, economics itself is a core competitive advantage.

For developers, the next steps may be quite simple: pay attention to the maturity of quantization toolchains, pay attention to the interface stability of inference frameworks, and factor cost into the same evaluation curve when selecting models. When hardware is no longer the ceiling, the gap shows up in who can use models efficiently.

One-line Summary

Huawei's quantization tool, llama.cpp 1.0, and other developments are jointly driving efficient operation of large models on ordinary hardware.

Paragraph Summary

This round of AI news centers on "getting large models to run on ordinary hardware": Huawei's open-source quantization tool MACaQ compresses 4-bit quantization of a 100B-parameter model to 64 minutes with peak VRAM of 32.5GB and over 2x inference speedup; llama.cpp releases version 1.0, promising interface stability and supporting over 150 architectures, with speculative decoding nearly doubling generation speed on a single RTX 5090; MiniMax's open-source 4K image generation model needs only 6.4GB of VRAM after community quantization, runnable on 8GB GPUs; the coding agent leaderboard introduces a cost-per-problem dimension, with DeepSeek V4.1 Flash standing out at about $0.30 per problem; Google's Gemini 4.2 agent helps mathematicians formalize century-old problems, automatically generating 1.8 million lines of code. These advances jointly level the VRAM and speed barriers, making economics the core competitive advantage for using AI at scale.

#Huawei MACaQ#llama.cpp 1.0#model quantization#local inference#MiniMax H3 Image#coding agent leaderboard#Gemini 4.2#speculative decoding
Share:

Related articles