Bonsai
An open-source project centered on lightweight machine learning, primarily written in Shell. It targets edge devices, embedded systems, and resource-constrained environments, orchestrating complete ML pipelines through shell scripts.
Timeline (last 90 days)
Paper evaluations show that, with hybrid decoding on NVIDIA H20, BitMTP achieves a 2.2x five-task average speedup on BitNet b1.58-2B and 2.1x to 2.4x on the 1.7B to 4B Bonsai model series, with comparable task quality.
Combining the Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading, peak decoding reached 748 tokens/s and prefill peaked at 3.8K tokens/s.
Looking at each component individually, the performance gain is only about 10%–15%, but combining all three produces a notable synergistic effect (nonlinear acceleration).
A 'Thunder Deep Think' anomaly occurred: after the chain of thought completed, the model output 'stop' and halted without entering the main text generation phase; the author mitigated this by patching in detection of 'stop' followed by a forced jump to the main text branch.
When the context was set to 180K, compression should only be triggered at 80%–90% capacity, but in practice abnormal compression triggered at 20%–30%, causing reasoning results to be lost and creating a dead loop.
In concise mode, Bonsai is more prone to falling into 'Thunder Deep Think' due to a lack of available tools; in standard mode with ample tools, it more easily converges to calling a tool or producing the main text output.
Providing small models with too many prompt tokens (tens of thousands to hundreds of thousands of specifications and tools) disperses their limited attention, and lightweight frameworks such as DSH are more small-model-friendly.
Bonsai compresses a 50-plus GB model down to 5–7GB while retaining considerable logical reasoning, coding, and multi-turn dialogue capabilities.
In the author's own testing, Bonsai processed a Zhihu-style question in 16 seconds per turn at an average of 235 tokens/s.
Bonsai uses 1-bit quantization, representing each weight with just one bit, making the entire model file only slightly over 1GB.
1 more timeline events
All Facts (11)
Paper evaluations show that, with hybrid decoding on NVIDIA H20, BitMTP achieves a 2.2x five-task average speedup on BitNet b1.58-2B and 2.1x to 2.4x on the 1.7B to 4B Bonsai model series, with comparable task quality.
50%UnverifiedCombining the Ninfer inference engine, Bonsai compressed model, and KVMem VRAM offloading, peak decoding reached 748 tokens/s and prefill peaked at 3.8K tokens/s.
50%UnverifiedLooking at each component individually, the performance gain is only about 10%–15%, but combining all three produces a notable synergistic effect (nonlinear acceleration).
50%UnverifiedA 'Thunder Deep Think' anomaly occurred: after the chain of thought completed, the model output 'stop' and halted without entering the main text generation phase; the author mitigated this by patching in detection of 'stop' followed by a forced jump to the main text branch.
50%UnverifiedWhen the context was set to 180K, compression should only be triggered at 80%–90% capacity, but in practice abnormal compression triggered at 20%–30%, causing reasoning results to be lost and creating a dead loop.
50%UnverifiedIn concise mode, Bonsai is more prone to falling into 'Thunder Deep Think' due to a lack of available tools; in standard mode with ample tools, it more easily converges to calling a tool or producing the main text output.
50%UnverifiedProviding small models with too many prompt tokens (tens of thousands to hundreds of thousands of specifications and tools) disperses their limited attention, and lightweight frameworks such as DSH are more small-model-friendly.
50%UnverifiedBonsai compresses a 50-plus GB model down to 5–7GB while retaining considerable logical reasoning, coding, and multi-turn dialogue capabilities.
50%UnverifiedIn the author's own testing, Bonsai processed a Zhihu-style question in 16 seconds per turn at an average of 235 tokens/s.
50%UnverifiedBonsai uses 1-bit quantization, representing each weight with just one bit, making the entire model file only slightly over 1GB.
50%UnverifiedIn the summarization task, 4 out of the 8 models failed: the 4-billion-parameter Qwen3.5 fabricated content, LFM2.5 altered facts, Bonsai added nonexistent information, and MiniCPM5 missed key points.
50%