Qwen3-4B

QLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.

The off-policy distillation experiment uses Qwen3 0.6B as the student model and Qwen3 4B as the teacher model.

In the final round, the averaged complementary answer method outperforms the better of the two original phrasings on only one of the three models.

In LLM agent opinion simulations, agents' answers to complementary questions (rewriting a statement into negated form and replacing probability p with 1-p) fail to satisfy the basic requirement that they sum to 1, and this test fails broadly on the three small open-source models Qwen3-4B, Llama-3.2-3B and Gemma-3-4B.

A controlled benchmark containing 24 worlds (each with exactly two possible outcomes) reproduces the failure of the complementary consistency test on all three models (Qwen3-4B, Llama-3.2-3B, Gemma-3-4B).

In the Qwen3-8B to Qwen3-4B communication setting, J-Comm achieves the highest average score across seven benchmarks.

A 4B model has limited knowledge capacity and performs relatively weakly on tasks requiring deep reasoning or specialized knowledge.

Qwen3-4B is a small language model with 4 billion parameters.

VASPO improves average accuracy by 2.59 percentage points over PPO on Qwen3-4B and by 1.81 percentage points on OLMo3-7B.

reef's OpenClaw-RL recipe uses 72 GSM8K problems as the task stream, with Qwen3-4B serving simultaneously as the policy model and the process reward model, Qwen3-32B as the student model, using seven GPUs, and setting a threshold that must be reached before an update is accepted.

When doing 50% pruning on Qwen3-4B, CACTUS raised the compressed-dense per-sample gradient cosine similarity on WikiText-2 from 0.670 to 0.755.

On Qwen3-4B, after pruning MLP layers to 2:4 sparsity and adding rank-64 correction, CACTUS combined with LoGRA still maintains 0.711 cosine alignment with the C4 gradients of dense LoGRA after projection.

When doing 50% pruning on Qwen3-4B, CACTUS reduced the gap between C4 perplexity and the dense model by 45%.

Gemma 4 E2B's KV cache capacity is 321,280 tokens, which is 2.6 to 6.3 times that of Granite 3.3 2B (125,184), SmolLM3-3B (120,320), Qwen3-1.7B (100,096), and Qwen3-4B (50,688).

On the GSM8K mathematical reasoning test, Qwen3-4B scores 0.915 and Gemma 4 E2B scores 0.911, a difference of +0.4 percentage points with a confidence interval crossing zero, making it the only model on par with Gemma.

On the BFCL tool-calling test, SmolLM3-3B scores 0.945, slightly above Gemma 4 E2B's 0.927, but the difference is not significant; Qwen3-4B (0.940), Qwen3-1.7B (0.935), Qwen2.5-1.5B (0.910), and Granite 3.3 2B (0.920) are all at the same level as Gemma.

In the scenario where 16 requests each carry a 3,072-token long prompt, Gemma 4 E2B maintains 1049 tokens/second, while Granite 3.3 2B drops to 732 (0.70x), SmolLM3-3B drops to 725 (0.69x), Phi-4-mini drops to 404 (0.39x), and Qwen3-4B reaches only 373 (0.36x).

In the Qwen3-4B comparative experiment, updating the model directly with unfiltered skills without using SGUID filtering decreased performance by 0.3 percentage points on HMMT25.

When using SGUID, Qwen3-4B improved by 0.5 percentage points on HMMT25 after the first round, and by a cumulative 1.1 percentage points after the second round.

The paper evaluates Qwen3.6-27B on five competition tasks from MLE-Bench Lite and Qwen3-4B on the text adventure game Zork I (Jericho framework).