Yalda-Embedding: A Breakthrough in Open-Source Persian Retrieval Models

Multilingual text representation has made significant progress, but dense retrieval for Persian remains constrained by the scarcity of language-specific training resources and compact embedding models. The release of Yalda Embedding offers a new solution to this gap—a 595.8 million parameter Persian retrieval model, fine-tuned from Qwen3-Embedding-0.6B and trained on over one million query-document pairs.

Model Architecture and Training Strategy

Yalda Embedding is initialized from Qwen3-Embedding-0.6B and fine-tuned via contrastive learning. The research team evaluated model performance on eight Persian retrieval tasks from FaMTEB and conducted incremental ablation experiments examining the effects of contrastive batch size, optimization stabilization, and relational knowledge distillation from F2LLM-v2-8B.

Under a single-round training setup, contrastive fine-tuning alone brought only a modest average improvement over the initialization, with regressions on two tasks. The final distilled configuration, however, outperformed the non-distilled model on all eight tasks, raising the average score from 0.4813 to 0.5488 without increasing inference-time model size. You may not have noticed that these gains include hard-negative evaluation tasks, even though hard negatives were not explicitly mined or provided during training.

Contrastive Learning and Contrastive Batch Size

Contrastive learning is the core paradigm for fine-tuning dense retrieval models: it pulls a query and its positive document closer in vector space while pushing the query away from other documents in the batch (negatives). Batch size plays a key role here—the larger the batch, the more negatives each query is exposed to, the richer the gradient signal, and the more discriminative the learned representation. Therefore, increasing the contrastive batch size typically improves retrieval performance, but it also imposes higher demands on memory and distributed training. Yalda Embedding's ablation experiments specifically examine this factor, showing that batch size is an important hyperparameter affecting the fine-tuning of small models.

Relational Knowledge Distillation

A common approach to knowledge distillation in retrieval models is to have a small student model mimic the similarity distribution or embedding vectors output by a large teacher model. Relational knowledge distillation goes a step further: instead of requiring the student to reproduce the teacher's embeddings dimension by dimension, it learns the relative relational structure among samples in the teacher model, such as the similarity ranking or distance relationships between a query and each document. This approach is more tolerant of differences in dimensionality or spatial scale, making it especially suitable when there is a large size gap between the student and teacher models. Yalda Embedding's distillation from F2LLM-v2-8B fits exactly this "large teacher, small student" configuration.

Performance: Leading Models of the Same Scale

On the eight Persian retrieval tasks, Yalda Embedding achieved the highest macro-average score of 0.5488 among the evaluated 0.6B models, surpassing Qwen3-Embedding-0.6B and F2LLM-v2-0.6B on every task. The model also approaches the performance of Qwen3-Embedding-4B, winning on four tasks, tying on one, and trailing by only 0.0075 in macro-average.

This result shows that through carefully designed fine-tuning and distillation strategies, small-scale models can approach the performance of larger models on language-specific tasks while maintaining inference efficiency.

Macro-Average Score and Hard Negatives

Macro-average is computed by first calculating the metric for each task individually (e.g., nDCG@10) and then taking an unweighted average of the scores across tasks. Compared with micro-average, which weights by the number of queries, macro-average gives each task equal weight, preventing one task from dominating the total score due to a large sample size, and is better suited to measuring a model's balanced performance across multiple tasks. Hard negatives refer to documents that are semantically close to the query but are not the correct answer; if a model cannot distinguish them, it is prone to errors in real retrieval scenarios. Even without explicitly mining hard negatives for training, Yalda Embedding still improved on hard-negative evaluation tasks, indicating that the representation improvements from distillation generalize well.

Open-Source Ecosystem and Reproducibility

The research team released the model weights, training and evaluation code, paired datasets, and the corresponding versions of teacher embeddings, making the data composition and training process inspectable, supporting independent evaluation and reuse, and avoiding costly repeated teacher inference. This open development model helps advance Persian retrieval research and provides a paradigm that other low-resource languages can learn from.

Key Takeaways

• Yalda Embedding is a 595.8 million parameter Persian retrieval model, fine-tuned from Qwen3-Embedding-0.6B and trained on over one million query-document pairs.
• On eight Persian retrieval tasks, its macro-average score of 0.5488 is the highest among models of the same scale, comprehensively surpassing Qwen3-Embedding-0.6B and F2LLM-v2-0.6B.
• The knowledge distillation strategy raised the average score from 0.4813 to 0.5488 without increasing the inference model size.
• The model weights, code, and datasets are all open source, supporting reproducible research and independent evaluation.

Background

FaMTEB and F2LLM-v2

FaMTEB is the Persian version of MTEB (Massive Text Embedding Benchmark), bringing together a variety of Persian embedding evaluation tasks, among which retrieval tasks measure a model's ability to match queries with documents. Yalda Embedding selected eight Persian retrieval tasks as its main evaluation set. F2LLM-v2 is a family of multilingual embedding models; F2LLM-v2-0.6B mentioned in the text is the same-scale comparison baseline, while F2LLM-v2-8B serves as the teacher model for knowledge distillation. Understanding these two names helps grasp the comparative framework of the experiments: the student model competes among peers of the same scale while learning from a larger teacher.