MTEB
MTEB (Massive Text Embedding Benchmark) is a large-scale evaluation benchmark for text embedding models, jointly proposed by Hugging Face and other organizations. It encompasses multiple natural language processing tasks including classification, clustering, semantic textual similarity, retrieval, and reranking, with multilingual support. MTEB provides a comprehensive and systematic framework for evaluating text embedding models, accompanied by a public leaderboard that enables researchers and developers to compare the overall performance of different embedding models.
Core Facts
Timeline (last 90 days)
FaMTEB is the Persian version of MTEB, gathering a variety of Persian embedding evaluation tasks, among which the retrieval tasks measure a model's ability in query-document matching.
nanoMTEB is a streamlined subset of MTEB used for low-cost evaluation of the text-matching capability of retrieval and reranking models; ViDoRe is a benchmark for visual document retrieval, and V3 is its newer version.
SpecMRL's advantage on MTEB is consistent, but no independent reproduction or third-party analysis has yet been seen.
Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions.
The researchers used the MTEB and MMTEB benchmarks, covering 18 languages, and conducted more than 200 monolingual evaluations.
MTEB ranks 162 embedding models on 58 datasets across 8 task types.
The top-ranked embedding model on MTEB often only ranks third or fourth in a specific domain, and sometimes worse.
In real-world RAG pipelines, switching between the top 15 MTEB-ranked embedding models produces performance fluctuations of 15 to 25 percentage points.
MTEB's retrieval tasks report nDCG@10.
Different MTEB tracks (English v1, English v2, multilingual, and the BEIR retrieval-specific data) use different test sets, and scores cannot be compared directly across tracks; it is recommended to rank by retrieval score rather than overall average and to compare models within the same track.
10 more timeline events
All Facts (20)
EmbeddingGemma 2's score on the MTEB Code benchmark improved from 68.76 in the first generation to 78.68, an increase of 9.92 points.
75%UnverifiedMTEB is an important reference leaderboard for evaluating the performance of embedding models.
70%UnverifiedFaMTEB is the Persian version of MTEB, gathering a variety of Persian embedding evaluation tasks, among which the retrieval tasks measure a model's ability in query-document matching.
50%UnverifiednanoMTEB is a streamlined subset of MTEB used for low-cost evaluation of the text-matching capability of retrieval and reranking models; ViDoRe is a benchmark for visual document retrieval, and V3 is its newer version.
50%UnverifiedSpecMRL's advantage on MTEB is consistent, but no independent reproduction or third-party analysis has yet been seen.
50%UnverifiedExperiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions.
50%UnverifiedThe researchers used the MTEB and MMTEB benchmarks, covering 18 languages, and conducted more than 200 monolingual evaluations.
50%UnverifiedThe top-ranked embedding model on MTEB often only ranks third or fourth in a specific domain, and sometimes worse.
50%UnverifiedMTEB ranks 162 embedding models on 58 datasets across 8 task types.
50%UnverifiedIn real-world RAG pipelines, switching between the top 15 MTEB-ranked embedding models produces performance fluctuations of 15 to 25 percentage points.
50%UnverifiedMTEB's retrieval tasks report nDCG@10.
50%UnverifiedDifferent MTEB tracks (English v1, English v2, multilingual, and the BEIR retrieval-specific data) use different test sets, and scores cannot be compared directly across tracks; it is recommended to rank by retrieval score rather than overall average and to compare models within the same track.
50%UnverifiedEmbeddingGemma 2 improves on code tasks by approximately 14% compared to the first-generation EmbeddingGemma (MTEB code score 78.68 vs. 68.76).
50%UnverifiedConTEB is a publicly available evaluation set specifically designed to assess contextual embedding capabilities, distinct from the more general MTEB.
50%UnverifiedThe accuracy of retrieval models is typically evaluated on standard benchmarks such as BEIR and MTEB.
50%UnverifiedMTEB covers multiple task types including retrieval, classification, clustering, and semantic similarity, and has become the mainstream standard for embedding model leaderboards.
50%UnverifiedVoyage AI's embedding models stand out in standard benchmarks such as MTEB, and are particularly strong in code retrieval and long-document understanding scenarios.
50%UnverifiedFastEmbed-rs所使用的量化模型在MTEB基准测试中相比全精度模型的性能下降通常控制在1-2%以内
50%UnverifiedFastEmbed-rs所使用的量化模型通常采用INT8动态量化或静态量化，在MTEB基准测试中相比全精度模型的性能下降通常控制在1-2%以内
50%UnverifiedBGE和E5系列在MTEB基准测试上表现出色，其中BGE-M3支持多语言、多粒度和多功能检索
50%