The Embedder's Dilemma: LLMs Are Better, but at What Cost?

arXiv:2608.128752026-08-12

大语言模型在整体表现上追平了专用文本嵌入模型,但成本最多贵1431倍

这篇论文对十个大语言模型(LLM)和二十六个文本嵌入模型进行了受控的、兼顾成本的对比测试,覆盖分类、语义相似度、聚类、句对判别和检索五类共37项任务。整体平均分上,表现最好的LLM(Gemini 3.1 Pro,77.6分)和表现最好的嵌入模型(77.2分)几乎打平,但LLM只在需要推理的检索任务上领先,而嵌入模型在分类任务上大幅领先。达到这种打平的代价是,LLM的成本最多是嵌入模型的1431倍(每次跑完基准测试154美元对0.11美元),在同一块GPU上处理速度也最多慢736倍。

他们做了什么

在完全相同的数据和条件下,比较了十个不做专门嵌入训练、仅靠提示词生成文本的前沿大语言模型,和二十六个专门训练的文本嵌入模型(参数量1.18亿到140亿)
构建了名为MTEB(LLM)的37项任务基准测试,覆盖五大类任务,并对每个模型都精确测量了实际API费用和GPU处理速度,而不只是准确率
整体平均分在统计上打平(77.6对77.2,差距在误差范围内),但按任务类型看优势相反:LLM在检索任务上领先(64.5对56.0),嵌入模型在分类任务上大幅领先(90.8对85.2)
Gemini 3.1 Pro每次跑完基准测试花费154美元,而性能相近的嵌入模型只需0.11美元,最多贵1431倍,且这一差距在五种不同硬件和定价方案下都成立
LLM推理成本的28%到81%来自内部的推理思考过程所产生的token;削减这部分推理预算后,大多数模型的检索质量保持不变甚至有所提升

Figure 1: Cost vs. performance across 36 models on MTEB(LLM). The frontier contains the leading embedding models and Gemini 3.1 Pro, which extends it by 0.4 points at 1,431× the cost of a comparable embedding.

Table 1: Model overview and per-category results. All ten LLMs and the ten highest-scoring embedding models. Scores are category means on a 0–100 scale; Overall is their mean. Cost is one MTEB(LLM) pass using API rates for LLMs and H100 throughput at $2.49/hr for embeddings (§3.3). Bold = best shown; full results are in Appendix B.2.
	Model	Params	Cls (8)	Clust (9)	STS (10)	PairCls (4)	Retr (6)	Overall (37)	Cost
LLM	Gemini 3.1 Pro	–	85.2	66.6	88.5	83.2	64.5	77.6	$154
Gemini 3 Flash	–	84.1	65.7	87.6	86.3	52.3	75.2	$56
Qwen3.6-27B	–	84.8	53.6	84.9	83.6	62.4	73.9	$103
Qwen3.6-35B-A3B	–	83.8	55.3	84.0	82.9	60.4	73.3	$34
Kimi-K2.6	–	84.5	55.4	83.9	78.0	56.7	71.7	$111
DeepSeek-V4-Flash	–	81.9	43.7	82.2	81.6	53.3	68.5	$3
MiniMax-M2.7	–	80.9	48.6	81.0	82.8	48.0	68.2	$25
GLM-4.7	–	84.2	36.5	83.5	79.7	54.4	67.7	$63
Gemini 3.1 Flash Lite	–	82.9	21.7	85.1	83.7	49.0	64.5	$7
DeepSeek-R1	–	82.5	31.3	83.7	79.5	44.7	64.3	$57
Embedding	Octen-8B	7.6B	90.1	65.1	88.7	86.1	56.0	77.2	$0.11
Qwen3-E-8B	7.6B	90.1	65.9	88.5	86.5	54.2	77.0	$0.11
Qwen3-E-4B	4.0B	89.3	64.6	88.8	86.5	50.3	75.9	$0.07
Nemotron-8B	7.5B	84.8	64.1	86.2	86.7	54.7	75.3	$0.11
KaLM-12B	11.8B	88.8	63.4	84.6	87.1	49.9	74.8	$0.16
Jina-v5-S	596M	90.4	61.5	86.9	85.1	48.0	74.4	$0.03
SFR-2	7.1B	90.8	66.7	77.9	85.3	48.8	73.9	$0.14
Jina-v5-Nano	212M	89.6	60.4	87.0	85.0	47.4	73.9	$0.01
F2LLM-14B	14.0B	77.7	66.5	84.1	86.3	52.1	73.3	$0.22
GTE-Qwen2-7B	7.1B	86.7	65.2	81.6	86.0	45.8	73.1	$0.09

Figure 2: Cost vs. performance by task category. Each panel plots score (0–100) against cost per benchmark pass (log scale) for all 36 models; the line is the Pareto frontier over all models and the star marks the best LLM in that category. Pro extends the retrieval frontier; embedding models define the frontiers for classification, clustering, STS, and pair classification.

Table 2: Statistical significance. Paired bootstrap test (10,000 resamples, seed 42). Δ = Gemini 3.1 Pro − best embedding; significance at α = 0.05. Pair classification uses Pro for consistency, although Flash scores higher.
Comparison	Δ	95% CI	p	Sig.
Overall (Gemini 3.1 Pro vs. Octen-8B, MTEB(LLM))
All tasks	+0.3	[−2.4, +3.1]	0.85	No
Per category (Gemini 3.1 Pro vs. best embedding)
Retrieval	+8.5	[+0.2, +16.8]	<0.05	Yes
Clustering	−0.2	[−5.6, +5.1]	0.96	No
STS	−0.3	[−2.2, +1.8]	0.75	No
Pair Classification	−3.9	[−13.6, +5.9]	0.50	No
Classification	−5.6	[−9.2, −2.4]	<0.01	Yes

Figure 3: Retrieve-then-rerank. LLM listwise reranking improves every first stage on BRIGHT. On BEIR, a strong embedding first stage outperforms all reranked configurations.

Table 3: Complete model listing with mean (macro) scores across 37 MTEB(LLM) tasks, total benchmark costs, and references. LLM costs reflect actual API usage; embedding costs from H100 throughput benchmarking ($2.49/hr).
Model	Params	Score	Cost	Reference
LLM Models
Gemini 3.1 Pro	–	77.6	$154.14	–
Gemini 3 Flash	–	75.2	$55.87	–
Qwen3.6-27B	–	73.9	$103.43	–
Qwen3.6-35B-A3B	–	73.3	$34.06	–
Kimi-K2.6	–	71.7	$110.70	–
DeepSeek-V4-Flash	–	68.5	$3.16	–
MiniMax-M2.7	–	68.2	$24.52	–
GLM-4.7	–	67.7	$63.31	–
Gemini 3.1 Flash Lite	–	64.5	$6.85	–
DeepSeek-R1	–	64.3	$57.38	–
Embedding Models (ranked by score)
Octen-8B	7.6B	77.2	$0.108	–
Qwen3-E-8B	7.6B	77.0	$0.108	78
Qwen3-E-4B	4.0B	75.9	$0.069	78
Nemotron-8B	7.5B	75.3	$0.115	37
KaLM-12B	11.8B	74.8	$0.158	80
Jina-v5-S	596M	74.4	$0.034	3
SFR-2	7.1B	73.9	$0.136	47
Jina-v5-Nano	212M	73.9	$0.010	3
F2LLM-14B	14.0B	73.3	$0.215	79
GTE-Qwen2-7B	7.1B	73.1	$0.089	42
Linq-Mistral	7.1B	72.8	$0.138	15
Qwen3-E-0.6B	596M	72.5	$0.018	78
EmbGemma-300M	308M	72.2	$0.008	39
F2LLM-8B	7.6B	72.2	$0.122	79
F2LLM-4B	4.0B	71.9	$0.079	79
F2LLM-1.7B	1.7B	71.7	$0.030	79
GTE-Qwen2-1.5B	1.5B	70.8	$0.024	42
F2LLM-0.6B	596M	69.9	$0.017	79

Figure 4: The thinking-token tax: what reasoning costs, and what it buys. (a) API cost per benchmark pass by token type for all ten LLMs. (b) Mean retrieval score with default vs. disabled reasoning for six models from five families; labels show the reduction in generated tokens. Four models preserve or improve retrieval with 54–96% fewer generated tokens; the two Qwen models lose ground.

Table 4: Category-level performance (best LLM vs. best embedding per category). Δ = best LLM − best embedding; bold = winner; task counts in parentheses.
Category	Gemini 3.1 Pro	Gemini 3 Flash	Best Emb.	Best Model	Δ
Retrieval (6)	64.5	52.3	56.0	Octen-8B	+8.5
Clustering (9)	66.6	65.7	66.7	SFR-2	−0.2
STS (10)	88.5	87.6	88.8	Qwen3-E-4B	−0.3
PairCls (4)	83.2	86.3	87.1	KaLM-12B	−3.9
Classification (8)	85.2	84.1	90.8	SFR-2	−5.6

Figure 5: How many documents each architecture reads jointly with the query. Top: the pipeline. Bottom: its attention mask, drawn over the same N documents at the same scale in every panel so the four are directly comparable. Each row is a token and each column a token it may read. The red region is what one forward pass reads jointly with the query; it cannot be computed before the query arrives, so cost grows with it.

Table 5: Complete per-category results for all 36 complete models, ranked by overall (macro) score. Bold = best in column.
Rank	Model	Type	Cls (8)	Clust (9)	STS (10)	PairCls (4)	Retr (6)	Overall
1	Gemini 3.1 Pro	LLM	85.2	66.6	88.5	83.2	64.5	77.6
2	Octen-8B	Emb	90.1	65.1	88.7	86.1	56.0	77.2
3	Qwen3-E-8B	Emb	90.1	65.9	88.5	86.5	54.2	77.0
4	Qwen3-E-4B	Emb	89.3	64.6	88.8	86.5	50.3	75.9
5	Nemotron-8B	Emb	84.8	64.1	86.2	86.7	54.7	75.3
6	Gemini 3 Flash	LLM	84.1	65.7	87.6	86.3	52.3	75.2
7	KaLM-12B	Emb	88.8	63.4	84.6	87.1	49.9	74.8
8	Jina-v5-S	Emb	90.4	61.5	86.9	85.1	48.0	74.4
9	SFR-2	Emb	90.8	66.7	77.9	85.3	48.8	73.9
10	Jina-v5-Nano	Emb	89.6	60.4	87.0	85.0	47.4	73.9
11	Qwen3.6-27B	LLM	84.8	53.6	84.9	83.6	62.4	73.9
12	F2LLM-14B	Emb	77.7	66.5	84.1	86.3	52.1	73.3
13	Qwen3.6-35B-A3B	LLM	83.8	55.3	84.0	82.9	60.4	73.3
14	GTE-Qwen2-7B	Emb	86.7	65.2	81.6	86.0	45.8	73.1
15	Linq-Mistral	Emb	82.5	61.3	83.4	86.1	51.0	72.8
16	Qwen3-E-0.6B	Emb	85.4	60.9	85.7	86.1	44.1	72.5
17	EmbGemma-300M	Emb	86.5	59.3	82.1	86.0	47.3	72.2
18	F2LLM-8B	Emb	75.2	65.8	83.9	86.1	50.0	72.2
19	F2LLM-4B	Emb	75.3	64.7	83.6	85.9	49.9	71.9
20	Kimi-K2.6	LLM	84.5	55.4	83.9	78.0	56.7	71.7
21	F2LLM-1.7B	Emb	74.7	65.1	83.8	86.1	48.5	71.7
22	GTE-Qwen2-1.5B	Emb	83.3	60.0	80.3	86.6	43.9	70.8
23	F2LLM-0.6B	Emb	72.5	62.9	83.1	85.8	45.4	69.9
24	mE5-L-Inst	Emb	74.4	59.1	83.9	86.1	44.8	69.7
25	DeepSeek-V4-Flash	LLM	81.9	43.7	82.2	81.6	53.3	68.5
26	MiniMax-M2.7	LLM	80.9	48.6	81.0	82.8	48.0	68.2
27	GLM-4.7	LLM	84.2	36.5	83.5	79.7	54.4	67.7
28	BGE-M3	Emb	76.2	45.9	80.4	85.9	44.3	66.5
29	mE5-L	Emb	73.3	48.3	80.3	83.9	43.6	65.9
30	Arctic-L-v2	Emb	71.4	49.7	77.2	83.9	44.6	65.3

Figure 6: Per-task performance. Horizontal bars show the full embedding score range (min to max across 26 models) for each MTEB(LLM) task, grouped by category. Circle = best embedding; diamond = best LLM on that task, taken over all ten (six different LLMs hold it across the suite, Gemini 3.1 Pro on 21 of 37 tasks). Where the diamond falls inside the bar, some embedding model already matches the best LLM: this holds on 7 of 8 classification tasks and 7 of 10 STS tasks, but on only 1 of 6 retrieval tasks.

Table 6: Per-task scores for representative models across all 37 MTEB(LLM) tasks. Bold = best in row (incl. best embedding). Metric per category in italics.
Task	Cat.	Gemini 3.1 Pro	Gemini 3 Flash	Qwen3.6-27B	Octen-8B	Qwen3-E-8B	Best Emb.
Classification (Accuracy)
AmazonCounterfactualClassification	Cls	84.8	81.2	90.1	93.0	92.9	93.0
Banking77Classification	Cls	85.0	83.1	80.5	87.3	87.2	91.6
ImdbClassification	Cls	98.0	97.6	97.4	97.7	97.8	97.8
MTOPDomainClassification	Cls	95.5	96.4	96.9	98.2	98.1	99.2
MassiveIntentClassification	Cls	84.9	85.4	84.6	85.7	85.8	88.9
MassiveScenarioClassification	Cls	79.4	75.9	76.4	89.3	89.4	93.0
ToxicConversationsClassification	Cls	89.6	90.0	89.2	91.9	92.0	94.4
TweetSentimentExtractionClassification	Cls	64.2	63.0	63.2	77.9	77.8	78.9
Clustering (V-measure)
ArxivClusteringP2P	Clust	60.1	60.3	54.5	61.9	62.5	64.1
ArxivClusteringS2S	Clust	62.5	60.1	41.6	60.5	61.1	61.5
BiorxivClusteringP2PV2	Clust	62.5	60.8	49.9	61.6	65.3	77.3
MedrxivClusteringP2PV2	Clust	52.5	50.6	48.0	55.9	55.9	60.7
MedrxivClusteringS2SV2	Clust	54.0	50.3	42.7	53.3	55.2	57.5
RedditClusteringP2P	Clust	93.9	90.5	60.1	78.7	79.8	80.8
StackExchangeClusteringP2PV2	Clust	42.9	49.0	46.1	59.7	59.5	60.7
StackExchangeClusteringV2	Clust	91.8	90.2	73.6	85.8	86.3	86.3
TwentyNewsgroupsClusteringV2	Clust	78.7	79.9	66.3	68.5	67.6	73.8
STS (Spearman ρ)
BIOSSES	STS	89.6	88.7	85.0	84.4	82.0	88.8
SICKR	STS	86.1	86.8	76.4	87.9	88.4	91.1
STS12	STS	82.6	79.5	77.1	87.3	87.3	87.4
STS13	STS	91.9	91.5	90.6	93.9	94.0	94.6
STS14	STS	90.5	89.3	88.0	90.9	91.0	91.4
STS15	STS	94.4	93.9	91.8	94.3	94.2	94.5
STS16	STS	89.0	88.5	87.1	92.5	92.5	92.9
STS17	STS	94.2	94.1	91.8	93.6	93.7	93.7
STS22v2	STS	73.6	71.7	72.0	69.5	68.9	69.5
STSBenchmark	STS	92.6	91.7	89.6	93.2	93.4	94.9

Figure 7: Same-hardware inference throughput. Two open-weight LLMs and seven representative embedding models (118M–14B) served on one H100 (tokens/second, log scale); the full 26-model embedding throughput is in Table 18. Even the slowest embedding runs ∼2.5× faster than the fastest LLM; the fastest runs ∼736× faster.

Table 7: Per-task scores: Classification (Accuracy).
Model	AmazonCF	Banking77	IMDB	MTOPDomain	MassiveIntent	MassiveScenario	ToxicConvs	TweetSent	Mean
LLMs
Gemini 3.1 Pro	84.8	85.0	98.0	95.5	84.9	79.4	89.6	64.2	85.2
Gemini 3 Flash	81.2	83.1	97.6	96.4	85.4	75.9	90.0	63.0	84.1
Qwen3.6-27B	90.1	80.5	97.4	96.9	84.6	76.4	89.2	63.2	84.8
Qwen3.6-35B-A3B	88.7	77.8	97.4	96.8	79.8	76.6	91.0	62.4	83.8
Kimi-K2.6	89.3	82.8	97.4	96.6	83.2	76.5	88.8	61.6	84.5
DeepSeek-V4-Flash	82.4	78.5	96.8	95.8	82.4	77.5	80.4	61.2	81.9
MiniMax-M2.7	83.6	78.4	97.0	95.2	75.1	67.0	87.6	63.0	80.9
GLM-4.7	88.7	80.4	97.2	96.8	83.0	77.1	87.6	63.2	84.2
Gemini 3.1 FLite	79.7	78.7	97.6	95.9	83.2	76.2	88.6	63.4	82.9
DeepSeek-R1	86.9	78.9	97.0	96.1	82.7	76.6	78.2	63.4	82.5
Embedding models
Octen-8B	93.0	87.3	97.7	98.2	85.7	89.3	91.9	77.9	90.1
Qwen3-E-8B	92.9	87.2	97.8	98.1	85.8	89.4	92.0	77.8	90.1
Qwen3-E-4B	92.6	86.3	97.4	97.6	85.2	86.0	92.3	77.1	89.3
Nemotron-8B	83.8	83.6	97.1	96.5	83.5	83.5	88.3	62.1	84.8
KaLM-12B	90.4	87.7	96.3	98.5	84.6	86.7	90.2	76.1	88.8
Jina-v5-S	91.8	91.5	95.9	99.2	88.9	92.6	94.4	68.5	90.4
SFR-2	92.7	90.1	97.6	98.3	86.3	90.5	91.9	78.9	90.8
Jina-v5-Nano	91.3	90.2	95.5	98.0	88.4	93.0	94.4	66.3	89.6
F2LLM-14B	64.1	84.7	91.2	98.8	77.6	89.4	58.7	56.7	77.7
GTE-Qwen2-7B	86.8	84.9	96.7	98.0	83.7	85.7	88.2	70.0	86.7
Linq-Mistral	83.9	87.7	94.9	97.0	82.7	84.5	70.2	59.2	82.5
Qwen3-E-0.6B	90.5	80.8	96.3	95.8	80.1	83.8	81.9	74.3	85.4
EmbGemma-300M	89.6	91.6	91.9	98.6	85.6	91.6	83.4	59.8	86.5
F2LLM-8B	63.1	83.6	87.3	99.0	72.2	85.8	59.3	51.4	75.2
F2LLM-4B	61.3	82.2	86.4	99.0	72.3	86.5	60.7	54.2	75.3
F2LLM-1.7B	61.2	77.6	83.4	98.3	74.6	87.5	61.1	54.3	74.7
GTE-Qwen2-1.5B	81.6	79.7	95.8	95.3	77.9	79.8	84.9	71.3	83.3
F2LLM-0.6B	59.5	73.8	79.7	97.3	73.4	87.3	59.7	49.0	72.5

Figure 8: Capability profiles across five task categories. Gemini 3.1 Pro (red), Qwen3.6-27B (amber), and Octen-8B (blue). The LLMs lead on retrieval, while Octen-8B leads on classification and remains competitive across categories.

Table 8: Per-task scores: STS (Spearman ρ).
Model	BIOSSES	SICK-R	STS12	STS13	STS14	STS15	STS16	STS17	STS22v2	STSBench	Mean
LLMs
Gemini 3.1 Pro	89.6	86.1	82.6	91.9	90.5	94.4	89.0	94.2	73.6	92.6	88.5
Gemini 3 Flash	88.7	86.8	79.5	91.5	89.3	93.9	88.5	94.1	71.7	91.7	87.6
Qwen3.6-27B	85.0	76.4	77.1	90.6	88.0	91.8	87.1	91.8	72.0	89.6	84.9
Qwen3.6-35B-A3B	84.0	75.5	75.1	89.6	85.1	91.4	85.1	92.7	73.4	87.8	84.0
Kimi-K2.6	86.4	82.1	72.2	89.3	85.1	91.1	83.9	90.1	70.9	87.8	83.9
DeepSeek-V4-Flash	78.5	75.3	75.6	87.3	84.5	89.2	82.4	91.5	69.2	87.9	82.2
MiniMax-M2.7	87.7	79.7	65.4	86.5	81.9	88.4	81.1	87.1	66.7	85.7	81.0
GLM-4.7	82.3	76.0	73.1	88.5	85.7	91.9	84.9	90.4	73.2	88.5	83.5
Gemini 3.1 FLite	89.8	78.0	76.0	90.5	86.3	91.4	86.5	91.4	71.2	90.3	85.1
DeepSeek-R1	85.0	79.7	71.9	88.7	85.6	89.9	84.2	91.5	71.4	89.2	83.7
Embedding models
Octen-8B	84.4	87.9	87.3	93.9	90.9	94.3	92.5	93.6	69.5	93.2	88.7
Qwen3-E-8B	82.0	88.4	87.3	94.0	91.0	94.2	92.5	93.7	68.9	93.4	88.5
Qwen3-E-4B	84.8	87.7	87.4	94.6	91.4	94.5	92.9	93.2	67.7	93.7	88.8
Nemotron-8B	86.3	85.4	83.4	91.5	87.9	92.1	88.7	91.4	64.3	90.8	86.2
KaLM-12B	87.0	81.9	81.4	89.2	85.9	90.6	86.7	89.0	65.1	89.1	84.6
Jina-v5-S	85.3	90.4	86.6	89.2	88.9	92.6	87.4	89.1	65.0	94.9	86.9
SFR-2	87.5	76.7	74.6	81.4	80.6	87.5	84.6	85.0	38.2	82.6	77.9
Jina-v5-Nano	87.4	91.1	86.2	89.9	89.2	93.1	85.4	88.4	64.9	94.2	87.0
F2LLM-14B	88.2	82.8	82.4	86.9	84.8	91.3	85.6	88.9	63.4	87.0	84.1
GTE-Qwen2-7B	83.0	79.7	78.1	88.6	83.4	89.5	85.5	87.1	54.7	86.2	81.6
Linq-Mistral	86.1	84.4	77.6	87.8	84.1	91.2	87.4	89.9	56.1	89.0	83.4
Qwen3-E-0.6B	84.6	84.6	83.6	92.1	87.0	91.7	89.7	87.9	64.6	91.3	85.7
EmbGemma-300M	83.2	81.9	79.1	85.5	84.5	89.2	84.7	86.6	57.3	88.8	82.1
F2LLM-8B	88.8	82.4	81.3	86.6	83.8	91.3	85.4	89.3	63.2	86.5	83.9
F2LLM-4B	87.5	81.9	82.3	86.1	83.8	91.5	85.2	87.9	62.8	87.1	83.6
F2LLM-1.7B	88.5	81.3	82.6	86.7	84.5	91.3	85.2	87.8	62.9	87.6	83.8
GTE-Qwen2-1.5B	81.4	80.7	72.9	84.4	81.7	88.7	84.6	86.1	56.6	85.6	80.3
F2LLM-0.6B	87.8	80.0	82.7	86.7	84.1	90.6	84.9	84.0	62.3	87.6	83.1

Figure 9: Cross-model task-behaviour correlation. Pearson correlations based on per-task MTEB(LLM) scores. LLMs and embedding models form visually distinct clusters: LLMs are strongly intercorrelated while embedding models cluster by architecture family. Lower inter-paradigm correlations are consistent with their different category profiles.

Table 9: Per-task scores: Clustering (V-measure).
Model	ArxivP2P	ArxivS2S	BioP2P	MedP2P	MedS2S	Reddit	SE-P2P	SE-Cl	20News	Mean
LLMs
Gemini 3.1 Pro	60.1	62.5	62.5	52.5	54.0	93.9	42.9	91.8	78.7	66.6
Gemini 3 Flash	60.3	60.1	60.8	50.6	50.3	90.5	49.0	90.2	79.9	65.7
Qwen3.6-27B	54.5	41.6	49.9	48.0	42.7	60.1	46.1	73.6	66.3	53.6
Qwen3.6-35B-A3B	48.5	60.4	53.2	49.4	46.2	69.2	42.2	72.7	55.7	55.3
Kimi-K2.6	52.6	44.2	51.9	50.6	39.1	57.4	52.1	81.8	68.9	55.4
DeepSeek-V4-Flash	37.4	45.5	29.3	27.0	40.9	55.2	40.3	53.5	64.4	43.7
MiniMax-M2.7	36.4	40.6	38.3	39.8	42.4	67.4	39.4	75.1	57.9	48.6
GLM-4.7	23.6	28.6	33.2	37.5	37.8	40.6	37.9	49.5	39.8	36.5
Gemini 3.1 FLite	14.7	18.9	17.4	19.7	21.6	28.5	21.5	25.8	27.2	21.7
DeepSeek-R1	22.0	25.3	32.4	31.7	33.4	26.6	33.1	42.1	34.9	31.3
Embedding models
Octen-8B	61.9	60.5	61.6	55.9	53.3	78.7	59.7	85.8	68.5	65.1
Qwen3-E-8B	62.5	61.1	65.3	55.9	55.2	79.8	59.5	86.3	67.6	65.9
Qwen3-E-4B	62.6	60.3	64.0	55.5	54.1	77.1	58.8	83.9	65.3	64.6
Nemotron-8B	59.2	59.1	65.2	52.9	52.6	79.7	58.7	84.6	65.2	64.1
KaLM-12B	61.6	57.4	64.1	54.7	53.2	76.1	58.5	83.3	62.0	63.4
Jina-v5-S	60.0	57.5	63.6	53.7	52.6	70.5	58.9	76.8	59.9	61.5
SFR-2	61.8	59.6	65.6	58.9	57.5	80.8	59.3	83.2	73.8	66.7
Jina-v5-Nano	59.1	55.4	61.6	54.6	52.1	67.4	57.3	76.8	59.7	60.4
F2LLM-14B	63.1	60.6	75.6	57.7	56.7	77.9	57.2	81.1	68.8	66.5
GTE-Qwen2-7B	64.1	61.5	66.7	57.0	54.1	80.5	60.7	84.5	58.1	65.2
Linq-Mistral	58.0	55.7	61.3	51.5	52.8	77.0	56.2	78.4	61.1	61.3
Qwen3-E-0.6B	60.2	57.6	61.8	52.9	51.1	69.9	56.0	77.3	61.7	60.9
EmbGemma-300M	57.9	53.9	62.8	54.4	51.0	75.9	51.9	73.7	51.8	59.3
F2LLM-8B	61.7	60.9	75.5	59.2	54.4	75.3	58.2	81.2	65.8	65.8
F2LLM-4B	61.3	59.2	74.2	59.8	54.7	74.4	55.2	79.9	63.9	64.7
F2LLM-1.7B	62.6	59.6	77.3	60.7	56.1	74.4	55.2	75.5	64.2	65.1
GTE-Qwen2-1.5B	58.5	57.5	62.1	53.2	52.3	71.1	52.3	73.6	59.1	60.0
F2LLM-0.6B	61.7	58.6	71.1	58.7	54.5	69.7	56.0	76.0	59.9	62.9

Figure 10: Overall rankings. Pro (77.6) narrowly leads Octen-8B (77.2); the difference lies within statistical noise (p = 0.85). Flash-Lite (64.5) ranks near the bottom.

Table 10: Per-task scores: Pair Classification (AP / Accuracy).
Model	LegalPC	RTE3	SprintDup	TwtURL	Mean
LLMs
Gemini 3.1 Pro	72.2	95.4	82.2	82.8	83.2
Gemini 3 Flash	77.4	96.7	83.4	87.6	86.3
Qwen3.6-27B	71.6	92.9	84.6	85.2	83.6
Qwen3.6-35B-A3B	69.4	93.6	85.2	83.4	82.9
Kimi-K2.6	69.2	88.8	78.8	75.4	78.0
DeepSeek-V4-Flash	67.2	90.4	86.6	82.0	81.6
MiniMax-M2.7	71.8	91.5	82.8	85.0	82.8
GLM-4.7	69.6	90.0	82.2	76.8	79.7
Gemini 3.1 FLite	72.4	95.0	86.6	80.8	83.7
DeepSeek-R1	68.8	90.0	79.6	79.6	79.5
Embedding models
Octen-8B	70.0	85.4	99.4	89.4	86.1
Qwen3-E-8B	70.4	85.7	99.6	90.2	86.5
Qwen3-E-4B	72.0	84.8	99.8	89.4	86.5
Nemotron-8B	72.6	84.8	99.4	89.8	86.7
KaLM-12B	73.6	85.2	99.8	89.6	87.1
Jina-v5-S	68.0	85.2	99.6	87.4	85.1
SFR-2	67.4	84.8	99.8	89.0	85.3
Jina-v5-Nano	68.8	85.0	99.4	86.6	85.0
F2LLM-14B	72.0	84.8	99.6	88.6	86.3
GTE-Qwen2-7B	71.2	85.2	99.6	88.0	86.0
Linq-Mistral	71.0	84.8	99.8	88.6	86.1
Qwen3-E-0.6B	70.4	84.8	99.8	89.4	86.1
EmbGemma-300M	71.0	84.8	99.6	88.6	86.0
F2LLM-8B	72.0	84.8	99.6	88.0	86.1
F2LLM-4B	70.6	84.8	99.8	88.4	85.9
F2LLM-1.7B	71.0	84.8	99.6	89.0	86.1
GTE-Qwen2-1.5B	73.0	84.8	99.4	89.2	86.6
F2LLM-0.6B	70.8	85.0	99.4	88.0	85.8

Figure 11: Classification rankings. SFR-2 ranks first (90.8); Pro scores 85.2 and ranks below ten embedding models.

Table 11: Per-task scores: Retrieval (Recall@1).
Model	AILA	FQuAD	HC3Fin	ConsumerQA	PubHealth	TwtHjerne	Mean
LLMs
Gemini 3.1 Pro	14.5	92.0	71.0	86.0	90.0	33.4	64.5
Gemini 3 Flash	5.7	88.0	60.0	79.0	49.0	32.4	52.3
Qwen3.6-27B	14.2	96.0	57.0	88.0	86.0	33.0	62.4
Qwen3.6-35B-A3B	12.9	95.0	55.0	86.0	83.0	30.5	60.4
Kimi-K2.6	13.2	91.0	56.0	80.0	67.0	33.0	56.7
DeepSeek-V4-Flash	13.3	83.0	50.0	76.0	64.0	33.4	53.3
MiniMax-M2.7	9.7	82.0	42.0	67.0	59.0	28.2	48.0
GLM-4.7	13.0	78.0	56.0	78.0	70.0	31.6	54.4
Gemini 3.1 FLite	10.2	84.0	46.0	77.0	45.0	31.9	49.0
DeepSeek-R1	11.9	57.0	43.0	62.0	64.0	30.6	44.7
Embedding models
Octen-8B	23.2	69.0	66.0	67.0	81.0	29.9	56.0
Qwen3-E-8B	21.8	68.0	59.0	70.0	78.0	28.6	54.2
Qwen3-E-4B	21.0	61.0	56.0	64.0	75.0	24.8	50.3
Nemotron-8B	9.1	72.0	67.0	70.0	80.0	29.9	54.7
KaLM-12B	10.3	70.0	55.0	63.0	71.0	30.2	49.9
Jina-v5-S	14.7	64.0	48.0	59.0	77.0	25.5	48.0
SFR-2	11.2	67.0	57.0	56.0	74.0	27.6	48.8
Jina-v5-Nano	12.6	65.0	44.0	61.0	74.0	27.6	47.4
F2LLM-14B	12.2	63.0	67.0	54.0	85.0	31.5	52.1
GTE-Qwen2-7B	9.1	64.0	51.0	55.0	77.0	19.0	45.8
Linq-Mistral	10.6	68.0	54.0	66.0	78.0	29.1	51.0
Qwen3-E-0.6B	20.2	55.0	36.0	63.0	68.0	22.4	44.1
EmbGemma-300M	5.5	72.0	42.0	62.0	78.0	24.2	47.3
F2LLM-8B	8.9	63.0	63.0	51.0	85.0	28.9	50.0
F2LLM-4B	8.1	64.0	62.0	53.0	83.0	29.4	49.9
F2LLM-1.7B	5.4	66.0	55.0	54.0	82.0	28.8	48.5
GTE-Qwen2-1.5B	5.3	58.0	44.0	59.0	76.0	21.2	43.9
F2LLM-0.6B	3.6	56.0	52.0	54.0	80.0	26.7	45.4

Figure 12: Clustering rankings. Statistical tie; SFR-2 (66.7) edges Pro (66.6). Flash-Lite scores 21.7.

Table 12: MTEB(LLM) task suite (37 tasks). N = held-out test samples (summed over languages for multilingual tasks); Q = queries, C = corpus documents. Multilingual tasks are evaluated per language and averaged. Held-out subsets (seed 42) derived from MTEB and MMTEB tasks (49; 23), hosted at mteb/llm-eval-*. Token counts: Table 13.
Task	Lang.	N	Cls.	Metric	Source
Classification (8 tasks)
ImdbCls	en	500	2	Acc.	45
Banking77Cls	en	3k	77	Acc.	11
AmazonCounterfactualCls	en, de, ja	809	2	Acc.	54
MTOPDomainCls	en, de, fr	2k	11	Acc.	41
MassiveIntentCls	en, de, fr, ja	4k	60	Acc.	24
MassiveScenarioCls	en, de, fr, ja	3k	18	Acc.	24
ToxicConversationsCls	en	500	2	Acc.	8
TweetSentimentCls	en	500	3	Acc.	49
Semantic Textual Similarity (10 tasks)
STSBenchmark	en	500	–	Spearman	12
SICK-R	en	500	–	Spearman	46
STS12	en	500	–	Spearman	2
STS13	en	500	–	Spearman	2
STS14	en	500	–	Spearman	2
STS15	en	500	–	Spearman	1
STS16	en	500	–	Spearman	1
BIOSSES	en	100	–	Spearman	60
STS17	en, de, es, fr	1k	–	Spearman	12
STS22v2	en, de, es, fr, ru, zh	2k	–	Spearman	14
Clustering (9 tasks)
RedditClustP2P	en	1k	–	V-meas.	49
TwentyNewsgroupsV2	en	1k	–	V-meas.	49
StackExchangeClustP2PV2	en	1k	–	V-meas.	49
StackExchangeClustV2	en	1k	–	V-meas.	49
ArxivClustP2P	en	1k	–	V-meas.	49
ArxivClustS2S	en	1k	–	V-meas.	49
BiorxivClustP2PV2	en	1k	–	V-meas.	49
MedrxivClustP2PV2	en	1k	–	V-meas.	49
MedrxivClustS2SV2	en	1k	–	V-meas.	49

Table 13: Token budget per task (GPT-4o tokenizer; raw text). Actual counts vary by ±15–40% across model vocabularies. *Train split is the kNN reference corpus processed only by embedding models; LLMs process only the test split. Corpus-in-context formatting adds ∼20 tokens per document for LLM retrieval.
Classification
Task	Test	Train (kNN)*
ImdbCls	132k	7,281k
Banking77Cls	38k	134k
AmazonCounterfactualCls	22k	258k
MTOPDomainCls	23k	393k
MassiveIntentCls	37k	433k
MassiveScenarioCls	30k	433k
ToxicConversationsCls	33k	3,239k
TweetSentimentCls	9k	483k
STS
STSBenchmark	12k
SICK-R	10k
STS12	14k
STS13	12k
STS14	12k
STS15	12k
STS16	14k
BIOSSES	7k
STS17	21k
STS22v2	1,866k
Clustering
RedditClustP2P	178k
TwentyNewsgroupsV2	8k
StackExchangeClustP2PV2	276k
StackExchangeClustV2	13k
ArxivClustP2P	226k
ArxivClustS2S	16k
BiorxivClustP2PV2	313k
MedrxivClustP2PV2	401k
MedrxivClustS2SV2	23k

Table 14: Reduced-thinking ablation (Gemini 3 Flash with reasoning_effort=low vs. default) on the MTEB(LLM) retrieval tasks. Think ↓ = reduction in thinking tokens vs. default. Reducing thinking by 54–94% improves all six retrieval scores in this ablation.
Task	Gemini 3 Flash	Gemini 3 Flash (low)	Δ	Think ↓	Best Emb.
Retrieval
AILAStatutes	5.7	12.0	+6.3	–	23.2
FQuADRetrieval	88.0	92.0	+4.0	54%	72.0
HC3FinanceRetrieval	60.0	66.0	+6.0	87%	67.0
LegalBenchConsumerContractsQA	79.0	83.0	+4.0	85%	70.0
PublicHealthQA	49.0	66.0	+17.0	94%	85.0
TwitterHjerneRetrieval	32.4	33.4	+1.1	–	31.5

Table 15: Few-shot classification ablation (Flash, 5 in-context examples vs. zero-shot). Bold = best score per task across all methods. Five-shot prompting matches zero-shot performance on the 2–3 class tasks and lowers Banking77 performance, where the prompt contains five examples for 77 labels.
Task	Classes	Zero-shot	5-shot	Δ	Best Emb.
IMDB	2	0.976	0.974	−0.002	0.976
ToxicConversations	2	0.900	0.838	−0.062	0.810
TweetSentiment	3	0.700	0.682	−0.018	0.710
Banking77	77	0.831	0.165	−0.666	0.960

Table 16: Retrieve-then-rerank matrix. Average nDCG@10 for first-stage retrievers crossed with cross-encoder and LLM listwise (†) rerankers over 7 BRIGHT and 5 BEIR tasks. Bold = best per row. An LLM reranker improves Qwen3-E-8B on BRIGHT (22.3→35.1); on BEIR, the embedding alone scores highest (63.1).
First stage	Pure	BGE-Gemma	Qwen3-RR-4B	Qwen3-RR-8B	Qwen3.6-27B†	Qwen3.6-35B†
BRIGHT (reasoning)
BM25	10.2	18.2	22.0	21.9	24.2	23.5
BGE-large	15.9	18.9	22.7	21.5	27.6	26.1
GTE-MC-v1	15.9	19.2	23.9	23.0	28.8	27.9
Qwen3-E-8B	22.3	20.6	26.4	24.6	35.1	33.6
BEIR (semantic)
BM25	39.8	50.5	53.2	52.5	52.1	51.1
BGE-large	52.7	55.3	58.6	57.6	57.6	56.2
GTE-MC-v1	53.7	55.0	58.5	58.5	57.5	56.9
Qwen3-E-8B	63.1	56.8	60.3	59.5	58.9	58.9

Table 17: Detailed LLM token usage and cost across all MTEB(LLM) tasks. Innc = non-cached input (cached billed at 10% of input rate); Think = reasoning tokens (billed at output rate); Out+Th combines standard output and thinking cost.
	Tokens (M)	Cost (USD)
Model	Innc	Cached	Out	Think	Innc	Cached	Out+Th	Total
Gemini 3.1 Pro	12.2	27.7	1.8	8.6	$24.39	$5.54	$124.21	$154.14
Gemini 3 Flash	12.4	27.5	1.8	14.3	$6.19	$1.38	$48.30	$55.87
Qwen3.6-27B	39.5	0.0	2.7	26.1	$11.46	$0.00	$91.97	$103.43
Qwen3.6-35B-A3B	39.5	0.0	2.5	26.0	$5.53	$0.00	$28.52	$34.06
Kimi-K2.6	40.1	0.0	2.6	21.7	$27.43	$0.00	$83.27	$110.70
DeepSeek-V4-Flash	16.8	22.6	2.1	4.5	$1.65	$0.22	$1.29	$3.16
MiniMax-M2.7	37.9	0.0	1.6	10.0	$10.58	$0.00	$13.93	$24.52
GLM-4.7	38.8	0.0	2.4	24.9	$15.54	$0.00	$47.77	$63.31
Gemini 3.1 Flash Lite	16.5	26.9	1.4	0.0	$4.13	$0.67	$2.05	$6.85
DeepSeek-R1	39.6	0.0	1.2	10.6	$27.73	$0.00	$29.65	$57.38

Table 18: Embedding throughput on a single NVIDIA H100 80GB (median tokens/s over the benchmark), per-MTok cost at $2.49/hr spot, and mean (macro) MTEB(LLM) score.
Model	Params	Tok/s	$/MTok	Score
mE5-S	118M	4,314,796	0.0002	63.1
mE5-B	278M	1,910,005	0.0004	64.4
mE5-L-Inst	560M	640,842	0.0011	69.7
BGE-M3	568M	640,425	0.0011	66.5
Arctic-L-v2	568M	640,405	0.0011	65.3
mE5-L	560M	640,126	0.0011	65.9
EmbGemma-300M	308M	374,988	0.0018	72.2
Jina-v5-Nano	212M	327,790	0.0021	73.9
F2LLM-0.6B	596M	189,653	0.0036	69.9
Qwen3-E-0.6B	596M	175,692	0.0039	72.5
GTE-Qwen2-1.5B	1.5B	133,567	0.0052	70.8
F2LLM-1.7B	1.7B	106,406	0.0065	71.7
Jina-v5-S	596M	91,658	0.0075	74.4
Qwen3-E-4B	4.0B	46,080	0.0150	75.9
F2LLM-4B	4.0B	40,157	0.0172	71.9
GTE-Qwen2-7B	7.1B	35,419	0.0195	73.1
Octen-8B	7.6B	29,313	0.0236	77.2
Qwen3-E-8B	7.6B	29,262	0.0236	77.0
SFR-2	7.1B	28,196	0.0245	73.9
Nemotron-8B	7.5B	27,808	0.0249	75.3
GritLM-7B	7.2B	27,730	0.0249	51.2
Linq-Mistral	7.1B	27,677	0.0250	72.8
E5-Mistral-7B	7.1B	27,522	0.0251	52.2
F2LLM-8B	7.6B	25,957	0.0266	72.2
KaLM-12B	11.8B	19,244	0.0359	74.8
F2LLM-14B	14.0B	14,665	0.0472	73.3

Table 19: Cost sensitivity analysis. LLM-to-embedding cost ratio under alternative hardware and pricing scenarios. Costs compare Octen-8B with Gemini 3.1 Pro at fixed API pricing; ratios range from 338–2,424×.
Hardware / Pricing Scenario	Emb. Cost	LLM Cost	Ratio
H100 spot $2.49/hr (our setup)	$0.108	$154.14	1,431×
H100 on-demand $3.99/hr	$0.173	$154.14	893×
A100 spot $1.49/hr (est. 1.5× slower)	$0.097	$154.14	1,594×
L4 spot $0.49/hr (est. 3× slower)	$0.064	$154.14	2,424×
Commercial API $0.10/MTok	$0.457	$154.14	338×

为什么重要

对于要搭建搜索、推荐或分类系统的人来说,这项研究给出了具体的成本与性能数据,帮助在便宜的专用嵌入模型和昂贵的通用大语言模型之间做出选择。它也提醒人们,只看准确率排行榜容易忽视背后数量级的成本差异。

本文术语

嵌入模型 · 把文本转换成数字向量,用于相似度比较、分类或聚类的模型
大语言模型(LLM) · 以生成文本为目标训练的模型,本文中仅通过提示词调用,没有专门的嵌入训练
推理token(thinking token) · 大语言模型在给出最终答案前内部生成的思考过程文本,这部分也计入推理成本
帕累托前沿 · 在成本和性能两个维度上都没有更优替代方案的一组最佳模型构成的边界
kNN(k近邻算法) · 通过比较新数据与已标注数据中最相近的几个样本来进行分类的简单方法

论文原文摘要（英文）

Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.

作者 · Adnan El Assadi

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Adnan El Assadi et al., arXiv:2608.12875, CC BY 4.0

The Embedder's Dilemma: LLMs Are Better, but at What Cost?

他们做了什么

为什么重要

本文术语

最新论文

METAL LAB 最新报道