The Embedder's Dilemma: LLMs Are Better, but at What Cost?
arXiv:2608.128752026-08-12
거대 AI 언어모델이 검색 전용 임베딩 모델과 성능은 비슷해도, 비용은 최대 1,431배 더 든다
이 논문은 열 개의 대형 언어모델(LLM)과 스물여섯 개의 텍스트 임베딩 모델을 문서 분류, 문장 유사도 비교, 군집화, 문장쌍 판별, 검색 등 다섯 가지 작업 37개 테스트로 비교했다. 전체 평균 점수로는 최고 성능 LLM(Gemini 3.1 Pro, 77.6점)과 최고 임베딩 모델(77.2점)이 거의 차이가 없었지만, LLM은 복잡한 추론이 필요한 검색에서만 앞서고 분류에서는 임베딩 모델이 크게 앞섰다. 문제는 비용으로, 같은 성능을 내는 데 LLM이 임베딩 모델보다 최대 1,431배 비싸고(154달러 대 0.11달러), 같은 GPU에서 처리 속도도 최대 736배 느렸다.
무엇을 했나
텍스트를 하나의 숫자 벡터로 바꿔 비교하는 '임베딩 모델'과, 문장을 직접 생성하는 '대형 언어모델(LLM)'을 정확히 같은 조건에서 비교했다
LLM 열 개와 임베딩 모델 스물여섯 개(1억1800만~140억 개 매개변수)를 분류·문장유사도·군집화·문장쌍판별·검색 다섯 범주 37개 과제로 평가했다
API 사용료와 실제 GPU 처리 속도를 직접 측정해 성능뿐 아니라 비용까지 함께 비교하는 '비용 대비 성능 그래프'를 만들었다
전체 평균은 사실상 동점이지만 과제별로는 갈렸다. LLM은 추론이 필요한 검색에서 앞섰고(64.5 대 56.0), 임베딩 모델은 분류에서 크게 앞섰다(SFR-2 90.8 대 Pro 85.2)
LLM 비용의 28~81%는 '생각하는 과정(추론 토큰)'에서 나가는데, 이 추론량을 줄여도 대부분의 모델에서 검색 성능이 떨어지지 않거나 오히려 좋아졌다
Figure 1: Cost vs. performance across 36 models on MTEB(LLM). The frontier contains the leading embedding models and Gemini 3.1 Pro, which extends it by 0.4 points at 1,431× the cost of a comparable embedding.
Table 1: Model overview and per-category results. All ten LLMs and the ten highest-scoring embedding models. Scores are category means on a 0–100 scale; Overall is their mean. Cost is one MTEB(LLM) pass using API rates for LLMs and H100 throughput at $2.49/hr for embeddings (§3.3). Bold = best shown; full results are in Appendix B.2.
Model
Params
Cls (8)
Clust (9)
STS (10)
PairCls (4)
Retr (6)
Overall (37)
Cost
LLM
Gemini 3.1 Pro
–
85.2
66.6
88.5
83.2
64.5
77.6
$154
Gemini 3 Flash
–
84.1
65.7
87.6
86.3
52.3
75.2
$56
Qwen3.6-27B
–
84.8
53.6
84.9
83.6
62.4
73.9
$103
Qwen3.6-35B-A3B
–
83.8
55.3
84.0
82.9
60.4
73.3
$34
Kimi-K2.6
–
84.5
55.4
83.9
78.0
56.7
71.7
$111
DeepSeek-V4-Flash
–
81.9
43.7
82.2
81.6
53.3
68.5
$3
MiniMax-M2.7
–
80.9
48.6
81.0
82.8
48.0
68.2
$25
GLM-4.7
–
84.2
36.5
83.5
79.7
54.4
67.7
$63
Gemini 3.1 Flash Lite
–
82.9
21.7
85.1
83.7
49.0
64.5
$7
DeepSeek-R1
–
82.5
31.3
83.7
79.5
44.7
64.3
$57
Embedding
Octen-8B
7.6B
90.1
65.1
88.7
86.1
56.0
77.2
$0.11
Qwen3-E-8B
7.6B
90.1
65.9
88.5
86.5
54.2
77.0
$0.11
Qwen3-E-4B
4.0B
89.3
64.6
88.8
86.5
50.3
75.9
$0.07
Nemotron-8B
7.5B
84.8
64.1
86.2
86.7
54.7
75.3
$0.11
KaLM-12B
11.8B
88.8
63.4
84.6
87.1
49.9
74.8
$0.16
Jina-v5-S
596M
90.4
61.5
86.9
85.1
48.0
74.4
$0.03
SFR-2
7.1B
90.8
66.7
77.9
85.3
48.8
73.9
$0.14
Jina-v5-Nano
212M
89.6
60.4
87.0
85.0
47.4
73.9
$0.01
F2LLM-14B
14.0B
77.7
66.5
84.1
86.3
52.1
73.3
$0.22
GTE-Qwen2-7B
7.1B
86.7
65.2
81.6
86.0
45.8
73.1
$0.09
Figure 2: Cost vs. performance by task category. Each panel plots score (0–100) against cost per benchmark pass (log scale) for all 36 models; the line is the Pareto frontier over all models and the star marks the best LLM in that category. Pro extends the retrieval frontier; embedding models define the frontiers for classification, clustering, STS, and pair classification.
Table 2: Statistical significance. Paired bootstrap test (10,000 resamples, seed 42). Δ = Gemini 3.1 Pro − best embedding; significance at α = 0.05. Pair classification uses Pro for consistency, although Flash scores higher.
Comparison
Δ
95% CI
p
Sig.
Overall (Gemini 3.1 Pro vs. Octen-8B, MTEB(LLM))
All tasks
+0.3
[−2.4, +3.1]
0.85
No
Per category (Gemini 3.1 Pro vs. best embedding)
Retrieval
+8.5
[+0.2, +16.8]
<0.05
Yes
Clustering
−0.2
[−5.6, +5.1]
0.96
No
STS
−0.3
[−2.2, +1.8]
0.75
No
Pair Classification
−3.9
[−13.6, +5.9]
0.50
No
Classification
−5.6
[−9.2, −2.4]
<0.01
Yes
Figure 3: Retrieve-then-rerank. LLM listwise reranking improves every first stage on BRIGHT. On BEIR, a strong embedding first stage outperforms all reranked configurations.
Table 3: Complete model listing with mean (macro) scores across 37 MTEB(LLM) tasks, total benchmark costs, and references. LLM costs reflect actual API usage; embedding costs from H100 throughput benchmarking ($2.49/hr).
Model
Params
Score
Cost
Reference
LLM Models
Gemini 3.1 Pro
–
77.6
$154.14
–
Gemini 3 Flash
–
75.2
$55.87
–
Qwen3.6-27B
–
73.9
$103.43
–
Qwen3.6-35B-A3B
–
73.3
$34.06
–
Kimi-K2.6
–
71.7
$110.70
–
DeepSeek-V4-Flash
–
68.5
$3.16
–
MiniMax-M2.7
–
68.2
$24.52
–
GLM-4.7
–
67.7
$63.31
–
Gemini 3.1 Flash Lite
–
64.5
$6.85
–
DeepSeek-R1
–
64.3
$57.38
–
Embedding Models (ranked by score)
Octen-8B
7.6B
77.2
$0.108
–
Qwen3-E-8B
7.6B
77.0
$0.108
78
Qwen3-E-4B
4.0B
75.9
$0.069
78
Nemotron-8B
7.5B
75.3
$0.115
37
KaLM-12B
11.8B
74.8
$0.158
80
Jina-v5-S
596M
74.4
$0.034
3
SFR-2
7.1B
73.9
$0.136
47
Jina-v5-Nano
212M
73.9
$0.010
3
F2LLM-14B
14.0B
73.3
$0.215
79
GTE-Qwen2-7B
7.1B
73.1
$0.089
42
Linq-Mistral
7.1B
72.8
$0.138
15
Qwen3-E-0.6B
596M
72.5
$0.018
78
EmbGemma-300M
308M
72.2
$0.008
39
F2LLM-8B
7.6B
72.2
$0.122
79
F2LLM-4B
4.0B
71.9
$0.079
79
F2LLM-1.7B
1.7B
71.7
$0.030
79
GTE-Qwen2-1.5B
1.5B
70.8
$0.024
42
F2LLM-0.6B
596M
69.9
$0.017
79
Figure 4: The thinking-token tax: what reasoning costs, and what it buys. (a) API cost per benchmark pass by token type for all ten LLMs. (b) Mean retrieval score with default vs. disabled reasoning for six models from five families; labels show the reduction in generated tokens. Four models preserve or improve retrieval with 54–96% fewer generated tokens; the two Qwen models lose ground.
Table 4: Category-level performance (best LLM vs. best embedding per category). Δ = best LLM − best embedding; bold = winner; task counts in parentheses.
Category
Gemini 3.1 Pro
Gemini 3 Flash
Best Emb.
Best Model
Δ
Retrieval (6)
64.5
52.3
56.0
Octen-8B
+8.5
Clustering (9)
66.6
65.7
66.7
SFR-2
−0.2
STS (10)
88.5
87.6
88.8
Qwen3-E-4B
−0.3
PairCls (4)
83.2
86.3
87.1
KaLM-12B
−3.9
Classification (8)
85.2
84.1
90.8
SFR-2
−5.6
Figure 5: How many documents each architecture reads jointly with the query. Top: the pipeline. Bottom: its attention mask, drawn over the same N documents at the same scale in every panel so the four are directly comparable. Each row is a token and each column a token it may read. The red region is what one forward pass reads jointly with the query; it cannot be computed before the query arrives, so cost grows with it.
Table 5: Complete per-category results for all 36 complete models, ranked by overall (macro) score. Bold = best in column.
Rank
Model
Type
Cls (8)
Clust (9)
STS (10)
PairCls (4)
Retr (6)
Overall
1
Gemini 3.1 Pro
LLM
85.2
66.6
88.5
83.2
64.5
77.6
2
Octen-8B
Emb
90.1
65.1
88.7
86.1
56.0
77.2
3
Qwen3-E-8B
Emb
90.1
65.9
88.5
86.5
54.2
77.0
4
Qwen3-E-4B
Emb
89.3
64.6
88.8
86.5
50.3
75.9
5
Nemotron-8B
Emb
84.8
64.1
86.2
86.7
54.7
75.3
6
Gemini 3 Flash
LLM
84.1
65.7
87.6
86.3
52.3
75.2
7
KaLM-12B
Emb
88.8
63.4
84.6
87.1
49.9
74.8
8
Jina-v5-S
Emb
90.4
61.5
86.9
85.1
48.0
74.4
9
SFR-2
Emb
90.8
66.7
77.9
85.3
48.8
73.9
10
Jina-v5-Nano
Emb
89.6
60.4
87.0
85.0
47.4
73.9
11
Qwen3.6-27B
LLM
84.8
53.6
84.9
83.6
62.4
73.9
12
F2LLM-14B
Emb
77.7
66.5
84.1
86.3
52.1
73.3
13
Qwen3.6-35B-A3B
LLM
83.8
55.3
84.0
82.9
60.4
73.3
14
GTE-Qwen2-7B
Emb
86.7
65.2
81.6
86.0
45.8
73.1
15
Linq-Mistral
Emb
82.5
61.3
83.4
86.1
51.0
72.8
16
Qwen3-E-0.6B
Emb
85.4
60.9
85.7
86.1
44.1
72.5
17
EmbGemma-300M
Emb
86.5
59.3
82.1
86.0
47.3
72.2
18
F2LLM-8B
Emb
75.2
65.8
83.9
86.1
50.0
72.2
19
F2LLM-4B
Emb
75.3
64.7
83.6
85.9
49.9
71.9
20
Kimi-K2.6
LLM
84.5
55.4
83.9
78.0
56.7
71.7
21
F2LLM-1.7B
Emb
74.7
65.1
83.8
86.1
48.5
71.7
22
GTE-Qwen2-1.5B
Emb
83.3
60.0
80.3
86.6
43.9
70.8
23
F2LLM-0.6B
Emb
72.5
62.9
83.1
85.8
45.4
69.9
24
mE5-L-Inst
Emb
74.4
59.1
83.9
86.1
44.8
69.7
25
DeepSeek-V4-Flash
LLM
81.9
43.7
82.2
81.6
53.3
68.5
26
MiniMax-M2.7
LLM
80.9
48.6
81.0
82.8
48.0
68.2
27
GLM-4.7
LLM
84.2
36.5
83.5
79.7
54.4
67.7
28
BGE-M3
Emb
76.2
45.9
80.4
85.9
44.3
66.5
29
mE5-L
Emb
73.3
48.3
80.3
83.9
43.6
65.9
30
Arctic-L-v2
Emb
71.4
49.7
77.2
83.9
44.6
65.3
Figure 6: Per-task performance. Horizontal bars show the full embedding score range (min to max across 26 models) for each MTEB(LLM) task, grouped by category. Circle = best embedding; diamond = best LLM on that task, taken over all ten (six different LLMs hold it across the suite, Gemini 3.1 Pro on 21 of 37 tasks). Where the diamond falls inside the bar, some embedding model already matches the best LLM: this holds on 7 of 8 classification tasks and 7 of 10 STS tasks, but on only 1 of 6 retrieval tasks.
Table 6: Per-task scores for representative models across all 37 MTEB(LLM) tasks. Bold = best in row (incl. best embedding). Metric per category in italics.
Task
Cat.
Gemini 3.1 Pro
Gemini 3 Flash
Qwen3.6-27B
Octen-8B
Qwen3-E-8B
Best Emb.
Classification (Accuracy)
AmazonCounterfactualClassification
Cls
84.8
81.2
90.1
93.0
92.9
93.0
Banking77Classification
Cls
85.0
83.1
80.5
87.3
87.2
91.6
ImdbClassification
Cls
98.0
97.6
97.4
97.7
97.8
97.8
MTOPDomainClassification
Cls
95.5
96.4
96.9
98.2
98.1
99.2
MassiveIntentClassification
Cls
84.9
85.4
84.6
85.7
85.8
88.9
MassiveScenarioClassification
Cls
79.4
75.9
76.4
89.3
89.4
93.0
ToxicConversationsClassification
Cls
89.6
90.0
89.2
91.9
92.0
94.4
TweetSentimentExtractionClassification
Cls
64.2
63.0
63.2
77.9
77.8
78.9
Clustering (V-measure)
ArxivClusteringP2P
Clust
60.1
60.3
54.5
61.9
62.5
64.1
ArxivClusteringS2S
Clust
62.5
60.1
41.6
60.5
61.1
61.5
BiorxivClusteringP2PV2
Clust
62.5
60.8
49.9
61.6
65.3
77.3
MedrxivClusteringP2PV2
Clust
52.5
50.6
48.0
55.9
55.9
60.7
MedrxivClusteringS2SV2
Clust
54.0
50.3
42.7
53.3
55.2
57.5
RedditClusteringP2P
Clust
93.9
90.5
60.1
78.7
79.8
80.8
StackExchangeClusteringP2PV2
Clust
42.9
49.0
46.1
59.7
59.5
60.7
StackExchangeClusteringV2
Clust
91.8
90.2
73.6
85.8
86.3
86.3
TwentyNewsgroupsClusteringV2
Clust
78.7
79.9
66.3
68.5
67.6
73.8
STS (Spearman ρ)
BIOSSES
STS
89.6
88.7
85.0
84.4
82.0
88.8
SICKR
STS
86.1
86.8
76.4
87.9
88.4
91.1
STS12
STS
82.6
79.5
77.1
87.3
87.3
87.4
STS13
STS
91.9
91.5
90.6
93.9
94.0
94.6
STS14
STS
90.5
89.3
88.0
90.9
91.0
91.4
STS15
STS
94.4
93.9
91.8
94.3
94.2
94.5
STS16
STS
89.0
88.5
87.1
92.5
92.5
92.9
STS17
STS
94.2
94.1
91.8
93.6
93.7
93.7
STS22v2
STS
73.6
71.7
72.0
69.5
68.9
69.5
STSBenchmark
STS
92.6
91.7
89.6
93.2
93.4
94.9
Figure 7: Same-hardware inference throughput. Two open-weight LLMs and seven representative embedding models (118M–14B) served on one H100 (tokens/second, log scale); the full 26-model embedding throughput is in Table 18. Even the slowest embedding runs ∼2.5× faster than the fastest LLM; the fastest runs ∼736× faster.
Figure 8: Capability profiles across five task categories. Gemini 3.1 Pro (red), Qwen3.6-27B (amber), and Octen-8B (blue). The LLMs lead on retrieval, while Octen-8B leads on classification and remains competitive across categories.
Table 8: Per-task scores: STS (Spearman ρ).
Model
BIOSSES
SICK-R
STS12
STS13
STS14
STS15
STS16
STS17
STS22v2
STSBench
Mean
LLMs
Gemini 3.1 Pro
89.6
86.1
82.6
91.9
90.5
94.4
89.0
94.2
73.6
92.6
88.5
Gemini 3 Flash
88.7
86.8
79.5
91.5
89.3
93.9
88.5
94.1
71.7
91.7
87.6
Qwen3.6-27B
85.0
76.4
77.1
90.6
88.0
91.8
87.1
91.8
72.0
89.6
84.9
Qwen3.6-35B-A3B
84.0
75.5
75.1
89.6
85.1
91.4
85.1
92.7
73.4
87.8
84.0
Kimi-K2.6
86.4
82.1
72.2
89.3
85.1
91.1
83.9
90.1
70.9
87.8
83.9
DeepSeek-V4-Flash
78.5
75.3
75.6
87.3
84.5
89.2
82.4
91.5
69.2
87.9
82.2
MiniMax-M2.7
87.7
79.7
65.4
86.5
81.9
88.4
81.1
87.1
66.7
85.7
81.0
GLM-4.7
82.3
76.0
73.1
88.5
85.7
91.9
84.9
90.4
73.2
88.5
83.5
Gemini 3.1 FLite
89.8
78.0
76.0
90.5
86.3
91.4
86.5
91.4
71.2
90.3
85.1
DeepSeek-R1
85.0
79.7
71.9
88.7
85.6
89.9
84.2
91.5
71.4
89.2
83.7
Embedding models
Octen-8B
84.4
87.9
87.3
93.9
90.9
94.3
92.5
93.6
69.5
93.2
88.7
Qwen3-E-8B
82.0
88.4
87.3
94.0
91.0
94.2
92.5
93.7
68.9
93.4
88.5
Qwen3-E-4B
84.8
87.7
87.4
94.6
91.4
94.5
92.9
93.2
67.7
93.7
88.8
Nemotron-8B
86.3
85.4
83.4
91.5
87.9
92.1
88.7
91.4
64.3
90.8
86.2
KaLM-12B
87.0
81.9
81.4
89.2
85.9
90.6
86.7
89.0
65.1
89.1
84.6
Jina-v5-S
85.3
90.4
86.6
89.2
88.9
92.6
87.4
89.1
65.0
94.9
86.9
SFR-2
87.5
76.7
74.6
81.4
80.6
87.5
84.6
85.0
38.2
82.6
77.9
Jina-v5-Nano
87.4
91.1
86.2
89.9
89.2
93.1
85.4
88.4
64.9
94.2
87.0
F2LLM-14B
88.2
82.8
82.4
86.9
84.8
91.3
85.6
88.9
63.4
87.0
84.1
GTE-Qwen2-7B
83.0
79.7
78.1
88.6
83.4
89.5
85.5
87.1
54.7
86.2
81.6
Linq-Mistral
86.1
84.4
77.6
87.8
84.1
91.2
87.4
89.9
56.1
89.0
83.4
Qwen3-E-0.6B
84.6
84.6
83.6
92.1
87.0
91.7
89.7
87.9
64.6
91.3
85.7
EmbGemma-300M
83.2
81.9
79.1
85.5
84.5
89.2
84.7
86.6
57.3
88.8
82.1
F2LLM-8B
88.8
82.4
81.3
86.6
83.8
91.3
85.4
89.3
63.2
86.5
83.9
F2LLM-4B
87.5
81.9
82.3
86.1
83.8
91.5
85.2
87.9
62.8
87.1
83.6
F2LLM-1.7B
88.5
81.3
82.6
86.7
84.5
91.3
85.2
87.8
62.9
87.6
83.8
GTE-Qwen2-1.5B
81.4
80.7
72.9
84.4
81.7
88.7
84.6
86.1
56.6
85.6
80.3
F2LLM-0.6B
87.8
80.0
82.7
86.7
84.1
90.6
84.9
84.0
62.3
87.6
83.1
Figure 9: Cross-model task-behaviour correlation. Pearson correlations based on per-task MTEB(LLM) scores. LLMs and embedding models form visually distinct clusters: LLMs are strongly intercorrelated while embedding models cluster by architecture family. Lower inter-paradigm correlations are consistent with their different category profiles.
Table 9: Per-task scores: Clustering (V-measure).
Model
ArxivP2P
ArxivS2S
BioP2P
MedP2P
MedS2S
Reddit
SE-P2P
SE-Cl
20News
Mean
LLMs
Gemini 3.1 Pro
60.1
62.5
62.5
52.5
54.0
93.9
42.9
91.8
78.7
66.6
Gemini 3 Flash
60.3
60.1
60.8
50.6
50.3
90.5
49.0
90.2
79.9
65.7
Qwen3.6-27B
54.5
41.6
49.9
48.0
42.7
60.1
46.1
73.6
66.3
53.6
Qwen3.6-35B-A3B
48.5
60.4
53.2
49.4
46.2
69.2
42.2
72.7
55.7
55.3
Kimi-K2.6
52.6
44.2
51.9
50.6
39.1
57.4
52.1
81.8
68.9
55.4
DeepSeek-V4-Flash
37.4
45.5
29.3
27.0
40.9
55.2
40.3
53.5
64.4
43.7
MiniMax-M2.7
36.4
40.6
38.3
39.8
42.4
67.4
39.4
75.1
57.9
48.6
GLM-4.7
23.6
28.6
33.2
37.5
37.8
40.6
37.9
49.5
39.8
36.5
Gemini 3.1 FLite
14.7
18.9
17.4
19.7
21.6
28.5
21.5
25.8
27.2
21.7
DeepSeek-R1
22.0
25.3
32.4
31.7
33.4
26.6
33.1
42.1
34.9
31.3
Embedding models
Octen-8B
61.9
60.5
61.6
55.9
53.3
78.7
59.7
85.8
68.5
65.1
Qwen3-E-8B
62.5
61.1
65.3
55.9
55.2
79.8
59.5
86.3
67.6
65.9
Qwen3-E-4B
62.6
60.3
64.0
55.5
54.1
77.1
58.8
83.9
65.3
64.6
Nemotron-8B
59.2
59.1
65.2
52.9
52.6
79.7
58.7
84.6
65.2
64.1
KaLM-12B
61.6
57.4
64.1
54.7
53.2
76.1
58.5
83.3
62.0
63.4
Jina-v5-S
60.0
57.5
63.6
53.7
52.6
70.5
58.9
76.8
59.9
61.5
SFR-2
61.8
59.6
65.6
58.9
57.5
80.8
59.3
83.2
73.8
66.7
Jina-v5-Nano
59.1
55.4
61.6
54.6
52.1
67.4
57.3
76.8
59.7
60.4
F2LLM-14B
63.1
60.6
75.6
57.7
56.7
77.9
57.2
81.1
68.8
66.5
GTE-Qwen2-7B
64.1
61.5
66.7
57.0
54.1
80.5
60.7
84.5
58.1
65.2
Linq-Mistral
58.0
55.7
61.3
51.5
52.8
77.0
56.2
78.4
61.1
61.3
Qwen3-E-0.6B
60.2
57.6
61.8
52.9
51.1
69.9
56.0
77.3
61.7
60.9
EmbGemma-300M
57.9
53.9
62.8
54.4
51.0
75.9
51.9
73.7
51.8
59.3
F2LLM-8B
61.7
60.9
75.5
59.2
54.4
75.3
58.2
81.2
65.8
65.8
F2LLM-4B
61.3
59.2
74.2
59.8
54.7
74.4
55.2
79.9
63.9
64.7
F2LLM-1.7B
62.6
59.6
77.3
60.7
56.1
74.4
55.2
75.5
64.2
65.1
GTE-Qwen2-1.5B
58.5
57.5
62.1
53.2
52.3
71.1
52.3
73.6
59.1
60.0
F2LLM-0.6B
61.7
58.6
71.1
58.7
54.5
69.7
56.0
76.0
59.9
62.9
Figure 10: Overall rankings. Pro (77.6) narrowly leads Octen-8B (77.2); the difference lies within statistical noise (p = 0.85). Flash-Lite (64.5) ranks near the bottom.
Table 10: Per-task scores: Pair Classification (AP / Accuracy).
Model
LegalPC
RTE3
SprintDup
TwtURL
Mean
LLMs
Gemini 3.1 Pro
72.2
95.4
82.2
82.8
83.2
Gemini 3 Flash
77.4
96.7
83.4
87.6
86.3
Qwen3.6-27B
71.6
92.9
84.6
85.2
83.6
Qwen3.6-35B-A3B
69.4
93.6
85.2
83.4
82.9
Kimi-K2.6
69.2
88.8
78.8
75.4
78.0
DeepSeek-V4-Flash
67.2
90.4
86.6
82.0
81.6
MiniMax-M2.7
71.8
91.5
82.8
85.0
82.8
GLM-4.7
69.6
90.0
82.2
76.8
79.7
Gemini 3.1 FLite
72.4
95.0
86.6
80.8
83.7
DeepSeek-R1
68.8
90.0
79.6
79.6
79.5
Embedding models
Octen-8B
70.0
85.4
99.4
89.4
86.1
Qwen3-E-8B
70.4
85.7
99.6
90.2
86.5
Qwen3-E-4B
72.0
84.8
99.8
89.4
86.5
Nemotron-8B
72.6
84.8
99.4
89.8
86.7
KaLM-12B
73.6
85.2
99.8
89.6
87.1
Jina-v5-S
68.0
85.2
99.6
87.4
85.1
SFR-2
67.4
84.8
99.8
89.0
85.3
Jina-v5-Nano
68.8
85.0
99.4
86.6
85.0
F2LLM-14B
72.0
84.8
99.6
88.6
86.3
GTE-Qwen2-7B
71.2
85.2
99.6
88.0
86.0
Linq-Mistral
71.0
84.8
99.8
88.6
86.1
Qwen3-E-0.6B
70.4
84.8
99.8
89.4
86.1
EmbGemma-300M
71.0
84.8
99.6
88.6
86.0
F2LLM-8B
72.0
84.8
99.6
88.0
86.1
F2LLM-4B
70.6
84.8
99.8
88.4
85.9
F2LLM-1.7B
71.0
84.8
99.6
89.0
86.1
GTE-Qwen2-1.5B
73.0
84.8
99.4
89.2
86.6
F2LLM-0.6B
70.8
85.0
99.4
88.0
85.8
Figure 11: Classification rankings. SFR-2 ranks first (90.8); Pro scores 85.2 and ranks below ten embedding models.
Table 12: MTEB(LLM) task suite (37 tasks). N = held-out test samples (summed over languages for multilingual tasks); Q = queries, C = corpus documents. Multilingual tasks are evaluated per language and averaged. Held-out subsets (seed 42) derived from MTEB and MMTEB tasks (49; 23), hosted at mteb/llm-eval-*. Token counts: Table 13.
Task
Lang.
N
Cls.
Metric
Source
Classification (8 tasks)
ImdbCls
en
500
2
Acc.
45
Banking77Cls
en
3k
77
Acc.
11
AmazonCounterfactualCls
en, de, ja
809
2
Acc.
54
MTOPDomainCls
en, de, fr
2k
11
Acc.
41
MassiveIntentCls
en, de, fr, ja
4k
60
Acc.
24
MassiveScenarioCls
en, de, fr, ja
3k
18
Acc.
24
ToxicConversationsCls
en
500
2
Acc.
8
TweetSentimentCls
en
500
3
Acc.
49
Semantic Textual Similarity (10 tasks)
STSBenchmark
en
500
–
Spearman
12
SICK-R
en
500
–
Spearman
46
STS12
en
500
–
Spearman
2
STS13
en
500
–
Spearman
2
STS14
en
500
–
Spearman
2
STS15
en
500
–
Spearman
1
STS16
en
500
–
Spearman
1
BIOSSES
en
100
–
Spearman
60
STS17
en, de, es, fr
1k
–
Spearman
12
STS22v2
en, de, es, fr, ru, zh
2k
–
Spearman
14
Clustering (9 tasks)
RedditClustP2P
en
1k
–
V-meas.
49
TwentyNewsgroupsV2
en
1k
–
V-meas.
49
StackExchangeClustP2PV2
en
1k
–
V-meas.
49
StackExchangeClustV2
en
1k
–
V-meas.
49
ArxivClustP2P
en
1k
–
V-meas.
49
ArxivClustS2S
en
1k
–
V-meas.
49
BiorxivClustP2PV2
en
1k
–
V-meas.
49
MedrxivClustP2PV2
en
1k
–
V-meas.
49
MedrxivClustS2SV2
en
1k
–
V-meas.
49
Table 13: Token budget per task (GPT-4o tokenizer; raw text). Actual counts vary by ±15–40% across model vocabularies. *Train split is the kNN reference corpus processed only by embedding models; LLMs process only the test split. Corpus-in-context formatting adds ∼20 tokens per document for LLM retrieval.
Classification
Task
Test
Train (kNN)*
ImdbCls
132k
7,281k
Banking77Cls
38k
134k
AmazonCounterfactualCls
22k
258k
MTOPDomainCls
23k
393k
MassiveIntentCls
37k
433k
MassiveScenarioCls
30k
433k
ToxicConversationsCls
33k
3,239k
TweetSentimentCls
9k
483k
STS
STSBenchmark
12k
SICK-R
10k
STS12
14k
STS13
12k
STS14
12k
STS15
12k
STS16
14k
BIOSSES
7k
STS17
21k
STS22v2
1,866k
Clustering
RedditClustP2P
178k
TwentyNewsgroupsV2
8k
StackExchangeClustP2PV2
276k
StackExchangeClustV2
13k
ArxivClustP2P
226k
ArxivClustS2S
16k
BiorxivClustP2PV2
313k
MedrxivClustP2PV2
401k
MedrxivClustS2SV2
23k
Table 14: Reduced-thinking ablation (Gemini 3 Flash with reasoning_effort=low vs. default) on the MTEB(LLM) retrieval tasks. Think ↓ = reduction in thinking tokens vs. default. Reducing thinking by 54–94% improves all six retrieval scores in this ablation.
Task
Gemini 3 Flash
Gemini 3 Flash (low)
Δ
Think ↓
Best Emb.
Retrieval
AILAStatutes
5.7
12.0
+6.3
–
23.2
FQuADRetrieval
88.0
92.0
+4.0
54%
72.0
HC3FinanceRetrieval
60.0
66.0
+6.0
87%
67.0
LegalBenchConsumerContractsQA
79.0
83.0
+4.0
85%
70.0
PublicHealthQA
49.0
66.0
+17.0
94%
85.0
TwitterHjerneRetrieval
32.4
33.4
+1.1
–
31.5
Table 15: Few-shot classification ablation (Flash, 5 in-context examples vs. zero-shot). Bold = best score per task across all methods. Five-shot prompting matches zero-shot performance on the 2–3 class tasks and lowers Banking77 performance, where the prompt contains five examples for 77 labels.
Task
Classes
Zero-shot
5-shot
Δ
Best Emb.
IMDB
2
0.976
0.974
−0.002
0.976
ToxicConversations
2
0.900
0.838
−0.062
0.810
TweetSentiment
3
0.700
0.682
−0.018
0.710
Banking77
77
0.831
0.165
−0.666
0.960
Table 16: Retrieve-then-rerank matrix. Average nDCG@10 for first-stage retrievers crossed with cross-encoder and LLM listwise (†) rerankers over 7 BRIGHT and 5 BEIR tasks. Bold = best per row. An LLM reranker improves Qwen3-E-8B on BRIGHT (22.3→35.1); on BEIR, the embedding alone scores highest (63.1).
First stage
Pure
BGE-Gemma
Qwen3-RR-4B
Qwen3-RR-8B
Qwen3.6-27B†
Qwen3.6-35B†
BRIGHT (reasoning)
BM25
10.2
18.2
22.0
21.9
24.2
23.5
BGE-large
15.9
18.9
22.7
21.5
27.6
26.1
GTE-MC-v1
15.9
19.2
23.9
23.0
28.8
27.9
Qwen3-E-8B
22.3
20.6
26.4
24.6
35.1
33.6
BEIR (semantic)
BM25
39.8
50.5
53.2
52.5
52.1
51.1
BGE-large
52.7
55.3
58.6
57.6
57.6
56.2
GTE-MC-v1
53.7
55.0
58.5
58.5
57.5
56.9
Qwen3-E-8B
63.1
56.8
60.3
59.5
58.9
58.9
Table 17: Detailed LLM token usage and cost across all MTEB(LLM) tasks. Innc = non-cached input (cached billed at 10% of input rate); Think = reasoning tokens (billed at output rate); Out+Th combines standard output and thinking cost.
Tokens (M)
Cost (USD)
Model
Innc
Cached
Out
Think
Innc
Cached
Out+Th
Total
Gemini 3.1 Pro
12.2
27.7
1.8
8.6
$24.39
$5.54
$124.21
$154.14
Gemini 3 Flash
12.4
27.5
1.8
14.3
$6.19
$1.38
$48.30
$55.87
Qwen3.6-27B
39.5
0.0
2.7
26.1
$11.46
$0.00
$91.97
$103.43
Qwen3.6-35B-A3B
39.5
0.0
2.5
26.0
$5.53
$0.00
$28.52
$34.06
Kimi-K2.6
40.1
0.0
2.6
21.7
$27.43
$0.00
$83.27
$110.70
DeepSeek-V4-Flash
16.8
22.6
2.1
4.5
$1.65
$0.22
$1.29
$3.16
MiniMax-M2.7
37.9
0.0
1.6
10.0
$10.58
$0.00
$13.93
$24.52
GLM-4.7
38.8
0.0
2.4
24.9
$15.54
$0.00
$47.77
$63.31
Gemini 3.1 Flash Lite
16.5
26.9
1.4
0.0
$4.13
$0.67
$2.05
$6.85
DeepSeek-R1
39.6
0.0
1.2
10.6
$27.73
$0.00
$29.65
$57.38
Table 18: Embedding throughput on a single NVIDIA H100 80GB (median tokens/s over the benchmark), per-MTok cost at $2.49/hr spot, and mean (macro) MTEB(LLM) score.
Model
Params
Tok/s
$/MTok
Score
mE5-S
118M
4,314,796
0.0002
63.1
mE5-B
278M
1,910,005
0.0004
64.4
mE5-L-Inst
560M
640,842
0.0011
69.7
BGE-M3
568M
640,425
0.0011
66.5
Arctic-L-v2
568M
640,405
0.0011
65.3
mE5-L
560M
640,126
0.0011
65.9
EmbGemma-300M
308M
374,988
0.0018
72.2
Jina-v5-Nano
212M
327,790
0.0021
73.9
F2LLM-0.6B
596M
189,653
0.0036
69.9
Qwen3-E-0.6B
596M
175,692
0.0039
72.5
GTE-Qwen2-1.5B
1.5B
133,567
0.0052
70.8
F2LLM-1.7B
1.7B
106,406
0.0065
71.7
Jina-v5-S
596M
91,658
0.0075
74.4
Qwen3-E-4B
4.0B
46,080
0.0150
75.9
F2LLM-4B
4.0B
40,157
0.0172
71.9
GTE-Qwen2-7B
7.1B
35,419
0.0195
73.1
Octen-8B
7.6B
29,313
0.0236
77.2
Qwen3-E-8B
7.6B
29,262
0.0236
77.0
SFR-2
7.1B
28,196
0.0245
73.9
Nemotron-8B
7.5B
27,808
0.0249
75.3
GritLM-7B
7.2B
27,730
0.0249
51.2
Linq-Mistral
7.1B
27,677
0.0250
72.8
E5-Mistral-7B
7.1B
27,522
0.0251
52.2
F2LLM-8B
7.6B
25,957
0.0266
72.2
KaLM-12B
11.8B
19,244
0.0359
74.8
F2LLM-14B
14.0B
14,665
0.0472
73.3
Table 19: Cost sensitivity analysis. LLM-to-embedding cost ratio under alternative hardware and pricing scenarios. Costs compare Octen-8B with Gemini 3.1 Pro at fixed API pricing; ratios range from 338–2,424×.
Hardware / Pricing Scenario
Emb. Cost
LLM Cost
Ratio
H100 spot $2.49/hr (our setup)
$0.108
$154.14
1,431×
H100 on-demand $3.99/hr
$0.173
$154.14
893×
A100 spot $1.49/hr (est. 1.5× slower)
$0.097
$154.14
1,594×
L4 spot $0.49/hr (est. 3× slower)
$0.064
$154.14
2,424×
Commercial API $0.10/MTok
$0.457
$154.14
338×
왜 중요한가
실무에서 검색·추천·챗봇 시스템을 만들 때 값비싼 LLM 대신 저렴한 임베딩 모델을 써도 되는 상황과, 반대로 LLM이 꼭 필요한 상황(복잡한 추론이 필요한 검색)을 명확히 구분해준다. 성능표만 보고 모델을 고르면 눈에 보이지 않는 비용·속도 격차를 놓치기 쉽다는 점을 데이터로 보여준다.
이 논문의 용어
임베딩 모델 · 텍스트를 숫자 벡터로 변환해 유사도 비교나 분류에 쓰는 모델
LLM(대형 언어모델) · 문장을 직접 생성하도록 훈련된 모델. 이 논문에서는 별도 임베딩 훈련 없이 프롬프트로만 사용
추론 토큰(thinking token) · LLM이 답을 내기 전 내부적으로 '생각'하며 만들어내는 텍스트 조각. 비용에 포함됨
파레토 프론티어 · 비용과 성능을 동시에 고려했을 때 더 나은 대안이 없는 최적의 모델들의 경계선
kNN(k-최근접 이웃) · 새로운 데이터를 가장 가까운 기존 데이터들과 비교해 분류하는 간단한 방법
논문 원문 초록 (영문)
Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.
