Figure 1: Construction of the public PruhaNLP/1C-Ebench benchmark and the dense-retrieval evaluation pipeline implemented in PruhaNLP/1C-RB.
Table 1: Public resource stack.
Component
Content
Scale
PruhaNLP/1C-Ebench
Test pairs: question → code, subsets forum/fastcode
3,413 pairs
PruhaNLP/1C-RB
Evaluation harness: dense, BM25, metrics
CLI/Python
PruhaNLP/1C-Code-Train
Triplets (q,d+,d−) with PII scrubbing
784,057 triplets
PruhaNLP/USER2-1C-code
Domain-adapted reference bi-encoder
768d, MRL
Figure 2: Construction of PruhaNLP/1C-Code-Train: from filtered public 1C code to synthetic queries, hard-negative triplets, PII scrubbing, and fine-tuning of PruhaNLP/USER2-1C-code.
Table 2: PruhaNLP/1C-Ebench subsets.
Subset
Queries
Share
Source
Characterization
forum
2,883
84.5%
Forum discussions
Long questions, context, errors, inline code
fastcode
530
15.5%
Snippet catalog
Short queries, long ready-to-use snippets
Total
3,413
100%
–
Closed-set, single-gold qrels
Figure 3: nDCG@10 leaderboard on PruhaNLP/1C-Ebench (balanced-subset macro).
Table 3: Train–benchmark overlap audit.
Metric
forum
fastcode
all
Exact match: code
0.10%
0.00%
0.09%
Exact match: question
0.03%
8.68%
1.38%
Exact match: pair
0.00%
0.00%
0.00%
13-gram overlap: code
6.24%
16.23%
7.79%
13-gram overlap: question
0.03%
8.68%
1.38%
Input-only
0.03%
6.60%
1.05%
Label-only
6.24%
14.15%
7.47%
Input-and-label
0.00%
2.08%
0.32%
Any 13-gram dirty flag
6.28%
22.83%
8.85%
Figure 4: Matryoshka truncation: balanced-subset macro nDCG@10 by embedding dimension.
Q: Как обойти ограничения ФО на реквизит в динамическом списке, не трогая константу ФО? Gold: ОтключенныеПоля = Новый Массив; … FO×dynamic-list pattern; little lexical overlap with gold.
Win (fastcode) Ours 1; others >10
Q: Описание строки в таблице значений Gold: ст90 = новый ОписаниеТипов("строка",,Новый КвалификаторыСтроки(90)); Colloquial «описание строки» → type API; lexical baselines miss it.
Fail (fastcode) Ours >10; base 1
Q: Произношение текста голосом Gold: Voice = Новый COMObject("SAPI.SpVoice"); Voice.Speak("Привет!"); Rare COM API; general models recover Speak/SAPI.
Fail (fastcode) Ours 32; base 1
Q: Сжатие длинной Строки Gold: СжатиеДанных = Новый СжатиеДанных(9); …Base64. Short generic title; surface «сжатие»+«строка» wins.
Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain-specific terminology, for which open datasets and specialized models have been virtually non-existent. We present a comprehensive pipeline for 1C code retrieval: an open benchmark of 3,413 real-world, PII-scrubbed query-code pairs, a reproducible evaluation harness, and a specialized bi-encoder. To overcome scarce labeled data, we fine-tune on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories, using Matryoshka Representation Learning (MRL) and a privacy-aware tokenizer. Because the benchmark subsets differ in size, we report balanced-subset macro, query-weighted micro, and forum-only results. Our model reaches 0.5992 balanced macro nDCG@10, 0.5044 micro, and 0.4617 on forum, versus 0.4932 macro for the baseline architecture and 0.5404 for google/embeddinggemma-300m. Removing every benchmark example flagged by the conservative exact/13-gram overlap audit leaves 0.6011 balanced macro (0.5010 micro), indicating that detected train-benchmark overlap does not explain the headline result. MRL truncation to 256 dimensions preserves 99.9% of retrieval quality while reducing dense-index storage and exact similarity arithmetic by a factor of three.
