Natural Language Code Retrieval for 1C:Enterprise: An Open Benchmark and Efficient Bi-Encoder
arXiv:2608.199572026-08-21
A first-of-its-kind search benchmark and AI model let you find 1C business-software code using Russian-language questions
1C:Enterprise is widely used enterprise software in Russian-speaking countries, but until now there was almost no open data or dedicated model for searching its code with natural-language questions. Researchers built a benchmark of 3,413 real question-code pairs plus over 780,000 synthetic training examples, then trained a specialized search model on top of them. The resulting model clearly outperforms general-purpose multilingual embedding models on this task.
What they did
The team collected 3,413 real Russian questions paired with 1C/BSL code from forums and the FastCode site, scrubbed personal information, and released it as the open 1C-Ebench benchmark
Using 784,057 filtered public GitHub code files, they had Google's gemma-4-26B-A4B-it language model auto-generate Russian queries to build 784,057 synthetic training triplets, then fine-tuned the deepvk/USER2-base model on this data
With a technique called Matryoshka Representation Learning, the model's embedding vectors (numeric representations of text) can be truncated to a shorter prefix with almost no quality loss, cutting storage and computation for similarity search by a factor of three
The team audited overlap between training data and the benchmark, and found that removing every flagged overlapping example did not hurt performance, showing the strong results aren't just due to data leakage
The final model reached a balanced macro nDCG@10 score of 0.5992, beating the unmodified base architecture (0.4932) and Google's embeddinggemma-300m (0.5404)
Figure 1: Construction of the public PruhaNLP/1C-Ebench benchmark and the dense-retrieval evaluation pipeline implemented in PruhaNLP/1C-RB.
Table 1: Public resource stack.
Component
Content
Scale
PruhaNLP/1C-Ebench
Test pairs: question → code, subsets forum/fastcode
3,413 pairs
PruhaNLP/1C-RB
Evaluation harness: dense, BM25, metrics
CLI/Python
PruhaNLP/1C-Code-Train
Triplets (q,d+,d−) with PII scrubbing
784,057 triplets
PruhaNLP/USER2-1C-code
Domain-adapted reference bi-encoder
768d, MRL
Figure 2: Construction of PruhaNLP/1C-Code-Train: from filtered public 1C code to synthetic queries, hard-negative triplets, PII scrubbing, and fine-tuning of PruhaNLP/USER2-1C-code.
Table 2: PruhaNLP/1C-Ebench subsets.
Subset
Queries
Share
Source
Characterization
forum
2,883
84.5%
Forum discussions
Long questions, context, errors, inline code
fastcode
530
15.5%
Snippet catalog
Short queries, long ready-to-use snippets
Total
3,413
100%
–
Closed-set, single-gold qrels
Figure 3: nDCG@10 leaderboard on PruhaNLP/1C-Ebench (balanced-subset macro).
Table 3: Train–benchmark overlap audit.
Metric
forum
fastcode
all
Exact match: code
0.10%
0.00%
0.09%
Exact match: question
0.03%
8.68%
1.38%
Exact match: pair
0.00%
0.00%
0.00%
13-gram overlap: code
6.24%
16.23%
7.79%
13-gram overlap: question
0.03%
8.68%
1.38%
Input-only
0.03%
6.60%
1.05%
Label-only
6.24%
14.15%
7.47%
Input-and-label
0.00%
2.08%
0.32%
Any 13-gram dirty flag
6.28%
22.83%
8.85%
Figure 4: Matryoshka truncation: balanced-subset macro nDCG@10 by embedding dimension.
Q: Как обойти ограничения ФО на реквизит в динамическом списке, не трогая константу ФО? Gold: ОтключенныеПоля = Новый Массив; … FO×dynamic-list pattern; little lexical overlap with gold.
Win (fastcode) Ours 1; others >10
Q: Описание строки в таблице значений Gold: ст90 = новый ОписаниеТипов("строка",,Новый КвалификаторыСтроки(90)); Colloquial «описание строки» → type API; lexical baselines miss it.
Fail (fastcode) Ours >10; base 1
Q: Произношение текста голосом Gold: Voice = Новый COMObject("SAPI.SpVoice"); Voice.Speak("Привет!"); Rare COM API; general models recover Speak/SAPI.
Fail (fastcode) Ours 32; base 1
Q: Сжатие длинной Строки Gold: СжатиеДанных = Новый СжатиеДанных(9); …Base64. Short generic title; surface «сжатие»+«строка» wins.
Table 12: MRL truncation on PruhaNLP/1C-Ebench.
Dimension m
macro nDCG@10
Retention
Relative index size
768
0.599
100.0%
1.00×
512
0.600
100.0%
0.67×
384
0.600
100.0%
0.50×
256
0.598
99.9%
0.33×
128
0.584
97.5%
0.17×
64
0.560
93.5%
0.08×
32
0.503
83.9%
0.04×
Why it matters
Code mixing Russian syntax with specialized business terminology previously had no public benchmark to even measure search quality, which stalled tool development for this domain. Because the benchmark, model, and training data are all released openly, anyone working on non-English or niche-domain code search can build on this work immediately.
Terms in this paper
bi-encoder · a search model architecture that encodes queries and documents into vectors independently, then compares them by similarity
nDCG@10 · a search quality metric measuring how highly the correct answer ranks among the top 10 results
Matryoshka Representation Learning (MRL) · a training method that lets an embedding vector be truncated to a shorter prefix while preserving most of its usefulness
BM25 · a classic keyword-matching search algorithm based on term overlap
PII scrubbing · automatically detecting and masking personal identifying information like emails or phone numbers
Original abstract (English)
Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly domain-specific terminology, for which open datasets and specialized models have been virtually non-existent. We present a comprehensive pipeline for 1C code retrieval: an open benchmark of 3,413 real-world, PII-scrubbed query-code pairs, a reproducible evaluation harness, and a specialized bi-encoder. To overcome scarce labeled data, we fine-tune on 784,057 synthetic triplets generated by google/gemma-4-26B-A4B-it from public code repositories, using Matryoshka Representation Learning (MRL) and a privacy-aware tokenizer. Because the benchmark subsets differ in size, we report balanced-subset macro, query-weighted micro, and forum-only results. Our model reaches 0.5992 balanced macro nDCG@10, 0.5044 micro, and 0.4617 on forum, versus 0.4932 macro for the baseline architecture and 0.5404 for google/embeddinggemma-300m. Removing every benchmark example flagged by the conservative exact/13-gram overlap audit leaves 0.6011 balanced macro (0.5010 micro), indicating that detected train-benchmark overlap does not explain the headline result. MRL truncation to 256 dimensions preserves 99.9% of retrieval quality while reducing dense-index storage and exact similarity arithmetic by a factor of three.
