Measuring the Partial-Credit Gap: A Strict Benchmark on Vietnam's 2025 Convex Marking Scheme
Grading AI exams the wrong way inflates their scores: a benchmark built on Vietnam's real high-school marking rules
Vietnam's 2025 reformed high-school graduation exam grades one section with a non-proportional step scale: getting 3 out of 4 true/false statements right earns 0.50 points, not the 0.75 a normal accuracy score would give. Existing AI benchmarks ignore this and just report plain accuracy, which overstates competence the government itself would not certify. The researchers built THPT-Ladder from 632 real exam items across 21 exams, scored eight models under the official rule, and found that accuracy alone cannot predict the actual mark.
What they did
- Part II of Vietnam's 2025-reformed exam asks candidates to judge four true/false statements per question, paying a convex ladder of 0, 0.10, 0.25, 0.50, or 1.00 points depending on how many are correct — three correct pays 0.50, not the proportional 0.75.
- Because this section is worth 4.00 of the exam's 10.00 points, scoring it with ordinary statement-by-statement accuracy — as standard benchmarks do — always inflates the score above what the ministry actually awards.
- The team built THPT-Ladder: 632 items from 21 official exams across 11 subjects, graded with the ministry's exact published answer keys and rules, and compared model scores to over a million real candidates' published score distributions to compute percentiles.
- Across eight models (three open-weight, five closed), the official ladder paid 0.020 to 0.159 points less per Part II question than proportional credit would. For Qwen3.5-27B on the 2025 History exam, this dropped its standing from the 90th to the 77th percentile.
- Models with identical accuracy could receive very different marks depending on how their errors were distributed across questions; at Claude Sonnet 5's accuracy level, scores ranged from 0.869 to 0.932 points per question.
- A fixed guess string (DDSS), chosen without reading any question, exploited imbalances in the official answer keys to earn 11.07% to 24.25% of an exam's total score.
| structure | items | ||||||
|---|---|---|---|---|---|---|---|
| Subject | I | II | III | rnd | 2025 | 2026 | Fig. |
| Biology | 18 | 4 | 6 | 2.350 | 28 | 28 | 28 |
| Chemistry | 18 | 4 | 6 | 2.350 | 28 | 28 | 9 |
| Economics & Law | 24 | 4 | 0 | 2.725 | 28 | 28 | 0 |
| English | 40 | 0 | 0 | 2.500 | 40 | 40 | 0 |
| Geography | 18 | 4 | 6 | 2.350 | 28 | 28 | 8 |
| History | 24 | 4 | 0 | 2.725 | 28 | 28 | 1 |
| Informatics | 24 | 4a | 0 | 2.725 | 30 | 30 | 0 |
| Mathematics | 12 | 4 | 6 | 1.975 | 22 | 22 | 14 |
| Physics | 18 | 4 | 6 | 2.350 | 28 | 28 | 4 |
| Technology (Agri.) | 24 | 4 | 0 | 2.725 | 56 | 28 | 4 |
| Technology (Ind.) | 24 | 4 | 0 | 2.725 | 28 | n.p. | 12 |
| total | 632 items | 80 |
| Model | % avail. | I | II stmt | II pts/q | shf/q | III |
|---|---|---|---|---|---|---|
| Qwen3.5-27B | 93.8 | 0.973 | 0.926 | 0.884 | 0.042 | 0.917 |
| Qwen3.5-9B | 90.7 | 0.965 | 0.896 | 0.830 | 0.066 | 0.850 |
| InternVL3.5-8B | 69.3 | 0.838 | 0.756 | 0.597 | 0.159 | 0.183 |
| Claude Opus 5 | 97.0 | 0.988 | 0.973 | 0.949 | 0.024 | 0.950 |
| GPT-5.5 | 97.0 | 0.988 | 0.967 | 0.948 | 0.020 | 0.950 |
| Claude Sonnet 5 | 93.6 | 0.977 | 0.935 | 0.881 | 0.054 | 0.917 |
| Claude Opus 4.8 | 92.9 | 0.975 | 0.920 | 0.867 | 0.052 | 0.917 |
| Claude Haiku 4.5 | 82.9 | 0.934 | 0.842 | 0.735 | 0.108 | 0.567 |
Why it matters
Simplifying an exam's grading rule into plain accuracy can make an AI model look like it passed an exam it would have actually failed under real rules. This work quantifies exactly how much that substitution distorts reported competence, which matters for anyone designing or trusting exam-based AI benchmarks.
Terms in this paper
- convex marking scheme · a grading rule where points don't scale proportionally with correct answers but jump or get compressed at certain thresholds
- partial-credit gap (shortfall) · the difference between the score a proportional accuracy metric would give and the actual score under the step-based official rule
- percentile · a rank showing what fraction of real test-takers a given score outperforms
- open-weight model · an AI model whose weight files are released publicly so anyone can download and run it
- Decision 764 · the official 2025 ruling by Vietnam's Ministry of Education and Training that defines the new exam formats and marking rules
Original abstract (English)
When evaluating language models on human exams, benchmarks typically score each response as right or wrong and report the overall accuracy. This approach assumes that partial knowledge is worth proportional credit, an assumption that fails when an examination uses a non-additive grading scheme. The 2025 reform of Vietnam's National High School Graduation Examination demonstrates the cost of this substitution. In Part II of the exam, candidates eval
Read on arXivLatest papers
- FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI SystemsAI often names the right cause of a financial mismatch without ever finding the proof for it
- Looped Language Models Improve Compositional Tool CallingAI models that rethink their own answers multiple times get better at chaining tools together
- FACET: Preserving Source Intent and Executable State in Terminal Task SynthesisFACET builds internally consistent terminal-task 'exam sets' to train command-line AI agents
- FM-Bench: A Benchmark for Long-Horizon Management with Competing AgentsHanding an AI a football club to run for 20 years reveals that winning comes from management habits, not raw model power
- Bridging Search and CRM: Productionizing AI Product Research Agents for Customer Re-EngagementAI agents win back window-shopping customers by chasing them down on WhatsApp
- Adversarial Review: Structured Disagreement for Grounded Agentic Code ReviewFor AI code review, one reviewer plus one critic beats piling on more agents
- GenEx: A Graph-Based Representational Paradigm for SARS-CoV-2 Variant Detection via Codon Co-occurrence NetworksTurning viral gene sequences into codon relationship maps to tell coronavirus variants apart
- Alignment Is All You Need: Instruction-Free Training for General Audio-Language ModelsA frozen language model plus one lightweight connector is enough to build a capable audio-understanding AI
Latest from METAL LAB
- NVIDIA's 300 Verified Skills Lift Correctness by 41 Points
- Wave your hand at a webcam, hear a theremin: browser instrument released
- Meta AI launches desktop app for Mac, can read an entire app window
- Factory Commits $100M to Partner Network, Pushes to Scale Software Factories
- SpaceX approached Cognition for acquisition four days after closing Cursor deal