hcai-lab-gt/capabilibara
Tracing where a language model's social reasoning ability actually comes from in its training data
This project builds a pipeline that traces which chunks of pretraining data support a given capability, like social reasoning, in a language model. It groups 5.68 million sampled documents into 576 topic-by-format regions, scores their influence on benchmark answers, then deletes the flagged regions to test if the model actually loses that ability. The work is by researchers including Georgia Tech's Experimental AI Lab and EleutherAI, published as a COLM 2026 paper.
- ai-research
- capability-provenance
- interpretability
- language-models
- machine-unlearning
- mechanistic-interpretability
- research
- training-data-attribution
What it does
- Starting from Dolma3, a deduplicated corpus of about 1.26 billion documents, the team sampled 5.68 million documents and classified them into a 24-topic by 24-format grid, giving 576 corpus regions
- Using gradient-based training-data attribution (TrackStar, via the Bergson tool) on OLMo3-7B, they traced which training documents most influenced answers on benchmarks like SocialIQA and MMLU, then aggregated that influence up to the 576 regions
- Regions flagged as influential were causally tested with selective unlearning, deleting their contribution from the model to see if performance actually dropped
- Removing the flagged region tied to SocialIQA caused a 1.60 percentage point drop in performance, a statistically significant result (p roughly 10 to the negative 5)
- The code, sampling manifests, the influence matrix, and unlearning checkpoints are planned for release alongside the camera-ready paper; currently only the repository structure and method description are public
Why it matters
Knowing which training data actually produced a specific model capability lets researchers fix or remove problematic data more precisely instead of guessing. This repo demonstrates a way to trace capabilities not document-by-document but at the level of broad corpus regions, and backs it up with real deletion experiments on a large open model, which matters for anyone working on interpretability or data governance.
Terms in this repo
- training-data attribution · a technique for tracing which part of the training data produced a model's specific behavior or answer
- machine unlearning · selectively removing the influence of specific data from an already-trained model
- WebOrganizer 24x24 taxonomy · a classification scheme sorting web documents into 24 topics crossed with 24 formats
- TrackStar (Bergson) · a tool that uses gradients to estimate which training documents influenced a model's output
- LoRA · a lightweight fine-tuning method that adjusts only a small set of added parameters instead of the whole model
Repository description (English)
Training-data attribution as a discovery method for capability provenance in language models. COLM 2026.
Open on GitHubProject siteCoverage
Trending repos
- openai/codexOpenAI's coding agent, now runnable straight from your terminal
- cordiverse/cordisA TypeScript framework for snapping feature pieces on and off whenever you need them
- ripienaar/free-for-devA single repo lists every free-tier cloud, API and collaboration tool developers actually need
- Wei-Shaw/sub2apiA relay server that lets one Claude/OpenAI/Gemini/Grok subscription be shared by many users
- multica-ai/andrej-karpathy-skillsA single instruction file that stops AI coding assistants from making silent bad guesses
- eneskirca/nodetermA workspace that puts scattered terminal tabs and AI coding agents on one draggable map
- affaan-m/ECCA toolkit that teaches AI coding agents to work like disciplined engineers
- n8n-io/n8nn8n lets you build automation workflows and AI agents by dragging nodes on a canvas, with code as a fallback