A Virtual Member of a Community of Practice for the Society of Petroleum Engineers: From Prototype to Deployment

What they did

1. The team built ATHENA to answer questions, capture field 'insights' shared by workers, and proactively surface relevant knowledge tailored to a user's expertise and task, inside a chat interface for the SPE professional community.
2. Instead of relying only on keyword search or vector embedding search like typical RAG systems, ATHENA rewrites a question into multiple interpretations, tags them by location, topic, and document type, and uses these as filters, raising retrieval accuracy (MRR) to 0.687 versus 0.412 and 0.376 for two common embedding models.
3. A new 'batch document reading' agent lets ATHENA aggregate information scattered across dozens to hundreds of documents, and a document viewer with page-level citations lets users jump straight to the source text behind any answer.
4. A new 'user competency vector' models each person's expertise across more than 200,000 topic entries, improving relevant-insight ranking (MRR) by 500% and cutting the number of insights needed to find critical ones by 84% compared to the earlier embeddings-only method.
5. In a 13-subject evaluation, ATHENA beat a state-of-the-art RAG baseline with a 216% higher average task score, a 100% reduction in failure rate, 215% higher productivity, and a 28% higher usability score (KM-SUS); it was deployed in June 2025 to 25 early adopters on the SPE Research Portal, with broader rollout planned for Q4 2025.

Why it matters

This work shows a rare case of a domain-specific AI assistant moving beyond a lab prototype into real use by a professional society with 132,000 members, handling messy, document-heavy expert knowledge. For anyone building or evaluating retrieval-based AI tools, it's a concrete example of how verification features, aggregation across many documents, and organizational trust-building matter as much as the underlying AI techniques for successful deployment.

Terms in this paper

• RAG (Retrieval-Augmented Generation) · An AI method that first retrieves relevant documents, then uses them to generate an answer
• MRR (Mean Reciprocal Rank) · A metric showing how high up the correct document appears in search results; higher is better
• embedding · A way of converting text into numerical vectors so similar meanings end up close together
• agentic approach · A system design where multiple specialized AI agents work together on parts of a task, rather than one single model
• KM-SUS · A usability scale newly developed specifically to evaluate knowledge management systems

Latest papers

• Specification-delta-driven data governance: an empirical study of the {\guillemotleft}spec-delta{\guillemotright} as the unit of change in lakehouse data platformsTreating data-platform changes like reviewable spec snippets instead of code diffs: an experiment design paperarXiv:2608.198382026-08-21
• Are LLMs becoming similarly creative? Evidence from three years of modelsNewer AI chatbots are giving increasingly similar answers to each other, three years of data showarXiv:2608.194372026-08-21
• Auditing Cross-Lingual Fairness in Language Model WatermarkingAI text watermarks that are supposed to catch machine-written content work far less reliably in many non-English languages, and the gap tracks language families, not individual languagesarXiv:2608.200472026-08-21
• TESTNAV: Pareto-Guided Search for Compositional Robustness TestingA smarter way to test AI models against combined real-world glitches, without checking every possible combinationarXiv:2608.198822026-08-21
• Optimal Skill Selection for LLM Agents with Provable Bicriteria GuaranteesA method that picks which 'skill documents' to feed an AI coding agent, with mathematically guaranteed near-optimal resultsarXiv:2608.199932026-08-21
• Reliable Financial Named Entity Recognition under Domain ShiftAn AI's confidence trained on formal filings turns unreliable once it reads tweetsarXiv:2608.195582026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy CorrectionWhen text input is missing or broken, this AI doesn't guess once and move on—it revises its guess step by step to read emotions more reliablyarXiv:2608.199712026-08-21

Latest from METAL LAB

• Google Discover adds chatbot that adjusts your feed based on spoken preferences
• OpenAI Closes In on Anthropic Again in Enterprise Spending Share
• Meta Unveils First 10 Tasks in WildArtifactBench, a Benchmark for AI Agents
• Musk: "Optimus + Grok will one day handle healthcare for all humanity"
• 35% of Web Pages Published Since ChatGPT Show Signs of AI Authorship