15–18 Dec 2026
Duke University
America/New_York timezone
Artificial Intelligence for the Electron Ion Collider (2026)

APS-RAG in Production: Lessons from Deploying an Agentic Knowledge Assistant for Accelerator Operations

Not scheduled
20m
LSRC B101 Love Auditorium (Duke University)

LSRC B101 Love Auditorium

Duke University

Durham, NC 27710
AI/ML in Production and Distributed ML AI/ML in Production, Distributed ML

Speaker

Rajat Sainju (member@anl.gov)

Description

Large scientific facilities accumulate decades of operational knowledge across logbooks, maintenance records, documents, chat, and control-system data, yet no single interface searches them together. Here, we present APS-RAG, a retrieval-augmented generation (RAG) platform deployed at the Advanced Photon Source (APS), and what it takes to run it in production. A daily diff-and-upsert pipeline keeps eight collections synchronized across dense, sparse, and knowledge-graph stores, while a Model Context Protocol (MCP) layer serves live EPICS Archiver data at query time. Staff use fast, corrective, and troubleshooting modes in a chat interface, averaging ~8 queries daily. An /aps-rag plugin and skill bring the corpus into coding agents (Claude Code, Codex): an MCP server over a read-only BM25 gateway returns ranked, citable passages for the host model to synthesize. Sustained operation required regression safeguards against infrastructure drift, such as gateway migrations and changing databases, backed by 818 automated tests. On a frozen, expert-adjudicated benchmark with cross-family LLM judging, a failing relevance scorer cost 32.8 percentage points of strict vital recall, the only significant component effect. Hallucination rates varied fivefold across ten generators (5.6–30.6%), with a 31B open-weight model the most faithful.

We discuss how these lessons could transfer to the Electron-Ion Collider.

Author

Rajat Sainju (member@anl.gov)

Co-authors

Presentation materials

There are no materials yet.