Speaker
Description
Large scientific facilities accumulate decades of operational knowledge across logbooks, maintenance records, documents, chat, and control-system data, yet no single interface searches them together. Here, we present APS-RAG, a retrieval-augmented generation (RAG) platform deployed at the Advanced Photon Source (APS), and what it takes to run it in production. A daily diff-and-upsert pipeline keeps eight collections synchronized across dense, sparse, and knowledge-graph stores, while a Model Context Protocol (MCP) layer serves live EPICS Archiver data at query time. Staff use fast, corrective, and troubleshooting modes in a chat interface, averaging ~8 queries daily. An /aps-rag plugin and skill bring the corpus into coding agents (Claude Code, Codex): an MCP server over a read-only BM25 gateway returns ranked, citable passages for the host model to synthesize. Sustained operation required regression safeguards against infrastructure drift, such as gateway migrations and changing databases, backed by 818 automated tests. On a frozen, expert-adjudicated benchmark with cross-family LLM judging, a failing relevance scorer cost 32.8 percentage points of strict vital recall, the only significant component effect. Hallucination rates varied fivefold across ten generators (5.6–30.6%), with a 31B open-weight model the most faithful.
We discuss how these lessons could transfer to the Electron-Ion Collider.