Multi-agent RAG in Microsoft Teams

01The problem

A large organisation needed employees to get answers grounded in internal documents (policies, procedures, technical specs) and to perform actions (open a ticket, look up a record, notify a channel) without leaving Teams. A single monolithic RAG did not scale: the retrieval quality collapsed as domains multiplied, tool calls became unreliable, and one prompt could not hold the entire policy surface.

02The approach

I split the assistant into a supervisor plus specialised agents, each owning one domain and one tool surface. The supervisor is a LangGraph state machine that decides which agent to activate, orchestrates multi-step reasoning, and keeps a bounded shared memory. Retrieval, structured tools and write actions live in separate agents so their prompts, evaluations and permissions stay independent.

03Architecture

04Key technical decisions

• LangGraph over a plain agent loop: explicit state, deterministic routing, replayable traces. Much easier to debug and to add human-in-the-loop later.
• Retrieval as its own agent: hybrid search (BM25 + embeddings) over Azure AI Search, chunking tuned per document type, per-source metadata for citation.
• Tool calling with typed schemas: every tool is a validated function signature. The model cannot invent arguments; invalid calls are refused before execution.
• Guardrails on input and output: prompt-injection screening on inbound messages, PII redaction and policy checks on outbound answers.
• Azure-native security: managed identity everywhere, no long-lived secrets, all traffic inside the tenant.

05What I would do differently

Invest in an evaluation harness from day one, not day thirty. A small labelled set per agent, replayed on every prompt change, would have caught two regressions that I caught only through user reports.