Agentic RAG: The LangGraph State Machine

TL;DR: Nine nodes, ten conditional edges, compiled with LangGraph. Fast paths short-circuit before retrieval (greeting, crisis, clarification). Normal path: retrieve → generate → END. Hallucination check and query-transform loops both exist, both disabled by default in production.

Code: packages/api/src/chat/agentic-rag.ts (~3 800 lines). Graph definition at line 3087.

The graph

Nodes

Node	Lines	What it does
detectGreeting	1250–1333	Regex of 39 patterns first. LLM fallback only if ambiguous. About 5ms hot path.
greetingResponse	1335–1430	Warm onboarding text. No RAG. Ends the graph.
detectAmbiguity	1605–1700	Deterministic pre-filter (SEND keywords, distress keywords, document-paste, short-message). LLM classifier only if all pre-filters are inconclusive.
generateClarification	1702–1800	Interactive clarification with suggestion chips. Skipped for distressed users.
retrieve	1832–2100	Hybrid search (vector + BM25 + RRF) with LA bias and chunk expansion. See Retrieval.
transformQuery	2102–2250	LLM rewrites the query when retrieval returns less than the minimum. Disabled by default.
generate	2252–2668	Main LLM inference: tool binding, phase guidance, context injection, streaming, post-filters, grounding pass.
checkHallucination	2676–2892	LLM factuality grader, separate model, different family. Disabled by default.
toolExecution	2989–3065	Executes LLM tool_calls. Appends ToolMessage results. Re-enters generate.

Edges (conditional routing)

From	Function	Routes
detectGreeting	decideGreetingOrRAG()	greetingResponse if greeting, else detectAmbiguity
detectAmbiguity	routeAfterAmbiguityCheck()	generateClarification if vague-and-not-distressed, else retrieve
retrieve	decideGenerate()	generate if enough docs or retries exhausted, else transformQuery
generate	routeAfterGenerate()	toolExecution if pendingToolCalls, else checkHallucination
checkHallucination	decideResponse()	END if grounded or retries exhausted, else generate

Entry point

// packages/api/src/chat/agentic-rag.ts:3168
export async function executeAgenticRAG(
  conversationId: string,
  userId: string,
  userRole: string,
  question: string,
  attachments?: Array<{ name: string; size: number; type: string; data: string }>,
  onStream?: (chunk: string) => void,
  onStage?: (stage: RAGStage) => void
): Promise<{
  response: string;
  suggestedQuestions: string[];
  relevantDocuments: RelevantDocument[];
  metadata: { responseMode, contextLevel, intent, isGrounded, retrievedChunks };
}>

Called from langchain.ts:generateResponse() when config.enableAgenticRAG === true (default). Before invocation, the question is sanitised and wrapped with [USER_MESSAGE_START]...[USER_MESSAGE_END] markers. Conversation history is loaded (last five turns). Initial state is populated.

State

The state object (about 35 fields, full list at agentic-rag.ts:170–360) groups into:

• Input and control. question, userId, userRole, conversationId, conversationHistory, attachments, streaming callbacks.
• Greeting and ambiguity. isGreeting, greetingConfidence, needsClarification, isDistressed, clarifyingQuestion, suggestions.
• Retrieval. documents, relevantDocuments, documentsRetrieved, noLaBroadFallbackUsed, transformedQuery.
• Intent and routing. intent, conversationIntent, responseMode (STANDARD, GATHER, DETAILED, TASK, SUPPORTIVE, CRISIS), contextLevel (LOW, MEDIUM, HIGH).
• Generation. generatedResponse, suggestedQuestions, binderSuggestions, reminderSuggestions.
• Quality. isGrounded, hallucinationReason, retryCount, loopCount.
• Tools. messages (BaseMessage[] for tool threading), pendingToolCalls.
• User profile. laCode, localAuthority, schoolUrn, schoolName, homeLaCode, homeLaName.
• Phase context. phaseContext, userRequestedDetail.

Defaults and env vars

These defaults control whether expensive nodes run:

Env var	Default in code	Effect
ENABLE_AGENTIC_RAG	true	Use the agentic graph vs the basic RAG fallback.
RAG_RELEVANCE_THRESHOLD	0.15 (retrieval floor)	Minimum cosine similarity to keep a chunk. The post-retrieval relevance filter uses 0.3 for the final relevantDocuments.
RAG_MIN_DOCS / MIN_DOCUMENTS_FOR_GENERATION	0	Trigger query transform if fewer docs. 0 disables transform entirely.
RAG_MAX_RETRIES / MAX_RETRIES	0	Max query-transform retries. 0 disables the retry loop.
RAG_SKIP_HALLUCINATION_CHECK / SKIP_HALLUCINATION_CHECK	"true"	Hallucination check is skipped by default. Saves about 4 seconds per request.
RAG_MAX_LOOP_ITERATIONS / MAX_LOOP_ITERATIONS	1	Maximum regeneration attempts on detected hallucination.
CHUNK_EXPANSION_ENABLED	true	Expand hit chunks with ±1 neighbours from the same parent doc.
LLM_TIMEOUT_MS	10000	Per-node LLM timeout, including the hallucination grader.

Older documentation claiming MAX_RETRIES=3 and SKIP_HALLUCINATION_CHECK=false is stale. Confirmed in agentic-rag.ts:50–67.

Greeting detection

Fast path. Function at agentic-rag.ts:1250–1333:

1. Regex of 39 patterns: /^(hello|hi|hey|welcome|what can you help)/i, /^(who are you|what are you|how does this work)/i, and so on.
2. If regex matches, return { isGreeting: true, greetingConfidence: 0.95 }, skip the LLM.
3. If regex is uncertain, single LLM call (max 10 tokens): "Is this a greeting?" yes or no.
4. If greeting, go to greetingResponse node, then END.

Typical latency: under 5ms (regex path), under 1.5 seconds (LLM path).

Ambiguity detection

detectAmbiguity at agentic-rag.ts:1605–1700, calls into clarification.ts:detectAmbiguity() at clarification.ts:35–153.

Hierarchical checks:

1. CONTINUATION. Is this a follow-up ("yes please", "tell me more")? Uses conversation history.
2. EMOTIONAL DISTRESS. Emojis, crying language, overwhelmed phrasing. Distressed users skip clarification and go to retrieve with a supportive tone.
3. CLASSIFICATION. SPECIFIC, VAGUE, or DISTRESSED.

Output:

{
  needsClarification: boolean,   // true iff VAGUE and not distressed
  isDistressed: boolean,
  reasoning: string,
  clarifyingQuestion: string,    // only when VAGUE
  suggestions: string[]          // 3 options when VAGUE
}

Distressed users bypass clarification. A parent in distress doesn't want an interactive menu, they want grounded, compassionate guidance.

Document-paste detection

Triggered by document-review.ts:detectDocumentPaste(). Requires both structural signal and length:

• Structural signals (line-start anchored or colon-delimited): "Individual Pupil Plan", "EHCP", "One-page profile", ^IEP\b, \bSection F:\b, field-value pairs like Name: X, Year Group: Y.
• Length thresholds (combined with signal): 40 or more non-empty lines, or 1 500 or more characters, or header + 300 or more characters, or field-pairs + 300 or more characters.

Without a structural signal, a long emotional narrative doesn't route to document-review. That's a post-owl-review regression: distressed parents write long messages, and treating those as "documents" loses the distress signal entirely.

See Safety & Grounding for the tests locking this down.

Retrieve node

The hybrid-search engine. Full treatment in Retrieval. In one paragraph: embed the query (with SEND-synonym expansion), vector search top-20 via HNSW, BM25 re-rank the top-20 via tsvector ts_rank_cd, Reciprocal Rank Fusion with k=60, apply LA bias (user's LA > home LA > national > cross-LA), apply temporal decay (older docs downweighted by up to 20%), filter by relevance threshold, expand each chunk with its ±1 neighbours from the same parent document.

Generate node

The heart of the system. Responsibilities:

1. Bind tools. llm.bindTools(registry.getTools(), { tool_choice: 'auto' }) (agentic-rag.ts:2273–2287).
2. Inject system prompt. Assembled from the prompt modules, see Prompt System.
3. Inject RAG context. Retrieved chunks formatted as [Source N] ${source}\n${content}\n(Relevance: X%).
4. Inject conversation history. Last five turns.
5. Inject phase guidance. Current response mode (STANDARD, GATHER, DETAILED, and so on) shapes the instruction.
6. Stream invocation. model.stream(...) via Azure OpenAI. Tokens flow back through the onStream callback.
7. Post-generation filters. Replace "unfortunately" with "currently". Strip security markers. Parse [SUGGESTIONS], [BINDER_SUGGESTIONS], [REMINDER_SUGGESTIONS] trailers if present.
8. Grounding pass. Hedge programme names and paragraph numbers not found in retrieved sources. See Safety & Grounding.
9. Statutory validation. Non-blocking check for invented deadlines. Logged to Prometheus.

Returns: generatedResponse, suggestedQuestions, binderSuggestions, reminderSuggestions, responseMode, contextLevel, loopCount. If the LLM called a tool, also messages with a pending AIMessage containing tool_calls.

Tools

The tool registry lives at tools/registry.ts. Currently one tool is registered:

search_schools

packages/api/src/chat/tools/school-search.ts

• Input schema: query (school name), postcode, localAuthority, isSpecialSchool, hasSenUnit, hasResourcedProvision, phase, limit.
• Output: JSON array of schools with URN, phase, SEND indicators, Ofsted rating.
• When called: the LLM decides, based on user intent. Typical trigger: "find a special school in Bradford".

Tools flow through generate → toolExecution → generate:

1. generate produces an AIMessage with tool_calls: [{ name: 'search_schools', args: {...} }].
2. routeAfterGenerate() sees pendingToolCalls: true, routes to toolExecution.
3. toolExecution (via tools/tool-node.ts:73–131) invokes the tool through the registry, builds ToolMessage results, appends them to state.messages.
4. Edge returns to generate. The LLM now sees tool results as chat context and produces the final human-readable response.
5. routeAfterGenerate() sees no new tool calls. Routes to checkHallucination.

Query transformation (disabled by default)

transformQuery at agentic-rag.ts:2102–2250. Only reached when relevantDocuments.length < MIN_DOCUMENTS_FOR_GENERATION and retries are available.

When enabled, it rewrites the query using an LLM with a SEND-domain-specific prompt:

You are a query rewriter for a SEND (Special Educational Needs) document retrieval system. ... Rewrite the query to (1) use precise SEND terminology, (2) add missing context if vague, (3) focus on the most actionable aspect, (4) keep under 20 words.

Returns { transformedQuery, retryCount: retryCount + 1 } and loops back to retrieve.

Default MIN_DOCUMENTS_FOR_GENERATION=0 disables this entirely. To enable: RAG_MIN_DOCS=1 and RAG_MAX_RETRIES=2.

Hallucination check (disabled by default)

checkHallucination at agentic-rag.ts:2676–2892. Separate LLM call. The prompt explicitly lists what to flag and what not to flag.

Flag as hallucination:

1. Actually incorrect facts (wrong dates, made-up legislation, wrong timelines).
2. Claims contradicting SEND law or the Code of Practice.
3. Made-up organisations or services.
4. Invented statistics.

Do not flag:

• Correct SEND terminology (EHCP, SENDIASS, Annual Reviews).
• Well-known legal timelines (6 weeks, 20 weeks).
• Standard LA processes.
• General best practices.
• References to CFA 2014 or the SEND Code of Practice.

A multimodal prompt handles images and PDFs in attachments.

Skipped when:

• SKIP_HALLUCINATION_CHECK=true (production default).
• The graph reaches END via the greeting or clarification fast paths.
• The grader times out (exceeds LLM_TIMEOUT_MS).
• Crisis intent + sparse retrieval. Hedging language would undercut the supportive tone.

Self-correction loop

Applied when the hallucination check is enabled and returns isGrounded: false:

1. Increment loopCount.
2. Append the hallucination reason to state.messages so the next generate pass can see what to fix.
3. Return to generate.
4. After MAX_LOOP_ITERATIONS attempts, accept the response regardless.

With MAX_LOOP_ITERATIONS=1 (default) and the hallucination check disabled, the loop is effectively never exercised in production.

Streaming and stage events

Tokens stream through the pipeline via onStream:

Azure OpenAI token → generate loop → onStream callback
  → langchain.ts buffers (strips security markers)
  → ai-service.ts pass-through
  → socket.ts emits message:ai:chunk

Stage events via onStage are emitted at node boundaries. See Message Flow: RAG stage events.

Latency by node

Node	Typical p50
detectGreeting	5ms (regex), 1–2s (LLM fallback)
greetingResponse	<100ms
detectAmbiguity	<50ms (pre-filter), 1–2s (LLM classifier)
generateClarification	1–2s
retrieve	200–500ms
transformQuery	1–2s (rarely reached)
generate	2–5s (streaming)
checkHallucination	2–4s (skipped in prod)
toolExecution	<1s per tool

Admin tracing

/admin/rag-workflow streams the full graph execution over Server-Sent Events: node entries, state diffs, stage transitions, retrieved doc IDs, token stream. Useful for debugging specific conversations, A/B testing prompt changes, or demonstrating the pipeline to stakeholders.

Further reading

• Retrieval. What actually happens inside the retrieve node.
• Safety & Grounding. Grounding pass and statutory validation in the generate node.
• Traces. End-to-end examples showing node sequences and state transitions.