Traces: Three Worked Examples

Three representative queries, end-to-end. These traces are reconstructed from reading the code and match the structure of the /admin/rag-workflow SSE stream in production. Where a specific value depends on database state at run time (chunk IDs, exact source URLs), a plausible example is given and flagged. Last verified against source on 17 April 2026 (commit 8acfbb72e).

Trace 1: Parent, T2 (specific)

Input

User: Sarah Thompson. Parent persona. Child James, age 10, Hackney.

Message:

My LA refused to assess our son after 6 weeks. They said they're still gathering evidence. Is that lawful?

Initial state (post-validation)

question: "[USER_MESSAGE_START]\nMy LA refused to assess our son after 6 weeks. ...\n[USER_MESSAGE_END]"
userId: <sarah's uuid>
userRole: "basic_user"
conversationId: <new uuid>
laCode: "E09000012"           # Hackney, resolved via child.schoolUrn → school.laCode
localAuthority: "Hackney"
schoolUrn: "100231"            # St Monica's Primary
homeLaCode: "E09000012"
homeLaName: "Hackney"
conversationHistory: ""         # new conversation

Node-by-node

1. detectGreeting (about 4ms)

• Regex check: no match. "My LA" ≠ "hello / hi / hey / etc."
• LLM fallback skipped (regex confident).
• Output: isGreeting: false, greetingConfidence: 0.02.

2. detectAmbiguity (about 40ms)

• Pre-filter: SEND terminology match (LA, refused to assess), specific.
• Distress keyword: none.
• Document-paste: no structural signals.
• Output: needsClarification: false, isDistressed: false. Routes to retrieve.

3. retrieve (about 380ms). Emits rag:stage: retrieve_start.

• Query expansion: send-synonyms.ts adds "Section 36" to the synonym set, prompted by "refused to assess".

• Topic tags extracted: ["ehcp", "assessment", "refusal", "statutory-timescale"].

• Embedding: 1536-d from Azure OpenAI (about 120ms round trip).

• SQL: vector top-20 via HNSW plus BM25 re-rank plus RRF fusion plus LA bias for E09000012 or NULL.

• Top chunks (illustrative, IDs anonymised):

  #	Source	la_code	similarity	BM25	RRF
  1	SEND Code of Practice 2015, §9.14	NULL	0.82	0.41	0.032
  2	IPSEA, Requesting EHC needs assessment	NULL	0.77	0.36	0.030
  3	Hackney Local Offer, EHCP process	E09000012	0.68	0.24	0.026
  4	CFA 2014 Section 36	NULL	0.74	0.28	0.025
  5	Hackney EHCP decision-making guidance	E09000012	0.65	0.18	0.022
  ...

• Post-threshold: 9 chunks survive the 0.3 floor.

• Chunk expansion: 3 chunks gain ±1 neighbours from parent docs.

• Output: relevantDocuments.length: 12. Emits rag:stage: retrieve_complete.

4. generate (about 3.2s streaming). Emits rag:stage: generate_start.

• System prompt assembled:
  • base_core (always).
  • confidence_signalling (always).
  • tools_awareness (always).
  • role_parent (Sarah is basic_user).
  • No task module: intent is GENERAL_QUERY.
• RAG context: 12 chunks, formatted as [Source N] <origin>\n<content>\n(Relevance: X%).
• Tools bound: search_schools. Not used here.
• Stream: about 540 tokens over about 3.2 seconds. Time-to-first-token about 420ms.
• Post-generation:
  • Statutory-fact pattern match on "6 weeks" plus "assessment", correct value, no mismatch logged.
  • Grounding pass: no programme names, one paragraph reference (paragraph 9.14) present in sources, no hedge applied.
• Output: generatedResponse, suggestedQuestions: ["Can you help me draft an appeal letter?", "What evidence should I gather?", "How long does tribunal take?"].

5. checkHallucination. Skipped (SKIP_HALLUCINATION_CHECK=true).

END. Emits message:ai:end.

What the user sees

No, the 6-week period is for the LA to make a *decision* on whether to assess.
It's not a window for gathering evidence. Under Section 36(8) of the Children
and Families Act 2014, the LA must tell you within 6 weeks whether it will
carry out an EHC needs assessment. "Still gathering evidence" is not a lawful
reason to delay that decision.

If they've told you they're refusing, you can:

1. Ask for a formal decision letter citing Section 36(8). Your appeal right is
   from the *decision*, not from silence.
2. Appeal to the First-tier Tribunal (SEND). You have 2 months from the
   decision date (or 1 month from mediation certificate, whichever is later).
3. Contact IPSEA (0800 018 4016) or Hackney SENDIASS for advocacy support.

Sources: SEND Code of Practice (paragraph 9.14), CFA 2014 Section 36, Hackney
Local Offer, EHCP process.

This isn't legal advice, it's SEND guidance. If you want a specialist review,
IPSEA can help.

Observability

Logs capture: node latencies, retrieved chunk IDs, prompt module list, stream chunk count, RAG metadata (responseMode: "STANDARD", contextLevel: "HIGH", intent: "general_query", isGrounded: true because the hallucination check was skipped), statutory-fact mismatches (none), grounding hedges applied (none).

Trace 2: Professional, T3 (profile-dependent)

Input

User: James Henderson. teacher_senco persona. Linked to pupil Oliver in Brent.

Message:

Oliver's Section F reads "access to speech and language therapy as required." Is that enforceable? How would I tighten it?

Initial state

userId: <james's uuid>
userRole: "teacher_senco"
laCode: "E09000005"       # Brent, from user.schoolUrn
localAuthority: "Brent"
schoolUrn: "100249"         # Wembley Primary
conversationHistory: "... (3 prior turns about Oliver's EHCP) ..."

Node-by-node

1. detectGreeting. Regex no match (about 4ms).

2. detectAmbiguity (about 45ms)

• Pre-filter: clear SEND terminology (Section F, enforceable, speech and language therapy). Specific claim (as required).
• Distressed: no.
• Document-paste detector: triggered on structural signal. The user pasted a Section F: fragment with a colon. But length is well below 300 chars, so it fails the header-plus-300 threshold. Does not route to document-review.
• Output: needsClarification: false, isDistressed: false. Routes to retrieve.

3. retrieve (about 410ms)

• Query expansion: adds "specified provision", "Section 42" to the synonym set.

• Topic tags: ["ehcp", "section-f", "provision", "enforceability", "quantification"].

• SQL retrieves chunks with la_code IN (E09000005, NULL), Brent plus national.

• Top chunks (illustrative):

  #	Source	la_code	RRF
  1	SEND Code of Practice §9.69, specific detailed provision	NULL	0.035
  2	CFA 2014 Section 42, duty to secure provision	NULL	0.031
  3	IPSEA, Section F quantification guide	NULL	0.028
  4	SEND Tribunal decision, M v LA (enforceability)	NULL	0.024
  5	Brent Local Offer, EHCP process	E09000005	0.019

• 11 chunks survive threshold. Chunk expansion adds 3 neighbours.

4. generate (about 3.8s)

• System prompt:

  • base_core.
  • confidence_signalling.
  • tools_awareness.
  • role_professional (James is teacher_senco).

• Because the user mentioned Section F, detectIntentFromMessage could return EHCP_WRITING. But the current code doesn't map that to task_ehcp_writing automatically. So no task module loads. Known gap, see Prompt Modules.

• Conversation history (last 5 turns about Oliver) is threaded in. The model knows Oliver's age, needs, and that he has an EHCP.

• Stream: about 620 tokens over about 3.8 seconds.

• Post-generation grounding pass: the response mentions "Nuffield Dyspraxia Programme" as an example of compliant wording. Not in retrieved sources, so a hedge is applied:

  ...e.g. "Nuffield Dyspraxia Programme" (this isn't in my retrieved sources, ask your SENCO before using it)...

  The hedge sits mid-sentence in the final response.

END.

What the user sees (abridged)

No, "access to SALT as required" isn't enforceable. The SEND Code of Practice
§9.69 says provision "must be detailed and specific and should normally be
quantified." Section 42 of the Children and Families Act 2014 makes Section F
legally enforceable, but only if the wording is specific enough to demonstrate
a breach.

To tighten this, aim for the quantification standard: **what, who, how often,
how long, what programme**. Concrete examples for Oliver:

- **What:** Speech and language therapy focused on [named targets from his
  assessment].
- **Who:** Qualified SALT.
- **How often:** 3 x 30-minute sessions weekly (direct) plus weekly liaison
  with teaching staff.
- **How long:** Throughout the school year.
- **Programme:** e.g. "Nuffield Dyspraxia Programme" (this isn't in my
  retrieved sources, ask your SENCO before using it) or whichever approach
  the SALT report specifies.

Anti-patterns to avoid (all refused by Tribunal): "access to", "support as
needed", "opportunities for", "where possible", "appropriate support".

Sources: SEND CoP §9.69, CFA 2014 s.42, IPSEA Section F quantification guide.

Note on a subtlety

The task_ehcp_writing module is not loaded despite being the perfect fit. That's a documented open gap. For now, the role_professional module covers the quantification standard well enough that the response is usable. Adding an intent mapping for Section F: plus EHCP keywords would improve consistency on these queries.

Trace 3: Crisis

Input

User: a parent (persona irrelevant, crisis overrides persona).

Message:

I just can't take this any more. My son's been refused again and I don't want to be here.

Node-by-node

0. Deterministic crisis check (in socket.ts, before the graph even starts, about 2ms)

• shouldTriggerCrisisResponse() scans the message against the CRISIS_PATTERNS regex set.
• Match: /\bcan'?t\s+take\s+(it\s+)?any\s*more\b/ AND /\bdon'?t\s+want\s+to\s+(be\s+here|live|exist)\b/.
• The helpline block is emitted as the first message:ai:chunk:

I can hear that things are incredibly hard right now. What you're feeling
is real, and you don't have to face this alone.

If you're in immediate danger, please phone 999.

For urgent support:
• Samaritans: 116 123 (free, 24/7)
• Shout: text SHOUT to 85258 (24/7)
• NHS mental-health: 111

When you're ready to return to SEND support, I'll be here.

• Logged at WARN with event: "crisis_helplines_triggered" and userId.

1. detectGreeting. No match (about 4ms).

2. detectAmbiguity (about 35ms)

• Distress keywords match (can't take, don't want to be here).
• Output: needsClarification: false, isDistressed: true. Routes to retrieve (distressed users bypass clarification).

3. Assembly override. Separately, the includeCrisis: true flag is set because the crisis detector also fires in the persona-mapping layer. The system prompt assembler:

• Loads base_core + confidence_signalling + tools_awareness + crisis_response.
• Skips role, task, and domain modules.
• Returns early from assemblePrompt().

4. retrieve (about 380ms). Proceeds with the user's LA context and SEND synonyms, because even in a supportive response we may want to reference their actual appeal rights once the emotional first-aid is done.

5. generate (about 2.8s). With the crisis_response module in the prompt, the LLM produces a warm, helpline-grounded follow-up. Example:

What you're going through sounds utterly exhausting. Being refused again
when you've been fighting this hard, it's such a heavy thing to carry.

Please, if you haven't yet, reach out to Samaritans (116 123) or Shout
(text 85258). They're there for exactly this.

When you're ready, and only when you're ready, we can look at the appeal
route together. You have 2 months from the refusal letter to appeal to the
First-tier Tribunal (SEND), and IPSEA (0800 018 4016) offers free
specialist advice.

But first, please take care of yourself tonight.

• Grounding pass: programmes, none mentioned. Paragraph references: "2 months" and "Section" mentions are all in retrieved sources. No hedges applied.
• Statutory-fact check: "2 months" for appeal, correct value, no mismatch.
• skipHedging: true was set (crisis intent plus sparse retrieval), moot here since nothing would have hedged anyway.

END. The user sees the deterministic helpline block first, followed by the warm, grounded response.

Key safety properties

• Helpline appears before any LLM output. Even if the Azure OpenAI call hung for 30 seconds or errored, the user has already seen helpline numbers.
• Helpline is belt-and-braces. If the socket.ts layer somehow skipped it (reconnection race, bug), ai-service.ts also prepends the helpline before persistence.
• Crisis response module takes priority. Role and task modules don't dilute the crisis framing.
• No SEND pivot without permission. The response explicitly says "when you're ready, and only when you're ready", rather than ploughing into tribunal mechanics.

What these traces tell you

Happy path is fast. Trace 1 is four graph nodes, about 3.6 seconds end-to-end, zero retries. That's typical.

The exceptional paths exist for a reason. Trace 3's double-layer crisis injection isn't paranoia. It's what separates "SEND chatbot" from "safe SEND chatbot for distressed families".

The brain is conservative about certainty. Trace 2's hedge around "Nuffield Dyspraxia Programme" is the grounding pass doing its job. The response is still useful. The hedge is just honest.

Profile context shapes retrieval and generation both. Trace 1 retrieves Hackney Local Offer chunks. Trace 2 retrieves Brent. Neither query mentioned their LA by name. The system resolved it from the child's school URN.

Capturing your own trace

To see live graph execution for a specific conversation:

1. Open /admin/rag-workflow in the web app.
2. Select the conversation.
3. Send a new message. The graph runs with SSE instrumentation and the admin UI streams node entries, state diffs, retrieved chunks, and token-by-token output.
4. Full trace persists under the conversation for replay.

For debugging, admins can also re-run any historical message through the pipeline without re-sending it from the user's account.