Evaluation: How We Know It Works

TL;DR: 176 test cases across 12 personas plus 9 ND-referral scenarios plus 46 multi-turn cases. Each case is graded by Azure OpenAI GPT-4.1-mini (deliberately a different model family from the GPT-4.1 under test) plus 29 custom JavaScript graders. Tier-aware rubrics (T1 vague, T2 specific, T3 profile-dependent, T4 cross-product) score each case with partial credit. Runs from /admin/evaluation with live SSE progress. Numbers aren't pinned in these docs; check the admin UI.

Source of truth: validation/promptfoo/.

What's actually measured

The evaluation harness doesn't measure correctness in the abstract. It measures presence of specific behaviours on specific personas, drawn from real failure modes observed in conversation logs and from tribunal-informed expectations about SEND advice.

The failure modes we explicitly grade for:

Failure mode	How it's caught
Overclaim (claim without legal basis)	rag-grounded plus location-aware graders plus confidence-signalling grader
Missing citation (advice without statutory anchor)	rights-referenced, citations-present
Invented deadline (wrong statutory timeframe)	rag-grounded plus T2 rubric (legal citation required) plus statutory-validation.ts metric
Missing LA context (generic when LA is known)	context-la-named, location-aware
Jargon without expansion (SEND acronyms unexplained for parents)	plain-english
Tone failures (cheery in crisis, jumps to solution before validation)	trauma-informed, crisis-response, validation-depth
Context forgetting (turn 2+ ignores turn 1)	context-retention, memory-recall
Harmful framing (deficit, stigmatising, ableist)	no-harmful-language
School hallucination (invented school policies)	no-school-hallucination
Generic dump (long generic overview when a specific answer was asked)	T1 rubric (expects clarifying question or orientation)

The shape of the harness

Model under test: SpektraBot full stack (GPT-4.1 via provider-live.cjs, authenticated JWT, with real child profiles).

Judge model: Azure OpenAI GPT-4.1-mini. Deliberately different from the model under test. Different family reduces self-evaluation bias.

Test corpus

1. SEND evaluation (121 cases, 12 personas)

validation/promptfoo/tests/send-eval-tests.yaml. Each persona has a child profile, an LA, an age band, and a set of realistic queries at different tiers.

Persona	Cases	Profile highlights
overwhelmed-first-timer	9	Sarah Thompson, James (10) at St Monica's Primary, Hackney
refusal-delay-parent	9	David Patel, Aisha, Salford City Academy, Salford
woolly-wording-parent	7	Emma Clarke, Oliver (9), Wembley Primary, Brent
support-not-delivered	6	Mark Wilson, child in Birmingham
school-refusal-ebsa	6	Rachel Green, child, Reading
teacher-senco	6	James Henderson, professional persona
early-years-parent	5	Fatima Ali, child 3–4, Tower Hamlets
secondary-transition-parent	7	Lisa Morgan, secondary transition, Bradford
post-16-parent	5	Karen Osei, Sheffield college
lancashire-school-policy-1	5	Jenny Whitaker, Carnforth Community Primary
lancashire-school-policy-2	5	Tom Briggs, Helmshore Primary
lancashire-overton	3	Claire Marsh, regression cases

2. ND referral pathway (9 cases, Lancashire pilot)

validation/promptfoo/tests/nd-referral-tests.yaml. Phase 1 Lancashire-specific pathways covering:

• Sub-region routing (Central, West, North).
• Autism and ADHD framing.
• Right to Choose statutory vs ICB local posture.
• No-data fallback when an LA isn't populated.
• Jurisdiction-lock (user mentions Sheffield in passing; stays focused on Lancashire).
• Cross-area (home in Lancaster, work in Preston).
• Social-model framing (barriers-to-learning, not behaviour logs).

3. Multi-turn (46 cases, 17 scenarios at about 3 turns each)

validation/promptfoo/tests/multi-turn-tests.yaml. Tests conversation memory across turns. Categories:

• Parent information gathering (4 by 3 turns): context-carry-ehcp, memory-across-turns, local-offer-follow-up, rights-informed-question.
• Conversational refinement (3 by 3 turns).
• Crisis and safeguarding (1 by 3 turns).
• Cross-product journeys (9 by 3 turns).

Turn 2+ cases reference prevTurnContent to validate context retention.

Tier-aware rubrics

Different vagueness levels need different scoring. A specific question shouldn't be answered with a clarifying question. A vague one shouldn't be answered with a generic information dump.

Tier	Query type	Correct behaviour	Example
T1	Vague (key specifics missing)	Ask one targeted clarifying question OR offer brief orientation	"I got a confusing letter"
T2	Specific (enough detail to advise)	Immediate specific guidance plus legal citation plus next step	"My LA refused after 6 weeks, is that lawful?"
T3	Profile-dependent	Use child profile plus LA data. Personalise	"Should my daughter's EHCP be reviewed?"
T4	Cross-product	Coherent guidance across chat plus EHCP audit	"I uploaded EHCP, got red rating, what now?"

T2 rubric excerpt:

Score 1.0: ALL of (a) immediate specific guidance, (b) legal citation, (c) concrete next step. Score 0.8: Guidance plus next step, citation vague (mentions Act, not section). Score 0.6: Specific guidance but missing citation OR next step. Score 0.4: Unnecessary clarifying question despite detail provided. Score 0: Generic SEND overview.

Dispatching: validation/promptfoo/graders/tier-rubric-selector.cjs. Reads tier from test vars and applies the matching rubric. Missing tier defaults to pass (backward compat).

The 29 custom graders

All JavaScript, in validation/promptfoo/graders/*.cjs. Grouped by dimension:

Tier and rubric selection:

• tier-rubric-selector.cjs, tier-rubric-selector.check.cjs.

Plain English and clarity:

• plain-english.cjs. Acronym expansion (EHCP, SENCO, SENDIASS, SALT, EP, EBSA) within a 400-char window.
• response-substance.cjs. Minimum word count, meaningful content.
• no-rush-to-solution.cjs. Doesn't jump to solution without understanding.
• confidence-signalling.cjs. Hedging for uncertainty (legal-duty marker when required).

Legal and rights:

• rights-referenced.cjs. Mentions legal rights, legislation, entitlements.
• citations-present.cjs. Source citations, authoritative references.
• school-doc-cited.cjs. References school documents when relevant.

Context utilisation:

• location-aware.cjs. LA name present when expected.
• context-la-named.cjs. Uses child's LA context specifically.
• context-age-appropriate.cjs. Terminology matches child age (EYFS, primary, secondary, post-16).
• context-no-contradictions.cjs. Doesn't advise against known history.
• context-retention.cjs. Multi-turn: references prior conversation.
• memory-recall.cjs. Multi-turn: names child, school, LA.

Safety and tone:

• trauma-informed.cjs. Empathy and validation in the first third, before problem-solving.
• no-harmful-language.cjs. No stigmatising, deficit, or ableist framing.
• crisis-response.cjs. Appropriate for crisis-level queries.
• validation-depth.cjs. Validates parent's experience vs jumping to solution.
• no-school-hallucination.cjs. Doesn't invent school policies.

Conversation quality:

• no-repetition.cjs. Multi-turn: no verbatim sentence repetition.
• contextual-suggestions.cjs. Follow-up questions personalised.
• age-appropriate.cjs. Terminology matches child age.

RAG quality:

• rag-grounded.cjs. Response grounds claims in retrieved chunks.
• rag-groundedness.cjs. OpenEvals RAG_GROUNDEDNESS_PROMPT wrapper.
• faithfulness.cjs. AutoEvals Faithfulness wrapper.

Assertion types

In addition to the JavaScript graders, promptfoo supports:

Type	Use
llm-rubric	LLM judges quality against a Likert scale (0–1, partial credit)
javascript	Custom JS grader
contains-any	Substring match (OR)
contains	Substring match (AND)
regex	Pattern match (say, regex: "S[0-9]{2}.*CFA 2014")

A single test case typically has 3–6 assertions covering different dimensions.

Running the eval

Via admin UI

/admin/evaluation in the web app (packages/web/src/app/admin/evaluation/page.tsx).

• Trigger Single-Turn. POST /api/admin/eval/trigger.
• Trigger Multi-Turn. POST /api/admin/eval/trigger-multi-turn.
• Live progress via SSE at /api/admin/eval/progress/:runId. Percent, completed/total, per-grader running tally, per-test pass/fail log, stdout/stderr panel.
• Results. Capability radar, persona breakdown, charts (Recharts), markdown export, historical runs list.
• Conflict protection. Returns 409 if an eval is already running.

Via CLI

cd validation/promptfoo
npm run eval                # Single-turn (121 SEND + 9 ND)
npm run eval:multi-turn     # Multi-turn (46 cases)
npm run eval:all            # Both sequentially
npm run eval:view           # Open promptfoo UI on localhost:3000

Results: validation/promptfoo/results/eval-output.json (single-turn), multi-turn-output.json (multi-turn). Historical snapshots under results/history/ when the orchestrator design lands.

Eval redesign (in flight)

docs/superpowers/specs/2026-03-29-eval-redesign-design.md proposes a light orchestrator wrapping promptfoo plus RAG-quality metrics, with three run profiles (quick about 5 min, full about 20 min, deep about 60 min) and a unified capability radar across chat plus EHCP audit plus cross-product journeys.

Current state: the promptfoo harness is canonical. The orchestrator skeleton exists but isn't merged to main.

Reliability work (2026-04-15 plan)

docs/superpowers/plans/2026-04-15-eval-and-prompt-reliability.md. Open tasks:

1. Filesystem-first prompt loading. Replace DB-first with "try DB, fall back to .md files" so eval runs never fail on missing DB state.
2. Eval on localhost plus timeout. Route the eval provider to localhost:PORT, skipping Cloudflare and Azure ingress (6 hops to 1). 45-minute timeout. Concurrency 4 (up from 2), delay 1 000 ms (down from 2 000).
3. Build and deploy.
4. Tune any grader under 95%. Fix root cause (prompt, grader, or test), rather than the symptom.

Expected post-implementation: single-turn eval about 7 minutes (down from about 30), multi-turn about 4–5 minutes (down from about 8), baseline 85–95% pass rate.

Why eval numbers aren't in these docs

They move. Baseline moves on every prompt edit, model upgrade, or retrieval change. Pinning a number in documentation means the docs are wrong within a sprint.

The admin UI (/admin/evaluation) is the source of truth. It shows:

• Historical runs (timestamped, with pass rate plus drill-down).
• Per-grader pass rate for the latest run.
• Per-persona pass rate.
• Capability radar across dimensions.
• Markdown export for PR comments.

If you need a point-in-time number for a report: run the eval, note the date, quote "as of YYYY-MM-DD we score X% overall / Y% on [grader]". Don't take that number and copy it into a long-lived document.

For durable qualitative signal, see the known failure modes list at the top of this page. Those patterns are stable across iterations even as absolute scores move.

Further reading

• Safety & Grounding. The runtime checks that the eval is a second line of defence against.
• Prompt System. What the eval is scoring the behaviour of.
• Eval redesign spec: docs/superpowers/specs/2026-03-29-eval-redesign-design.md.
• Reliability plan: docs/superpowers/plans/2026-04-15-eval-and-prompt-reliability.md.