Development Timeline

A short history of how SpektraBot was built, drawn from the project's git history.

The first commit landed on 15 November 2025. Just over five months later, on 26 April 2026, the platform sits on Azure Container Apps in UK South, retrieves from a curated knowledge base of over a million chunks, and runs a phased pilot with structured Local Authority routing for Lancashire. What follows is the path between those two points.

This is a single-author project (Joel Merrick) of 1,397 commits across 86 active development days. The arc, broadly, is six phases: get something working, find the right audience, get serious about who the user is, build out the platform, harden it for trust, then clean up the knowledge base it depends on.

Phase 1: Foundations (15 to 30 November 2025)

The first three weeks were about getting the bones in place. The initial commit shipped a working chat surface, a Postgres database, an authentication flow, and the core idea: retrieval-augmented generation over a SEND-specific knowledge base. By the end of the second week the agentic RAG workflow was running on LangGraph.js, with multimodal support so a parent could upload a PDF (an EHCP, a school report) and have it processed alongside their question.

AWS Bedrock provided the model layer at this stage, using Nova Lite for chat and Claude for document handling. PDF processing went through Bedrock's native document API. Deployment ran on Civo's managed Kubernetes (K3s) with Helm charts and TLS via cert-manager, with automated PostgreSQL backups to S3.

Notable commits from this phase:

• 0fe0f0423 (25 Nov 2025): RAG database schema with pgvector
• 48c6aa54a (27 Nov 2025): Phase 2 Agentic RAG with LangGraph
• 8b25c3f28 (25 Nov 2025): Admin console with user management
• 6f563b02f (21 Nov 2025): anti-hallucination measures for factual accuracy

What the platform could not yet do: it had no proper sense of who its user was. There were no child profiles, no Local Authority awareness, no schools data, no memory of past conversations. It was a generic SEND assistant that happened to know the SEND Code of Practice.

Phase 2: Pivot to Parents, Pivot to Europe (December 2025)

December reshaped the product in two ways at once.

The bigger pivot was strategic. The original positioning had been ambiguous about the primary user. Was this for SENCOs? Teachers? Parents? On 6 December the homepage was rewritten as a parent-first advocacy model (6a07ca37a), the schema followed shortly after (bff05af24), and the waitlist (64b4df613, 73e39948e) was built specifically for England-only families seeking SEND support. The teacher and SENCO surfaces stayed, but as secondary professional user types layered on top of a parent-centred core.

The smaller pivot was technical, but legally consequential. AWS Bedrock processed inference outside the EU, which created a real problem for a service handling sensitive data about disabled children in the UK. On 4 December the project moved off Bedrock to Mistral AI, hosted in the EU (76244c48d). This was framed in the commit as "GDPR compliance", and that is exactly what it was: the only realistic path to a defensible Data Protection Impact Assessment.

December also brought the modular prompt system (b24e02d5e), with prompt modules stored in the database and editable through an admin UI (65033fe47). This is the foundation of how prompts are still managed today: role-based routing, intent detection, per-module versioning, and live editing without a redeploy.

Other shipped work: a real-time RAG workflow visualiser using React Flow (48086fca3, c75ccb4f0), conversation metadata persistence (60eef54e7), an email service via Resend (20ff04d26), and a 74% reduction in Docker image size by stripping Bedrock dependencies (9b3c17747).

Phase 3: Who Are You? (January 2026)

January was the month the platform learnt to ask "who is asking, on behalf of which child, in which school, in which Local Authority?" Without that context, retrieval is generic and advice is shallow. With it, you can route a question about a Lancashire EHCNA to the actual Lancashire portal, and weight school-specific policy ahead of a national document.

The work split four ways:

• Child profile management (3a69be9cb, 19 Jan): families add their children, with date of birth, year group, primary need, current setting
• Schools data via the GIAS feed (14c7d53b2, 21 Jan): all 30,000+ England schools with URN, type, and local authority, plus a school search API and an autocomplete that lets parents pick the right setting
• Local Authority awareness in retrieval (6f19a02e8, e087d1ec7, 2 Feb): every knowledge-base document gained an la_code, and the agentic RAG node started filtering by it
• Professional setup flow for teachers and SENCOs (db369c4e6, 26 Jan), so the same chat surface could behave differently depending on who logged in

Two pieces of infrastructure moved forward in parallel. CloudNativePG took over Postgres operations, with automated backups to Civo Object Storage (a3d126295). Mistral fine-tuning was wired up as a path for SEND-specific knowledge transfer (f15ee5b20, dc213d88e), though it was later overtaken by retrieval improvements and the model swap to Azure OpenAI.

Phase 4: The Build-Out (February 2026)

February was, by some distance, the busiest month of the project. 737 commits landed, roughly 26 a day, and the platform stopped feeling like a prototype and started feeling like a product. The commit messages from this stretch read like a feature release log for a company of ten.

A few of the larger pieces:

Memory and continuity. The child-specific memory system (361094f92, 17 Feb) introduced persistent context: discoveries, extractions, and a per-child memory store that the assistant could draw on across sessions. The conversation engine (08a38e1c4, 24 Feb) added phase tracking and topic continuity, so a conversation about EHCP timelines that drifted into a question about school transport could come back to the original thread without losing place.

The Battle Binder. Borrowed from the parent-advocate vocabulary, this is the structured evidence pack a family builds when preparing for a tribunal or annual review. February shipped both the data model (97a1d26d2, EHCP golden thread metadata) and the user-facing flows (0cf41a368, Brain Dump and Detective Mode).

Knowledge base growth. The LGSCO SEND decision crawler (a42aaeddd, 20 Feb) and the Lancashire digital EHCNA portal ingestion (2e1e83b82) brought structured statutory and decision data into the corpus. Five idle SEND data sources were activated and wired into the RAG pipeline (ef55e2aac).

Evaluation infrastructure. A multi-turn eval suite (d2bf33264), a comprehensive grader expansion (77925e500), live SSE streaming for eval observability (c4f17d88e), and an admin system health page (f5b627666). By the end of the month, prompt and model changes could be tested against a held-out test set with real graders rather than smoke tests.

Observability. Langfuse for LLM traces (7b842822d), Prometheus metrics for the AI pipeline with three Grafana dashboards (ceb751719), and a Grafana web analytics dashboard replacing Umami (277fb3679).

Accessibility, properly. A WCAG 2.2 AA accessibility test suite using axe-core and Playwright (22617bf50), Universal Design for Learning principles applied to the chat surface, and a Quiet HUD redesign (e9b5efa16) that moved status, suggestions, and pill actions out of the way of the conversation itself.

The brain hardening sprint. SEND synonym expansion, statutory validation, and memory-enhanced retrieval shipped together as one unit (c281af4bd). This is the work that lifted answer specificity above the surface-paraphrase threshold.

Other notable February work: the advocate and specialist role with referral system (0ba60fe69), zero-downtime WebSocket deployments with sticky sessions (781e3dfbc), an admin blog CMS with a WYSIWYG editor (317d7e7d7), and a complete SEO build (2d92f1fad) with sitemap, structured data, and per-page metadata.

Phase 5: Trust, Safety, Evidence (March 2026)

If February was about building, March was about earning the right to put the platform in front of real families.

The headline change was the AI provider swap. On 5 March, Mistral AI was retired and Azure OpenAI GPT-4o (UK South) took over (ce536e424). Mistral had served well, but Azure offered a UK-South region with a defensible data residency story, a higher token-per-minute ceiling, and access to text-embedding-3-large. The same commit shipped the LangChain integration, with two real gotchas captured in project memory: AzureOpenAIEmbeddings ignores azureOpenAIEndpoint and needs azureOpenAIBasePath, and the Azure AI Services endpoint uses cognitiveservices.azure.com rather than openai.azure.com. Both cost a working evening to find.

On the compliance side, March brought a cluster of work that had been deferred from the build-out:

• DSAR (Data Subject Access Request) data export and self-service account deletion (9c9b36e59)
• Cookie and localStorage consent banner (5279749d1)
• Tiered data retention with a CronJob (f0a2c6250)
• AI response disclaimers for SEND legal guidance (43af90444)
• A DPIA template with tiered retention references (bb11f1847)
• Children's Code assessment, GDPR controls documentation, and a non-discrimination policy (all in /docs/compliance/)

Safety hardening for the RAG pipeline arrived in the same window. Prompt module integrity monitoring (20fd3191d), RAG content sanitisation with boundary markers (0e9ee39f5), an injection barrier in the query transform node (39a4f2280), Trivy container image scanning (b9f358ab5), and ETag-based change detection on the school policy crawler so that a modified document supersedes its predecessor rather than duplicating it (b8a1c27ac).

The user-facing platform kept moving. The Insights dashboard for parents (216ad59e7), an EHCP audit pipeline (ba88f60f5), markdown rendering during streaming (b1322a93b), the Spectrum Toolbox of AI writing tools (1d40dc1ee), assistant-ui adoption with a tool-calling framework (5e070ac0c), and a richer source attribution showing school, LA, and URL in citations (50aeeeb74).

The eval framework took a significant step. On 30 March the LLM judge was switched from a generic comparison to Azure OpenAI GPT-4.1-mini with proper context (aafb4d425), and the pass rate jumped from 39% to 62%. A few days later the chat model itself moved from GPT-4o to GPT-4.1 (5188d8d09), bringing further quality gains. Plan B and Plan C eval orchestrators followed (bc9d66823, 1635807c5), with specialist runners for the EHCP pipeline, dialogue engine, and RAG quality.

The end of the month saw the start of the infrastructure migration that would finish in early April. Azure Container Apps was investigated as an AKS alternative on 22 March (8c41eb17f), and a full investigation and migration plan landed three days later (0d549f1c8). Civo K8s, the deployment substrate since the very first weeks, was on its way out.

Child memory continuity (f6ac21acb, 29 Mar) deserves a paragraph of its own. The four-tier memory model (working, child-specific, conversation, narrative capsule) ships semantic retrieval over a child's history alongside per-session summaries. This is what allows the assistant to come back to a conversation a fortnight later and still know that this child has selective mutism, refuses school on Tuesdays, and is awaiting an autism assessment. Without it, every chat starts from zero.

Phase 6: Brain and Knowledge Base Trust (April 2026)

April had two themes that worked in tension. On the application side, it was about precision: how the assistant frames difficult content, when it should hedge, and how it routes a structured task like a neurodevelopmental referral. On the data side, it was about taking honest stock of what was actually in the knowledge base, then doing something about it.

The application work shipped largely on 17 April. The ND referral pathway introduced structured Local Authority routing (8acfbb72e), with Lancashire as the Phase 1 LA, a behaviour-module prompt that activates on intent detection, and a curator guide for the data. The document-paste detector (91c357472, 9065c2d0f) recognises when a user pastes an EHCP or IPP and routes it to a dedicated review path with trauma-informed framing (085216afd). National-source bias when no LA context is present (1900203c1), hedged citations for unverified programmes and paragraph references (4c38d7440), retention of distress signals through the document-paste path (2d32c2c62), and an overclaim audit with confidence signalling (a3270fc94). All of these are small in lines of code and large in how the assistant feels.

The knowledge base work began with a measurement. On 22 April the rag_retrieval audit log started capturing per-turn provenance for every chat: the top retrieved chunks, their tier projection, score components, position in context, and the query context that produced them (34d2b7790). A baseline pulled the next day was uncomfortable: 91% of the corpus was Tier 3 (general web), 8% was Tier 2A (NHS, official), and just 0.06% was Tier 1 (legislation, statutory guidance). The platform was answering statutory questions from peer-support content because that was most of what it had.

D1, the school-policies split, ran across 23 April in five staged phases (Phase A through Phase D.2). It moved 971,909 chunks identified as school behaviour, safeguarding, child protection, anti-bullying, attendance, and admissions policies out of the general knowledge_base table into a dedicated school_policies table, with retrieval logic that augments general results with school-specific policy context behind a feature flag. Two gotchas were learnt the hard way and committed to project memory: Postgres has no native MAX(uuid) aggregate (caught by the smoke-test gate, 13888c03a), and the first pass missed dash-variant titles, requiring a second pass (7e794fc71).

On 23 and 24 April the Tier 1 and 2A seed corpus work began. Curated URL lists for legislation, the SEND Code of Practice, tribunal decisions, NHS guidance, official sources, and the Local Government and Social Care Ombudsman were committed alongside an admin endpoint that drives bulk crawl ingestion through a Container Apps Job (a524c396e). The first run on legislation revealed that the readability extractor returns short stubs from legislation.gov.uk's SPA shell, which is a real but fixable problem for the next phase.

By the end of April the infrastructure migration was complete. Civo K8s and Helm were retired. All services ran on Azure Container Apps in spektrabot-rg, UK South, with images in ACR (spektrabotacr.azurecr.io) and a database operations pattern using Container Apps Jobs in place of kubectl exec. The CLAUDE.md file was updated with a banner making the new reality unmistakable (0ab173bb0).

What's next

The 26 April commit history closes with an implementation plan for a RAG lab: an admin debug surface and an eval interface that should make the next round of retrieval improvements measurable rather than vibes-based. Phase 0.5 corpus curation continues against the trust-architecture plan committed on 22 April. The flag that gates school-policy split retrieval is on in production but not yet removed; the cleanup phase is deferred at least seven days to observe behaviour first.

Five months in, the platform handles ten thousand-plus chunks of statutory and policy content per query, holds memory across sessions, routes by Local Authority, knows which school a child attends, audits its own retrieval choices, and is ready for a structured pilot. The path here was not a straight line, and the next stretch is unlikely to be either.

Reading the timeline

Commit hashes in this document are stable references into the repository on GitHub. Paste any of them into the GitHub commit URL or use git show <hash> locally to see the change in full. PR numbers (for example #634) appear in commit messages where the work landed via pull request.

Phase boundaries are descriptive, not bureaucratic. They reflect where the work shifted shape, not where a milestone closed. Real project history is messier than a timeline suggests, and several pieces of work named in one phase had their first commits in the previous one.