Knowledge Base: Ingestion, Crawlers, Validation

TL;DR: About 451 000 chunks from about 27 700 documents across 143 English LAs plus national charity sources and school policies. Three ingestion paths (direct upload, ZIP, URL crawl). Chunk (1 000 chars, 200 overlap), embed (Azure OpenAI 1536-d), store (pgvector HNSW), validate (manual or LLM-assisted). Two crawlers: LA crawler (per-LA seed URLs) and school-policy crawler (per-GIAS-school).

Code: packages/api/src/knowledge-base/ (service, routes, auto-validation) and packages/api/src/crawler/ (LA crawler, school-policy crawler, document processor, change detection).

What's in the knowledge base

Content snapshot (2026-03-11, per memory): 451 000 chunks, 27 713 documents, 143 LAs with content.

Sources, in priority order for retrieval:

1. National statutory. SEND Code of Practice, Children and Families Act 2014 regulations, DfE guidance. Always available (la_code IS NULL).
2. LA Local Offers. Per-LA seed URLs crawled on a schedule. 152 English LAs configured, 143 with ingested content.
3. National charity. IPSEA, SENDIASS, Contact, NSPCC, and about 10 other charity domains. Hardcoded in NATIONAL_DOMAINS to avoid per-LA duplication.
4. School SEND policies. Scraped from school websites for about 52 000 GIAS-registered schools when available. Tagged with school_urn.

Storage model

Table: knowledge_base (schema.prisma: KnowledgeBase model).

Column	Type	Purpose
id	UUID	Primary key
embedding	vector(1536)	Cosine-similarity search (HNSW index)
search_vector	tsvector	BM25 full-text search (GIN index)
content	TEXT	Chunk text
source	VARCHAR	Human-readable source (URL or title)
send_tags	TEXT[]	Topic labels (ehcp, tribunal, ...). GIN indexed
la_code	VARCHAR(5)	GSS code for LA filtering (NULL = national)
school_urn	VARCHAR(10)	GIAS URN for school-specific docs
validation_status	VARCHAR	pending, approved, rejected, flagged, superseded
parent_doc_id	UUID	Link chunks back to their original document for expansion
chunk_index	INT	Position within parent document
document_type	VARCHAR	pdf, docx, txt, html, school_policy, crawled
metadata	JSONB	contentHash (SHA-256), extraction timestamp, source URL, page count, headings
created_at	TIMESTAMPTZ	Feeds the temporal-decay weight in retrieval

Three ingestion paths

1. Direct upload

Endpoint: POST /api/admin/knowledge-base/upload (knowledge-base/routes.ts:109–181).

• Multer with memory storage (no disk temp files).
• Accepted MIME types: application/pdf, application/vnd.openxmlformats-officedocument.wordprocessingml.document, text/plain.
• Max size: 50 MB per file.
• Admin form supplies laCode, localAuthority, sendTags, and (optionally) schoolUrn.

2. ZIP upload

Endpoint: POST /api/admin/knowledge-base/upload-zip (routes.ts:189–230).

• Max size: 200 MB per archive.
• Directory structure becomes SEND tags: SEND/Autism/guide.pdf becomes tags ["SEND", "Autism"].

3. URL ingestion (asynchronous)

Endpoint: POST /api/admin/knowledge-base/ingest-urls (routes.ts:732–821).

• Returns 202 Accepted immediately. Work runs in the background.
• Redis-backed batch tracking with 24-hour TTL.
• Poll status at GET /api/admin/knowledge-base/ingest-status?batchId=....
• Internally calls knowledgeBaseService.ingestFromUrl() per URL, which reuses the crawler's fetch and parse path.

Parsing

knowledge-base/service.ts:181–330. Parser by MIME type:

Format	Parser	Library
PDF	unpdf.extractText()	unpdf npm package
DOCX	mammoth.extractRawText()	mammoth npm package
TXT	Native UTF-8 decode	(n/a)

Content must be non-empty post-parse, otherwise the upload is rejected with a parse error.

Chunking

packages/api/src/crawler/chunking.ts:26–72.

• Chunk size: 1 000 characters.
• Overlap: 200 characters.
• Minimum chunk length: 50 characters (filters noise).
• Algorithm:
  1. Normalise line endings (CRLF to LF).
  2. Collapse 3+ newlines to 2.
  3. Try to split at paragraph breaks (\n\n) within ±50% of the chunk-size boundary.
  4. Fall back to sentence breaks (. , .?\n, ! , !) if no paragraph break within range.
  5. Apply 200-char overlap between adjacent chunks.
  6. Drop chunks under 50 characters.

Headings are preserved inside chunk text but not extracted to separate metadata fields.

Embedding

Per chunk:

1. langChainService.generateEmbedding(text). Calls Azure OpenAI text-embedding-3-large. Returns number[1536].
2. Validate dimensions === 1536. Warn on mismatch.
3. L2-normalise to unit length (validateAndNormalizeEmbedding at langchain.ts:51–89). Cosine similarity on unnormalised vectors gives wrong rankings.
4. Insert via raw SQL with ::vector(1536) cast on the embedding array string.

Retries rely on LangChain's default backoff (exponential, capped). No custom retry logic in the ingestion path.

Deduplication

Three-level check (service.ts:216–239 plus document-processor.ts:293–390):

1. Content-hash dedup (upload). SHA-256 hash of full parsed content. If the hash exists in any row's metadata.contentHash, reject with 409 Conflict plus existing document ID. No two copies of the same file.
2. URL-based dedup (crawl, global). If the source URL already exists anywhere in the knowledge base with the same content hash, skip re-embedding. If same URL but different hash, replace chunks atomically.
3. Per-LA URL dedup. Tracked separately in CrawledUrl. Allows change detection without scanning knowledge_base on every crawl.

Hash stored at metadata.contentHash in the knowledge_base JSONB column.

Validation workflow

States: pending, approved, rejected, flagged, superseded.

State	Set by	Mechanism
pending	System on upload	Default status after chunk insertion
approved	Admin	Manual review via UI. PATCH /api/admin/knowledge-base/:id/validate with status: "approved"
rejected	Admin	Same endpoint, status: "rejected". Excluded from retrieval
flagged	LLM auto-validator or admin	For human re-review when auto-validation is uncertain
superseded	Auto (kb-supersession.ts)	After ingesting a newer version. Old doc retained for audit but excluded from retrieval

Admin UI paths

• List. GET /api/admin/knowledge-base?validationStatus=pending&laCode=...&tags=...&minQuality=... (routes.ts:250–304). Filters, paginated.
• Chunks. GET /api/admin/knowledge-base/:documentId/chunks. All chunks for one parent, for human review.
• Bulk validate. POST /api/admin/knowledge-base/bulk-validate. Up to 50 at once. Backed by a keyboard-shortcut UI at /admin/knowledge-base/validate.
• Auto-validate. POST /api/admin/knowledge-base/auto-validate (routes.ts:593–627) with a dryRun option. Runs asynchronously. Check status at /auto-validate/status.

Auto-validation

packages/api/src/knowledge-base/auto-validation.service.ts:110–150:

• Fetches pending documents (chunk_index 0 only, to avoid duplicate scoring).
• Optional LA filter.
• Batches: 50 documents per batch, 5 concurrent batches.
• Calls Azure OpenAI with the validation prompt (validation-prompt.ts).
• Parses response for { status, confidence }.
• If confidence ≥ threshold, updates the database. Otherwise marks flagged for human review.
• dryRun logs intended changes without persisting.

Quality scoring

Computed dynamically on list and filter queries. Not persisted (service.ts:369–388). Formula:

qualityScore =
    content_length_score (0–40)    -- longer chunks = more substantive
  + document_type_score (0–30)     -- PDF=30, DOCX=25, HTML=20, TXT=15
  + send_tags_score (0–30)         -- 0 tags=0, 1–2=15, 3+=30

Score scale: 0 to 100.

Use case: admin filtering in /admin/knowledge-base?minQuality=60. Not fed into RAG retrieval. Retrieval relies on similarity, BM25, and freshness.

LA tagging

Source depends on ingestion path:

Path	Mechanism
Direct upload	Admin specifies laCode plus localAuthority in the form
ZIP upload	Admin specifies via form fields. Directory structure captures tags
LA crawler	Automatic from crawl-job metadata (localAuthorityId to LA row)
National domains	Automatic: la_code = NULL, localAuthority = "National" when URL matches NATIONAL_DOMAINS
School-policy crawler	Inherits from school.laCode

National domain list (document-processor.ts:72–86): gov.uk, legislation.gov.uk, NHS resources, major SEND charities. Prevents the same legislation being indexed 152 times (once per LA crawl).

LA crawler

Queue: Bull (packages/api/src/crawler/crawl-job.queue.ts:17–35).

Parameter	Value	Purpose
Queue name	la-crawl	Identifies LA crawl jobs
Backoff	Exponential	Retry on transient failures
Lock duration	300 s	How long a worker holds a job
Max retries	CRAWLER_CONFIG.maxRetries	Before permanent failure

Seed URLs per LA: Stored on LocalAuthority rows. seedUrl (primary) plus additionalSeedUrls[].

Depth: CRAWLER_CONFIG.maxDepth (typically 2–3).

Browser fetching: browser-fetcher.ts uses Playwright headless Chrome for:

• HTTP 403, 406, or 429 responses (WAF or rate-limit blocks).
• Pages where plain HTML returns over 512 bytes but extracts to empty content (JavaScript-rendered).

Robots.txt compliance: url-validator.ts fetches, parses User-Agent: * rules, respects Disallow, extracts Crawl-Delay and Request-Rate. URLs matching Disallow are flagged blockedByRobots and never re-visited.

Change detection: change-detection.ts:68–99. Four signals in priority order:

1. HTTP 304 (Not Modified). Skip.
2. ETag match. Skip.
3. Content-hash match (SHA-256). Skip.
4. Last-Modified stale. Fall back.

CrawledUrl rows track etag, lastModified, contentHash, and version (incremented when content changes, old versions archived via isArchived = true).

Operational notes

Per memory, from earlier incidents:

• Bull stale active jobs after a container restart leave "active" entries blocking concurrency. Flush with a Redis-keys script and restart the worker.
• VACUUM ANALYZE on knowledge_base takes 30+ minutes due to the 2 GB HNSW index.
• Trigger all LA crawls via POST /api/admin/crawler/trigger-all with an admin JWT. The endpoint deduplicates against running and pending jobs.

School-policy crawler

Parallel crawler for school SEND policies. Differences vs the LA crawler:

Aspect	LA crawler	School crawler
Trigger	Manual click or scheduled cron per LA	Manual per-school OR batch by LA
Seed URL	LA domain from LocalAuthority.seedUrl	schools.website from GIAS
Content	Gov.uk, council policies, Local Offer	School SEND, accessibility, safeguarding policies
LA tag	Direct from crawl job	Indirect via school.laCode
Job model	CrawlJob	SchoolCrawlJob
Document type	crawled	school_policy

Schools enumerated from GIAS: SELECT * FROM schools WHERE la_code = ? AND is_open = true AND website IS NOT NULL. Count varies by LA (London: 1 000+, rural: 50–200).

Indexing

Main indexes (prisma/migrations/20260212100000_add_hnsw_vector_index/):

• HNSW on knowledge_base.embedding. m=16, ef_construction=64, vector_cosine_ops.
• HNSW on document_chunks.embedding. Same parameters.
• GIN on search_vector. BM25 tsvector.
• GIN on send_tags. Topic filter.
• Compound (la_code, validation_status). Retrieval filter.
• Compound (parent_doc_id, chunk_index). Chunk expansion.

Dimension change from 1024 to 1536 (2026-03-05 migration) dropped the HNSW indexes. Current procedure: drop, then re-embed, then recreate CONCURRENTLY to avoid OOM on the managed Postgres.

Document model (richer structure)

Some newer document types use a two-table pattern:

• Document. One row per parent (metadata, source URL, permissions).
• DocumentChunk. N rows per parent (content plus embedding plus chunk_index).

Used for user-uploaded documents where sharing semantics matter (DocumentShare rows gate visibility). The legacy knowledge_base table still serves for curated national, LA, and school content.

Configuration constants

Constant	Value	File
CHUNK_SIZE	1 000 chars	chunking.ts:13
CHUNK_OVERLAP	200 chars	chunking.ts:14
MIN_CHUNK_LENGTH	50 chars	chunking.ts:20
EMBEDDING_DIMENSIONS	1 536	langchain.ts:1354
EMBEDDING_MODEL	text-embedding-3-large	langchain.ts:1353
MAX_UPLOAD_SIZE	50 MB	routes.ts:22
MAX_ZIP_SIZE	200 MB	routes.ts:49
HNSW m	16	migration 20260212
HNSW ef_construction	64	migration 20260212

Supersession

lib/kb-supersession.ts (extracted post-2026-03-04). After ingesting a new document, searches for older docs with the same normalised title and marks them superseded. RAG retrieval excludes superseded content via the validation-status filter.

Benefit: when a council publishes an updated Local Offer page, yesterday's version automatically drops out of retrieval without manual re-validation.

Further reading

• Retrieval. How the indexed content gets queried.
• Safety & Grounding. Document sanitiser for crawled content.