Backfill and Fix NULL Columns in fact_records

Audit of fact_records revealed several columns systematically NULL due to pipeline gaps and missing wiring.

Challenges

Wave 1: Schema & Type Expansion

#ChallengePASS Criteria
1.1Add notability fields to ImportFactsMessageSchemaZod schema includes notability_score, notability_reason, context as optional fields
1.2Update createImportFactsMessage TS typeTS type includes challenge_title?, notability_score?, notability_reason?, context?
1.3Add generationCostUsd to insertFactRecordFunction accepts and writes generationCostUsd to DB

Wave 2: Pipeline Wiring Fixes

#ChallengePASS Criteria
2.1Enqueue GENERATE_CHALLENGE_CONTENT after validationvalidate-fact.ts enqueues both RESOLVE_IMAGE and GENERATE_CHALLENGE_CONTENT for validated facts
2.2Remove duplicate enqueue from import-facts.tsimport-facts.ts no longer imports or enqueues GENERATE_CHALLENGE_CONTENT; explanatory comment present
2.3Wire notability fields through import pipelineimport-facts.ts passes notability_score, notability_reason, context from message to insertFactRecord; explode-entry.ts passes notability_score and context in IMPORT_FACTS items
2.4Wire generationCostUsd in extraction handlersextract-facts.ts computes cost via estimateCost() and passes to insertFactRecord; generate-evergreen.ts splits cost across records

Wave 3: Backfill Existing Data

#ChallengePASS Criteria
3.1Backfill notabilityScore for NULL records--dry-run shows counts; --notability sets source-type defaults; verified with SELECT COUNT(*) WHERE notability_score IS NULL AND status = 'validated' returns 0
3.2Enqueue challenge content for gaps--challenge-content enqueues GENERATE_CHALLENGE_CONTENT for validated facts with zero fact_challenge_content rows
3.3Audit data integrity--audit reports anomalies: NULL validation JSONB, NULL publishedAt on validated records

Verification

bun run typecheck
bun run test
bun run lint
bun scripts/seed/backfill-fact-nulls.ts --dry-run

Files Modified

FileChange
packages/shared/src/schemas.tsAdded optional notability fields to ImportFactsMessageSchema
packages/queue/src/index.tsUpdated createImportFactsMessage TS type
packages/db/src/drizzle/fact-engine-queries.tsAdded generationCostUsd to insertFactRecord
apps/worker-validate/src/handlers/validate-fact.tsEnqueue GENERATE_CHALLENGE_CONTENT after validation
apps/worker-facts/src/handlers/import-facts.tsWire notability fields, remove duplicate challenge enqueue
apps/worker-facts/src/handlers/explode-entry.tsPass notability/context to IMPORT_FACTS message
apps/worker-facts/src/handlers/extract-facts.tsWire generationCostUsd
apps/worker-facts/src/handlers/generate-evergreen.tsWire generationCostUsd
scripts/seed/backfill-fact-nulls.tsNEW: Backfill script