#Observability

#Purpose

Make it easy to answer: What happened? Why? Which pipeline step? Where did it fail? How much did it cost?

---

#Package

packages/observability — Shared structured logging utilities.

• Custom structured JSON logging
• Component-scoped loggers
• Child loggers for request/task context

---

#Core Concepts

#Structured Logging

All logs are JSON-formatted for machine parsing:

{
  "timestamp": "2026-03-24T12:00:00.000Z",
  "level": "info",
  "message": "Extracted facts from story",
  "component": "worker-facts",
  "storyId": "abc123",
  "factsCreated": 3
}

Log levels (configurable via LOG_LEVEL env var):

• debug - Verbose development info
• info - Normal operations
• warn - Potential issues
• error - Failures requiring attention

#Component Loggers

Each component creates a scoped logger:

import { createComponentLogger } from '@eko/observability'

const logger = createComponentLogger('worker-facts')

logger.info('Processing story cluster', { storyId, articleCount })
logger.warn('Notability below threshold', { factId, score: 0.4 })
logger.error('Extraction failed', { error: String(error), model })

#Child Loggers

Create child loggers for request-scoped context:

const taskLogger = logger.child({ storyId, provider: 'newsdata' })

taskLogger.info('Starting extraction')   // includes storyId and provider
taskLogger.info('Extraction complete', { factsCreated: 3 })

---

#Error Tracking (Sentry)

Sentry is integrated across all apps and workers for error tracking and performance monitoring.

#Configuration

App	Config Files
apps/web	sentry.server.config.ts, sentry.edge.config.ts, sentry.client.config.ts, instrumentation.ts
apps/admin	Same pattern as web
apps/public	Same pattern as web
Workers	Initialized in entry point (src/index.ts) via @eko/observability

#Environment Variables

Variable	Description
ERROR_TRACKING_PROVIDER	none or sentry
SENTRY_DSN	Sentry project DSN

Workers initialize Sentry at startup and report unhandled errors automatically.

---

#AI Cost Observability

All AI calls are tracked via the ai_cost_tracking table for budget monitoring and optimization.

#How It Works

1. Every AI SDK call records model, tokens, and cost via recordAICost()
2. Costs aggregate daily per model + feature combination
3. The daily cost report cron emails admins a breakdown with 7-day trend

#Key Tables

Table	Purpose
ai_cost_tracking	Per-model, per-feature daily cost aggregation
ai_model_tier_config	Model routing rules per tier (default/mid/high)

#Environment Variables

Variable	Description	Default
ANTHROPIC_DAILY_SPEND_CAP_USD	Daily budget cap for all AI providers	5
OPUS_ESCALATION_ENABLED	Allow routing to expensive models	false
OPUS_MAX_DAILY_CALLS	Hard cap on expensive model invocations	20

#Querying Costs

-- Today's cost by model
SELECT model, feature, SUM(total_cost) as cost, SUM(total_calls) as calls
FROM ai_cost_tracking
WHERE date = CURRENT_DATE
GROUP BY model, feature ORDER BY cost DESC;

-- 7-day trend
SELECT date, SUM(total_cost) as daily_cost
FROM ai_cost_tracking
WHERE date >= CURRENT_DATE - INTERVAL '7 days'
GROUP BY date ORDER BY date;

---

#Pipeline Observability

#Ingestion Runs

The ingestion_runs table tracks each cron invocation:

Column	Purpose
trigger_type	Which cron or manual trigger
status	running, completed, failed
facts_created	Output count
duration_ms	Execution time
error	Error message if failed

#Content Operations Log

The content_operations_log table records pipeline debugging events — archival, promotion, quota violations, etc.

---

#Usage Examples

#Worker startup

import { createComponentLogger } from '@eko/observability'

const logger = createComponentLogger('worker-ingest')
logger.info('Starting worker', { queues: ['INGEST_NEWS', 'CLUSTER_STORIES'] })

#News ingestion

logger.info('Fetching articles', { provider: 'newsdata', topicSlug: 'science' })
logger.info('Articles fetched', { provider: 'newsdata', count: 25, newCount: 12 })

#Fact extraction

const taskLogger = logger.child({ storyId, model: 'gpt-5.4-nano' })
taskLogger.info('Extracting facts')
taskLogger.info('Extraction complete', { factsCreated: 3, costUsd: 0.002 })

#Validation

logger.info('Validation complete', {
  factId,
  phase: 4,
  result: 'validated',
  evidenceSources: ['wikipedia', 'wikidata'],
  confidence: 0.92,
})

---

#Invariants

• Don't store raw article content in logs
• Don't add logs without a question they answer
• Always include component context
• Use structured data (objects) not string interpolation
• Always track AI costs for every LLM invocation

---

#Configuration

Env Var	Description	Default
LOG_LEVEL	Minimum level to output	info
ERROR_TRACKING_PROVIDER	Error tracking backend	none
SENTRY_DSN	Sentry project DSN	—
ANTHROPIC_DAILY_SPEND_CAP_USD	AI daily budget	5

---

#Related

• Architecture Overview - System components
• APP-CONTROL.md - Operational manifest with all env vars
• Runbooks - Use logs for debugging
• Queue Runbook - Queue-specific logging patterns