5. Recover Stuck Queues

Purpose: Recover stuck or stalled queue processing by diagnosing the issue and re-enqueuing unprocessed messages.

Prerequisites:

  • Bun installed with bun install completed
  • Workers running (or ready to be restarted)
  • Upstash Redis credentials in .env.local

Cost / Duration: $0 | 10-30 minutes

Prompt

Recover stuck queue processing. This handles cases where queue depths are
not decreasing despite workers running, or where stalled entries are reported.

Step 1 -- Check queue status to see pending, processing, and stalled counts:
```bash
bun run scripts/queue-recovery.ts --action=status
```

Step 2 -- Retry any dead letter queue messages first:
```bash
bun run scripts/queue-recovery.ts --action=retry-dlq
```

Step 3 -- Catch up on stalled messages (re-enqueues messages stuck in processing state):
```bash
bun run scripts/queue-recovery.ts --action=catch-up --batch=500
```

Step 4 -- If needed, clean up orphaned queue entries and reset processing locks:
```bash
bun scripts/queue-cleanup.ts
```
Note: Only run cleanup when workers are stopped to avoid removing active entries.

Step 5 -- Verify recovery:
```bash
bun scripts/diagnose-pipeline.ts
```

After recovery, queue depths should be decreasing normally with no stalled
entries and DLQ counts at 0. If messages re-stall, check worker logs for
the specific error causing the handler to hang.

Verification

  • Queue status inspected (pending, processing, stalled counts)
  • DLQ messages retried
  • Stalled messages caught up and re-enqueued
  • No stalled entries in final diagnostic
  • Queue depths are decreasing normally
  • DLQ counts are 0

Back to index