Why We Chose Postgres Over Redis for Our AI Job Queue

When we started building DocPilot's documentation generation pipeline, the queue decision seemed obvious. AI jobs are bursty, latency-sensitive, and need a fast in-memory store. Redis was the default. Six months and a few production incidents later, our entire job queue runs on Postgres. This is the story of why — and what we'd do differently if we started over.

The Setup: What a Documentation Generation Queue Actually Does

Every time someone connects a GitHub repo to DocPilot, we kick off a pipeline that looks roughly like this:

1. Clone the repository.
2. Walk the file tree and identify source files.
3. Chunk large files and send them to an LLM for summarization.
4. Generate a README, an architecture overview, and API references.
5. Write the docs back to the connected GitHub repo via a webhook.

Steps 3 and 4 are the expensive ones — anywhere from 30 seconds to 10 minutes depending on repo size. They're also the steps that fail in interesting ways: rate limits, token limits, transient API errors, partial completions. So every job needs:

• A durable record of state (pending, running, completed, failed).
• Visibility into retries and attempt counts.
• The ability to cancel or reprioritize a job.
• Auditing — who triggered what, when, and why it failed.

That last requirement is the one that quietly killed Redis for us.

Why Redis Looked Right — And Wasn't

Redis is genuinely excellent at what it does. Push a job onto a list, have a worker BRPOP from it, and you're processing thousands of jobs per second with single-digit-millisecond latency. For a simple producer/consumer setup, nothing touches it.

The problems start the moment you need anything more than that.

Durability is a feature you bolt on, not get for free

Redis persistence is optional and awkward. RDB snapshots can lose minutes of in-flight jobs. AOF helps but tanks performance unless you tune fsync policy carefully. For us, "a webhook failed and we lost the user's job" was an unacceptable failure mode. We needed every job to survive a pod restart, a node failure, and ideally a region failure.

Postgres gives us this by default. A row in a jobs table is durable the moment COMMIT returns.

Queues need queries, not just pops

The moment we shipped the first version, support asked: Can you show me all failed jobs for user X in the last week? With Redis, this means either keeping a separate index in Postgres anyway, or doing painful SCAN operations over keyspaces. With Postgres, it's:

SELECT id, repo_url, error_message, created_at
FROM jobs
WHERE user_id = $1
  AND status = 'failed'
  AND created_at > NOW() - INTERVAL '7 days'
ORDER BY created_at DESC;

That query runs in milliseconds on millions of rows with the right indexes. No second system to keep in sync.

Operational surface area matters

Two stateful services is more than twice the work of one. Backups, monitoring, failover, version upgrades, security patches — every additional component is a thing on fire at 3am. Postgres is already in our stack for application data. Adding Redis meant new expertise, new dashboards, new runbooks.

The Architecture We Landed On

We use Postgres with a simple but battle-tested pattern: a jobs table with SELECT ... FOR UPDATE SKIP LOCKED for worker coordination. Here's the schema:

CREATE TABLE jobs (
    id            BIGSERIAL PRIMARY KEY,
    user_id       TEXT NOT NULL,
    repo_url      TEXT NOT NULL,
    status        TEXT NOT NULL DEFAULT 'pending',
    payload       JSONB NOT NULL,
    attempts      INT NOT NULL DEFAULT 0,
    max_attempts  INT NOT NULL DEFAULT 3,
    scheduled_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    started_at    TIMESTAMPTZ,
    completed_at  TIMESTAMPTZ,
    error_message TEXT,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_jobs_pending
    ON jobs (scheduled_at)
    WHERE status = 'pending';

CREATE INDEX idx_jobs_user_status
    ON jobs (user_id, status, created_at DESC);

How a worker picks up a job

WITH next_job AS (
    SELECT id
    FROM jobs
    WHERE status = 'pending'
      AND scheduled_at <= NOW()
    ORDER BY scheduled_at
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
UPDATE jobs
SET status = 'running',
    started_at = NOW(),
    attempts = attempts + 1
FROM next_job
WHERE jobs.id = next_job.id
RETURNING jobs.*;

SKIP LOCKED is the magic. Multiple workers can run this query simultaneously without blocking each other — each one gets a different row. It's been supported in Postgres since 9.5 and it's the reason this pattern works at all. No Redis Streams, no advisory locks, no separate coordination layer.

Retries with exponential backoff

When a job fails, we don't immediately requeue it. We set scheduled_at = NOW() + INTERVAL '2^attempts minutes' and bump attempts. The partial index on scheduled_at WHERE status = 'pending' keeps the worker query fast even with millions of completed jobs sitting in the table.

UPDATE jobs
SET status = 'pending',
    error_message = $2,
    scheduled_at = NOW() + (INTERVAL '1 minute' * POWER(2, attempts))
WHERE id = $1;

Visibility for users

The same table powers the DocPilot dashboard. Users see real-time status updates (poll every 2s) and a full history of every job we've ever run against their repos. We didn't have to build a separate audit log — the jobs table is the audit log.

The Numbers, Honestly

Let's talk about throughput, because that's where Redis wins on paper.

On a single small Postgres instance (db.t3.medium equivalent, 2 vCPU, 4 GB RAM), we're comfortably handling:

• 50–80 worker processes polling the queue.
• 200–400 jobs claimed per second under steady load.
• Sub-50ms p95 latency on the claim query.

For comparison, the same box running a single Redis instance and a Redis worker can claim 10,000+ jobs per second. If you're doing pure producer/consumer with no joins, no filtering, no auditing — it's not close.

But we are doing all those things. And if we'd built Redis, we'd need Postgres anyway as the system of record. So the apples-to-apples comparison is Redis + Postgres vs Postgres alone. We chose the latter.

What about the bad days?

The one thing Redis genuinely handles better is long-tail spikes. If 10,000 jobs land in the queue at once, Redis will chew through them as fast as you can spin up workers. Postgres will eventually catch up — but if you're not careful, you'll exhaust connections, bloat the table, or start vacuuming during peak.

Mitigations we've added:

• Connection pooling with PgBouncer in transaction mode.
• Aggressive partitioning by week for the jobs table — old partitions get detached and archived.
• Rate limiting at the API layer so a single user can't flood the queue.
• Worker concurrency limits so we don't over-subscribe to upstream LLM APIs.

None of this is exotic. It's all standard Postgres operations work. But it is work that didn't exist for us when we thought "just use Redis."

Tradeoffs We Accept

This isn't a universal recommendation. There are real costs to using Postgres as a queue:

Concern	Impact	Mitigation
Index bloat on busy tables	Slow vacuum, query degradation	Partition by time, run pg_repack monthly
Connection limits	Bottleneck under load	PgBouncer, keep workers stateless
Hot row contention on popular tables	Lock waits	Shard by hash(user_id), separate write DB
Backup size	Larger snapshots	Logical replication for analytics, partition pruning

For a startup running 50 jobs a minute, none of this matters. For us running thousands per hour across thousands of users, it's the operational tax we pay for not having two systems to babysit.

What We'd Still Use Redis For

To be clear: we use Redis. It's just not the job queue. We use it for:

• Rate limiting — INCR with a TTL is the canonical pattern.
• Session caching — short-lived, evictable, ephemeral by design.
• Real-time websocket fanout — pub/sub is genuinely the right primitive here.

The lesson we keep learning is that Redis is great when your data is transient and uniform. Postgres is great when your data is durable and queryable. A job queue sits awkwardly between those — and we decided durable and queryable mattered more.

The Takeaway

Defaulting to Redis for a queue is one of those decisions that feels smart and turns out to be a tax. Every property Redis gives you out of the box — speed, atomicity, simple data structures — you'll eventually need to add to whatever system of record you're also running. Better to start with the system of record and skip the second hop.

If you're building a queue that needs retries, visibility, durability, and any kind of "show me what happened" UI, Postgres with SKIP LOCKED is a remarkably capable starting point. You can always add Redis later for the parts that need it. We haven't needed to.

---

If you're building pipelines that need durable, observable job processing, DocPilot handles a lot of this for you. The docs you're reading right now were probably generated by the same Postgres-backed queue described above — and the architecture diagram in our README was rendered from the actual jobs table schema. If you're spending engineering time reinventing this for your own AI workflows, DocPilot might save you a sprint or two.