The failure that has no error message
Most failures announce themselves. A queue that will not connect throws. A provider that rejects a message returns a code you can log.
The one that does not is this: you write the message row inside a transaction, commit, then enqueue the job — and between those two steps the process dies. The row exists, status queued. No job exists. Nothing retries it, because nothing knows it should. The customer never gets their password reset, your dashboard shows a message that has been queued for nine days, and no alert fired.
Why choosing BullMQ creates it
A Postgres-backed queue such as pg-boss can enqueue inside the same transaction as your write. Commit succeeds and both exist, or it fails and neither does. That property removes this bug entirely.
BullMQ stores jobs in Redis, which cannot participate in a Postgres transaction. In exchange you get better rate limiting, better delayed-job handling, and a system that scales further before it complains. That is a reasonable trade — but it is a trade, and the outbox pattern is the price.
The outbox pattern
Write the intent to enqueue in the same transaction as the message. Then attempt the real enqueue. A sweeper picks up anything the attempt missed.
const message = await db.transaction(async (tx) => {
const [m] = await tx.insert(messages).values({ ...input, status: 'queued' }).returning();
await tx.insert(outbox).values({ messageId: m.id, queue: 'send.transactional' });
return m;
});
// Outside the transaction. If this throws, the sweeper handles it.
await sendQueue.add('send', { messageId: message.id }, { jobId: message.id });
await db.delete(outbox).where(eq(outbox.messageId, message.id));jobId is what makes it safe to re-run
Setting jobId to the message ID means BullMQ treats a duplicate add as a no-op. The sweeper can re-enqueue anything in the outbox without checking whether it already succeeded, because enqueueing twice is harmless.
This is the detail that turns the outbox from "mostly works" into something you can reason about. Without it, your recovery mechanism becomes a duplicate-email generator.
// Repeatable sweeper, every 15 seconds
const stale = await db.select().from(outbox)
.where(lt(outbox.createdAt, new Date(Date.now() - 30_000)));
for (const row of stale) {
await sendQueue.add('send', { messageId: row.messageId }, { jobId: row.messageId });
await db.delete(outbox).where(eq(outbox.messageId, row.messageId));
}The Redis setting that deletes your jobs
BullMQ stores job state in Redis hashes. If Redis is configured with any allkeys eviction policy, it will delete those hashes under memory pressure — silently, with no error on either side.
Redis running a queue is durable infrastructure, not a cache, and must be configured as such.
- maxmemory-policy noeviction. This is the critical one.
- appendonly yes, with appendfsync everysec.
- A managed instance with backups, not a container beside the app.
- Separate from your rate-limit cache — disposable high-churn data should not share memory pressure with queue state.
- Alert on used_memory above 70%. With noeviction, a full Redis stops accepting jobs.
Per-tenant fairness needs building
BullMQ open source gives you a per-queue rate limiter and per-worker concurrency, but not per-tenant groups — that is a Pro feature. On a shared platform one customer bursting a hundred thousand messages should not delay another customer's password resets.
A Redis token bucket checked in the processor covers this. The important detail is returning the job to the queue without consuming an attempt.
const allowed = await tokenBucket.take(message.accountId);
if (!allowed) {
await worker.rateLimit(1_000);
throw Worker.RateLimitError(); // requeued, attempt not burned
}Verify it by breaking it
None of this is trustworthy until it has been tested adversarially, and all of it is testable in CI.
- Kill a worker mid-job and assert the message is neither lost nor duplicated.
- Flush Redis and assert the outbox sweeper recovers every queued message.
- Replay the same idempotency key fifty times and assert exactly one send.
- Point the provider adapter at a failing stub and assert backoff, then a clean drain when it recovers.