Engineering notes — 2026-10-27

NestJS + BullMQ at Scale: What Breaks First and How to Fix It

BullMQ breaks in predictable places. The first failures I see at scale are retry storms from unbounded backoff, jobs lost to consumers dying mid-flight, and short jobs starving behind long ones. All three are fixed by design decisions you make before the first deploy, not after an outage.

This article comes from building RealStream, an AI media orchestration platform where every user submission is an inference job that can run from seconds to tens of minutes. The production MVP shipped in 60 days on NestJS and BullMQ, and it has been scheduling thousands of concurrent jobs since. Here is what I got right early, and what I have watched break in systems built without these decisions.

What breaks first, in order

• Retry storms. A provider goes down, every failed job retries on the default schedule, and the retry volume at peak failure is a multiple of your normal load. Your outage becomes a self-inflicted DDoS against your own queue and the already-degraded provider.
• Jobs lost mid-flight. A worker process dies during a long inference call. Without explicit handling, the job sits in an uncertain state: either stuck active forever, or duplicated if someone re-enqueues it blindly.
• Starvation. Batch renders occupy every worker while interactive users wait. Fairness inside a single queue does not exist; it has to be designed.
• Silent queues. Nothing breaks, but jobs slowly pile up because consumers are throughput-bound and nobody has a number that says so.

Fix 1: interview every job at design time

A job is not a fire-and-forget function call. Before writing the processor, answer four questions for every job type: Is the operation idempotent? What is its timeout? What does one retry mean, and how many retries make sense? What state does the job leave behind if it dies halfway?

Fix 2: bounded, tiered retry policy

Default BullMQ retry behavior is fine for demos and dangerous for incidents. Set explicit attempts with capped exponential backoff, and separate failure types: a 429 from the provider should back off in minutes, a malformed payload should fail on the first attempt.

const inferenceQueue = new Queue('inference', {
  connection, // shared ioredis connection
})

await inferenceQueue.add('render', payload, {
  priority: 3, // lower number = higher priority; batch renders run behind default jobs
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 2_000, // 2s, 4s, 8s, 16s, 32s
  },
  removeOnComplete: { count: 1_000 },
  removeOnFail: { age: 24 * 3600 },
})

const worker = new Worker('inference', processor, {
  connection,
  // hard ceiling on concurrent AI calls per worker
  concurrency: 4,
})

Fix 3: multiple queues, not one

One queue for everything is the single most common mistake. Separate queues per workload class, so interactive and batch traffic physically cannot starve each other. In RealStream, status-critical jobs and long batch renders run in different lanes with different concurrency and priorities. This is the cheapest scaling decision you will ever make.

Fix 4: make death boring

Workers will die: deploys, OOM, node crashes. Design so that a dead worker is a non-event. State lives in Redis, artifacts live in object storage (S3 in our case), and a stalled-job detector re-queues work from workers that stopped heartbeating. The result: any worker can be killed and replaced at any time, at 3 a.m., without a runbook.

Fix 5: events are the product

Users of a pipeline care about one thing: where is my job. Publish stage transitions as BullMQ job events, stream them through Redis pub/sub to the WebSocket gateway, and the frontend renders truth instead of a spinner. This also gives you free operational insight: queue event streams are the cheapest monitoring you will ever wire.

What this buys you

In RealStream's numbers: production in 60 days from first commit, thousands of concurrent inference jobs, and job status visible to every user in real time. No Kafka, no extra infrastructure: one Redis instance carries the whole queue layer. BullMQ's speed-of-development profile is exactly right for teams that need production-grade event-driven systems without operating a streaming platform.

When not to use BullMQ

Be honest about the ceiling. BullMQ assumes Redis is durable enough for your requirements, and that one Redis node (or a managed one) can absorb your write rate. If you need replayable event logs, exactly-once semantics across services, or multi-dc fan-out, you want Kafka-class infrastructure, not more retries. Most product teams debating this on paper are not in either regime yet; measure, then choose.

I write about real production systems at safesync.dev. If your team is designing a queue layer right now and wants a second opinion on the architecture, I consult on exactly this: reach out on Telegram or by email.