Engineering notes — 2026-11-03

Production AI Agents in Node.js: What Actually Breaks at Scale

The gap between an AI demo and a production AI system is not model quality; it is everything around the model. This is what actually breaks when agent-like AI workloads go to production, drawn from building RealStream, an event-driven platform that schedules thousands of concurrent jobs on Node.js.

There is a lot of advice about building AI agents, and most of it covers the 10% that works on the first try: calling a model, streaming a response, wiring a tool. The hard 90% is what happens after that, and almost none of it is about prompts. It is queues, failure contracts, cost accounting, and state. Here is the production checklist I wish I had before shipping RealStream, an AI media orchestration platform running thousands of concurrent jobs on NestJS and BullMQ.

Break #1: treating long AI calls like HTTP requests

A generative video job can run from seconds to tens of minutes. If your web handler awaits that call, you have built a system that times out under normal operation. The first architectural decision in any serious AI backend is the boundary between synchronous requests and queued jobs: user input becomes a job with an ID immediately, and everything after that is asynchronous. In RealStream, every submission is a BullMQ job whose lifecycle has a defined timeout, retry policy, and failure state. The API route returns a job ID in milliseconds; the WebSocket channel tells the user what stage the job reached.

Break #2: assuming the provider is up

Model providers fail mid-flight, rate-limit exactly during your traffic peak, and change latency profiles without notice. A production integration layer assumes failure as the default case:

• Bounded, tiered retries: a 429 backs off in minutes, a malformed payload fails on the first attempt. Unbounded default retries turn one provider outage into a queue collapse.
• Artifact cleanup: half-finished generations must not leak storage or block retries forever.
• A provider-agnostic seam: swapping or adding a model provider must not touch the orchestration pipeline.

Break #3: no per-job cost accounting

AI unit economics are invisible unless you record them per job: which prompt, which model version, how many tokens or GPU-seconds, what the artifacts cost in storage. Without that ledger, the first time you learn your cost per successful output is when the invoice arrives. In RealStream the queue job is the natural accounting unit: it already carries the payload, the status, and the artifacts, so it carries the cost line too.

Break #4: state that evaporates when a worker dies

Workers will die: deploys, OOM kills, node crashes. If job state lives in worker memory, every deploy loses everything in flight. The pattern that survives production: state in Redis, artifacts in object storage (S3), and a stalled-job detector that re-queues work from workers that stopped heartbeating. Then you can kill any worker at 3 a.m. and nothing is lost. That is the test, and you should run it on purpose before your users run it by accident.

Break #5: a spinner as the status UX

For jobs that can run longer than a coffee break, a spinner is a lie. Users need the actual stage: queued, in progress, which step, failed with a reason. Publish stage transitions as queue events and stream them through Redis pub/sub to the WebSocket gateway, and the frontend becomes a live dashboard of truth. The same event stream is the cheapest monitoring you will ever get: job throughput and failure reasons are already flowing through it.

The minimal production shape

// NestJS service: submit a job — the synchronous part ends here
@Post('renders')
async submit(@Body() dto: RenderDto) {
  const job = await this.renderQueue.add('render', dto, {
    priority: 3,
    attempts: 5,
    backoff: { type: 'exponential', delay: 2_000 },
    timeout: 30 * 60 * 1_000, // 30 min ceiling
    removeOnComplete: { count: 1_000 },
  });
  return { jobId: job.id, status: 'queued' };
}

// Worker: one stage of the pipeline, assumes it can die anytime
new Worker('render', async (job) => {
  const artifacts = await this.provider.generate(job.data, {
    onStage: (stage) => publishStage(job.id, stage), // live status
  });
  await this.s3.put(key(job.id), artifacts);       // state outside worker
  return { key: key(job.id), costCents: this.costMeter(job) };
}, { connection, concurrency: 4 });

That is the whole shape: a queue job with a contract, a stateless worker that assumes death, events that become user-facing status, and a cost meter on every job. Nothing here is exotic; the discipline is treating every AI call like the distributed-system operation it actually is. If your team is moving an AI feature from demo to production and wants a second opinion on the architecture, I consult on exactly this: reach out on Telegram or by email.