Ray Nirola
All posts

Queues · 6 min read

Lanes, fan-out, and the job ID that swallowed our retries

Splitting one BullMQ worker into lanes, fanning work out per unit, and the deduplication bug that silently dropped re-scheduled jobs.

Ray Nirola

Founding Engineer at Roger, 2025–2026

Most of what Roger does happens in the background. Sending scheduled email steps, polling thousands of inboxes for replies, generating personalized sequences with LLMs, rolling up campaign stats, rendering ad creative: all of it runs as BullMQ jobs on Redis, in a Node.js worker that sits beside the web app.

It started as one process that ran every worker. This post covers how that grew into something we could reason about, and the handful of queue mistakes that taught me the most along the way.

One codebase, many lanes

A single process running every worker is the right place to start, and it stops being right sooner than you expect. A video render and a batch of LLM calls compete for the same memory and event loop as the job that sends email. When the generation queue backed up, sending slowed down with it, and a memory spike in one workload could restart all of them.

The fix was to keep one codebase but run it as separate lanes. Each deployed service sets a QUEUE_TYPE, and the runtime loads only the workers for that lane:

const lanes = {
  outbound: () => import("./lanes/outbound.js"),
  inbound: () => import("./lanes/inbound.js"),
  generation: () => import("./lanes/generation.js"),
  // …
};

const lane = process.env.QUEUE_TYPE;
if (!lane || !(lane in lanes)) {
  throw new Error(`QUEUE_TYPE must be one of: ${Object.keys(lanes).join(", ")}`);
}
await lanes[lane]();

A lane file is nothing more than a list of worker imports. Two details made this work well:

Each lane deploys as its own service from the same repository, so heavy dependencies such as a headless browser and FFmpeg only ship in the image that needs them. Concurrency is set per lane from measurement: when inbox polling fell behind, we compared completed jobs per minute with the number of mailboxes per cycle and raised it to match, rather than guessing.

Fan out, don't loop

The first version of the sender was a single repeatable job that walked every due sequence step. It worked until it didn't: one run could hold the entire send workload for hours, one slow mailbox delayed everyone behind it, and a crash meant starting again from the top.

Now nearly every scheduled workload is a parent that fans out. A scheduled parent job queries the database, then adds one small child job per unit of work in bulk: one per sequence, per mailbox, or per campaign. The parent runs with a concurrency of one and finishes quickly. The children are independently retried, independently observable, and spread across as many processes as the lane has.

Where a child's failure shouldn't fail its parent, such as one mailbox's sync breaking, we use BullMQ flows with failParentOnFailure: false.

Send children are a special case: they run with a single attempt. If a send can't go out, the step returns to SCHEDULED and the next planner run picks it up again. Whether and when to retry an email is a business decision with rules about pacing and capacity, so it lives in the domain model, not in the queue's backoff settings.

Producers follow one rule too. The web app and the worker enqueue through a single queue API that validates every payload against a schema. The only exception is a running worker adding its own child jobs. That costs some duplicated payload types across repositories, and it buys exactly one place to look when asking "who puts jobs on this queue?".

Waiting is not failing

Generating a sequence needs a mailbox with room to send from. Early on, when none was available, the job threw. BullMQ retried it, the retries ran out, and the failure alert fired, for a situation that would resolve on its own in a few minutes.

BullMQ has a better tool for this. A job can move itself back to the delayed set and throw DelayedError, which neither consumes an attempt nor counts as a failure:

if (!mailbox) {
  await job.moveToDelayed(Date.now() + 15 * 60_000, token);
  throw new DelayedError();
}

The distinction matters more than it looks. Once "waiting for capacity" stopped showing up as a failure, the failure alerts went back to meaning something.

The job ID that swallowed our retries

Custom job IDs are the usual way to deduplicate in BullMQ: give the job for step 42 the ID send-step-42, and a second add while the first is queued is ignored.

The catch is that BullMQ ignores the add whenever a job with that ID exists, and that includes completed and failed jobs you're keeping for history. Our step failed, the job stayed in the failed set for inspection, the step was rescheduled, and the next add for send-step-42 silently did nothing. No error, no log line, just a step that never sent until the old job aged out of retention.

The fix came in two stages. First, removeOnComplete on those children, so successful runs stopped holding their IDs. Later, BullMQ's deduplication option, whose ID is released as soon as the job finishes. That gives deduplication while a job is in flight, which is what we actually wanted, without colliding with history.

A smaller lesson from the same area: don't put colons in custom job IDs. BullMQ uses colons to build its Redis keys, and we had to rename every ID to use dashes.

The general rule I took away: a deduplication key needs a lifetime that matches the window you want to deduplicate over. "Forever, or until retention cleans it up" is almost never that window.

The database is the source of truth

BullMQ delivers jobs at least once. A stalled job is retried, deploys overlap old and new processes for a few seconds, and a parent can run again before its last batch of children finishes. So every side effect claims its work in PostgreSQL before doing it:

That foundation is also why the workers have very little shutdown ceremony. Deploys drain with an overlap window, stalled jobs are recovered, and the claims make a second run harmless. Each piece is simple; together they mean a killed process is an inconvenience, not an incident.

Transactions needed the same care. One serializable transaction was configured to retry on conflict up to a hundred times. Under contention it replayed for minutes and held database connections while doing it, until the pool ran dry and unrelated jobs started timing out. We capped the retries, moved the hot shared-row update out of the transaction, and shortened its timeout. Retry loops need a budget, especially the ones inside your database.

What I'd keep

If I were starting the same system again, I'd do these from day one:

None of this is exotic. Most of it is what you learn the first time a queue does exactly what you told it to.

All posts