Building a job queue with BullMQ v5 to defend against LLM API quota overruns — concurrency limits, priorities, and exponential backoff in real code

If you only attach setTimeout retry logic to try/catch on a server that calls the LLM API directly, 429 Too Many Requests errors will flood in as soon as traffic picks up even slightly. Scaling Workers horizontally actually worsens quota overruns, synchronous retry loops burn through Lambda costs, and priority-handling code fails to prevent starvation — where batch jobs crowd out real-time requests.

BullMQ v5 has tools that address each of these problems individually. Global Concurrency atomically caps the total number of concurrent requests at the Redis level regardless of how many Workers you scale to, a priority queue prevents premium user requests from being pushed back by batch jobs, and exponential backoff + Worker.RateLimitError() handles 429 responses gracefully.

We'll walk through everything in one flow: from queue setup to dynamic 429 backoff, stalled job defense, and a multi-queue separation architecture. The examples are in TypeScript, but apply equally to JavaScript.

Core Concepts

Why You Need a Queue in Front of the LLM API

LLM providers impose limits along two axes: RPM (requests per minute) and TPM (tokens per minute). Even if you reduce the number of concurrent requests, sending long prompts back-to-back will hit the TPM limit first.

When you run multiple Worker instances, lowering each Worker's local concurrency doesn't help — the total combined count still exceeds your quota. Before v5, there was no way to prevent this at the BullMQ level.

queue.setGlobalConcurrency(N), added in v5, atomically limits the total number of concurrent jobs across the entire Worker cluster at the Redis level. No matter how many Workers there are or which servers they run on, the combined total never exceeds N.

Concurrency Limits: Local vs. Global

There are two concurrency controls, and they serve different roles.

Both limits apply simultaneously. The actual processing count is determined by whichever is lower. Even if 3 Workers each have concurrency: 5, setting setGlobalConcurrency(5) means only 5 jobs are processed concurrently across all of them. Global concurrency is essential for external services like the LLM API that have a global quota.

import { Queue, Worker, Job } from 'bullmq';
import { Redis } from 'ioredis';
 
// ioredis automatically retries failed commands by default,
// which conflicts with BullMQ's blocking commands and causes unexpected timeouts.
// When using ioredis with BullMQ, this must be set to null.
const connection = new Redis({ maxRetriesPerRequest: null });
 
const llmQueue = new Queue('llm-requests', {
  connection,
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: 'exponential',
      delay: 2000, // 2s → 4s → 8s → 16s → 32s (5 retries)
    },
    removeOnComplete: { count: 500 },
    removeOnFail: { count: 200 },
  },
});
 
// Process at most 5 Jobs concurrently across the entire cluster
await llmQueue.setGlobalConcurrency(5);

Priority: Splitting SLAs Within a Single Queue

BullMQ priorities are integers between 1 and 2,097,152, where lower numbers mean higher priority. Think of it as "1 is first place." Jobs added without a priority are generally processed first due to BullMQ's internal implementation, but this is not officially guaranteed behavior.

Priority is closer to a weighting than a strict ordering guarantee. If batch jobs must never be processed ahead of real-time requests without exception, the multi-queue separation approach covered later is safer.

Exponential Backoff Retry: Gracefully Surviving 429s

Exponential backoff increases the retry interval using the formula 2^(attemptsMade-1) × delay. With delay: 2000, the sequence is 2s → 4s → 8s → 16s → 32s.

When multiple Workers retry at the same interval simultaneously, a thundering herd occurs — retries all pile up at once, triggering another wave of 429s in a vicious cycle. You can register a custom strategy via settings.backoffStrategy to give each Job a different delay.

// The processor function is declared before Worker creation; the worker variable is assigned below.
let worker: Worker;
 
const processor = async (job: Job<{ prompt: string; model: string }>) => {
  try {
    return await callLLMApi(job.data);
  } catch (err: any) {
    if (err.status === 429) {
      const retryAfterRaw = err.headers?.['retry-after'] ?? '30';
      const retryAfter = parseInt(retryAfterRaw, 10);
      // The Retry-After header can also arrive in HTTP-date format,
      // in which case parseInt returns NaN.
      const waitMs = Number.isNaN(retryAfter) ? 30_000 : retryAfter * 1000;
      await worker.rateLimit(waitMs);
      throw Worker.RateLimitError(); // Only works when thrown inside the processor
    }
    throw err;
  }
};
 
worker = new Worker('llm-requests', processor, {
  connection,
  concurrency: 3,
  settings: {
    backoffStrategy: (attemptsMade: number) => {
      const base = Math.pow(2, attemptsMade) * 1000;
      const jitter = Math.random() * 1000; // 0–1 second random jitter
      return base + jitter;
    },
  },
});
 
await llmQueue.add('generate', data, {
  attempts: 6,
  backoff: { type: 'custom' },
});

When a 429 response arrives, the Worker itself is paused with worker.rateLimit(ms), and Worker.RateLimitError() is thrown to return the current Job to the front of the queue.

Important: Worker.RateLimitError() must be thrown inside the processor function. Throwing it inside a worker.on('failed', ...) event listener has no effect, because by that point the Job has already been marked as failed.

Practical Application

Basic Queue Setup and OpenAI Integration

Here's a Worker example with a real OpenAI API integration. The limiter option provides a first line of defense on requests per minute, and setGlobalConcurrency caps concurrent processing.

import { Queue, Worker, QueueEvents, Job } from 'bullmq';
import OpenAI from 'openai';
import { Redis } from 'ioredis';
 
const connection = new Redis({ maxRetriesPerRequest: null });
const openai = new OpenAI();
 
const llmQueue = new Queue('llm-requests', {
  connection,
  defaultJobOptions: {
    attempts: 6,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: { count: 500 },
    removeOnFail: { count: 200 },
  },
});
 
await llmQueue.setGlobalConcurrency(5);
 
let worker: Worker;
 
const processor = async (job: Job<{ prompt: string; model: string }>) => {
  const { prompt, model } = job.data;
  try {
    const response = await openai.chat.completions.create({
      model,
      messages: [{ role: 'user', content: prompt }],
    });
    return response.choices[0].message.content;
  } catch (err: any) {
    if (err.status === 429) {
      const retryAfterRaw = err.headers?.['retry-after'] ?? '30';
      const retryAfter = parseInt(retryAfterRaw, 10);
      // If Retry-After is in HTTP-date format, parseInt returns NaN.
      const waitMs = Number.isNaN(retryAfter) ? 30_000 : retryAfter * 1000;
      await worker.rateLimit(waitMs);
      throw Worker.RateLimitError();
    }
    throw err;
  }
};
 
worker = new Worker('llm-requests', processor, {
  connection,
  concurrency: 3,
  limiter: {
    max: 60,          // Max 60 requests per minute
    duration: 60_000,
  },
  lockDuration: 90_000,
});

The 429 Response Flow — What Actually Happens

Worker.RateLimitError() does not consume a retry attempt (attempts). This is because the 429 was caused by exceeding a quota limit, not a failure of the Job itself.

A note on multi-Worker environments: worker.rateLimit(ms) only pauses that specific Worker instance. If multiple Workers receive a 429 simultaneously, each pauses independently, and if their resume timings overlap, another burst of 429s can follow. A custom backoffStrategy with jitter spreads out retry timings during this window as well.

Priority-Based User Tier Processing

// Premium users: processed immediately
await llmQueue.add('generate', { userId, prompt }, {
  priority: 1,
  attempts: 6,
  backoff: { type: 'exponential', delay: 1000 },
});
 
// Free user real-time requests
await llmQueue.add('generate', { userId, prompt }, {
  priority: 10,
  attempts: 6,
  backoff: { type: 'exponential', delay: 2000 },
});
 
// Nightly batch jobs
await llmQueue.add('batch-generate', { userId, prompt }, {
  priority: 100,
  attempts: 3,
  backoff: { type: 'exponential', delay: 3000 },
});

Stalled Job Defense — Handling LLM Timeouts

LLM inference can take a long time. If a response doesn't arrive within lockDuration, BullMQ marks the Job as stalled and lets another Worker pick it up — which is where the same Job can end up being processed twice.

const worker = new Worker('llm-requests', processor, {
  connection,
  concurrency: 3,
  lockDuration: 90_000,    // Maximum model response time + buffer
  maxStalledCount: 2,      // Force fail after 2 stalled occurrences
  stalledInterval: 30_000, // Check for stalled jobs every 30 seconds
});
 
worker.on('stalled', (jobId: string) => {
  console.warn(`Job ${jobId} stalled — returning to retry queue`);
});

lockDuration should be set based on the maximum expected response time for your model. Too short causes false stalled positives; too long delays detection of actual Worker crashes.

Multi-Queue Separation Architecture and Monitoring

The most reliable way to eliminate slow batch jobs blocking real-time requests is to separate them into distinct queues entirely. Jobs that exhaust their maximum retries are handled via a Dead Letter Queue pattern — listening for the failed event and manually adding them to a separate analysis queue.

const priorityQueue = new Queue('llm-priority-queue', { connection });
const dlqQueue = new Queue('llm-dlq', { connection });
 
const priorityWorker = new Worker('llm-priority-queue', processor, {
  connection,
  concurrency: 3,
  limiter: { max: 40, duration: 60_000 },
  lockDuration: 90_000,
  maxStalledCount: 2,
});
 
const batchWorker = new Worker('llm-batch-queue', processor, {
  connection,
  concurrency: 1,
  limiter: { max: 10, duration: 60_000 },
  lockDuration: 120_000,
  maxStalledCount: 2,
});
 
// Attach a DLQ handler to each queue using the same pattern
const priorityQueueEvents = new QueueEvents('llm-priority-queue', { connection });
 
priorityQueueEvents.on('failed', async ({ jobId, failedReason }) => {
  const job = await Job.fromId(priorityQueue, jobId);
  if (job && job.attemptsMade >= (job.opts.attempts ?? 1)) {
    await dlqQueue.add('failed-job', {
      originalData: job.data,
      reason: failedReason,
      source: 'llm-priority-queue',
    });
  }
  alertSlack(`Job ${jobId} failed: ${failedReason}`);
});
 
priorityQueueEvents.on('stalled', ({ jobId }) => {
  alertSlack(`Job ${jobId} stalled in llm-priority-queue — please investigate`);
});

Pros and Cons

Advantages

Drawbacks and Caveats

Two Common Mistakes in Production

1. Throwing Worker.RateLimitError() inside worker.on('failed')

The failed event fires after the Job has already been marked as failed, so throwing inside it has no effect on the queue.

// Wrong — has no effect
worker.on('failed', async (job, err: any) => {
  if (err.status === 429) {
    throw Worker.RateLimitError(); // Does not work here
  }
});
 
// Correct — handle inside the processor
const processor = async (job: Job<{ prompt: string; model: string }>) => {
  try {
    return await callLLMApi(job.data);
  } catch (err: any) {
    if (err.status === 429) {
      await worker.rateLimit(30_000);
      throw Worker.RateLimitError();
    }
    throw err;
  }
};

2. Using exponential backoff without jitter

Setting only type: 'exponential' causes all Jobs that failed simultaneously to retry at the exact same moment. If 10 Workers all receive a 429 and retry 8 seconds later at once, another wave of 429s hits immediately. Adding jitter via settings.backoffStrategy spreads out the retry timing.

Conclusion

Three things are key.

• setGlobalConcurrency(N): No matter how many Workers you add, the total concurrent requests will never exceed N. This is the cornerstone of LLM API quota defense.
• Worker.RateLimitError() inside the processor: The correct way to handle a 429 — returns the Job to the front of the queue without consuming a retry attempt.
• Custom backoffStrategy + jitter: Spreads out retry timing across multiple Workers to prevent thundering herd.

1. Wrap your existing synchronous LLM call code in a BullMQ Queue + Worker. Start by verifying basic quota defense with setGlobalConcurrency(5) and limiter settings.
2. Add a 429 handling branch inside the processor. Read the retry-after header value and pass it to worker.rateLimit().
3. Spin up a dashboard with @bull-board/express. Visualizing Job status lets you immediately see which Jobs are piling up and where failures occur.

TPM limit handling cannot be solved with a request-count-based limiter alone. A realistic approach is to pre-estimate token counts from prompt length and use that as a weight in a custom limiter — and this logic must be applied before the processor is entered to be effective.

References

• BullMQ Official Docs — Concurrency
• BullMQ Official Docs — Global Concurrency
• BullMQ Official Docs — Prioritized Jobs
• BullMQ Official Docs — Rate Limiting
• BullMQ Official Docs — Retrying Failing Jobs
• BullMQ Official Docs — Stalled Jobs
• BullMQ v5 Migration Notes
• BullMQ Parallelism and Concurrency — Official Guide
• BullMQ Custom Backoff Strategy
• Advanced BullMQ Features: Rate Limiting, Job Retries, and Concurrency — GoPenAI Blog
• LLM API Rate Limiting Best Practices — ClawPulse Blog
• Rate-Limit Recipes in Node.js using BullMQ — Taskforce Blog
• BullMQ Production Architecture: Patterns That Hold at 500 Jobs/Second — Markaicode
• Reliable Job Queue with BullMQ and Redis: AI Task Processing — Markaicode
• How to Implement Dead Letter Queues in BullMQ — OneUptime Blog
• BullMQ 5 Background Jobs in Node.js 2026 Guide — 1xAPI
• GitHub: taskforcesh/bullmq — Releases