Retries and exponential backoff in BullMQ

Building PriceWatch meant accepting an uncomfortable reality: scraping fails, often. A store changes its HTML, a proxy drops, a timeout happens mid-request. The question isn't how to avoid failures, it's how to make the system recover on its own.

Why a queue instead of a plain cron

A cron job that runs every 10 minutes and scrapes sequentially breaks as soon as you add the 41st URL: the previous job hasn't finished when the next one should start. A queue with concurrent workers decouples "when work gets scheduled" from "when it actually runs".

import { Queue } from "bullmq"
 
const scrapeQueue = new Queue("scrape-product", { connection: redisConnection })
 
await scrapeQueue.add(
  "scrape",
  { productId },
  {
    attempts: 5,
    backoff: { type: "exponential", delay: 2000 },
  },
)

Exponential backoff in practice

With delay: 2000 and exponential backoff, retries end up looking like this:

This matters because a transient failure (a network timeout, a momentary rate limit) usually resolves itself within seconds. Retrying immediately in a tight loop only makes the rate limit worse; waiting longer each time gives the problem a chance to go away.

Telling recoverable errors apart from permanent ones

Not every failure should be retried. If a store changed its HTML structure and the selector no longer exists, retrying 5 times just delays the inevitable.

worker.on("failed", (job, err) => {
  if (err instanceof SelectorNotFoundError) {
    // Structural error: retrying automatically won't help.
    job.discard()
    notifyMaintainer(job.data.productId, err)
  }
})

In practice I split errors into two categories from the start: transient (retryable, let BullMQ do its job) and structural (flagged for manual review and triggers an alert). That distinction kept the system from endlessly retrying jobs that were never going to succeed.