Queues & background workers

01Why queues: producers, consumers and brokers

Picture a restaurant ticket rail. The waiter (producer) pins an order ticket and goes back to customers; cooks (consumers) take tickets when they are free. If ten orders arrive at once, tickets wait on the rail instead of the waiter freezing. In software the rail is a broker: Redis, a database table, RabbitMQ or SQS. Your web app is the producer; worker processes are consumers. Benefits: fast responses, smoothing traffic spikes, isolating failures of slow services, and scaling workers independently from web servers. The cost: work becomes asynchronous, so you need to think about retries, duplicates, ordering and monitoring.

• Job: a small message with a type and data, ideally just ids, not whole objects.
• Queue names separate priorities: critical, default, emails, reports.
• Concurrency: how many jobs a worker runs at the same time.

02Retries, backoff and dead-letter queues

Jobs fail: a payment API times out, a database deadlocks, a third party returns 503. Retry, but carefully. Exponential backoff waits longer each time (for example 10 seconds, 1 minute, 5 minutes), giving the other side time to recover; jitter spreads retries so they do not all hit at once. Limit attempts. Distinguish errors: a timeout is worth retrying; a validation error ('customer does not exist') will fail forever, so mark it failed immediately. After the last attempt, move the job to a dead-letter queue (in Laravel, the failed_jobs table; in BullMQ, the failed set). Somebody must look at it: alert on failure counts and build a way to inspect and replay jobs after fixing the cause.

1class SyncOrderToErp implements ShouldQueue
2{
3    use Queueable;
4 
5    public int $tries = 5;
6    public int $timeout = 60;
7 
8    public function __construct(public int $orderId) {}
9 
10    public function backoff(): array
11    {
12        return [10, 60, 300, 900];
13    }
14 
15    public function handle(ErpClient $erp): void
16    {
17        $order = Order::findOrFail($this->orderId);
18        $erp->upsertOrder($order); // idempotent by order id on the ERP side
19    }
20}

03Idempotent jobs: safe to run twice

Most queues guarantee at-least-once delivery: a job can run twice, for example if a worker crashes after doing the work but before acknowledging it. So every job must be idempotent, meaning running it again has no extra effect. Techniques: use a natural unique key (create the invoice for order 123 only if it does not exist), store a processed-marker with a unique constraint, use upserts instead of inserts, and pass an idempotency key to payment providers. Check state before acting ('already sent?'). Exactly-once delivery is largely a myth across systems; exactly-once effect comes from idempotent handlers. Also avoid a classic race: dispatch jobs only after the database transaction commits, otherwise a worker may pick up a job for a row that does not exist yet.

04BullMQ in Node

BullMQ is the standard Redis-based queue for Node. A Queue adds jobs, a Worker processes them with a set concurrency, and options control attempts, backoff, delays, priority and cleanup. Set a jobId to deduplicate: adding a job with an existing id is ignored. Repeating work uses job schedulers (upsertJobScheduler) with a cron pattern or interval; in BullMQ v6 the old repeat option was removed in favour of schedulers. Run workers as separate processes from the API so heavy jobs never slow down requests, and give each queue its own concurrency. Redis must be configured with maxmemory-policy noeviction, otherwise Redis may silently delete jobs.

1import { Queue, Worker } from 'bullmq';
2const connection = { host: '127.0.0.1', port: 6379 };
3 
4export const webhooks = new Queue('webhooks', { connection });
5 
6await webhooks.add('deliver', { eventId }, {
7  jobId: 'evt-' + eventId,              // dedupe
8  attempts: 8,
9  backoff: { type: 'exponential', delay: 5_000 },
10  removeOnComplete: 1000,
11  removeOnFail: 5000,
12});
13 
14new Worker('webhooks', async (job) => deliverWebhook(job.data.eventId), {
15  connection,
16  concurrency: 10,
17});

05Scheduling and running workers in production

Scheduled work (nightly reports, expiring subscriptions, hourly syncs) runs on a timer. Classic cron runs a command at fixed times; Laravel's scheduler needs one cron entry and defines the rest in code; BullMQ schedulers store schedules in Redis. Make scheduled jobs safe to overlap or prevent overlap explicitly, and run them on one server only. Workers are long-running processes, so a process manager must keep them alive: supervisor or systemd for Laravel workers, pm2 or systemd for Node, or one container per worker type in Docker. Restart workers on every deploy so they load new code, and give them graceful shutdown so a job in progress can finish.

1; /etc/supervisor/conf.d/app-worker.conf
2[program:app-worker]
3command=php /var/www/app/artisan queue:work redis --queue=critical,default --sleep=3 --tries=3 --max-time=3600
4process_name=%(program_name)s_%(process_num)02d
5numprocs=2
6user=app
7autostart=true
8autorestart=true
9stopwaitsecs=3600
10stdout_logfile=/var/log/app-worker.log

06Webhooks in and out

Incoming webhooks (from payment providers, ERPs, delivery services): verify the signature, store the raw event, respond 200 quickly, and process it in a job. Providers retry, so deduplicate by their event id. Never trust the payload alone for money: re-fetch the payment status from the provider's API. Outgoing webhooks (to your partners): write the event to an outbox table inside the same transaction as the business change, let a worker deliver it, sign the body with HMAC-SHA256 and a timestamp so receivers can verify it and reject replays, retry with backoff for a day or more, and show delivery logs to the partner. This outbox pattern guarantees you never notify about something that was rolled back, nor lose an event that was committed.

07How a job is 'reserved': visibility timeouts, locks and stalled jobs

When a worker takes a job, the broker does not delete it. It hides it for a while and waits for a confirmation. If no confirmation arrives in time, it assumes the worker died and hands the job to someone else. In SQS this is the visibility timeout. In Laravel it is retry_after in config/queue.php: if a job is still running after that many seconds, another worker starts it again, so the job's timeout must always be shorter than retry_after, or long jobs run twice in parallel. BullMQ uses a lock that the worker keeps renewing while it processes; if the lock expires, for example because CPU-heavy code blocked the event loop, the job is marked stalled and moved back to waiting, and after too many stalls it fails. All three designs give at-least-once delivery, which is exactly why idempotency is required.

1// config/queue.php
2'redis' => [
3    'driver' => 'redis',
4    'connection' => 'default',
5    'queue' => env('REDIS_QUEUE', 'default'),
6    'retry_after' => 960,     // longer than the slowest job's timeout
7    'block_for' => 5,
8    'after_commit' => true,   // push only after the DB transaction commits
9],
10 
11// Slow export job: public $timeout = 900; run it on a worker
12// started with --timeout=900, on its own queue.

08Controlling concurrency: uniqueness, overlap and rate limits

Retries and parallel workers create new problems: two syncs for the same customer racing each other, or a burst of jobs hitting a partner's rate limit. Laravel gives you building blocks. ShouldBeUnique ignores a new dispatch while an identical job, by uniqueId, is still pending. The WithoutOverlapping job middleware lets only one job per key run at a time and releases the others back to the queue. RateLimited applies a named rate limiter to jobs, and ThrottlesExceptions backs off after repeated failures, acting like a circuit breaker for a flaky API; pair it with retryUntil so time, not attempts, limits retries. Unique and overlap locks need a cache store with atomic locks, such as Redis or the database. In BullMQ, the deduplication option ignores repeated jobs with the same id, and a worker limiter caps jobs per time window.

1class SyncCustomerToCrm implements ShouldQueue, ShouldBeUnique
2{
3    use Queueable;
4 
5    public function __construct(public int $customerId) {}
6    public function uniqueId(): string
7    {
8        return (string) $this->customerId;
9    }
10 
11    public function middleware(): array
12    {
13        return [
14            new WithoutOverlapping('crm-'.$this->customerId),
15            new ThrottlesExceptions(10, 5 * 60), // 10 errors, then pause 5 min
16        ];
17    }
18 
19    public function retryUntil(): DateTime { return now()->addHours(6); }
20}

09Workflows: chains, batches and parent-child flows

Real work is often several jobs with dependencies. A chain runs jobs one after another and stops if one fails: generate invoice, then send it, then notify the accountant. A batch runs many jobs in parallel and calls back when they all finish, which suits imports and exports split into thousands of small jobs; Laravel tracks progress in a job_batches table and jobs in it use the Batchable trait. In BullMQ, a FlowProducer creates parent-child trees where the parent runs only after all children complete, and the parent can read the children's results. Design each step to be idempotent and small, so retrying one step does not redo the whole workflow. When a later step fails after earlier ones changed external systems, you need explicit compensation, such as cancelling a reservation, because a queue cannot roll back a payment.

1$batch = Bus::batch(
2    $orderIds->map(fn (int $id) => new ExportOrderRow($exportId, $id))->all()
3)
4    ->name('orders-export-'.$exportId)
5    ->allowFailures()
6    ->then(fn (Batch $batch) => BuildExportFile::dispatch($exportId))
7    ->catch(fn (Batch $batch, Throwable $e) => report($e))
8    ->finally(fn (Batch $batch) => CleanupExportChunks::dispatch($exportId))
9    ->onQueue('reports')
10    ->dispatch();
11 
12// Progress for the UI: Bus::findBatch($batch->id)->progress()

10Capacity and observability: depth, age and Little's law

How many workers do you need? Little's law gives a first answer: the average number of jobs in progress equals the arrival rate times the average processing time. If 30 jobs arrive per second and each takes 200 ms, you need at least 6 concurrent slots just to keep up, and more for spikes and retries. Queue depth alone is misleading: 10,000 tiny jobs may clear in seconds, while 50 slow ones may be hours behind. The most useful single metric is the age of the oldest waiting job per queue, alerted against a target such as 'emails within 60 seconds'. Also track throughput, failure rate and processing time percentiles. Horizon can balance workers between queues automatically, and php artisan queue:monitor can fire an event when a queue grows beyond a threshold. In multi-tenant systems, watch fairness: one tenant's import should not delay everyone's receipts.

• Autoscale workers on oldest-job age, not CPU.
• Separate queues per latency target: critical, default, bulk.
• Cap per-tenant concurrency for heavy job types.