Skip to main content

Queue consumers & background processing

Asynchronous work on the B2B Platform — bulk edits, search re-indexing, webhook delivery — runs through durable, Redis-backed queues consumed by workers. This page explains how to run those consumers and how the platform's current queue-backed operations behave.

The design follows the platform's scalable queue consumer rule. The binding invariant is:

  • the queue is a durable, distributed substrate (Redis / BullMQ-class), not an in-memory list bound to one process;
  • every job is claimed atomically so that N ≥ 2 consumer instances never double-process a job, and handlers are idempotent under retries;
  • the component that enqueues (an HTTP handler, an event subscriber, a scheduler) is the producer — it only enqueues and returns, never inline-executing the job on the request path;
  • the consumer is a separable worker entrypoint — a process that can be started on its own and scaled to multiple instances without code changes.

Running the consumers​

Workers and the API share one composition (the same module wiring), so the consumer needs no separate service graph. Which role a process plays is chosen by the BACKEND_ROLE environment variable:

BACKEND_ROLEProcess behaviour
unset / all (default)API and co-located workers — the single-VPS default
apiHTTP only; runs no consumers (pair it with a separate worker process)
workerConsumers only; serves no HTTP (set automatically by the worker entrypoint)

Single deployable (default)​

For a single VPS, run only the API process — it co-locates the workers:

# In the root of an instance:
pnpm run start # production (built)
pnpm run dev # development (watch)

# In a checkout of the Endora Commerce repository:
pnpm --filter backend run start
pnpm --filter backend run dev

This is the posture the rule allows for low-volume work: the worker stays a separable entrypoint but is hosted in the API process to keep the deployment simple.

Separate, independently scalable workers​

When bulk edits, imports, or re-indexing get heavy enough to contend with request latency, split the workers into their own process(es):

# In the root of an instance.
# Web tier — HTTP only, no in-process consumers.
BACKEND_ROLE=api pnpm run start

# Worker tier — consumers only, no HTTP listener.
pnpm run worker

# In a checkout of the Endora Commerce repository:
BACKEND_ROLE=api pnpm --filter backend run start
pnpm --filter backend run worker # production (built)
pnpm --filter backend run worker:dev # development (tsx watch)

An instance has no watch-mode worker: pnpm run dev runs the API with the workers co-located, which is what a development machine needs. An instance scaffolded before its root declared worker has no pnpm run worker; pnpm -C backend run worker is the same command there.

Scale the worker tier horizontally by starting more worker processes (or containers). BullMQ's atomic job claim guarantees a job is processed by exactly one of them; no configuration change is required to add instances.

Both tiers must point at the same Redis (REDIS_URL, default redis://localhost:6379) and the same PostgreSQL database. Redis is the queue substrate; Postgres holds the operation rows that are the source of truth.

Graceful shutdown​

The worker entrypoint (backend/src/worker.ts) handles SIGINT / SIGTERM: it closes the BullMQ workers (letting in-flight jobs finish), then disconnects Redis and the ORM. Send SIGTERM and wait for the process to exit before replacing it during a deploy.

Current queue-backed operations​

Catalog bulk operations — catalog.bulk-operation​

Backs the admin Bulk actions (Akcje masowe) page. Two job types share one queue:

  • product_bulk_update — a product bulk-edit larger than the synchronous threshold (50 products). The admin request persists a pending BulkOperation row, enqueues its id, and returns 202 immediately.
  • search_reindex — a full Meilisearch re-index (the CLI pnpm run cli search reindex equivalent), enqueued automatically when an attribute's searchable flag is flipped.

The consumer claims a row by flipping it pending → running with a conditional UPDATE (atomic claim), runs the handler, and persists live progress plus a final completed / failed status. On finish, the requester gets an in-app bell notification and an email. Re-delivery of an already-claimed job is a no-op, so the handler is safe at N ≥ 2 workers.

At worker startup a boot reconciliation re-enqueues any rows left pending (e.g. created while Redis was briefly unreachable). This only enqueues onto the durable queue — it is not a draining sweeper.

Webhook delivery — webhook.deliver​

Domain events on the in-process event bus are bridged onto a BullMQ queue; the worker signs each payload (HMAC-SHA-256) and POSTs it to the subscriber URL. Up to 8 attempts with exponential backoff; a final failure is dead-lettered and replayable from the admin Webhooks → Deliveries view. See the webhooks module for the delivery contract and retry model.

Price surface refresh — price_lists.surface​

Keeps the prices that price-sorted and price-filtered product listings are ordered by up to date. Those listings cannot be ordered from the search index — every buyer may have their own prices — so the price_lists module precomputes each product's price per group of buyers that share the same price books (a pricing profile). Every committed price change — a price book, its rows, channels or members, base prices, a status change by the sweep below, a product, category or attribute change, a change to a sales channel — marks what it can move: the products it concerns in the profiles it reaches, or whole profiles when it concerns every product of them (a sales-channel change, a line for all products or for an attribute value alone, more than 10 000 products at once). It then enqueues one coalesced job (a fixed job id, so a burst of writes costs one job; three attempts with exponential backoff).

Each pass of the consumer builds never-built and wholly marked profiles in full and re-resolves only the marked products of the others, the everyone profiles before the buyer ones, and swaps each change in one transaction. Passes repeat while writes keep marking, so a change made during a pass is picked up by the next. A request never waits for it: a marked profile keeps serving its previous order, and the prices on the product cards are always resolved live, so only the order of a price-sorted listing lags a change — by the consumer's turnaround, about half a second for a single change in the platform's measurements and under about a minute and a half, with some 200 profiles, for a change that rebuilds many profiles whole (85 s at 202 profiles on 50 000 products, issue #416). The job is idempotent (a refresh recomputes from source), so a retry or a second worker instance is harmless.

If no worker consumes this queue — a process tier without consumers, or no Redis — the profiles are never built: price-sorted listings are then ordered from a live resolution when they are narrowed to at most 5 000 products (a category, a search), and otherwise served in the catalogue's default order with correct prices on the cards. Run the worker wherever price sorting matters.

Table maintenance after bulk writes — catalog.table-maintenance, price_lists.table-maintenance​

Right after a bulk write, Postgres has no fresh statistics and no visibility map for the rows it wrote until autovacuum's first pass — and for a write below autovacuum's scale factor that pass may not come. Until then the admin's index-only reads (the product list's ordered walk, a price book's lines and searches) fetch every row from the table, which on a 50 000-product catalogue costs the broadest searches about 25 ms each (issue #409). So each module asks for VACUUM (ANALYZE) of its own tables once a bulk write of them has committed: catalog for products and product_variants after a products import (Import / Export, catalogBulkImportPort), and price_lists for price_list_matrix_entries and price_list_product_lines after a price-book row batch (the editor's save, a CSV import), a base-price write (the PIM price import) and a book duplicate. VACUUM cannot run inside a transaction, so it is a job, never part of the write.

Each request is a job delayed by one minute and deduplicated by its table: every write while it waits is absorbed (the vacuum still runs after them), and a write while it runs keeps exactly one follow-up. One consumer per module runs them one at a time; a failed job is logged, retried once, and dropped. If no worker consumes these queues, nothing breaks — autovacuum remains the floor, and the reads above are slower until its first pass.

Command outbox — command_outbox.drain​

Delivers the afterCommit handlers that installed modules register on another module's write — telling an ERP that an Organization registered, say. Each handler's delivery is a row written in the same transaction as the write, so a committed write never loses it. A job scheduler drains the rows every 10 seconds, and a write that produced rows also wakes the drain at once. A failed delivery is retried with a growing delay, up to 10 attempts, and is then dead; a handler may also give up on a delivery at once.

Dead deliveries are listed in the admin under System → Command hooks → Outbox (platform.command_outbox.read), with the handler, the attempts and the failure's code — never the data the delivery carried. An administrator holding platform.command_outbox.manage retries one there, and the retry is recorded in the audit log. A delivery whose handler's module is switched off waits, without spending an attempt, until the module is switched back on. Delivered rows are removed after 7 days, dead ones after 90. See the Command Bus for the delivery contract.

If no worker consumes this queue, nothing is lost: the rows wait in the database and are delivered once a worker runs.

Cart abandonment sweep — carts.abandonment.sweep​

Marks active carts idle past carts.abandonment.inactivity_minutes as abandoned (and sends the abandonment notification a deployment supplies), and removes working baskets left behind by an order placement that never finished. A job scheduler runs it every 15 minutes. A tick first asks whether any cart is due and runs a pass only when one is, so an idle instance does no work and writes no audit row for it. A pass works in batches of 500, committing each batch's status flips together with the audit rows for them, so an interrupted pass leaves whole batches behind rather than a partial one; the next tick picks up the rest. Passes are idempotent, so a second worker instance is harmless. While carts is unavailable on the platform no tick runs.

To run one pass by hand, use pnpm run cli carts abandonment-sweep in the root of an instance (pnpm --filter backend run cart:abandonment-sweep in a checkout of the Endora Commerce repository). The command reads module presence from module_registrations before it does anything and exits non-zero with MODULE_DISABLED when carts is not installed on the deployment.

If no worker consumes this queue, carts are not marked abandoned and orphaned working baskets stay in the database — harmlessly, since nothing reads them — until a worker runs or the command is run by hand.

Other background jobs (not yet on a shared queue)​

A few periodic/maintenance jobs still run as in-process timers or manual scripts. These predate the scalable queue consumer rule and are tracked for migration to the same worker model; document and operate them as follows in the meantime:

JobHow it runs todayInvocation
Full search re-indexManual CLIpnpm run cli search reindex
Price-book status sweep (moves Scheduled books to Active and Active ones to Expired for the admin's display; prices follow a book's dates at the exact instant regardless)In-process setInterval (every 5 min)starts with the API process
RFQ expiryService method, invoked on a scheduleRfqExpiryWorker.sweep()

The command is run in the root of an instance. In a checkout of the Endora Commerce repository it is pnpm --filter backend run search:reindex.

note

The in-process setInterval sweepers above are the legacy pattern the scalable queue consumer rule replaces. New asynchronous, queue-backed work MUST use the durable-queue + separable-worker model described here, never a request-process timer.

Monitoring & troubleshooting​

  • Queue depth / failed jobs — inspect the BullMQ keys in Redis, e.g. redis-cli keys 'bull:catalog.bulk-operation:*' and redis-cli keys 'bull:webhook.deliver:*'. Failed jobs are retained (removeOnFail) for inspection.
  • Operation status — for catalog work, the Bulk actions admin page lists pending / running / completed / failed operations with per-item counters; the underlying rows live in the catalog_bulk_operations table.
  • A queued operation never leaves pending — confirm a worker is running (BACKEND_ROLE includes consumers) and that it shares the same REDIS_URL as the API. Restarting a worker re-enqueues orphaned pending rows via boot reconciliation.
  • Workers idle while jobs pile up — check the owning module is enabled; workers registered through the module lifecycle are paused when their module is disabled and resumed when it is re-enabled.