Message Queues: The Boundary Between Receiving Work and Processing It.

Five hundred webhook events arrive within a few seconds.

Each request must be authenticated, validated, stored, transformed, and delivered to a browser watching the event stream.

In a fully synchronous design, the webhook sender remains connected while WebStream completes that entire workflow.

That approach may work while traffic is low and every dependency is healthy. It begins to fail when the database slows down, an external API becomes unavailable, or incoming events exceed the application’s immediate processing capacity.

The problem is not that these operations exist.

The problem is that the external sender depends on every one of them completing before the request can succeed.

A message queue creates a boundary between accepting the event and completing the work triggered by it.

Instead of performing every downstream operation inside the original HTTP request, the ingestion service validates the event, safely places it onto a queue, and returns an acknowledgement. Separate workers then retrieve and process the event.

Without a queue

Webhook sender
↓
WebStream endpoint
↓
Authenticate
↓
Validate
↓
Write to database
↓
Call external API
↓
Transform event
↓
Notify dashboard
↓
Return HTTP response

With a queue

Webhook sender
↓
WebStream endpoint
↓
Authenticate
↓
Validate
↓
Durably enqueue
↓
Return HTTP response

Meanwhile:

Message queue
↓
Worker
↓
Process downstream operations

The queue does not make the work disappear.

It changes when the work happens, where unfinished work waits, and how the system behaves when demand temporarily exceeds processing capacity.

What happens inside a synchronous webhook request?

In a fully synchronous webhook workflow, the HTTP request remains open while the application completes every operation required by the pipeline.

The server first accepts the connection and reads the incoming request body.

Before trusting the event, it may need to:

• verify the provider’s signature using the original payload;
• check that the timestamp falls within an acceptable replay window;
• validate an API key or authentication token;
• enforce request-size limits;
• and confirm that the sender is permitted to access the endpoint.

After authentication and integrity checks pass, the service parses the request body and validates the event structure.

It determines whether required fields are present, whether the event type is supported, and whether the values can be processed safely.

The endpoint then executes the core workflow. Depending on the application, this may include:

• writing data to a database;
• calling external APIs;
• performing calculations;
• transforming the payload;
• updating analytics;
• or publishing the event to connected clients.

Only after that execution chain completes does the service construct the response, set the appropriate status code and headers, and transmit the result to the sender.

Receive request
↓
Authenticate and verify
↓
Parse and validate
↓
Execute business workflow
↓
Prepare response
↓
Return status

The weakness is not that any individual step is unreasonable.

The weakness is that every dependency called before the response becomes part of the request’s latency and failure surface.

Engineering insight
In a synchronous pipeline, every dependency executed before the response can delay or fail the original request.

External APIs expand the failure surface

The dependency I would be most concerned about is an external API.

WebStream does not control the external service’s infrastructure, capacity, deployment schedule, or latency.

The service may experience:

• complete outages;
• intermittent 5xx errors;
• strict rate limiting;
• DNS or TLS failures;
• overloaded infrastructure;
• or responses that are technically successful but unacceptably slow.

Timeout configuration introduces another risk.

An HTTP client with an excessively long timeout—or without a deliberate timeout policy—can leave the WebStream request waiting on a downstream service long after the webhook event has already been received and validated.

In a synchronous pipeline, downstream latency passes directly through the system.

Webhook sender
↓
WebStream
↓
External API
↓
Wait for response
↓
Return webhook acknowledgement

If the external API takes eight seconds to respond, WebStream also takes at least eight seconds to respond.

If the webhook provider expects an acknowledgement within five seconds, it may assume delivery failed and send the event again.

That creates a retry loop.

Slow external API
↓
WebStream request times out
↓
Webhook provider retries
↓
More concurrent external calls
↓
Greater downstream pressure

The external API’s slow response has now become a WebStream failure, even though WebStream does not control the external service.

What must happen before WebStream responds?

Moving work onto a queue does not mean returning success as soon as the server sees the request.

The event must first be accepted safely.

Before returning a successful response, WebStream should normally complete the following:

1. Read the request body.
2. Enforce payload-size limits.
3. Verify authentication, signatures, and timestamp integrity.
4. Parse and validate the required event structure.
5. Identify an existing duplicate when an event identifier is available.
6. Preserve the event durably.
7. Return any response body or headers required by the sending platform.

The crucial step is durable acceptance.

WebStream should not return 200 OK and only then attempt to preserve the event in temporary application memory.

If the process crashes immediately after responding, the sender believes the event was accepted even though WebStream has lost it.

Durable acceptance can take different forms:

1Validated event ↓ Written to durable storage

or:

1Validated event ↓ Published successfully to a durable queue

Once this step succeeds, WebStream has accepted responsibility for completing the work.

The downstream workflow can then happen asynchronously.

Synchronous path

Receive
↓
Verify
↓
Validate
↓
Durably enqueue
↓
Acknowledge

Asynchronous path

Consume
↓
Transform
↓
Call integrations
↓
Store derived data
↓
Notify connected clients

Tasks that can usually happen after the response include:

• external API calls;
• heavy data processing;
• payload enrichment;
• analytics updates;
• real-time dashboard delivery;
• sending notifications;
• and creating secondary database records.

Database work requires a distinction.

A write that proves WebStream accepted the event is critical. It should happen before acknowledgement unless the durable queue itself is the authoritative record.

A derived analytics record or formatted display record can usually be reconstructed and retried later.

Architecture decision
The webhook response confirms that WebStream has safely accepted responsibility for the event—not that every downstream operation has finished.

What is a message queue?

A message queue is an intermediary that accepts messages from producers, stores them temporarily, and delivers them to consumers when those consumers are ready to process them.

The basic flow contains five parts:

1Producer → Message → Queue or broker → Consumer → Acknowledgement

Producer

The producer creates and publishes the message.

In WebStream, the ingestion service acts as the producer after it has authenticated and validated the incoming webhook.

Message

The message represents an event or a unit of work.

For example:

1{2  "eventId": "evt_8412",3  "inboxId": "inbox_123",4  "receivedAt": "2026-07-21T09:42:13Z",5  "method": "POST",6  "eventType": "payment.completed",7  "payload": {8    "paymentId": "pay_7281",9    "status": "completed"10  }11}

A strong message contains enough information for a consumer to understand what happened or what work must be performed.

It should not require the consumer to reconstruct critical context from temporary application memory.

Queue or broker

The queue holds messages until they can be delivered.

A broker may also handle routing, acknowledgements, redelivery, persistence, consumer coordination, and failure policies.

Consumer

The consumer retrieves and processes the message.

In WebStream, one consumer might save the event into a searchable store while another prepares it for real-time delivery to the dashboard.

Acknowledgement

An acknowledgement tells the broker that processing completed successfully.

Without an acknowledgement, the broker cannot reliably distinguish between a completed operation and a worker that crashed midway through processing.

Depending on the system’s delivery policy, an unacknowledged message may later be delivered again.

Work queues and publish-subscribe

Not every messaging workflow distributes messages in the same way.

Two common models are work queues and publish-subscribe.

Work queue

A work queue distributes tasks across multiple workers.

Each message is normally processed by one available consumer.

1┌──→ Worker A Task queue ────────┼──→ Worker B └──→ Worker C

This model is useful when several workers share responsibility for completing the same kind of task.

Examples include:

• resizing images;
• generating reports;
• sending emails;
• processing webhook events;
• or creating invoice PDFs.

Adding consumers can increase processing capacity, provided the work can safely run in parallel.

Publish-subscribe

In publish-subscribe systems, one event may be delivered to several interested consumers.

1Event published ├──→ Store the event ├──→ Update analytics └──→ Notify the dashboard

Each consumer reacts independently to the same event.

A failure in the analytics consumer should not necessarily prevent the storage consumer from succeeding.

Engineering insight
A work queue answers, “Which worker should process this?” Publish-subscribe answers, “Which systems need to know this happened?”

WebStream may eventually use both patterns.

A work queue could distribute payload-processing jobs across workers, while a publish-subscribe layer could send the accepted event to storage, analytics, and real-time delivery consumers.

What happens when events arrive too quickly?

Suppose WebStream receives events at the following rate:

1Incoming rate: 1,000 events per minute Processing rate: 600 events per minute

The system receives 400 more events each minute than it can process.

Without a queue acting as a controlled buffer, that unfinished work remains active inside the application as:

• open HTTP connections;
• pending database operations;
• waiting external API calls;
• request bodies held in memory;
• scheduled processing tasks;
• and retrying requests.

As concurrent work accumulates, finite resources become saturated.

These may include:

• database connections;
• sockets;
• memory;
• CPU time;
• worker capacity;
• and the application’s ability to schedule additional work.

Users initially experience increased latency.

Dashboard requests must compete with webhook processing for the same resources. Even though those users are not sending webhooks directly, they may see:

• delayed event delivery;
• failed page loads;
• slow navigation;
• and an increasingly unresponsive interface.

Internally, WebStream may experience:

• memory and CPU saturation;
• exhausted database connections;
• increased garbage-collection pressure;
• cascading timeouts;
• failed health checks;
• process or container restarts;
• and duplicate deliveries caused by upstream retries.

Webhook burst
↓
More concurrent synchronous processing
↓
Resource saturation
↓
Slower responses
↓
Timeouts
↓
Provider retries
↓
Additional system pressure

This creates a feedback loop.

Slower processing causes timeouts. Timeouts cause retries. Retries create more work, which makes processing even slower.

Engineering insight
Overload rarely remains isolated to the busiest endpoint. Shared compute, memory, and connection pools allow one saturated workflow to degrade the rest of the application.

A message queue changes where the excess work waits.

Instead of keeping hundreds of HTTP requests and downstream operations active, the ingestion endpoint can validate and durably enqueue events, respond quickly, and allow workers to process the backlog at a controlled rate.

A queue absorbs bursts—it does not create capacity

A queue gives the system time.

It does not permanently solve insufficient processing capacity.

Using the previous example:

1Incoming rate: 1,000 events per minute Processing rate: 600 events per minute Backlog growth: 400 events per minute

The queue grows continuously:

1After 1 minute: 400 waiting After 2 minutes: 800 waiting After 3 minutes: 1,200 waiting

A temporary traffic burst may be absorbed and processed later.

But if producers consistently create work faster than consumers can complete it, the backlog will continue to grow.

The system eventually needs one or more of the following:

• additional consumers;
• faster processing;
• reduced work per message;
• workload prioritization;
• producer throttling;
• or admission control.

Warning
A growing queue is delayed failure. The backlog still requires processing capacity, storage, monitoring, and a plan for recovery.

The queue is therefore a backpressure mechanism, not an unlimited warehouse.

It makes excess demand visible and manageable instead of allowing it to consume the entire application immediately.

Delivery guarantees

Once work moves into a messaging system, the application must decide what delivery behaviour it can tolerate.

At-most-once delivery

A message is processed zero or one time.

The system avoids intentional redelivery, but a message may be lost if failure occurs before processing completes.

This may be acceptable for low-value telemetry. It is usually not acceptable for payments or critical business events.

At-least-once delivery

A message is retried until processing is acknowledged.

This reduces the risk of permanent loss, but it means the same message may be processed more than once.

A consumer could complete a database write and crash before sending its acknowledgement. The broker then sees an unacknowledged message and delivers it again.

The consumer must therefore tolerate duplicates.

Exactly-once processing

Exactly-once behaviour is often treated as the ideal, but it becomes difficult when a workflow crosses databases, networks, message brokers, and external services.

For many applications, the safer design assumption is:

Messages may be delivered more than once, and consumers must be idempotent.

An idempotent consumer produces the same final result even if it processes the same message repeatedly.

We will examine idempotency in the third article when introducing queues into a complete workflow.

Message ordering

A queue does not automatically guarantee the business order the application expects.

A system must distinguish between:

• the order in which messages were created;
• the order in which they entered the queue;
• the order in which consumers received them;
• and the order in which processing completed.

These may differ.

Consider two messages:

1Message 1: Account created Message 2: Account suspended

If several consumers process messages concurrently, the suspension could complete before account creation.

Retries can also disrupt order. A failed earlier message may return to the queue while later messages continue processing.

Ordering becomes even more complex when a broker divides messages across partitions or queues.

The application must decide whether strict ordering is required globally, per customer, per account, or not at all.

Global ordering limits concurrency. Partitioned ordering allows more parallel processing but requires a reliable routing key.

Trade-off
Stronger ordering guarantees usually reduce the amount of work that can be processed concurrently.

When should a queue be considered?

A message queue becomes useful when one or more of these conditions are present:

• work takes longer than the original request should remain open;
• incoming traffic arrives in unpredictable bursts;
• failed operations need retries;
• downstream systems may be temporarily unavailable;
• work can be completed independently from the request;
• multiple consumers need to react to the same event;
• or processing capacity must be controlled separately from ingestion capacity.

WebStream satisfies several of these conditions.

Webhook traffic may arrive in bursts. Downstream APIs can become slow. Real-time delivery should not block event acceptance. Failed processing may need to be retried without asking the original sender to understand WebStream’s internal architecture.

When is a queue unnecessary?

Queues introduce real operational cost.

They add:

• another infrastructure component;
• message persistence and retention policies;
• duplicate-delivery handling;
• consumer lifecycle management;
• queue-depth monitoring;
• retry behaviour;
• dead-letter handling;
• and more difficult debugging across asynchronous boundaries.

A queue may be unnecessary when:

• the workflow is simple and reliably synchronous;
• immediate completion is required;
• traffic is low and predictable;
• there is no need for buffering or retries;
• or the operational complexity outweighs the expected benefit.

A queue should solve a specific reliability or scaling problem. It should not be introduced merely because distributed architecture appears more advanced.

Trade-off
A queue reduces temporal coupling between services, but replaces a simple call stack with asynchronous state that must be observed, retried, and reconciled.

The architectural boundary

In WebStream, the queue creates a boundary between accepting an external event and completing the internal work caused by that event.

The ingestion endpoint remains responsible for:

1Receive → Verify → Validate → Durably accept → Respond

Workers become responsible for:

1Consume → Transform → Integrate → Store derived data → Notify

This separation allows WebStream to acknowledge webhook delivery quickly, process events at a controlled rate, and retry failed work without forcing the sender to understand its internal architecture.

It also prevents temporary traffic bursts from immediately consuming every resource required to keep the wider application responsive.

But placing a message onto a queue is only the beginning.

The system must still decide:

• which consumer receives each message;
• how messages should be routed;
• how often failed messages should be retried;
• what happens when a message repeatedly fails;
• and whether some work should be processed before other work.

Those are the concerns of routing, retry policies, dead-letter queues, and priority—the subject of the next article in this series.