Ingesting Webhooks with AWS Serverless

I’ve recently been working on ingesting webhooks using AWS.

It sounds simple enough, right? We’ve been receiving payloads over HTTP for decades, so surely this is a solved problem. Someone, somewhere, must have already written a clean and satisfying serverless implementation.

This turned out to be a very fun and interesting rabbit hole, and I would like to share my journey.

Webhook basics

Let’s start with the basics: a webhook is simply an HTTP request sent by one system to another when something happens. Instead of asking the provider every few minutes whether anything changed, the provider calls an endpoint we expose.

The provider decides the rules of the game. Slack expects an HTTP 2xx response within three seconds and retries failed deliveries up to three times. Other providers have different rules: GitHub, notably, can send webhook payloads up to 25 MB in size and does not automatically retry failed deliveries.

Scale

“Webhook ingestion” can mean very different things.

One team may receive a few events every minute. Another may receive thousands of events per second when a provider retries a backlog, sends a bulk update, or has an incident. The architecture should be driven by both average traffic and the peak we must absorb.

Case	Ingress rate	Rough daily volume if sustained	Example	Serverless fit
A	1 event/second	86,400 events/day	A small integration or internal tool. Long periods without traffic are common.	Excellent. Paying only when traffic arrives is exactly the point.
B	10 events/second	864,000 events/day	A healthy SaaS integration with occasional bursts.	Excellent. This is a natural serverless workload.
C	100 events/second	8.6 million events/day	A meaningful production workload. Failures, retries, and observability now matter more than the endpoint itself.	Good, provided that quotas and the complete cost per event are understood.
D	1,000 events/second	86 million events/day	A high-volume platform. A short outage can create a large backlog very quickly.	Maybe. Bursty traffic can still fit, but sustained traffic needs cost testing.
E	10,000 events/second	864 million events/day	Infrastructure-scale traffic. A one-minute burst is 600,000 events.	Usually poor. Per-request and per-hop charges become expensive at this scale.
F	30,000 events/second	2.6 billion events/day	Very large bursts or a major multi-tenant platform. One minute is 1.8 million events.	Poor by default. Evaluate provisioned streaming and worker platforms instead.

We should make a distinction between accepting an event and processing it.

Accepting an event means validating the request, recording it durably, and returning a response before the provider’s deadline. Processing begins afterward and may involve database writes, third-party APIs, file downloads, media encoding, or other comparatively expensive work.

These two stages have very different performance requirements and failure modes.

A, B, and C: workloads that may look simple

At lower event rates, accepting and processing events can appear closely coupled. If requests arrive slowly and each event requires little work, the processing layer may keep up without accumulating a meaningful backlog. For very low event rates, and when there is little work to do, the system may not need external buffers or workers.

Cases A through C may allow a simpler implementation, but they still need explicit limits, retries, idempotency, and visibility into any growing backlog.

These are also the workloads where serverless is usually at its best. Traffic is low or irregular, idle periods are common, and paying per request is often cheaper than keeping servers and workers running all day. Cases A, B, and C are the main target for the architectures in this article.

D, E, and F: not your serverless usecase.

Acknowledging 30,000 HTTP requests per second is an easier problem than performing 30,000 units of downstream work per second.

A system may continue returning successful responses while its processing queue grows by millions of events. From the provider’s perspective, every delivery succeeded. Internally, however, processing latency may have increased from seconds to hours.

Case D can still be a reasonable serverless workload, particularly when the traffic is bursty rather than sustained, but it deserves a careful quota and cost model. At this point, every additional managed hop is multiplied by tens of millions of daily events.

For sustained cases E and F, serverless services can technically scale to very high request rates, but request-based pricing, repeated message operations, Lambda duration, logs, metrics, and data transfer compound quickly. At that volume, a continuously provisioned stream-processing or worker platform, such as Kinesis, Kafka, ECS, or Kubernetes, can be substantially cheaper and more predictable.

This is not a claim that serverless stops working at an exact number. The crossover depends on payload size, processing time, burstiness, consumer count, and operational constraints. For cases E and F, benchmark the complete cost per event before committing to the architecture.

The classic solution

Let’s start with the classic solution that everyone used before serverless was cool: a traditional web framework and long-lived workers.

The main idea is to accept the webhook as a standard HTTP request, validate it, store it somewhere durable, and process it asynchronously. We can break down the process into three elements:

• HTTP receiver
  An always-available service exposing the webhook endpoint. It authenticates the sender, performs minimal validation, writes the event durably, and returns a success response.
• Durable buffer
  Storage independent of the web process, such as a message queue, stream, or database table. It prevents events from being lost if the web service restarts and absorbs bursts when processing is slower than ingestion.
• Asynchronous processor
  One or more workers that consume stored events and run the actual business logic, including retries and failure handling.

Common combinations include:

HTTP receiver	Durable buffer	Async processor	Typical environment
Django on ECS	Amazon SQS	Django worker on ECS	AWS, container-based
FastAPI on Kubernetes	Kafka	Python consumer on Kubernetes	High-volume event platform
Laravel on-premise	RabbitMQ	Laravel queue workers	Traditional on-premise
Laravel	Redis	Laravel Horizon workers	Simple Laravel deployment
Express.js	PostgreSQL queue table	Node.js worker	Small system with existing database
Spring Boot	Kafka	Spring Kafka consumer	Enterprise event streaming
Rails	Redis	Sidekiq	Conventional Ruby stack

Serverless

Serverless feels appealing because it promises that infrastructure becomes a utility rather than a product you must design, own, patch, and monitor.

The dream of using serverless looks like this:

• no servers to manage
• scale automatically
• pay only when used
• built-in retries and integrations
• less operational burden
• IaC: infrastructure described as code

Ingesting webhooks using a serverless approach could be as simple as defining some infrastructure using Terraform and writing a piece of business logic for the Lambda function.

Generally speaking, writing a system that ingests webhooks consists of providing a solution to the following:

• We need to accept untrusted, at-least-once, potentially out-of-order notifications from external systems.
• We do not control the event distribution: bursts of events can happen, as well as minutes without any delivery.
• We need to acknowledge receipt quickly and process the information with reasonable latency, often below 500 ms and even lower for some use cases.
• We should keep track of historical events, both for rebuilding history and for compliance.
• We should create a simple yet effective infrastructure that aims to save costs.
• Operations around this process might involve:
  • Reprocessing failed events.
  • Rebuilding the historical chain of events.
  • Monitoring latencies and deliveries to find anomalies.
• Bonus: can we design a solution general enough that we can easily reuse it for different, independent ingestions?

We should only use AWS serverless products, and the operational complexity and cost should make the trade-off of using serverless worthwhile compared with deploying servers, load balancers, and workers, and maintaining and patching them.

Toolbox

AWS gives us a suspiciously large collection of services for moving JSON from one place to another. We will not use all of them, but it helps to know who is attending the party.

API Gateway is the public entrance. It accepts HTTP requests and can invoke Lambda or integrate directly with services such as SQS and S3. HTTP APIs are simpler and cheaper, while REST APIs offer more transformations, service integrations, and controls.

Lambda Function URLs are the tiny alternative to API Gateway. They attach an HTTPS endpoint directly to a Lambda function. They are convenient for simple or internal endpoints, but public webhooks usually benefit from the additional control provided by API Gateway.

Lambda is our compute engine. It can verify signatures, transform events, and run business logic without requiring us to manage servers.

SQS is the durable work queue. It holds events while consumers are unavailable, retries failed processing, absorbs traffic spikes, and can move repeatedly failing messages to a dead-letter queue. Lambda integrates with it particularly well through automatic polling, batching, and concurrency controls.

SNS provides simple fan-out. We publish an event once, and SNS delivers a copy to every subscribed consumer. In practice, each consumer usually gets its own SQS queue so that it can fail, recover, and build a backlog independently.

EventBridge is the more opinionated routing option. It can inspect an event and send it to different targets according to rules. It is useful when routing depends on event type or content, while SNS is often simpler when every subscriber should receive the same event.

S3 is where large or replayable payloads go. Instead of pushing the complete webhook through every service, we can store the raw bytes once and pass around a small pointer containing the bucket and object key. S3 is also useful for audit trails, retention, and disaster recovery.

Firehose continuously delivers events to destinations such as S3. It is handy for building an append-only operational or analytics archive without asking another Lambda to spend its life copying messages between services.

DynamoDB is useful for small pieces of state, especially idempotency records, processing status, and event indexes. It is not necessarily the event pipeline itself, but it often keeps the pipeline from doing the same expensive thing twice.

Step Functions coordinates workflows with multiple steps, retries, waits, and branches. It is probably excessive for “receive event, update row,” but useful when the processing logic starts looking like a flowchart drawn by someone who has had too much coffee.

Mapped back to the three components of the classic solution, the AWS serverless toolbox looks like this:

Classic component	AWS serverless tools	Role in the webhook ingestion system
HTTP receiver	API Gateway HTTP API, Lambda synchronous invocation	Exposes the public webhook endpoint, performs authentication and lightweight validation, persists or forwards the event, and returns the provider-facing 2xx response quickly.
Durable buffer	SQS, EventBridge, SNS, DynamoDB, S3	Decouples acceptance from processing. SQS is the most direct queue-shaped replacement for a classic worker queue. EventBridge is useful when events need routing. SNS is useful for fan-out. DynamoDB or S3 can store the raw event body, especially when payload size, replay, or auditability matter.
Asynchronous processor	Lambda asynchronous invocation, Lambda triggered by SQS/EventBridge/SNS, Step Functions	Runs the business logic after the event has been accepted. Lambda handles the common worker case, while Step Functions becomes useful when processing is a multi-step workflow with branching, retries, or human-readable orchestration.

This is the same architecture as before, just with different ownership boundaries. Instead of running a web service, queue, and worker fleet ourselves, we compose managed services that each cover one responsibility.

Building the pipeline, one assumption at a time

Now that we have the Lego building blocks, let’s have some fun, shall we?

We will build the pipeline incrementally. Each step starts with the smallest design that satisfies the current requirements, then adds one capability when the previous design reaches its limit.

Stage 1: Direct invocation, in a peaceful fictional universe

Let’s start with deliberately narrow assumptions:

• The sender is an internal AWS service, so we trust it.
• It sends exactly one event at a time and each payload is smaller than 1 MB.
• The processing Lambda never fails. A very peaceful fictional universe.

There are two simple ways for the sender to call Lambda. They look similar in a diagram, but their failure behaviour is completely different.

Stage 1a: invoke Lambda asynchronously through the AWS SDK

Internal service -> AWS SDK Invoke -> Lambda

The internal service uses its IAM role to call Lambda’s Invoke API with InvocationType: Event. Lambda accepts the event, places it in its own asynchronous queue, and returns 202 Accepted to the caller. The caller does not wait for the processing code to finish.

Even though our fictional Lambda never fails, this option comes with useful built-in behaviour for when reality returns: Lambda retries a failed asynchronous invocation twice by default. We can also configure an onFailure destination to retain events that fail all attempts.

Stage 1b: call a Lambda Function URL

Internal service -> HTTPS Function URL -> Lambda

We said that a webhook is HTTP-based, so calling Lambda directly through the AWS SDK is slightly outside the usual webhook shape. Let’s return to an HTTP endpoint. How can we call Lambda over HTTP? Function URLs.

A Function URL gives a Lambda function an HTTPS endpoint. It can use AWS_IAM authentication or no authentication.

This invocation is synchronous. The sender waits for the Lambda to run and receives the HTTP response. That is pleasantly simple when the function is fast and never fails. Everyone is happy, and nobody has had to draw a queue.

The important difference is that a Function URL does not put the request into Lambda’s asynchronous queue. If the function times out, runs out of memory, or cannot reach its database, Lambda returns an error to the caller. Retrying is the caller’s responsibility.

Stage 2: Give it a durable buffer

Of course, life is not usually this simple. When the worker can fail and an event must survive that failure, we need to think about a durable buffer.

The goal of a durable buffer is to accept an event durably and hold it until a consumer has processed it. SQS is a natural fit for Lambda because the two services integrate through automatic polling, batching, visibility timeouts, and controlled concurrency.

flowchart LR
    P[Producer] --> Q[SQS queue]
    Q --> L[Lambda worker]
    L -->|success| D[Message deleted]
    L -->|maxReceiveCount reached| DLQ[(Optional DLQ)]

If the Lambda invocation fails, the message becomes visible again after the visibility timeout and can be processed again. If it reaches the queue’s configured maxReceiveCount, SQS moves it to the DLQ.

We now have a durable buffer between the HTTP endpoint and the processing logic.

We have glossed over the Producer -> SQS step. We want the producer-facing side to remain HTTP-based, so we have two main options:

• Use a front-door Lambda Function URL whose only job is to validate the request and send the event to SQS.
• Use API Gateway REST, a managed HTTP front door that can forward requests directly to AWS services, including SQS.

At a high level the main differences are the following:

Option	Best for	Trade-off
Lambda Function URL	The simplest possible HTTP endpoint for one Lambda	Lowest complexity, but fewer edge controls and the Lambda is directly on the request path
API Gateway HTTP API	A lightweight, lower-cost HTTP front door for Lambda	More control than a Function URL, but fewer advanced integrations and API-management features
API Gateway REST API	A fully controlled public API or webhook edge	More expensive and complex, but supports richer transformations, direct AWS service integrations, usage plans, and WAF-oriented designs

Stage 3: More than one consumer wants this event

So far, we have assumed that one Lambda worker is interested in each event. Let us relax that assumption.

There is also a subtle detail in the SQS flow: once the consumer successfully processes a message, the message is deleted from that queue. That is exactly what we want for a work queue, but it does not help when several independent consumers need to process the same webhook.

For example, a payment refund event might need to:

• send an email to the customer support team;
• update the business system;
• be copied into an analytics pipeline.

Each consumer should be able to process the event independently. A slow analytics job should not delay an email, and a temporary failure in the business system should not prevent the other consumers from making progress.

The usual fan-out pattern is one shared topic with one queue per consumer:

flowchart LR
    P[Producer] --> T[SNS topic]
    T --> Q1[SQS support queue]
    T --> Q2[SQS business queue]
    T --> Q3[SQS analytics queue]
    Q1 --> L1[Support Lambda]
    Q2 --> L2[Business Lambda]
    Q3 --> L3[Analytics Lambda]

Each queue has its own retry policy, visibility timeout, concurrency limit, and optional DLQ. The queues also give each consumer its own backlog. This is the important part: fan-out should duplicate the event at the boundary, not make every consumer call the others.

There are two main AWS services we can use for the fan-out layer.

SNS: simple publish and subscribe

SNS is the straightforward option. The producer publishes one message to a topic, and SNS delivers a copy to each subscription. When the subscribers are Lambda workers, I would usually place an SQS queue between SNS and each Lambda. That gives every consumer durable buffering and independent retries.

SNS is a good fit when the routing is simple and the main requirement is: “send this event to these consumers.” It is inexpensive, has low latency, and supports subscription filter policies for basic message-attribute filtering.

SNS is not a long-term archive, however. Queue retention and replay are separate concerns. If we need compliance retention or historical replay, we should store the event in a durable archive such as S3 as well.

EventBridge: richer event routing

EventBridge is the more expressive option. Rules can inspect event fields and route different event types to different targets. One event can match multiple rules, which is useful when the routing logic becomes more complicated than a list of subscriptions.

EventBridge also provides features such as event archives and replay, and it integrates with a wider range of AWS targets. The trade-off is an extra routing layer, additional cost, and slightly more operational complexity.

The choice is therefore fairly simple:

Requirement	Natural fit
Simple fan-out to a known set of consumers	SNS
Basic subscription filtering	SNS with filter policies
Content-based routing and many rules	EventBridge

Stage 4: Untrusted producers

So far, we have designed a simple architecture that accepts webhooks from trusted sources and distributes them to multiple consumers independently. It gives us a useful foundation for future development and pipeline evolution.

Let’s now focus on external producers. What changes? As soon as we expose our endpoint to the internet, bad actors will start sending it messages. Some may be looking for leaked credentials or unprotected resources. Others may simply want to launch a DDoS attack, prevent legitimate webhooks from reaching us, or trigger internal services and make us spend money.

The first line of defence is the edge. AWS WAF can allow or block IP ranges, apply rate-based rules, inspect the beginning of a request body, and use managed reputation rules. For API Gateway, WAF inspects 16 KB by default and can be configured up to 64 KB. Requests larger than that require an explicit oversize-handling policy. WAF does not enforce an arbitrary 256 KB or 10 MB application payload limit; API Gateway and the integration must enforce the architecture’s actual ceiling separately.

The second line of defence is authenticating the request itself:

• A producer can send a static authentication token in a header.
• A producer can sign the request body with HMAC and send the signature in a header.

These two mechanisms should not be confused. A Lambda authorizer can check a token in a header, but it cannot see the request body. Therefore, it cannot verify an HMAC calculated over that body. HMAC verification must happen in a Lambda integration or processing step that receives the raw request bytes.

flowchart LR
    P[External producer] --> W[WAF]
    W --> G[API Gateway REST]
    G -. optional header-token check .-> A[Lambda authorizer]
    G --> V[Verification Lambda]
    A -. authorized request .-> V
    V -->|valid signature| Q[SQS queue]
    V -->|invalid request| R[Reject with 4xx]
    Q --> L[Lambda worker]

The authorizer path is optional and applies to simple header-based authentication. The verification Lambda is the important path for body-based signatures. Only verified events should reach the durable work queue and downstream consumers.

Stage 5: Someone sends you a genuinely large payload

We finally have a complete architecture. Surely it is time to shave a few milliseconds from the happy path and declare victory.

Not quite. Every service in the pipeline has a payload limit, and the smallest limit on the chosen path wins:

• API Gateway accepts requests up to 10 MB.
• A synchronous Lambda invocation accepts 6 MB, while an asynchronous invocation accepts 1 MB.
• SQS accepts messages up to 1 MiB.
• EventBridge PutEvents requests must remain below 1 MB.
• SNS accepts messages up to 256 KiB.

These limits apply to the complete message seen by each service, not only to the provider’s raw JSON. An event envelope, message attributes, escaping, and base64 encoding all consume space. Base64 alone adds roughly one third, which is an impressively expensive way to make bytes look polite.

Most webhook events are small, often well below 100 KB, so the direct path is a good default. Large events force us to stop moving the body through every service. Instead, we store the bytes once in S3 and send a small pointer containing the bucket and object key.

flowchart LR
    P[Webhook producer] --> G[API Gateway]
    G --> D{Fits every service<br/>on the direct path?}
    D -->|yes| V[Verify request]
    V --> T[SNS or EventBridge]
    D -->|no| S[(S3 raw payload)]
    S --> A[Verify asynchronously]
    A --> PT[Publish small pointer]

This introduces an important trust trade-off. If the payload is no larger than 6 MB, API Gateway can invoke Lambda synchronously, Lambda can verify the signature, and only valid bodies are written to S3. Invalid requests can still be rejected while the producer is waiting.

The result is two final designs: one that carries small verified events directly, and one that stores the raw body and carries a pointer.

The two final architectures

We have now collected the main building blocks. From here, there are two useful shapes:

• Direct processing architecture: the webhook body travels through the synchronous request and messaging path, with an enforced limit safely below SNS’s 256 KiB maximum.
• Pointer architecture: the raw body is stored in S3, and the rest of the system moves only a pointer to it. This ingress path is still limited by API Gateway’s 10 MB maximum.

The direct processing architecture is the simpler and faster option when payloads are small enough. The pointer architecture is the more flexible option when payload size, replay, and long-term governance matter. We will start with the direct version.

Direct processing architecture

Let us make this concrete with a Stripe webhook endpoint. This is a deliberately small-payload design: the complete Stripe event travels through API Gateway, the verification Lambda, SNS, and the consumer queues. It is easy to understand and has fewer moving parts than the pointer architecture.

The same shape can support other small webhook providers, but the verification details change from provider to provider. Stripe is our example because it uses a signed raw request body and has a well-defined event ID that we can carry through the pipeline.

flowchart LR
    P[Stripe] --> W[WAF]
    W --> G[API Gateway REST]
    G --> V[Verification Lambda]
    V -->|verified event| T[SNS verified-events topic]
    T --> Q1[SQS business queue]
    T --> Q2[SQS notification queue]
    T --> F[Firehose delivery stream]
    Q1 --> L1[Business Lambda]
    Q2 --> L2[Notification Lambda]
    F --> S[(S3 observability and governance bucket)]

The intended workload for this example is modest but bursty: fewer than 50 requests per second during normal operating hours, short spikes of 300 or 400 requests per second, and little or no traffic outside business hours. The system should absorb the spikes without making the provider wait for downstream processing.

Why Stripe is a toy example here

Stripe is a useful example because its events are usually small and its signatures use HMAC. It is also a toy example for this article: Stripe supports delivering events directly to EventBridge, so a custom API Gateway endpoint is not necessarily the best real-world Stripe integration in 2026.

For the toy version, the flow is:

Stripe -> WAF -> API Gateway REST -> verification Lambda -> SNS -> SQS -> business Lambda

The verification Lambda reads the raw request body and checks Stripe’s signature using an endpoint secret stored in Secrets Manager. API Gateway cannot perform this HMAC-over-body verification itself, and a Lambda authorizer cannot do it either because authorizers do not receive the request body.

For a provider without body signatures, this stage may be simpler. A static header token can be checked by a Lambda authorizer, while providers using asymmetric signatures need a public-key verification strategy instead of Stripe’s HMAC calculation.

Component responsibilities and configuration

WAF

WAF is the first filter at the public edge. Configure it to:

• allow Stripe IP ranges when they are stable and published. This can be the only configuration set, however there are a few nice to have ones:
  • apply a rate-based rule per source, with thresholds high enough for the expected 300 to 400 request-per-second bursts;
  • configure WAF oversize handling to reject bodies beyond the WAF inspection limit only when that limit is acceptable for the endpoint;
  • enable IP reputation and anonymous-IP managed rules where appropriate;
  • log every rule and start new rules in count mode before promoting them to block.

For a general provider, the IP allowlist is optional because many providers do not publish stable source ranges. The rate and reputation controls still apply. When the architecture allows bodies larger than WAF’s inspection limit, enforce that larger ceiling in API Gateway or the integration rather than implying that WAF inspected the complete body.

WAF is an abuse filter, not webhook authentication. A request that passes WAF still needs signature or token verification.

API Gateway REST

For the Stripe example, API Gateway exposes one regional route such as /webhooks/stripe. Its job is to provide a stable HTTPS endpoint, reject requests that are obviously malformed, and invoke the verification Lambda. It is not the component that authenticates Stripe.

At the API layer, we can require POST, require the Stripe-Signature header and the expected content type, and apply throttling. API Gateway enforces its hard service limit, while the verification Lambda rejects bodies above the lower direct-processing ceiling chosen by the application.

The integration must preserve the raw request body. Stripe calculates its signature over those exact bytes, and parsing the JSON before verification can change the input and make a valid event appear invalid. The Lambda receives the body and the Stripe-Signature value, loads the endpoint secret from Secrets Manager, and performs the verification using Stripe’s signing rules. This is the Stripe-specific part of the design. Other providers may use a different header, HMAC preimage, timestamp tolerance, or an asymmetric signature scheme.

API Gateway invokes the verification Lambda synchronously in this architecture. The Lambda should publish the verified event to SNS before returning success. Only after that publish succeeds should API Gateway return 2xx to Stripe. If the signature is invalid, return a terminal client error and record the rejection. If the Lambda, secret store, or SNS publish is unavailable, return a server error so Stripe can retry the delivery. Stripe retries live-mode deliveries for up to three days with exponential backoff, so this response distinction is operationally important.

For a general webhook provider, the same shape applies, but the provider-specific verification strategy belongs in configuration and code, not in the API Gateway authorizer. A Lambda authorizer can check a static token in a header, but it cannot verify an HMAC calculated over the request body because it does not receive that body.

Verification Lambda

For Stripe, this Lambda is the trust boundary. It receives the raw body and the Stripe-Signature header, loads the endpoint secret, checks the timestamp and signature, and extracts the Stripe event ID. It should:

• receive the raw body without parsing and reserialising it first;
• load secrets from Secrets Manager or another protected secret store;
• compare the expected signature using a constant-time comparison;
• enforce Stripe’s timestamp tolerance to reduce replay risk;
• publish only verified events to SNS;
• return a non-2xx result for transient verification or publishing errors so the provider can retry;
• classify a bad signature as a terminal rejection and record it for investigation.

We should keep this function small and fast. Give it enough memory for predictable startup and a timeout below Stripe’s response deadline. Include the Stripe event ID in every log and message attribute, and make publishing idempotent so a Stripe retry cannot create an accidental duplicate downstream event.

For other providers, the same Lambda boundary may select a different strategy: another HMAC header and preimage, a static token check, or verification against a public key.

SNS verified-events topic

SNS provides the fan-out boundary. Publish the verified Stripe event once, then let each consumer receive its own copy. Add message attributes such as provider=stripe, event_type, and event_id so subscriptions can use simple filter policies.

This topic should contain verified events only. Unverified traffic belongs in rejection logs or a separate quarantine path, never in a topic consumed by human-facing systems. For another provider, the topic can carry the same envelope while the provider name and event metadata change.

SQS consumer queues

Create one SQS queue per consumer. Each queue should have its own:

• visibility timeout, longer than the Lambda timeout;
• batch size and batching window;
• maximum receive count;
• DLQ and redrive policy;
• reserved or maximum concurrency;
• partial batch failure reporting when batch size is greater than one.

The queue is the consumer’s shock absorber. A database outage should increase that consumer’s backlog without stopping the other consumers.

Consumer Lambdas

Consumer Lambdas contain Stripe-related business logic, not provider authentication. One might update the payment state, another might notify support, and a third might feed analytics. They should be idempotent, classify transient and terminal failures, and emit structured logs with the Stripe event ID. A consumer should acknowledge a message only after its own work has completed successfully.

Firehose to S3

Subscribe an Amazon Data Firehose delivery stream to the verified SNS topic and deliver the Stripe events to an S3 bucket. This branch is not the processing path. It is the observability and governance path.

Use it for:

• an append-oriented copy of verified events;
• long-term reporting and audit queries;
• independent investigations when a consumer has a bug;
• compressed and partitioned data for analytics.

Partition the S3 objects by provider, event type, and date. Configure encryption, lifecycle rules, and a retention policy that reflects the governance requirement. Keep WAF logs and API Gateway access logs in their own log archive because Firehose receives the events published to SNS, not requests blocked at the edge.

Illustrative cost analysis

The exact bill depends on payload size, Lambda duration, log volume, number of consumers, retention, and region. For a rough estimate, assume:

• 50 requests per second for 8 business hours per day;
• negligible traffic outside those hours;
• 43.2 million events per month;
• two SQS consumers;
• 10 KB average event size;
• 256 MB Lambdas running for 100 ms on average;
• one Firehose stream delivering compressed data to S3;
• no unusually large DLQ or replay workload.

Component	Main cost driver	Rough monthly order of magnitude
API Gateway REST	43.2 million requests	~$150
Verification Lambda	Requests plus short execution time	~$25
SNS	Publishes plus delivery to two queues and Firehose	~30–70
SQS	Send, receive, delete, and visibility operations for two consumers	~100–150
Consumer Lambdas	Two workers, request count and execution time	~50–100
Firehose	Approximately 432 GB ingested before compression	~15–30
S3	Stored audit data, requests, and lifecycle retention	Usually tens of dollars or less at this volume
WAF, CloudWatch, Secrets Manager, KMS	Rules, logs, metrics, keys, and secrets	Highly variable; often a meaningful part of the bill

The illustrative total is roughly 400–550 per month before detailed logging, data transfer, and unusual retry costs. The important cost lesson is that the payload itself is not the only driver. API Gateway request volume, SQS operations, consumer count, and especially observability can dominate the bill.

This estimate also explains why the direct architecture is attractive for small events and a modest number of consumers. It is simple, fast, and reasonably cheap. Once payloads become large or replay and raw-byte retention become first-class requirements, the pointer architecture earns its extra S3 hop.

This estimate omits several costs that can add up quickly, including KMS, data transfer, WAF, NAT gateways, and detailed logging.

Pointer architecture

The direct architecture is elegant until the body becomes larger than the messaging path can carry, or until raw-byte retention and replay become requirements rather than nice-to-haves. The pointer architecture separates the payload from the event that describes it.

The raw webhook is written to S3 first under a server-generated ingestion ID. Before verification, any provider event ID or event type in the request is an untrusted claim. After the signature succeeds, the verifier can publish a small pointer containing the bucket, object key, authenticated provider, trusted provider event ID, and event type.

This is a significant trade-off. The API Gateway to S3 integration is deliberately very simple: it writes the request body directly to an incoming/ prefix before any signature or payload validation has happened. Nothing in that integration knows whether the request is genuine. WAF can reduce abuse at the edge, but WAF is not webhook authentication and it does not verify an HMAC over the body.

That means the incoming/ prefix is a staging area, not a trusted event store. An asynchronous Lambda must validate every object after it arrives. Only a valid object is published to SNS. An invalid object is copied to a dedicated invalid/ prefix, and that prefix must be monitored. Keeping invalid attempts is useful for forensics, but treating them as trusted events would be a fairly spectacular security bug.

flowchart LR
    P[External producer] --> W[WAF]
    W --> G[API Gateway REST]
    G --> S[(S3 incoming raw payload)]
    S --> V[Async verification Lambda]
    V -->|verified pointer| T[SNS verified-events topic]
    V -->|invalid payload| I[(S3 invalid prefix)]
    T --> Q1[SQS business queue]
    T --> Q2[SQS analytics queue]
    T --> F[Firehose]
    Q1 --> L1[Business Lambda]
    Q2 --> L2[Analytics Lambda]
    L1 -->|GetObject| S
    L2 -->|GetObject| S
    F --> A[(S3 audit bucket)]

The target workload is the same as before: fewer than 50 requests per second during business hours, short spikes of 300 or 400 requests per second, and no meaningful traffic overnight. The difference is that the payloads may be large, and we want the original bytes to remain available for verification, replay, and governance.

If a maximum payload size of 6 MB is acceptable, there is a simpler alternative. API Gateway can invoke the verification Lambda synchronously, and the Lambda can verify the request before writing valid bodies to incoming/. Invalid requests can be rejected immediately and recorded in logs or a quarantine store. This restores validation before storage, but puts Lambda back on the provider-facing path and inherits the synchronous Lambda payload limit.

The pointer architecture accepts the larger operational trade-off in exchange for archive-first durability and a larger practical payload envelope. The 6 MB alternative accepts the smaller payload limit in exchange for a stronger trust boundary before anything reaches S3.

Real-world examples

GitHub, SendGrid’s Inbound Parse, DocuSign, Salesforce, HubSpot, and some data-rich Shopify webhook topics can all produce payloads that do not fit comfortably within SNS’s 256 KiB limit. Not every event from these providers is large, but payload size may grow with attachments, line items, custom fields, or batched records. If the provider does not guarantee a suitably small upper bound, designing around an S3 pointer avoids finding that limit for the first time in production, which is rarely where one wants to meet new constraints.

Ingestion identity and storage envelope

API Gateway REST writes each request to a key such as incoming/github/<ingestion-id>. The ingestion ID must be generated by our infrastructure, not copied from an unverified header or body field. API Gateway’s $context.extendedRequestId is suitable because API Gateway generates it and the client cannot override it. A claimed GitHub delivery ID can be stored for later verification and correlation, but it must not control the object key or overwrite another request.

This means ingestion deliberately does not deduplicate by provider event ID. Two retries create two raw objects. After verification, the provider event ID becomes a trusted idempotency input and can be used to suppress duplicate publication or duplicate business effects. Consumers must remain idempotent because publication and queue delivery are still at least once.

The S3 object is the storage envelope:

Part	Stored value	Trust before verification
Object key	Server-generated ingestion ID under the provider route prefix	Trusted as an ingestion identifier only
Object body	Exact request bytes as received by the integration	Untrusted payload
provider-route metadata	Provider selected by API configuration, such as github	Trusted routing context, not proof of sender identity
signature metadata	Value of the one provider-specific signature header	Untrusted, required for verification
delivery-id metadata	Optional provider delivery ID	Untrusted correlation hint
received-at metadata	Server-generated receipt time	Trusted ingestion context
Content type and encoding	Allowlisted request values	Untrusted but required to decode the body correctly

Use S3 user metadata for this small, fixed envelope. API Gateway maps only the expected headers into explicit x-amz-meta-* fields. Do not copy every incoming header. Cookies, authorization headers, forwarding headers, and arbitrary attacker-controlled values do not belong in the object metadata.

S3 user-defined metadata is limited to 2 KB in total and must use ASCII-compatible keys and values. Set strict per-field length limits and reject a required signature header that is missing, malformed, or too large. If a provider requires more context than fits in this fixed envelope, the direct API Gateway to S3 integration is no longer a good fit. Use an ingestion Lambda or another explicit envelope store instead.

Byte preservation is part of the contract. The S3 integration must pass the request body through without a JSON mapping template that parses or reserialises it. Configure API Gateway binary media types and contentHandling deliberately. For a JSON-only webhook endpoint, the simplest contract is to accept the expected JSON content type, reject unsupported content encodings, and verify through integration tests that whitespace, Unicode, and trailing newlines arrive byte for byte. If gzip or binary webhook bodies are supported, store the encoding state explicitly and test the decode path against real signed fixtures.

This is the first important difference from the direct architecture: the provider receives success once this raw envelope is durable, not once verification and downstream publishing have completed. We have therefore accepted responsibility for replaying objects that fail later in the pipeline.

Asynchronous verification

S3 invokes the verification Lambda asynchronously for objects under the incoming/ prefix. The Lambda reads the raw object, loads the provider secret, and verifies the signature against the exact stored bytes. For GitHub this means using the relevant HMAC header and secret; for another provider it may mean a different HMAC scheme or public-key verification.

A bad signature is a terminal result. Copy the object to the invalid/ prefix, record the reason, and return success from the Lambda so it is not retried forever. Monitor the prefix and alert on unusual increases. An unexpected error, such as a GetObject, secret-store, or publish failure, should fail the invocation so Lambda retries it. Configure an onFailure destination and an alarm because the provider has already received a successful storage response and will not necessarily resend the event.

The S3 object remains in incoming/ even after an invalid copy is created. This preserves a complete raw log and keeps replay possible without relying on an atomic move.

Consumer processing

SNS or EventBridge carries only the pointer. Each consumer still receives its own SQS queue, DLQ, retry policy, and concurrency settings. The worker calls GetObject, checks that the object is still within its retention window, and processes the raw payload.

The S3 lifecycle policy is now part of the reliability design. incoming/ must live at least as long as the longest consumer DLQ retention, verification failure-destination retention, and expected replay window.

Firehose and governance

Firehose can subscribe to the verified topic and write a compact pointer or verified-event envelope to an audit bucket. The raw payload remains in the incoming bucket, while the Firehose stream provides a convenient append-oriented dataset for analytics, reporting, and governance.

For strict compliance, apply encryption, access logging, lifecycle policies, and optionally Object Lock to a curated archive. Keep WAF and API Gateway logs separate because they describe requests at the edge, including requests that never became S3 objects.

Latency and operational trade-offs

The provider-facing path is still quick because it ends at the S3 write. Processing latency is higher and less predictable than in the direct architecture because verification waits for the S3 notification, and consumers perform an additional GetObject. In exchange, the system survives verification outages, supports replay from the raw object, and is no longer limited by the 256 KB message buses downstream.

This architecture also changes the failure contract. A 2xx response means “the raw event is safely stored,” not “the event has been verified and processed.” That distinction must be visible in dashboards, alarms, runbooks, and documentation.

Illustrative cost analysis

Use the same 43.2 million events per month as the direct example, but assume a 100 KB average payload, two consumers, and a 30-day S3 retention period. That produces roughly 4.3 TB of raw ingress before compression and lifecycle expiry.

Component	Main cost driver	Rough monthly order of magnitude
API Gateway REST	43.2 million requests	~$150
S3 raw storage	4.3 TB-month before lifecycle expiry	~$100
S3 requests	PUT, verification GET, and consumer GET operations	~250–350
Verification Lambda	Async invocations and short reads	~25–50
SNS and SQS	Pointer fan-out and two consumer queues	~130–200
Consumer Lambdas	Two workers and their execution time	~50–100
Firehose audit branch	Pointer/envelope volume and delivery	~15–30
WAF, CloudWatch, KMS, and logs	Security and operations	Highly variable

The illustrative total is roughly 700–1,000 per month before data transfer, retries, and detailed log volume. The direct architecture is cheaper when events are small because it avoids the S3 PUT and GET operations. The pointer architecture becomes attractive when payload size, replayability, and raw-byte governance are worth paying for.

Operating the two architectures

Architecture diagrams are unusually calm places. Production brings duplicate deliveries, cold starts, poison messages, expired objects, broken secrets, and third-party outages.

Both architectures use managed services with at-least-once delivery semantics. They are designed to avoid losing accepted events, but neither provides exactly-once business processing. Operational reliability comes from idempotent handlers, explicit failure sinks, measurable state transitions, and rehearsed recovery procedures.

At-least-once delivery and idempotency

An event can appear more than once in either architecture:

• the webhook provider can retry when it does not receive 2xx;
• the verification Lambda can publish successfully and then time out before returning;
• SNS and SQS can redeliver messages;
• an SQS batch can be retried after one record fails;
• S3 notifications are delivered at least once and can occasionally arrive more than once or out of order;
• an operator can replay or redrive an event manually.

The system should therefore treat duplicates as normal input, not as an unlikely accident.

After verification, use a stable idempotency key such as <provider>:<event-id>:<consumer>. Each consumer records that key using a conditional database write before performing an irreversible side effect. If the key already exists with a completed status, the consumer acknowledges the message without repeating the work. If the consumer calls an external API, pass the same idempotency key when that API supports it.

Idempotency must exist at every side-effect boundary. The pointer architecture intentionally stores every unverified request under a server-generated ingestion ID, so it does not deduplicate at ingress. After verification, a trusted provider event ID can suppress duplicate publication, but that still does not protect a consumer from an SQS retry. There is no single magic deduplication table that can politely solve every boundary for us.

When a database update and event publication must succeed together, use an outbox or another transactional pattern. Otherwise, a function can commit the database transaction, fail before publishing, and leave the system in a state that no retry can infer safely.

What does 2xx mean?

The acknowledgement has a different meaning in each architecture:

Architecture	Meaning of 2xx	Who owns recovery afterward?
Direct processing	The signature was verified and the event was published to the durable fan-out boundary	SNS, SQS, and the consumer teams
Pointer	The raw request was stored in the S3 incoming/ prefix	The verification pipeline and then the consumer teams

The pointer architecture acknowledges earlier, which makes ingestion resilient to verification outages. The price is that the provider will not retry when verification later fails. The S3 object, asynchronous failure destination, alarm, and replay procedure become part of the delivery guarantee.

Latency distributions

Latency should be measured as two separate journeys:

1. Acknowledgement latency: from the provider opening the HTTP request until it receives the response.
2. End-to-end latency: from the request arriving until a consumer completes its business work.

The following figures are planning ranges for a healthy, same-region deployment with warm Lambdas, no batching window, small messages, and approximately 50 to 100 ms of business logic. They are not AWS guarantees and should be replaced with measurements from a load test and production telemetry.

Architecture and journey	Average	p75	p90	p99
Direct: provider acknowledgement	150–300 ms	250–400 ms	400–700 ms	1–1.5 s
Direct: consumer completed	400–700 ms	600–900 ms	1–1.5 s	2–3 s
Pointer: provider acknowledgement	80–200 ms	150–300 ms	250–500 ms	0.8–1.5 s
Pointer: consumer completed	0.7–1.2 s	1–1.5 s	2–3 s	4–10 s

The direct architecture pays for signature verification and SNS publishing before returning to the provider, but reaches consumers with fewer hops. The pointer architecture returns after the S3 write, so acknowledgement can be faster, but end-to-end processing waits for an S3 notification, asynchronous verification, fan-out, queue polling, and GetObject.

The tail matters more than the average. S3 notifications usually arrive within seconds but can occasionally take a minute or longer. Cold starts, Lambda throttling, non-zero SQS batching windows, large S3 reads, and downstream APIs all stretch p99. Provisioned concurrency can reduce Lambda cold-start latency, but it cannot tune away S3 notification or SQS pickup time.

Track these distributions separately by provider, event type, and consumer. An average of 400 ms can look healthy while one percent of payment events quietly takes thirty seconds.

The operational funnel

The dashboard should describe the event’s journey rather than presenting a wall of unrelated AWS charts:

received -> accepted/stored -> verified -> published -> queued -> processed
                              |                         |
                              v                         v
                           invalid                    DLQ

Emit semantic counters at each transition:

• received: API Gateway accepted the HTTP request;
• stored: the pointer architecture wrote the S3 object;
• signature_valid: verification succeeded;
• signature_invalid: verification rejected the payload;
• verification_error: verification failed for an operational reason;
• published: the verified event reached SNS or EventBridge;
• processed: a consumer completed successfully;
• processing_failed: a consumer exhausted its retries.

Dimension these metrics with low-cardinality fields such as provider, event_type, consumer, stage, and outcome. Never use the event ID as a metric dimension. That creates one metric per webhook and turns CloudWatch billing into its own incident.

The funnel should reconcile over a time window rather than request by request. At-least-once delivery means retries can make raw service counts larger than the number of unique provider events.

Service metrics and alarms

The semantic funnel sits on top of the native AWS metrics:

Layer	Metrics to watch	Useful alarm
WAF	allowed requests, blocked requests, rate-rule matches	Sudden block spike or unexpected allowed-traffic spike
API Gateway	Count, 4XXError, 5XXError, Latency, IntegrationLatency	5xx rate or p99 acknowledgement latency above the provider deadline
Verification Lambda	Invocations, Errors, Throttles, Duration	Error rate, throttles, or duration near timeout
Async verification	AsyncEventAge, AsyncEventsDropped, failure-destination depth	Any dropped event or growing event age
SNS	messages published and delivery failures	Delivery failure to a subscriber
Consumer SQS	ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage, in-flight messages	Oldest-message age above the processing SLO
Consumer Lambda	errors, throttles, duration, concurrency	Sustained errors, throttles, or exhausted concurrency
DLQ	ApproximateNumberOfMessagesVisible	Greater than zero
S3 pointer store	PUT/GET errors, object count, bytes stored	Storage growth outside the expected retention envelope

Traffic naturally falls to zero outside business hours in our example, so a simple low-volume alarm will be noisy. Use a business-hours schedule or anomaly detection to alert when expected traffic disappears. A sudden drop can indicate a provider outage, an expired certificate or a stale IP allowlist.

DLQs are workflows

Every consumer queue should have a DLQ, but creating one is only the beginning. A DLQ without an alarm, owner, and redrive procedure is just a quiet place where events go to retire.

When a DLQ receives a message:

1. Alert the owning team immediately.
2. Inspect one message using the event ID, without copying sensitive payloads into tickets or chat.
3. Classify the failure as transient, a code defect, bad data, or an expired dependency.
4. Confirm that the handler is idempotent and, for the pointer architecture, that the S3 object still exists.
5. Fix or mitigate the cause.
6. Redrive at a limited velocity while watching errors, queue age, and downstream capacity.
7. Confirm that the funnel reaches processed before resolving the incident.

Do not automatically redrive terminal business rejections or invalid signatures. The invalid/ prefix is a quarantine and forensic record, not a retry queue.

The pointer architecture has two additional failure paths. Verification errors go to the Lambda onFailure destination after asynchronous retries are exhausted. Operators replay those events by re-invoking verification with the stored S3 pointer. Invalid signatures go to invalid/ and should trigger a rate-based alarm, particularly when the normal invalid rate is close to zero.

AWS recommends setting the SQS visibility timeout to at least six times the Lambda timeout, plus any batching window, to leave room for throttling and retries. Enable ReportBatchItemFailures for batches larger than one so a single poison message does not replay the successful records in the same batch. Keep the source queue and DLQ retention periods shorter than the S3 raw-object retention period.

Out-of-order events

At-least-once delivery is only half of the fun. Events can also arrive out of order.

A provider can generate two related events close together, retry the older one, and deliver it after the newer one. SNS and SQS Standard preserve high throughput but do not guarantee strict ordering. Parallel Lambda consumers make reordering even more visible. In the pointer architecture, S3 notifications are also not guaranteed to arrive in order.

The safest rule is to make ordering a domain concern. A consumer should compare a provider sequence number, resource version, or monotonic business state before applying an update. An event timestamp is useful evidence, but timestamps alone are not always enough because clocks, retries, and independently generated events can disagree.

For Stripe, an event describes something that happened, but it should not always be treated as the unquestionable current state of the object. When order matters, retrieve the latest resource state from Stripe or apply transitions through a domain state machine. For example, an old payment_intent.processing event should not move a payment backward after payment_intent.succeeded has already been applied.

SQS FIFO can preserve the order in which messages are placed on the queue within a message group. A useful message group might be the payment, subscription, or customer ID. This allows unrelated resources to process in parallel while serialising events for the same resource.

FIFO does not repair events that were already produced out of order, and it reduces concurrency for hot message groups. Use it only when ordered application is a real requirement. Most consumers are better served by Standard queues plus version-aware, idempotent business logic.

Reference table

AWS service	Maximum payload / item size	Representative unit cost
API Gateway HTTP API	10 MB	Approximately $1.00 per million requests for the first 300 million requests. Payloads are metered in 512 KB increments.
AWS Lambda, synchronous invocation	6 MB request and response	0.20 per million invocations**, plus approximately **0.0000166667 per GB-second for standard x86 execution.
AWS Lambda, asynchronous invocation	1 MB	$0.20 per million invocations, plus execution duration. Retries generate additional invocation and compute charges.
Lambda Function URLs	6 MB	A tiny HTTP front door for Lambda. Nice for simple things, but API Gateway provides more control for public webhooks.
Amazon SQS Standard	1 MiB per message	Approximately $0.40 per million API requests. Send, receive, delete, and visibility operations are billed separately. Each 64 KiB payload chunk counts as one request unit.
Amazon EventBridge PutEvents	1 MB per event entry	Approximately $1.00 per million custom events ingested. Each 64 KB chunk counts as one event.
EventBridge Pipes	256 KB	Glue between a source and a target. It can filter, enrich, and move events without another small Lambda whose only job is connecting services.
Amazon SNS Standard	256 KiB	Approximately $0.50 per million API requests. Each 64 KB payload chunk counts as one request unit. Delivery charges may also apply.
AWS Step Functions	256 KiB	Standard workflows: approximately 0.025 per 1,000 state transitions**. Express workflows: approximately **1.00 per million executions, plus duration and memory charges.
Amazon Kinesis Data Streams	1 MiB	More Kafka-ish than queue-ish. Useful when ordering, replay, high throughput, or multiple consumers really matter. Otherwise, SQS is usually the boring and good choice.
Amazon DynamoDB	400 KB per item	On-demand Standard tables: approximately $0.625 per million write request units, plus storage. Larger items consume multiple request units.
Amazon S3	5 TB per object	S3 Standard: approximately 5.00 per million PUT requests**, **0.40 per million GET requests, and $0.023 per GB-month for the first storage tier.

So which one would I actually build?

If I had to pick today, for the kind of small-to-medium integration this post is scoped to: start with the direct architecture. It’s cheaper, it’s faster end-to-end, and it’s easier to reason about during an incident. Only reach for the pointer architecture once you have a provider whose payloads genuinely blow past 256 KB, or a compliance requirement demanding raw-byte retention and replay.

Useful further reading

The following posts explore related architectures from slightly different angles. Some predate current AWS quotas and features, but the design lessons remain useful:

• Sending and receiving webhooks on AWS: Innovate with event notifications AWS reference architectures for both sides of the webhook boundary, including the claim-check pattern for larger payloads.
• Beyond webhooks: Event-driven payment architectures with Amazon EventBridge Shows when a payment provider’s native EventBridge integration can remove the public webhook endpoint entirely.
• Reliable Webhooks Using Serverless Architecture Square’s experience using one SQS queue per application, Lambda workers, retries, rate limits, and operational metrics.
• Webhook Processing with API Gateway and SQS A focused walkthrough of the smallest API Gateway-to-SQS ingestion pattern.
• Lessons Learned From Sending Millions of Serverless Webhooks Production lessons from Dwolla on per-customer queues, concurrency, cold starts, log retention, quotas, and operating many serverless resources.

What comes next

In the next post, we will implement the Pointer Architecture with Terraform and deal with the tiny details where the devil has rented a very comfortable apartment.

We will move beyond the clean architecture diagram and focus on the parts that usually decide whether the system works in production: SQS batching and partial failures, visibility timeouts, DLQ redrive, cold starts, Lambda concurrency, RDS Proxy and database connections, NAT costs, IAM permissions, KMS and encryption, S3 lifecycle policies, retention, Glacier transitions, replay tooling, alarms, runbooks, and dashboards.

In other words, we will turn the boxes and arrows into infrastructure that can survive real payloads, real failures, and real production incidents.