Building the Pointer Architecture with Terraform

In the previous post, we worked our way from a Lambda behind an HTTP endpoint to a complete serverless webhook ingestion pipeline. We compared direct processing with the pointer architecture, where the raw payload lives in S3 and the messaging layer carries only a small reference to it.

The complete implementation is available in the serverless webhook ingestion repository.

I am going to use Stripe because it is the go-to example for webhook ingestion discussions and tutorials. Let me repeat something from the previous post, though: this is a toy example. Stripe is nice enough to send payloads that are generally smaller than 256 KB, so this architecture may be overkill.

Before we build it, there is one tempting variation worth getting out of the way: what if we validate the webhook before storing it?

The tempting version: verify before storage

It looks like this:

flowchart LR
    P[Webhook provider] --> W[WAF]
    W --> A[API Gateway REST]
    A --> V[Synchronous validation Lambda]
    V -->|Valid payload| S[(S3 verified payloads)]
    V -->|Invalid signature| X[4xx response]
    S --> T[SNS pointer topic]
    T --> Q1[SQS business queue]
    T --> Q2[SQS analytics queue]
    Q1 --> C1[Business Lambda]
    Q2 --> C2[Analytics Lambda]

It feels like the safer design. The validation Lambda sits at the entrance, checks the signature, and only lets authenticated payloads reach S3. Invalid requests receive a 4xx, our bucket stays clean, and everyone is happy.

There is, however, a small problem. It is the sort of small problem that can make an event disappear forever.

API Gateway invokes the validation Lambda synchronously and does not retry it. If the function is throttled, runs out of memory, times out, cannot read its secret, or crashes before PutObject finishes, API Gateway returns a 5xx. At this point the payload has not reached S3, SQS, or any other durable service.

If the webhook provider retries non-2xx responses, we get another chance and everything may still work. If the provider does not retry, the event is lost.

This is the flaw: verification happens before the first durable write. The design gives us a clean trust boundary, but its delivery guarantee depends on behavior outside our system. It can still make sense for a provider with reliable retries or a redelivery API, but it is not a safe general-purpose architecture.

The architecture we will build

The pointer architecture we are going to build accepts the uncomfortable part instead. API Gateway stores the raw request in an untrusted S3 staging prefix before returning 2xx. S3 sends an object-created notification to an SQS verification queue, and the validation Lambda polls that queue. Valid payloads continue through the pointer pipeline; invalid ones are quarantined or expired. Operational failures return to the queue and eventually move to its DLQ if they keep failing.

From this point on, when I say pointer architecture, I mean this archive-first version.

flowchart LR
    P[Webhook provider] --> W[WAF]
    W --> A[API Gateway REST]
    A --> S[(S3 untrusted staging)]
    S --> Q[SQS validation queue]
    Q --> V[Validation Lambda]
    V -->|Valid pointer| T[SNS verified pointer topic]
    V -->|Invalid signature| I[(S3 quarantine)]
    Q -->|Repeated operational failure| D[Validation DLQ]
    T --> Q1[SQS business queue]
    T --> Q2[SQS analytics queue]
    Q1 --> C1[Business Lambda]
    Q2 --> C2[Analytics Lambda]
    C1 -->|GetObject| S
    C2 -->|GetObject| S

Ingestion

For this post, ingestion is everything between the webhook provider and the verified SNS notification: WAF, API Gateway, S3, the SQS verification queue, the validation Lambda, and finally SNS. SNS is the interface between ingestion and the rest of the processing pipeline.

SNS might be replaced by EventBridge Pipes, but the core idea remains the same.

Let’s set WAF aside for the time being. It is important, but it is also the least interesting part of this particular journey. Let’s focus instead on what we are trying to produce.

The first goal is to store the raw request durably. The final goal is to publish a small SNS message telling the rest of the system that there is a new, verified event to process.

Naively, we could publish a pointer containing only the bucket and key:

{
  "bucket": "growing-bits-webhooks",
  "key": "incoming/stripe/1722470400000/ingestion-id"
}

This works, but it gives consumers very little information. Every subscriber receives the same opaque pointer, and filtering by event type becomes difficult.

Let’s make this a little more concrete. If we are ingesting Stripe webhooks, one consumer may care about payment_intent.* and charge.*, while another may care only about refund.* or charge.dispute.*. SNS can filter those subscriptions, but only if the validation Lambda publishes the trusted event type as a message attribute or as part of the verified envelope.

Storage governance is another interesting detail. Ideally, we might like a beautifully organised S3 path such as:

s3://growing-bits-webhooks/verified/stripe/2026/08/01/event_type=payment_intent.succeeded/<event-id>.json

Unfortunately, API Gateway cannot safely create that path during ingestion. At that point, the event type and provider event ID are still untrusted values from the request body. Using either of them in the object key would allow an attacker to control our storage layout or overwrite another event. API Gateway is also fairly limited in how it can construct calendar-based paths without transforming the request.

The raw path therefore has to be a little more boring:

s3://growing-bits-webhooks/incoming/stripe/<request-time-epoch>/<ingestion-id>

Both the receipt timestamp and ingestion ID are generated by API Gateway.

Preserving the exact bytes

There is also an hidden details that we need to take care of: Webhook signatures are calculated using the exact bytes sent by the provider, not the parsed JSON object.

We want to store the unchanged payload in S3 together with the information needed to verify it later. The payload itself becomes the S3 object body, while headers such as Stripe-Signature must be mapped into S3 object metadata.

The flow is fairly simple:

1. The producer sends the JSON payload and the signature (example Stripe-Signature ) header to API Gateway.
2. API Gateway passes the body to S3 without parsing or transforming it.
3. API Gateway maps Stripe-Signature to an S3 metadata field such as x-amz-meta-stripe-signature.
4. S3 stores the original body and the additional metadata together.
5. The validation Lambda downloads the object and reads both the raw bytes and the stored signature.
6. It uses those two values to verify that the payload really came from Stripe.

We should map only the small set of headers we actually need, rather than trying to preserve the entire HTTP request. For Stripe, that is mainly Stripe-Signature, together with useful context such as the content type, ingestion ID, and receipt timestamp. The important part is that API Gateway never parses and rebuilds the JSON before storing it: the bytes read by the validation Lambda must be the same bytes Stripe originally signed.

Now that we have found a way to encode the headers and body of an HTTP request into an S3 object, we are ready to focus on the processing phase.

The simplest solution: keep the original key

If S3 is mostly a durable buffer and our consumers rarely inspect it directly, there is no need to reorganise anything. API Gateway writes the request under the incoming prefix, S3 notifies the validation pipeline, and the validation Lambda verifies the exact stored bytes before publishing the pointer to SNS.

Every ingestion key is write-once. API Gateway uses a conditional S3 write that fails if the generated key already exists, and the processing roles can read or tag the object but cannot replace its body. The pointer deliberately contains no S3 version identifier: bucket and key identify immutable bytes until the retention policy expires them. Tags may change after verification, but the payload they describe may not.

flowchart LR
    A[API Gateway] --> S[(S3 incoming)]
    S --> Q[SQS validation queue]
    Q --> V[Validation Lambda]
    V --> T[SNS verified pointer]

If every subscriber processes every event, the SNS message can remain deliberately boring:

{
  "bucket": "growing-bits-webhooks",
  "key": "incoming/stripe/1722470400000/ingestion-id"
}

The consumers already know the provider from their configuration and can download the object when they need it. No second copy, no tags, no catalogue, and very little to explain. This is the option I would start with unless we have a concrete requirement for something more elaborate.

If SNS filtering is useful, the Lambda can still extract the event type after verification and add it to the verified envelope. That does not require changing the S3 layout.

When S3 is a real output of the pipeline

Sometimes S3 is more than a temporary home for a large webhook. Imagine a hybrid streaming-and-batch system: we process events continuously from SNS and SQS, but we also run a daily batch over the same history. In that case, the organisation of the objects matters. Making every batch job scan incoming, download objects, and rediscover their event types would be wasteful.

After verifying the signature, the Lambda can extract the event type and provider event ID and make a server-side copy into a partitioned prefix:

s3://growing-bits-webhooks/verified/
  provider=stripe/
  event_type=payment_intent.succeeded/
  year=2026/
  month=08/
  day=01/
  <event_id>.json

The date partition should normally come from our trusted receipt timestamp. A timestamp inside the webhook may be old, absent, or simply wrong. The event type must also be normalised before we let it influence an S3 key.

The sequence is now:

verify signature
    -> extract and normalise metadata
    -> copy the object to the verified prefix
    -> publish the verified pointer to SNS

CopyObject performs the copy inside S3, so the Lambda does not need to upload the payload again. The SNS message points to the verified location and includes provider, event_type, provider_event_id, received_at, and ingestion_id, giving SNS enough trusted metadata for subscription filtering.

Keeping both copies forever would, of course, mean paying for both. We can put a short retention policy on incoming and expire the original after the verified copy has had enough time to settle. I would keep a safety window rather than deleting the source immediately. An S3 “move” is really a copy followed by a delete, not an atomic operation, and the original is also the exact request we used for signature verification.

There is an important reliability consequence here: if the batch layer depends on the verified prefix, the copy is no longer a nice-to-have. A failure to create it must fail the SQS attempt and be retried. Otherwise the streaming path may see an event that is permanently missing from the batch dataset.

The middle ground: tag the incoming object

There is also a useful compromise. Keep the immutable payload in incoming, but tag the object after verification:

verification_status=verified
provider=stripe
event_type=payment_intent.succeeded
provider_event_id=evt_123

Then publish the same trusted values in the SNS envelope. This avoids copying a potentially 10 MB object while still giving us metadata for lifecycle policies, access control, cost allocation, and operational inspection.

Tags do not automatically create an index: the normal S3 API still lists objects by key prefix, not by tag. There is, however, a useful AWS-native option. If we enable an S3 Metadata inventory table for the bucket and integrate its table bucket with the Glue Data Catalog, the object tags become queryable from Athena:

SELECT
  key,
  object_tags['provider_event_id'] AS stripe_event_id,
  last_modified_date AS received_at
FROM "s3tablescatalog/aws-s3"."b_growing_bits_webhooks"."inventory"
WHERE object_tags['provider'] = 'stripe'
  AND object_tags['event_type'] = 'payment_intent.succeeded'
  AND object_tags['verification_status'] = 'verified'
  AND last_modified_date >= date_add('day', -1, current_timestamp)
ORDER BY last_modified_date DESC;

That makes the tagging option much more interesting for discovery, audits, and replay tooling. It is not a real-time index: changes are typically visible in the live inventory table within an hour. It also does not give our batch jobs the same efficient physical layout as objects already partitioned by event type and date. Athena can find the keys, but the batch still has to read the payloads from their scattered incoming locations.

Why SQS sits between S3 and Lambda

S3 can invoke the validation Lambda directly, and for a small pipeline that can be a reasonable design. It has one less component and slightly less latency. It is not, however, the architecture we are building here.

In this architecture, S3 sends each object-created notification to an SQS verification queue. Lambda polls that queue through an event source mapping. The queue gives us a visible backlog, a measurable age for the oldest unverified webhook, explicit concurrency control, and a straightforward redrive path after we fix a problem.

Terraform has to wire both sides of that hand-off. The SQS queue policy allows our webhook bucket to call sqs:SendMessage, restricted by the bucket ARN and our AWS account. The S3 bucket notification targets the queue only for ObjectCreated:Put events under incoming/, so copying an invalid object into quarantine does not create a verification loop. The Lambda event source mapping then connects the verification queue to the function.

The retry contract is now unambiguous. If verification fails for an operational reason, the Lambda reports that SQS record as failed. The message becomes visible again after the visibility timeout and receives another attempt. After maxReceiveCount failed receives, SQS moves it to the verification DLQ. There is no Lambda asynchronous retry queue and no Lambda onFailure destination in this path.

The extra queue costs us another hop. Lambda has to poll it and, if we configure a batching window, may deliberately wait for more messages before invoking the function. This adds some latency. In our case that is a fair trade: we already returned 2xx to the provider and chose asynchronous processing, so shaving a little time off verification matters less than knowing exactly where our events are when something goes wrong.

SQS also gives us batching, although we should be a little careful with it. The SQS messages are tiny because they contain only S3 pointers, but the work behind each pointer is not tiny: every one of them may require downloading and parsing a 10 MB object. A batch size of ten could therefore mean downloading up to 100 MB in one Lambda invocation, depending on how the Lambda code reads it.

I would start with a batch size of one, keep the behaviour predictable, and increase it only after measuring the function. If we do process batches, we need partial batch responses; otherwise one bad object makes Lambda retry all the successful objects that happened to be in the same batch.

The verifier also has to unwrap the S3 event from the SQS message body before it can read the bucket, key, and object metadata. That extra envelope is easy to overlook when moving from a direct S3 trigger to a queue.

Finally, SQS gives us control over concurrency. S3 and SNS are unlikely to be the bottlenecks here, so we can give the validation Lambda fairly high concurrency and drain the queue quickly. This is the path implemented by the Terraform in this repository: S3 to SQS to Lambda, with an SQS DLQ for records that exhaust their receives.

Is the validation Lambda doing too much?

It is doing several operations, but they still belong to one coherent responsibility: turning an untrusted stored request into a trusted event. Verifying the signature, extracting routing metadata, optionally classifying or promoting the object, and publishing the trusted pointer are not four unrelated applications hiding in one function.

I would not introduce Step Functions for this short, linear sequence. It would make the individual states and retries visible, but it would not give us a transaction across S3 and SNS, nor would it remove the need for idempotency. It would also carry only the pointer: Step Functions cannot carry our 10 MB webhook through its state.

Consumers

Once the event is in the SNS topic, we can send it almost anywhere: SQS, Lambda, HTTP endpoints, Firehose, and a few other places. I am not going to discuss every type of subscriber and all its quirks. For this architecture, I care about two common cases:

• a worker built from SQS, Lambda, and a DLQ for business processing;
• delivery to an internal or external HTTP service.

An SNS topic can fan out the same event to all of them:

flowchart LR
    T["SNS verified-events topic"] --> Q1["Business SQS"]
    T --> Q2["Analytics SQS"]
    T --> F["Data Firehose"]
    T -. optional .-> H["External HTTPS endpoint"]

    Q1 --> L1["Business Lambda"]
    Q2 --> L2["Analytics Lambda"]
    F --> S["Snowflake or S3"]

Worker consumers

I use worker as a small abstraction made of three pieces:

• SQS is the todo list;
• Lambda is the brain doing the work;
• the DLQ is the “I tried several times and I have no idea how to fix this” list.

flowchart LR
    T["SNS verified-events topic"] --> S["SNS subscription"]
    S --> Q["SQS worker queue"]
    Q --> L["Lambda worker"]
    Q -->|"maxReceiveCount reached"| D["SQS processing DLQ"]
    L -->|"GetObject"| B["S3 payload"]

Let’s start from SNS. We have a verified event there and want our SQS queue to receive it. Luckily, AWS provides a direct SNS-to-SQS integration, so we do not need a Lambda whose only purpose is to move one JSON document from one AWS service to another.

There are still a few things to configure. We create the queue, subscribe it to the topic, and add a queue policy allowing that specific SNS topic to call sqs:SendMessage. The “specific” part matters: the queue should not accept messages from every SNS topic in the account just because configuring Principal = "*" was convenient at the time.

This is also where SNS filtering becomes useful. A payment worker may subscribe only to payment_intent.*, while an analytics worker may receive everything. The verifier publishes trusted values such as provider, event_type, and schema_version as message attributes, and each subscription decides which ones it wants. Filtering before SQS is both cheaper and easier than waking up a Lambda only for it to say “not my event” and return.

I would enable raw message delivery for the subscription. Without it, SNS wraps our pointer inside its own notification envelope, and the worker has to decode an SQS message containing an SNS message containing our actual message. Raw delivery lets SQS receive the pointer envelope directly. There is one small limit to remember: an SNS message sent to SQS with raw delivery can have at most ten message attributes. We only need a few, so this should not be a particularly difficult constraint to live with.

Once the message is in SQS, Lambda polls the queue for us. When Lambda receives a message, SQS hides it for the duration of the visibility timeout. If the function succeeds, Lambda deletes it. If the function crashes, times out, or reports the record as failed, the message becomes visible again and gets another attempt. After maxReceiveCount attempts, SQS moves it to the DLQ.

The visibility timeout must be longer than the Lambda timeout. AWS recommends at least six times the function timeout, plus any batching window. This may sound excessive, but it gives Lambda enough room for throttling and retries without making the same message visible while the previous invocation may still be running.

For pointer messages, I would start with a batch size of one. We can increase the batch later if measurements show that invocation overhead matters. When the batch size is greater than one, partial batch responses are mandatory in practice; otherwise one failed record causes every successful record in the same batch to be retried.

The worker should then follow a deliberately sequence:

1. Validate the pointer envelope and its schema version.
2. Check that the bucket and prefix are ones this worker expects.
3. Download the immutable S3 object identified by the pointer’s bucket and key.
4. Parse the payload and run the business operation.
5. Return success only after the operation has completed.

The IAM role should enforce most of this. A worker that consumes Stripe events should receive s3:GetObject only for the relevant bucket and prefix, not permission to read every bucket in the account and not permission to replace or delete the raw object. If the object is encrypted with a customer-managed KMS key, the role also needs the corresponding decrypt permission.

Idempotency is part of the worker

The same event can reach the worker more than once. Stripe can retry it, SNS or SQS can redeliver it, and Lambda can finish the database update but time out before SQS deletes the message. None of these situations is especially exotic.

A useful idempotency key is:

<consumer>:<provider_event_id>

The consumer name is important because two independent consumers should both process the same provider event. For a database-only operation, store the idempotency record and make the business update in the same transaction. For an external API, pass the same idempotency key when that API supports it. The ingestion ID is still useful for tracing one delivery, but it is not enough for deduplication because a provider retry creates a new ingestion ID.

It is tempting to make the infrastructure enforce idempotency for every worker, perhaps with a shared DynamoDB table and a conditional write for each processed event. I would not make that part of the default architecture.

The webhook provider, S3 notifications, SNS, and SQS all permit duplicate delivery, so an infrastructure ledger does not create exactly-once processing; it only moves the deduplication decision into another distributed component and leaves us to coordinate that write with the real side effect. Idempotency is usually strongest inside the worker, where the domain knows what “already processed” means and can commit the idempotency record with the business operation.

A DynamoDB ledger remains a useful option for a stateless worker with no natural transactional store, but in practice I tend to leave it out unless that worker genuinely needs it. I am rarely enthusiastic about adding one more piece of infrastructure to conceal a property the system does not actually provide.

How much concurrency?

SQS gives us a convenient place to control how quickly the worker runs. We can set a maximum concurrency on the Lambda event source mapping and let the queue absorb short spikes.

For a worker that reads S3 and calls a scalable external API, this limit can be fairly high. For a worker writing to PostgreSQL, the answer is different. One thousand concurrent Lambdas opening one thousand database connections is a creative way to turn automatic scaling into an outage.

RDS Proxy helps by pooling and reusing database connections, but it does not make the database infinitely scalable. We should still derive the worker concurrency from the number of connections and queries that the database can actually sustain. If the queue grows, that is backpressure doing its job: processing becomes slower without taking the database down with it.

VPC and non-VPC workers

Most workers do not need to be attached to our VPC. A Lambda that reads S3 and calls Stripe, Slack, or another public API already has the network connectivity it needs when it runs outside our VPC. This should be the default because it requires no subnets, security groups, NAT gateway, or VPC endpoints.

A worker that connects to a private RDS database is different. It must run in the VPC and needs security-group access to the database or RDS Proxy. Once we attach the Lambda to private subnets, we also become responsible for its outbound network path:

• add an S3 gateway endpoint so the pointer can be downloaded without crossing a NAT gateway;
• add interface endpoints for private access to services such as Secrets Manager when needed;
• provide NAT egress if the same worker must also call a public API such as Slack or Stripe.

This is why VPC configuration belongs to the worker module and not to the shared ingestion Lambda. One consumer may update RDS, another may call an external API, and there is no reason to give both of them the same networking complexity.

Two different failure boundaries

There is a slightly annoying detail in the SNS-to-SQS integration: the processing DLQ only helps after the message has reached the worker queue. SNS could fail before that because the queue policy is wrong, the KMS policy does not allow delivery, or the queue no longer exists.

SNS subscriptions can therefore have their own delivery DLQ:

SNS subscription --cannot deliver--> SNS delivery DLQ
SQS worker queue --cannot process--> SQS processing DLQ

These queues answer different questions. The SNS DLQ means “the worker never received this.” The SQS DLQ means “the worker received this several times and kept failing.” Delivery failures between SNS and SQS should be rare, but a subscription DLQ is cheap insurance against a bad deployment or permission change.

Neither DLQ fixes anything by itself. Both need an alarm, an owner, and a redrive procedure. Their retention also has to fit the pointer architecture: the S3 payload must live longer than the source queue, the DLQs, and the realistic amount of time it takes us to notice and repair a failure.

HTTP consumers

SNS can also deliver a message directly to an HTTP or HTTPS endpoint. This is useful when another internal service, another AWS account, or an external partner already exposes an endpoint capable of consuming SNS notifications.

The endpoint needs to do a little more than accept a generic POST. It must confirm the SNS subscription, validate that notifications really came from SNS, understand the SNS message format, return quickly, and be idempotent because deliveries can be retried. If the endpoint needs time-consuming processing, it should acknowledge the notification and put the work into its own queue rather than keep the SNS request open.

The pointer adds another complication. Receiving the SNS message does not automatically grant the remote service access to our private S3 object. A service in another AWS account can use an explicit cross-account role or bucket policy. A service outside AWS may need a small authenticated payload API. I would avoid putting a short-lived presigned URL in the SNS message: the notification or DLQ may outlive the URL, leaving us with a perfectly replayable message pointing to an expired credential.

Observability: follow the webhook

Let’s start from the principle rather than from CloudWatch. We want to monitor every step. We want to build a funnel that shows how many webhooks make it from one step to the next. And we want an alert as soon as something gets stuck, disappears, or ends up in a failure queue.

That sounds obvious, but it is easy to get distracted by all the boxes in the diagram. API Gateway has metrics, Lambda has metrics, every SQS queue has metrics, and CloudWatch will happily give us forty colourful charts. We could monitor every AWS service and still fail to answer the only question that matters:

We returned 202 to the provider. Where is the webhook now?

The 202 tells us that S3 stored the raw bytes. It does not tell us that the signature was valid, that SNS received the pointer, or that a worker updated its database. Those promises happen later, one after another, and each one can fail on its own.

So we are going to follow the webhook through the pipeline:

flowchart LR
    A["WebhookStored"] --> Q["SQS verification queue"]
    Q --> V{"Signature"}
    V -->|"valid"| P["PointerPublished"]
    V -->|"invalid"| I["SignatureInvalid"]
    P --> D["SNS delivered"]
    D --> C["ConsumerProcessed"]
    C --> R["Business result"]

    V -. "operational error: retry record" .-> Q
    Q -. "maxReceiveCount reached" .-> VF["Verification DLQ"]
    D -. "delivery error" .-> SD["Subscription DLQ"]
    C -. "processing error" .-> CD["Consumer DLQ"]

This gives us our funnel: stored, queued for verification, verified, published, delivered, and processed. The numbers will not balance perfectly every minute, and that is fine. S3 notifications, SNS, SQS, and Lambda all have at-least-once behaviour. A retry may increase one counter, an event may fall on the other side of a dashboard window, and one pointer may fan out to several consumers. This is a smoke detector, not an accounting ledger. We are looking for gaps that persist long enough to mean something.

Start with the 202

The first step is the only one the webhook provider can see. We want to know:

• how many requests reached API Gateway;
• how many received 4xx, 5xx, or exactly 429;
• how long the provider waited for the response;
• how long the S3 integration took;
• how many requests received the 202 that represents durable storage.

WebhookStored should come from the API Gateway access log, not from the verifier. A metric filter counts entries whose response status is exactly 202. That is the moment we made our promise to the provider. If we emitted the metric from the verifier instead, we would be measuring the next step and completely ignoring objects still waiting in the verification queue.

The same access log gives us exact 429 counts and the p99 API and integration latencies. Native API Gateway metrics already give us request volume, 4XXError, and 5XXError; the log-derived metrics fill in the semantic details that the generic service metrics cannot express.

There is a price for this: log-based metrics need access logging, and logs cost money. We can keep the access log deliberately boring. It needs the server-generated request ID, static route, response status, integration status, latency, response length, source IP, and perhaps a trace ID. It does not need the body, signature, authorization header, cookies, or every request header the provider happened to send. API Gateway data tracing and verbose execution logging stay off.

We want enough context to investigate a failure, but we do not want to quietly build a second copy of every webhook in CloudWatch.

Give each step a name

AWS already tells us quite a lot. Lambda can tell us that an invocation failed, SQS can tell us that a queue is growing, and SNS can tell us that a delivery failed. What AWS cannot tell us is what those failures mean to our pipeline. It does not know whether a webhook had a bad signature or whether a consumer deliberately skipped a duplicate.

Our Lambdas therefore emit a small vocabulary of metrics for the steps in our funnel:

Metric	Meaning
WebhookStored	API Gateway returned 202 after S3 accepted the object
SignatureValid	The verifier authenticated the stored bytes
SignatureInvalid	The request was terminally invalid
VerificationOperationalError	Verification failed for a retryable infrastructure reason
PointerPublished	SNS accepted the verified pointer
ConsumerProcessed	One consumer completed its work
ConsumerFailed	One consumer record failed
DuplicateSkipped	The consumer found already-completed work
ReplayRequested	An operator started a replay
ReplaySucceeded	A replayed record completed successfully

The Python runtime writes these metrics using CloudWatch Embedded Metric Format. In practice, it writes structured JSON to the normal log stream, and CloudWatch extracts the metric asynchronously. That is cheaper and safer than putting a synchronous PutMetricData call in the processing path.

Metric emission is best-effort. If formatting the telemetry fails, we log that problem. We do not retry a perfectly valid webhook because our dashboard had a bad day. Monitoring describes whether the pipeline is working; it does not get to decide whether the pipeline works.

Dimensions need the same restraint as logs. A small, controlled set such as Environment, Provider, EventType, Consumer, Stage, and Outcome is enough. We also emit a series without dimensions for global totals. Event IDs, ingestion IDs, S3 keys, email addresses, and error strings never become dimensions. Otherwise every webhook creates a new custom metric, and the CloudWatch bill becomes the most reliable alarm in the system.

Those unique values still belong in structured logs. That is where they help us follow one event without creating a custom metric for every event.

Bad signatures and broken infrastructure are different problems

The verifier has three meaningful outcomes:

valid signature
invalid signature
operational failure

A missing signature, mismatch, or timestamp outside the allowed tolerance is hostile or malformed input. The verifier records a short reason, quarantines or classifies the object, emits SignatureInvalid, and intentionally finishes the SQS record.

An S3 read error, missing secret, KMS throttle, SNS failure, timeout, or unexpected exception is a different story. The verifier emits VerificationOperationalError and fails the SQS record so that it is retried and, if it keeps failing, reaches the verification DLQ.

If we put both cases into Lambda’s generic Errors metric, a burst of bad signatures would look as if our infrastructure were down. Keeping them separate tells us whether somebody is throwing rubbish at the endpoint or whether we broke our own delivery pipeline.

The verifier logs the ingestion ID, provider, trusted event ID when available, event type, S3 key, duration, outcome, and bounded failure reason. It never logs the body, signature, or secret.

Watch how long the oldest message has been waiting

Every worker gets one dashboard row with:

• visible and in-flight SQS messages;
• age of the oldest message;
• Lambda errors and throttles;
• Lambda duration and concurrency;
• ConsumerProcessed, ConsumerFailed, and DuplicateSkipped;
• processing DLQ depth.

Queue depth alone is easy to misread. Ten thousand fresh messages may be a normal burst that disappears in two minutes. Ten messages with one that has been waiting for an hour usually mean something is stuck. ApproximateAgeOfOldestMessage is therefore our main consumer-latency alarm. Queue depth still helps us understand the shape of a spike, but age tells us when the spike has become a problem.

There is one Lambda detail waiting to trip us up. With partial batch responses, the invocation can succeed while declaring one SQS record as failed. Lambda’s normal Errors metric stays green while that event goes around the retry loop. The worker therefore emits ConsumerFailed for each failed record before returning batchItemFailures. We care about failed events, not only failed invocations.

Alert as soon as a message reaches a failure queue

We have three kinds of failure queues:

1. The verification DLQ means the stored request could not be verified because an operational error exhausted its retries.
2. The SNS subscription DLQ means a verified pointer never reached one consumer’s source queue.
3. The consumer DLQ means the pointer reached the consumer but business processing repeatedly failed.

It is tempting to hide them all behind one friendly “pipeline unhealthy” alarm. I would not. One visible message in any of these queues is one real piece of unfinished work. Each DLQ gets a direct alarm after one 60-second period, an owner, and its own playbook. We want to know as soon as something arrives there, not after enough failures have accumulated to look statistically interesting.

This can be noisy, but a failed event cannot hide behind healthy aggregate traffic. An incident that affects several components may still produce several warnings. That is where composite alarms help with the less concrete symptoms.

Page on facts

Some alarms describe a fact that already happened:

• a message reached any DLQ;
• SNS permanently failed to deliver a pointer;
• a consumer reported a failed record.

Those alarms notify us directly. Other alarms are clues that something is becoming unhealthy:

• API Gateway 5xx, exact 429, and excessive acknowledgement latency;
• verification queue age, verifier errors, throttles, and semantic operational failures;
• SNS delivery and redrive failures;
• per-consumer queue age, errors, throttles, and record failures;
• WAF allowed or blocked traffic spikes.

Composite alarms group those clues into incidents such as:

Ingress unhealthy
Verification unhealthy
Consumer unhealthy
WAF traffic spike

Only the composite alarm needs to notify the paging system for grouped symptoms. If verifier throttling causes Lambda errors and a growing validation queue at the same time, we receive one incident instead of three versions of the same bad news.

DLQ alarms remain direct because they describe actual failed records, not supporting evidence. SNS terminal delivery failures also page directly. A pointer that never reached a consumer has already broken the delivery promise.

The initial thresholds can be intentionally sensitive: one DLQ message, one verifier failure, a small number of 429s, or five minutes of queue age. They still need to be Terraform variables. A threshold that makes sense for five events per minute is absurd for five thousand events per second. The module creates the alarm, but the application owner decides what is acceptable.

The observability module should not create our company’s paging system either. It accepts alarm, recovery, and insufficient-data action ARNs. Those can point to an SNS alarm topic, PagerDuty, or whatever unpleasant sound the organisation has standardised on.

Keep the WAF logs useful and small

WAF traffic helps when a provider disappears after an IP-range change, a rate rule becomes too aggressive, or somebody starts throwing garbage at the endpoint. It is also extremely easy to log too much of it.

The practical configuration is:

• retain BLOCK and COUNT records;
• drop routine ALLOW records;
• redact authorization and cookie headers;
• automatically redact configured signature, timestamp, and delivery-ID headers;
• keep a short, configurable retention period;
• optionally encrypt the log group with a customer-managed KMS key.

Dropping routine allowed records saves money and limits sensitive data, although it removes some forensic detail. API Gateway access logs still tell us which requests reached the API. WAF logs concentrate on the traffic that our security rules actually found interesting. That feels like a sensible split for this system.

And one reminder worth repeating: WAF ALLOW means “no WAF rule blocked this request.” It does not mean that the webhook signature was valid.

Use traces for speed, not history

We can enable tracing on API Gateway, the verifier Lambda, the verified SNS topic, and the managed workers. It helps when we want to understand where one sampled request spent its time.

Tracing has two limitations here. First, it is sampled, so it will never cover every webhook. Second, S3, SNS, and SQS create asynchronous boundaries. The basic Lambda trace does not magically give us one beautiful waterfall across the whole architecture. Detailed S3, secret-store, and SNS SDK calls need OpenTelemetry or ADOT instrumentation.

The server-generated ingestion ID therefore remains our main correlation key. It appears in the S3 key, pointer, metric context, logs, and replay tooling. A trace helps us investigate performance. The ingestion ID is how we find the event at three in the morning.

An alarm still needs instructions

A dashboard helps when we already know that something is wrong. An alarm tells us that something is wrong. Neither tells the unlucky person holding the pager what to do next.

An alert should create a piece of owned operational work, not merely make a noise. The responder acknowledges it, opens the linked runbook, and inspects a small bounded sample from the DLQ without deleting or redriving anything. Start with message IDs, timestamps, receive counts, pointer metadata, and the ingestion ID; use those to find the structured logs and S3 object rather than pasting the raw webhook into a ticket. We then decide whether the messages are terminal bad data, a transient infrastructure failure, or a defect in the consumer. Terminal messages stay quarantined with a recorded reason. Retryable messages move only after the cause is fixed, first through a dry run and then through a slow, monitored redrive. The incident closes when the DLQ is empty or its remaining messages are deliberately accounted for, and the event has reached the expected processing milestone—not merely when the alarm turns green.

Every alarm description should include an owner, a short impact statement, and a link to a playbook. At minimum, we need separate playbooks for:

• provider acknowledgement failures;
• a growing verification queue;
• the verification DLQ;
• the SNS subscription-delivery DLQ;
• a growing consumer queue;
• the consumer processing DLQ;
• a WAF traffic spike;
• a pointer whose S3 object no longer exists.

The details change, but the recovery loop is usually the same:

1. Identify the affected provider, consumer, queue, and time window.
2. Inspect the pointer and structured logs without copying the payload into tickets or chat.
3. Classify the problem as bad data, a code defect, a permission change, throttling, or a dependency outage.
4. Confirm that the S3 object still exists and that the handler is idempotent.
5. Fix or mitigate the cause.
6. Run the replay in dry-run mode and confirm the exact source and destination.
7. Redrive at a low, configurable rate while watching queue age, errors, and dependencies.
8. Confirm that the event reaches ConsumerProcessed before closing the incident.

Replay should leave an audit trail. A verification replay records a replay ID, timestamp, AWS caller identity, incident reason, exact source queue, expected bucket and prefix, maximum record count, rate limit, and whether it was a dry run or a real execution. It emits ReplayRequested and ReplaySucceeded. We delete successful messages only after the verifier returns the success result we expect.

Consumer recovery can use native SQS redrive, but it should still start in dry-run mode, ask the operator to confirm the exact queue name, and move slowly. Sending fifty thousand previously failing events straight back to a database is less of a recovery and more of a second incident.

Test the monitoring too

Terraform can successfully create a completely useless dashboard. Our monitoring therefore needs tests just like the rest of the infrastructure.

Useful invariants include:

• WebhookStored matches exact 202 access-log records;
• one verification DLQ message alarms;
• one SNS subscription DLQ message alarms;
• one consumer DLQ message alarms;
• record-level consumer failures have a semantic alarm;
• WAF logs drop routine allowed traffic and redact sensitive headers;
• the dashboard contains ingress, verification, consumer, and failure-queue views;
• high-cardinality dimensions are rejected by the runtime metric helper.

These tests do not prove that production is healthy. They prove that a refactor did not quietly remove the alarm that was supposed to warn us when production became unhealthy. That is still a rather useful property.

Start with CloudWatch, then change the screen if we need to

For an AWS-only pipeline, CloudWatch is enough for a reliable first version. The service metrics already live there, Embedded Metric Format keeps our metrics close to the runtime, and we can create the dashboards, alarms, Logs Insights queries, and tracing configuration with Terraform. We avoid another agent, another export path, and another bill based on log volume and custom metrics.

CloudWatch becomes awkward when we want to correlate many applications, accounts, or clouds. If the organisation already uses Grafana, Datadog, or another platform, exporting the metrics and logs may give everyone a better place to work. The dashboard product is not the important part. The important part is that we measure stored, verified, published, delivered, and processed, and that we keep the correlation IDs and playbooks behind those steps.

I would keep the instrumentation AWS-native and provider-neutral. Start with CloudWatch, expose alarm destinations as Terraform inputs, and export the data later if the organisation wants a different front end. Replacing a dashboard is easy. Reconstructing an event lifecycle that we never measured is not.

Terraform, shall we?

Terraform module

The root module is mostly glue around three smaller modules, because one enormous Terraform module is how innocent evenings disappear.

• ingestion owns the front door: API Gateway, the raw S3 bucket, the verification queue and Lambda, the verified-pointer SNS topic, and WAF.
• worker is instantiated once for every consumer in the workers map and gives each one its own SQS queue, DLQs, Lambda, filters, and narrowly scoped access to the raw payloads.
• observability adds the CloudWatch dashboard, alarms, metrics, and WAF logging; it can be disabled, although doing that in production is a rather optimistic monitoring strategy.

The strict minimum is small: project_name, environment, and an ingestion object containing bucket_name and provider_routes. A real deployment will normally also provide the verifier package and handler, verifier_environment_variables with the names of its SSM parameters, a workers map, and the observability settings.

The Stripe example therefore wires these module inputs:

• project_name and environment, so every resource gets a predictable name;
• ingestion, with the S3 bucket, the /stripe route, Stripe-Signature, verifier zip and handler, API limits, retention, WAF allowlist, and optional KMS keys;
• verifier_environment_variables, pointing the verifier at the Stripe signing-secret parameter in SSM;
• workers, with one entry per consumer: Lambda zip and handler, queue settings, SNS filter, raw-object prefix, and any SSM parameter ARNs it may read;
• observability, with dashboards, alarms, notification ARNs, and an optional runbook URL;
• enable_xray_tracing, if we want traces as well as logs and metrics.

Most of the knobs inside those objects have sane defaults. We should change them because the workload requires it, not because Terraform has generously given us forty opportunities to invent configuration. Outside the module we still need an AWS provider and credentials; the Stripe example also configures the Stripe, SOPS, and HTTP providers, supplies STRIPE_API_KEY, and decrypts secrets.sops.yaml before putting the values into SSM.

Stripe example

The complete toy implementation lives in examples/example-stripe in the companion repository. There is a little ceremony before we can press the exciting button: we need

• Terraform,
• aws-vault : the nicest utility to work with aws credentials locally imho.
• SOPS: I like to handle secrets directly with sops, i usually generate a kms key or a pgp key and i encrypt the secrets directly in a plain textfile in the repository.
• KMS key that SOPS can use, optionally a pgp key or others.
• The Stripe CLI: This is useful to send webhooks directly to our endpoint to test it.
• Basic tools such as uv, zip, and make
• Checkov is optional, but rather useful if we want the security scan too.

Stripe module

The stripe module is useful since it automatically createas the webhook with the api gateway endpoint and we do not need to update it manually.

We also fetch the stripe ips directly from stripe to create the allowlist for WAF.

Secrets

I do not like to use secret manager, it’s too expensive for what it offers, i prefer using ssm secure parameters.

make secrets-edit opens the SOPS-encrypted secrets.sops.yaml, and Terraform then writes the Stripe signing secret and the two fake worker secrets into SSM as SecureString parameters. The Lambda functions receive only the parameter names and load the values at runtime, so we do not smuggle secrets into environment variables and then act surprised when they appear in a deployment diff.

CI pipeline

The Python Lambda code is in this repository only so the example can be deployed from beginning to end. I would not normally keep verifier or business-worker code beside the Terraform module: those functions should live in their application repositories, where a real CI/CD pipeline tests them, builds immutable zip files, publishes Lambda versions, and promotes the live alias. The included build and deployment scripts are a small mock of that hand-off.

From the example directory, the happy path looks like this:

export STRIPE_API_KEY=sk_test_...
 
make init AWS_VAULT_PROFILE=terraform TF_WORKSPACE=dev \
  TF_BACKEND_BUCKET=your-terraform-state-bucket \
  TF_BACKEND_REGION=eu-central-1
make secrets-edit AWS_VAULT_PROFILE=terraform
make validate AWS_VAULT_PROFILE=terraform TF_WORKSPACE=dev
make security-scan
make plan AWS_VAULT_PROFILE=terraform TF_WORKSPACE=dev
make apply AWS_VAULT_PROFILE=terraform TF_WORKSPACE=dev

After the apply, make endpoint prints the webhook URL, make send-stripe-webhook STRIPE_EVENT=payment_intent.succeeded asks Stripe to deliver a real signed test event, and make send-wrong-webhook sends an invalid signature so we can watch the quarantine path do its job. When only the verifier code changes, make deploy-verifier publishes a new immutable Lambda version and moves the live alias without running another Terraform apply. make help lists the available commands, which is generally nicer than learning the Makefile by archaeological excavation.