Loop
Sign inGet started
← All posts
May 14, 2026·8 min read

How we built exactly-once webhook delivery

A practical look at how Loop delivers signed webhooks exactly once — covering idempotency keys, the outbox pattern, retries with backoff, and consumer-side dedup.

DA
Devin Acharya
Engineering

"Exactly-once delivery" is a phrase that invites a fair amount of skepticism, and it should. In a distributed system, the network can drop, duplicate, or delay any message, and a sender that crashes mid-send cannot know whether the receiver got the payload. So when we say Loop delivers webhooks exactly once, we mean something specific and achievable: every event is produced once, retried until acknowledged, and deduplicated so your handler observes it a single time.

This post walks through how that works, because the guarantee is only useful if you understand where it lives.

The honest framing: effectively-once

Pure exactly-once delivery over an unreliable network is not possible. What is possible is at-least-once delivery plus idempotent processing, which together produce an effect that is indistinguishable from exactly-once. We lean on three layers:

  • A durable outbox so an event is never lost between our database and the wire.
  • Retries with backoff so transient failures resolve themselves.
  • An idempotency key on every event so duplicates collapse into one observed delivery.

If any one of those is missing, the guarantee breaks. Together, they hold.

Producing the event: the outbox pattern

The first failure mode people hit is writing the event to a queue separately from the database change that caused it. If the database commits and the queue write fails, the event is lost. If the queue write succeeds and the database rolls back, you emit an event for something that never happened.

We avoid that by writing the event into the same transaction as the state change. When a feedback item is created or its status flips to resolved, the event row lands in an outbox table atomically with the data it describes. A separate dispatcher reads the outbox and delivers it.

{
  "id": "evt_3kfa92",
  "type": "feedback.created",
  "created_at": "2026-05-14T09:21:04Z",
  "data": {
    "user": { "id": "u_8f2c" },
    "message": "The export dialog hangs on large CSV files.",
    "sentiment": "negative",
    "source": "in-app-widget",
    "status": "open"
  }
}

Because id is generated once, at the moment the event is written, it stays stable across every retry. That stable id is the seam the whole guarantee hangs on.

Delivering it: signing and retries

The dispatcher signs each payload with your endpoint secret and sends it with the event id in a header. Signing matters for two reasons: it proves the request came from Loop, and it lets you reject anything that was tampered with in transit.

If your endpoint returns a 2xx, we mark the event delivered. Anything else — a timeout, a 500, a connection reset — schedules a retry with exponential backoff and jitter. The jitter is deliberate; without it, a brief outage on your side turns into a synchronized stampede of retries the moment you recover.

We keep retrying for hours, not seconds. A deploy that takes your webhook handler offline for ten minutes should never cost you an event.

Retries are why duplicates exist at all. If we send an event, your server processes it successfully, and then the acknowledgement is lost on the way back, we have no way to know you got it. So we send it again. That is correct behavior — and it is exactly why the consumer side needs to be idempotent.

Observing it once: consumer-side dedup

The contract is simple: every delivery carries a stable event id, and you should treat that id as the unit of work. Record the ids you have processed, and ignore repeats.

import { Loop } from '@loop/sdk';

const loop = new Loop(process.env.LOOP_API_KEY);

export async function handleWebhook(req: Request) {
  const event = await loop.webhooks.verify(req); // throws on bad signature

  // `seen` is any durable store with an atomic insert: Redis, Postgres, etc.
  const isNew = await seen.add(event.id);
  if (!isNew) {
    return new Response('duplicate ignored', { status: 200 });
  }

  if (event.type === 'feedback.created' && event.data.sentiment === 'negative') {
    await openTriageTicket(event.data);
  }

  return new Response('ok', { status: 200 });
}

The verify call does two jobs: it checks the signature and parses the payload into a typed event. If the signature is wrong, it throws before your logic runs. The dedup check should use an atomic operation — an insert that fails on conflict, a SET NX — so two concurrent deliveries of the same id cannot both pass.

Why we push dedup to the edge

We could try to guarantee single delivery entirely on our side, but that would require us to know your handler succeeded, and the network denies us that knowledge. By giving you a stable id and a tiny dedup check, the guarantee becomes robust against the failures we genuinely cannot prevent.

A few practical notes for consumers:

  • Make the handler idempotent even beyond the id check. Opening the same ticket twice should be a no-op anyway.
  • Acknowledge fast, work async. Return 2xx quickly and do slow work in the background, so retries are not triggered by your own latency.
  • Keep dedup records long enough. Retain processed ids well past the retry window so a very late duplicate still collapses.

The result

What you get is a delivery layer you can build on without writing defensive plumbing. Events are never silently lost, transient failures heal themselves, and duplicates are harmless. That is what we mean by exactly-once — not a promise that the network behaves, but a system designed so that it does not have to.

engineeringwebhooksreliability