Webhook delivery operations

Webhook systems are asynchronous and retry-heavy. A useful analytics model records each delivery attempt plus an explicit terminal outcome, while the delivery database remains the authority for scheduling, signing, and retry state.

Separate attempts from outcomes#

Emit webhook_delivery_attempted after each HTTP attempt:

{
  "event": "webhook_delivery_attempted",
  "event_id": "attempt_01J...",
  "delivery_id": "del_91",
  "account_id": "acct_7",
  "provider": "customer_endpoint",
  "event_type": "invoice.paid",
  "attempt_number": 2,
  "status_code": 503,
  "outcome": "retryable_failure",
  "duration_ms": 842,
  "region": "us-west",
  "schema_version": 1
}

When the delivery succeeds or exhausts its retry policy, emit one terminal event:

{
  "event": "webhook_delivery_completed",
  "event_id": "outcome_del_91",
  "delivery_id": "del_91",
  "account_id": "acct_7",
  "provider": "customer_endpoint",
  "event_type": "invoice.paid",
  "final_outcome": "exhausted",
  "attempt_count": 6,
  "total_elapsed_seconds": 3720,
  "schema_version": 1
}

The attempt stream explains reliability and latency. The terminal stream makes final success rate easy to calculate without inferring the latest attempt.

Keep payloads out of analytics#

Do not send:

• webhook request or response bodies
• signing secrets
• authorization headers
• complete endpoint URLs with credentials or query parameters
• arbitrary response headers

Log a controlled provider or endpoint ID and retain raw payloads only in the operational system designed to secure them. Truncate or classify response errors before analytics; an upstream service may echo submitted personal data.

Make delivery instrumentation durable#

Write the attempt record after the HTTP result is known. Use the same job system as the webhook dispatcher or a database outbox, but isolate analytics failure from the customer webhook retry policy.

The source application owns:

• next-attempt scheduling
• exponential backoff
• signing and secret rotation
• endpoint disablement
• authoritative delivery state

GraphJSON observes those facts. It should not decide whether the next webhook fires.

Build reliability metrics#

Final success rate:

SELECT
  JSONExtractString(json, 'provider') AS provider,
  countIf(
    JSONExtractString(json, 'final_outcome') = 'delivered'
  ) AS delivered,
  count() AS completed,
  round(100 * delivered / nullIf(completed, 0), 2)
    AS success_rate_pct
FROM webhook_events
WHERE
  JSONExtractString(json, 'event') = 'webhook_delivery_completed'
  AND toDateTime(timestamp) >= subtractDays(now(), 30)
GROUP BY provider
ORDER BY completed DESC

Attempt latency by provider:

SELECT
  JSONExtractString(json, 'provider') AS provider,
  quantileExact(0.50)(
    JSONExtractFloat(json, 'duration_ms')
  ) AS p50_ms,
  quantileExact(0.95)(
    JSONExtractFloat(json, 'duration_ms')
  ) AS p95_ms,
  count() AS attempts
FROM webhook_events
WHERE
  JSONExtractString(json, 'event') = 'webhook_delivery_attempted'
  AND toDateTime(timestamp) >= subtractDays(now(), 7)
GROUP BY provider
ORDER BY attempts DESC

Retry distribution:

SELECT
  JSONExtractInt(json, 'attempt_count') AS attempts,
  count() AS deliveries
FROM webhook_events
WHERE
  JSONExtractString(json, 'event') = 'webhook_delivery_completed'
  AND toDateTime(timestamp) >= subtractDays(now(), 30)
GROUP BY attempts
ORDER BY attempts

Split customer-caused failures, provider failures, timeouts, and internal errors. A 410 that disables an obsolete endpoint should not be grouped with a transient 503.

Create an operations dashboard#

Use:

• deliveries completed per hour
• final success rate by provider
• retryable failures by status family
• p50 and p95 attempt latency
• deliveries by attempt count
• exhausted deliveries by account
• time from enqueue to terminal outcome

Add filters for provider, event type, region, and customer tier only when those fields have controlled values. delivery_id belongs in investigation queries, not a dashboard split.

Alert on actionable conditions#

Create alerts for sustained changes, not every failed attempt:

• final success rate below the expected floor
• p95 latency above the operational target
• exhausted delivery count above baseline
• no completed events from a normally active region

Also alert on the analytics queue itself. If tracking stops, a perfect-looking success rate may simply be stale data.

Link alerts to a runbook that checks the dispatcher queue, provider status, recent deployments, GraphJSON delivery worker, and a sample of affected operational records.

Retention and customer views#

Attempt-level diagnostics are high-volume and often need short retention. Terminal outcomes may be useful for longer customer success and reliability trends. Put them in separate collections when their lifecycle or access differs substantially.

For a customer-facing delivery page, enforce account_id on your server and return only that customer’s results. Avoid exposing internal exception text or endpoint identifiers that reveal infrastructure.

Launch checklist#

• Attempt and terminal events have distinct purposes.
• Bodies, secrets, and raw headers cannot enter analytics.
• The dispatcher—not GraphJSON—owns retry state.
• Failure classes and terminal outcomes are controlled enums.
• Metrics use terminal events for final success.
• Dashboard and pipeline health are both monitored.
• Diagnostic retention is intentionally short.
• Customer views use server-authorized account filters.