Incident analytics explains how reliability failures are detected, managed, resolved, communicated, and prevented from recurring. It complements service metrics by treating the incident itself as a governed lifecycle.
Keep the incident-management system, pager, service telemetry, status page, and postmortem repository authoritative. GraphJSON can analyze normalized incident facts and customer impact; it does not page responders, acknowledge alerts, coordinate response, or preserve the authoritative incident record.
Start with SLOs and error budgets for service-level measurement.
Define the incident grain#
One incident_id identifies one coordinated reliability event. Separate:
- alert or detector evaluation
- page or notification
- incident
- affected service
- customer-impact interval
- responder action
- status communication
- postmortem action item
Several alerts may belong to one incident. One incident may affect several services and customer segments.
Emit the declaration#
{
"event": "incident_declared",
"incident_id": "inc_91",
"severity": "sev_2",
"incident_type": "availability",
"primary_service": "query_api",
"detected_at": "2026-07-25T18:14:00Z",
"declared_at": "2026-07-25T18:19:00Z",
"detection_source": "independent_synthetic",
"incident_policy_version": 4
}
Use controlled service, severity, type, and detection-source values. Keep responder names, chat transcripts, raw alert payloads, and private postmortem notes out of GraphJSON.
Preserve lifecycle transitions#
{
"event": "incident_status_changed",
"incident_id": "inc_91",
"previous_status": "investigating",
"status": "mitigating",
"effective_at": "2026-07-25T18:32:00Z",
"reason": "traffic_shift_started",
"transition_id": "inc_91:status:3"
}
Recommended controlled states:
declared
→ investigating
→ identified
→ mitigating
→ monitoring
→ resolved
Your process may differ. Preserve regressions and reopenings rather than overwriting the previous state.
Define the clocks#
Common intervals:
time to detect = detected_at − impact_started_at
time to declare = declared_at − detected_at
time to acknowledge = acknowledged_at − page_sent_at
time to mitigate = mitigation_effective_at − impact_started_at
time to recover = impact_ended_at − impact_started_at
time to resolve = resolved_at − declared_at or impact_started_at
The denominator and starting point matter. “MTTR” can mean time to repair, recover, resolve, or restore.
Store approved terminal timestamps or durations from the incident authority:
{
"event": "incident_resolved",
"incident_id": "inc_91",
"severity": "sev_2",
"primary_service": "query_api",
"impact_started_at": "2026-07-25T18:09:00Z",
"detected_at": "2026-07-25T18:14:00Z",
"mitigation_effective_at": "2026-07-25T18:47:00Z",
"impact_ended_at": "2026-07-25T18:51:00Z",
"resolved_at": "2026-07-25T19:10:00Z",
"duration_policy_version": 3
}
Do not derive contractual incident clocks from analytics arrival time.
Model customer impact separately#
An internal incident can have no customer impact, and customer impact can begin before declaration.
{
"event": "incident_impact_evaluated",
"incident_id": "inc_91",
"service": "query_api",
"impact_type": "elevated_errors",
"impact_scope": "subset",
"affected_account_count": 184,
"affected_request_count": 9120,
"impact_started_at": "2026-07-25T18:09:00Z",
"impact_ended_at": "2026-07-25T18:51:00Z",
"evaluation_version": 2,
"data_complete_through": "2026-07-25T19:30:00Z"
}
Do not put customer names in broad incident dashboards. For authorized support workflows, store pseudonymous account IDs in a dedicated bounded collection or query the authoritative impact service.
Account impact estimates can change as late telemetry arrives. Version or restate them explicitly.
Track status communication#
{
"event": "incident_communication_published",
"incident_id": "inc_91",
"channel": "public_status_page",
"communication_type": "initial",
"published_at": "2026-07-25T18:26:00Z",
"minutes_since_declaration": 7
}
Keep message contents in the communication system. Analyze whether the initial notice, updates, and resolution notice met the documented communication policy.
Do not send a public-status update based solely on a GraphJSON query; use the incident authority and approval workflow.
Query finalized incident durations#
WITH resolved AS (
SELECT
JSONExtractString(json, 'incident_id') AS incident_id,
argMax(JSONExtractString(json, 'severity'), timestamp) AS severity,
argMax(JSONExtractString(json, 'primary_service'), timestamp) AS service,
argMax(
parseDateTimeBestEffort(JSONExtractString(json, 'impact_started_at')),
timestamp
) AS impact_started_at,
argMax(
parseDateTimeBestEffort(JSONExtractString(json, 'impact_ended_at')),
timestamp
) AS impact_ended_at
FROM incident_events
WHERE JSONExtractString(json, 'event') = 'incident_resolved'
GROUP BY incident_id
)
SELECT
toStartOfMonth(impact_started_at) AS month,
severity,
service,
count() AS resolved_incidents,
quantileExact(0.5)(
dateDiff('minute', impact_started_at, impact_ended_at)
) AS median_impact_minutes,
quantileExact(0.9)(
dateDiff('minute', impact_started_at, impact_ended_at)
) AS p90_impact_minutes
FROM resolved
GROUP BY month, severity, service
ORDER BY month, severity, service
Use medians and distributions. A mean can be dominated by one extreme event, while a median can hide the same event. Show the largest incidents separately.
Connect incidents to service evidence#
Join by service and a fixed impact interval to inspect:
- error and availability SLIs
- latency and saturation
- deploys and configuration changes
- queue backlog
- integration failures
- customer support volume
Do not assume the nearest deployment caused the incident. Record the approved contributing-factor category after review and preserve uncertainty.
Use Release-impact analytics for deployment comparisons.
Classify causes without oversimplifying#
Postmortem categories may include:
- software defect
- configuration
- capacity
- dependency
- data quality
- security
- operational procedure
- unknown
Keep root cause, contributing factors, trigger, and detection gap separate. Avoid “human error” as a terminal explanation; it rarely describes the system conditions that allowed impact.
GraphJSON should receive controlled categories, not the confidential narrative.
Track postmortem follow-through#
{
"event": "incident_action_item_status_changed",
"incident_id": "inc_91",
"action_item_id": "act_81",
"action_category": "detection",
"priority": "high",
"previous_status": "open",
"status": "completed",
"due_at": "2026-08-15T00:00:00Z",
"effective_at": "2026-08-10T17:00:00Z"
}
Track completion, overdue work, and verified effectiveness. Do not reward teams for producing many action items; measure whether the selected actions reduce the identified risk.
Measure recurrence carefully#
Define recurrence using reviewed relationships:
- same failure mechanism
- same missing safeguard
- same service dependency
- repeated customer symptom
Do not classify incidents as recurring merely because they share a service or severity. Preserve a recurrence-group ID and review version in the incident authority.
Build the incident program dashboard#
Show:
- incidents by severity, service, and type
- impact duration and affected-account distributions
- detection, declaration, mitigation, and recovery intervals
- detection source and alert effectiveness
- communication timeliness
- recurrence groups and contributing factors
- postmortem and action-item completion
- service SLO and support-volume context
- incident-data freshness and missing clocks
Avoid public or executive dashboards that expose customer identities or unreviewed root-cause claims.
Production checklist#
- Alert, page, incident, impact, communication, and action-item grains are distinct.
- The incident system and pager remain authoritative.
- Every clock states its exact start and end.
- Customer impact is separate from internal incident status.
- Late impact evaluations create explicit revisions.
- Communication analytics excludes message contents.
- Deploy proximity is not labeled causal without evidence.
- Cause categories remain distinct from confidential narratives.
- Recurrence uses reviewed relationships.
- Action items track verified completion, not just creation.
- Immediate paging remains independent of GraphJSON.