Create and operate alerts

GraphJSON alerts watch a Single Line visualization and notify email or SMS recipients when completed chart values cross an absolute threshold or move materially relative to a comparison period.

An alert is useful only when its metric is trustworthy, its timing is understood, and its recipient knows what to do. This guide covers the complete lifecycle: creation, evaluation, notification, investigation, recovery, editing, and removal.

Alert lifecycle at a glance#

Single Line visualization
        ↓
saved alert rule
        ↓
enough completed data points?
        ├─ no  → remain inactive
        └─ yes → evaluate latest completed window
                    ├─ condition false → inactive
                    └─ condition true  → active + notify
                                             ↓
                                  later evaluation is normal
                                             ↓
                                       resolved + notify

GraphJSON sends a trigger notification when an alert moves from inactive to active. Repeated evaluations that still violate the rule keep the alert active but do not send another trigger notification. When the metric returns to normal, GraphJSON marks the alert resolved and sends a recovery notification.

Start with a trustworthy metric#

Build and verify a Single Line visualization before creating the alert. A good alert has:

• one clear business or operational meaning
• a range and granularity appropriate to the response time
• enough normal data to establish a baseline
• filters that exclude known test or internal traffic
• an owner and a written response

Examples:

An event-volume alert can detect a broken producer as easily as a product incident. Check instrumentation health before concluding that customer behavior changed.

Create an alert#

1. Open the collection and select Visualize.
2. Choose Single Line.
3. Configure the aggregation, metric, range, granularity, filters, and optional comparison.
4. Confirm that the chart represents the intended metric.
5. Select Export → As Alert.
6. Add a unique name and an actionable description.
7. Choose an absolute or relative rule.
8. Add at least one email or SMS recipient.
9. Save the alert.

Put the condition, owner, and first response in the description:

Checkout failures exceed 100 per hour.
Owner: payments on-call.
Check provider status, deploy history, and the checkout queue.

The description appears on the alert page. It is the closest thing to a runbook link in the current workflow, so make it useful during an incident.

Understand the evaluation window#

An alert configured with a window of N evaluates N completed chart points. GraphJSON requires at least N + 1 result points and leaves the newest point out of the calculation.

For a three-point window:

point 1    point 2    point 3    newest point
└──────── evaluated window ────┘ └─ not evaluated yet

This prevents an in-progress chart bucket from being treated like a complete interval. It also means a new alert cannot evaluate until the visualization returns enough history.

Relative alerts require N + 1 points in both the primary and comparison result. If either side lacks enough data, the alert remains unevaluated rather than treating missing history as zero.

Warning: A five-point window does not mean five minutes. It means five values at the visualization’s granularity. Five hourly points cover five completed hours; five daily points cover five completed days.

Alert rules#

GraphJSON supports two rule families.

Absolute thresholds#

An absolute alert calculates the Avg, Min, or Max of the completed window and compares the result with a fixed watermark.

Examples:

Avg of last 3 values greater than 100
Min of last 2 values less than 1
Max of last 5 values greater than 2500

Use absolute thresholds for hard operating boundaries:

• errors exceed a known tolerance
• throughput falls below a minimum
• latency exceeds a customer-facing objective
• a queue or balance approaches a fixed capacity

Choose the aggregate deliberately:

Relative thresholds#

A relative alert calculates the selected aggregate for the completed primary window and compares it with the equivalent comparison window.

Examples:

Avg of last 3 values is less than the comparison by 30%
Max of last 5 values is greater than the comparison by 50%

Use relative rules when the expected level changes by hour, weekday, or season:

• signups down from the preceding week
• API errors up from a prior period
• usage down from a comparable business day

Relative rules are only as meaningful as their comparison. Confirm that the visualization’s comparison offset aligns weekdays, release periods, or seasonal boundaries appropriately.

Evaluation timing#

The alert worker checks for alert work approximately once per minute. Individual alerts may be skipped between checks based on the spacing of the visualization’s returned points.

Evaluation is therefore asynchronous and approximate. It is not:

• a per-second scheduler
• a guaranteed notification deadline
• a substitute for an infrastructure paging system
• proof that every notification provider accepted a message

Choose a granularity that matches how quickly the underlying metric becomes complete. A daily metric is not made operationally immediate by checking it every minute.

Notification state#

The current state machine is:

GraphJSON does not repeatedly notify while the same alert remains active. If you require reminders, escalations, or paging policies, route the GraphJSON notification into a workflow you operate or use a dedicated incident-management system.

Email and SMS are the currently supported recipient types. Treat recipient addresses and phone numbers as operational data: keep the list current and remove former team members promptly.

Investigate an active alert#

Open Alerts, select the active alert, and use this order:

1. Read the rule and description. Confirm what the alert actually measures.
2. Inspect the chart. Compare the highlighted alert interval with surrounding values.
3. Review anomaly history. Determine whether the same pattern has occurred recently.
4. Select Dive In. Open the underlying visualization and widen the range.
5. Remove filters one at a time. Determine whether the issue affects one segment or the whole collection.
6. Check Samples. Verify that events and field types are arriving.
7. Check the producer. Review response codes, retries, queue age, and dead letters.
8. Check releases and dependencies. Compare the alert start with deploys and provider incidents.

Do not change the rule during an incident merely to clear the red state. Record the investigation first, then tune the alert after the incident is understood.

Understand anomaly history#

When an inactive alert first becomes active, GraphJSON records an anomaly containing the evaluated value, rule configuration, recipients, and highlighted interval.

When the alert returns to normal, GraphJSON extends that anomaly’s highlighted interval through recovery. The alert detail page displays recent anomaly history.

An anomaly record means the configured rule was violated. It does not prove:

• the underlying product was unavailable
• every recipient received a message
• the alert caused an incident
• the displayed metric is authoritative

Keep the incident narrative, decisions, and remediation in the system your team uses for incident management.

Tune an alert#

Tune one variable at a time:

1. confirm the metric definition
2. adjust the chart granularity
3. adjust the number of evaluated points
4. choose Avg, Min, or Max
5. adjust the threshold or comparison percentage
6. review recipients

Typical symptoms:

Every change modifies future evaluation. Record material threshold changes in the alert description or an external change log.

Edit or delete an alert#

Open the alert detail page to edit:

• name
• description
• aggregate
• window size
• comparison direction
• watermark or percentage
• recipients

After renaming, use the new alert URL generated by the dashboard rather than an old bookmark.

Delete alerts that no longer represent an owned decision. Before deleting, confirm that the alert is not the only documented monitoring control for a production workflow.

Troubleshoot notifications#

If an expected alert did not notify:

1. confirm that the alert has enough primary and comparison points
2. confirm that it was inactive before the rule became true
3. confirm the alert currently shows Active
4. inspect the anomaly history
5. verify the recipient value and type
6. check email spam, filtering, and SMS delivery restrictions
7. confirm that the chart itself returns current data
8. contact support with the alert name, approximate transition time, time zone, and recipient type

Do not send a complete phone number, API key, private payload, or generated embed URL in the initial support message.

Current limitations#

GraphJSON currently does not document:

• a test-notification button
• notification delivery logs
• acknowledgement or ownership state
• snoozes or scheduled maintenance windows
• repeated escalation while an alert remains active
• Slack, PagerDuty, or webhook recipients
• a notification-delivery service-level agreement

Use email or SMS alerts for focused product and operational signals. Use a dedicated observability or incident-management system when the workflow requires paging guarantees, routing policies, acknowledgements, or escalations.

Production checklist#

• The visualization definition is written and reviewed.
• The range contains at least N + 1 points.
• Relative alerts have enough comparison history.
• The granularity matches the required response time.
• The description includes an owner and first response.
• Recipients are current and authorized.
• Missing data is distinguishable from a true zero.
• Instrumentation health is monitored separately.
• The alert has been observed through one normal evaluation cycle.
• A separate incident system owns acknowledgement and escalation when required.

Continue with Monitor instrumentation health, Measure release impact, and the GraphJSON troubleshooting hub.