Contain, correct, and replay bad data

Bad analytics data can look credible while changing a decision. Treat a material instrumentation defect as a data incident.

Examples:

• amount changed from cents to dollars
• staging events entered production
• one retry created duplicates
• a producer used milliseconds as occurrence time
• an identity field became empty
• a backfill mapped every row to the import date
• a webhook began forwarding secrets

The safe response is:

detect
  → contain
    → preserve safe evidence
      → scope
        → choose correction
          → replay if needed
            → reconcile
              → prevent recurrence

Classify severity#

Do not publish a corrected number until the correction method and authoritative source are known.

Contain the source#

First stop expansion:

1. disable the faulty producer or feature flag
2. pause its queue and replay workers
3. stop the related historical import
4. disable provider event types when safe
5. prevent a deployment rollback from re-enabling the old code
6. keep unrelated healthy producers running

If the event is important to the product transaction, continue committing the authoritative business state. Queue the analytical copy for later rather than blocking the customer action.

Protect affected consumers#

Inventory:

• saved SQL queries
• visualizations
• dashboards
• alerts
• shared dashboard links
• customer embeds
• Data API consumers
• CSV exports
• external decisions already made

For a customer-facing metric:

• show an unavailable or maintenance state
• remove the affected tile
• freeze a previously verified value only when clearly labeled and approved
• communicate the affected interval and scope

Do not silently show zero. Zero is a valid analytical value, not an error state.

Preserve safe evidence#

Record outside ordinary analytics:

incident ID
detection time
first suspected bad occurrence time
last suspected bad occurrence time
producer version
event name
collection
schema version
safe reason
affected dashboards and consumers

Do not copy a leaked secret or complete sensitive payload into the incident ticket. Record an opaque event ID and keep restricted evidence in an approved system.

Define the affected population#

Use dimensions that existed on the event:

• event name
• schema version
• application version
• deployment ID
• source
• environment
• import ID
• occurrence interval
• event ID range
• provider delivery ID

Example:

SELECT
  JSONExtractString(json, 'app_version') AS app_version,
  JSONExtractInt(json, 'schema_version') AS schema_version,
  count() AS rows,
  min(timestamp) AS first_time,
  max(timestamp) AS last_time
FROM product_events
WHERE JSONExtractString(json, 'event') = 'checkout_completed'
GROUP BY app_version, schema_version
ORDER BY first_time

Avoid defining scope only by detection time. Delayed delivery and old clients can produce affected rows later.

Compare with an authoritative source#

Choose the system that decides the fact:

GraphJSON can reveal and analyze the discrepancy, but it should not become the authority merely because its number is convenient.

Choose a correction strategy#

Strategy 1: Fix future events#

Use when the historical defect is immaterial or clearly labeled.

Actions:

• deploy corrected producer
• increment schema version when needed
• monitor adoption
• annotate affected dashboards or metric definitions

Do not blend old and new semantics without a query boundary.

Strategy 2: Exclude invalid rows in queries#

Use when affected rows are precisely identifiable:

WHERE NOT (
  JSONExtractString(json, 'event') = 'checkout_completed'
  AND JSONExtractString(json, 'app_version') = '4.8.0'
  AND timestamp >= 1785000000
  AND timestamp < 1785086400
)

Centralize the exclusion in a saved query or documented metric definition. Copying a different filter into every dashboard creates inconsistent truth.

Record:

• incident ID
• exclusion rule
• effective interval
• owner
• retirement condition

Strategy 3: Interpret versions explicitly#

When units changed:

SELECT
  sum(
    if(
      JSONExtractInt(json, 'schema_version') = 1,
      JSONExtractFloat(json, 'amount_dollars') * 100,
      JSONExtractFloat(json, 'amount_minor')
    )
  ) AS amount_minor
FROM order_events
WHERE JSONExtractString(json, 'event') = 'order_completed'

Use only when the old conversion is unambiguous. Do not guess a currency or unit from magnitude.

Strategy 4: Emit correction facts#

For an append-only analytical copy of an authoritative ledger:

{
  "event": "usage_corrected",
  "correction_id": "corr_71",
  "corrects_event_id": "usage_91",
  "quantity_delta": -1200,
  "unit": "api_call",
  "reason_code": "duplicate_import",
  "occurred_at": 1785081600
}

Queries sum original facts plus correction deltas.

Do not use a correction event to hide a security or privacy exposure. Follow the relevant deletion and incident process.

Strategy 5: Rebuild a dedicated collection#

Use when:

• the collection has an authoritative source
• most of its history is affected
• dependencies can tolerate a controlled rebuild
• the source can reproduce the complete intended interval

GraphJSON self-service controls operate at collection or workspace scope; a general row-level mutation API is not documented.

For a dedicated collection:

1. stop all producers
2. export or preserve the authoritative source
3. inventory dependencies
4. build a clean replacement collection
5. canary and reconcile
6. switch saved queries, dashboards, alerts, and embeds
7. retire the affected collection deliberately

Prefer building a replacement over deleting first. It preserves the ability to compare and roll back the consumer cutover.

Never improvise deletion#

Do not:

• send negative copies of non-additive events
• rename an event and assume old dashboards stop counting it
• clear all_events to fix one isolated producer
• delete a collection before confirming the source can rebuild it
• promise row-level removal that has not been tested

For personal data or secrets, follow the data lifecycle and privacy procedure and contact GraphJSON with safe metadata.

Build the replacement dataset#

Create a new collection such as:

orders_rebuilt_2026_07

The name should be operationally clear. Do not expose an internal incident identifier in a customer-facing title.

Apply:

• executable event contract
• approved transformation version
• stable original event IDs
• original occurrence timestamps
• corrected units and identity
• explicit import ID

Keep the import ID if it is useful for reconciliation and safe to retain.

Canary the replay#

Replay a narrow interval first:

1. validate Samples
2. compare counts by event and day
3. compare financial or operational totals
4. run saved SQL
5. render every graph type used by consumers
6. verify time zone and retention
7. test an alert without depending on an immediate notification
8. verify personalized tenant filters

Do not use the easiest day. Include the interval where the defect occurred and at least one known edge case.

Cut consumers over#

Use a dependency checklist:

saved query
  → visualization
    → dashboard
    → shared link
    → embed
    → alert
    → external API consumer

For every consumer, record:

• old collection/query
• replacement
• owner
• validation result
• cutover time
• rollback action

Changing a dashboard does not update an application-owned Data API request.

Reconcile recovery#

Completeness#

authoritative expected facts
= corrected GraphJSON facts + documented exclusions

Time#

Compare first and last occurrence times and counts per day.

Semantics#

Verify:

• units
• currency
• identity grain
• schema version
• environment
• deduplication
• late-arrival behavior

Consumers#

Have owners verify:

• executive dashboard
• customer-facing analytics
• alerts
• downstream reports

Monitor after reopening#

Track:

• invalid-event rate
• old schema-version traffic
• duplicate IDs
• missing required fields
• delivery delay
• unexplained source/destination count difference
• affected metric versus authoritative total

Keep the incident open through one full expected reporting interval when the metric is daily, weekly, or monthly.

Prevent recurrence#

Add the missing control:

Do not close with “be more careful.” Change the system that allowed the defect.

Incident checklist#

• Faulty source and replay workers are contained.
• Affected customer views are safe.
• Evidence excludes active secrets and private payloads.
• Scope uses event dimensions and occurrence time.
• An authoritative source is named.
• Correction method is documented.
• Collection deletion is not assumed to be row-level repair.
• Replays preserve IDs and occurrence times.
• Replacement data passed a canary.
• Every dependency has a cutover owner.
• Business and destination totals reconcile.
• Preventive contract and monitoring changes shipped.

Continue with Monitor instrumentation health, Troubleshoot GraphJSON, and High-volume ingestion and backfills.