Sampling, approximation, and statistical uncertainty

Sampling reduces the observations processed. Approximate algorithms reduce the memory or work needed to aggregate them. Both can be useful at scale, but they create different uncertainty and must not become invisible implementation details.

GraphJSON does not currently provide a managed sampling policy or statistical analysis layer. Events are not automatically a representative sample merely because they were accepted. Any source sampling, deterministic filtering, approximate ClickHouse function, weighting, or confidence interval is part of your metric definition.

Distinguish four limits#

A result-row limit does not turn a total into a sample estimate. It truncates the displayed result set. Aggregate before the outer result and keep the output grain bounded.

Keep decision-critical facts exact#

Do not sample:

• billing, invoice, refund, entitlement, or usage-ledger facts
• security and legal audit records
• experiment assignment and exposure
• rare failures or abuse decisions
• account creation, deletion, consent, or suppression transitions
• small denominators used for customer-specific decisions

Sampling is better suited to high-volume, repeated observations such as page performance, scroll depth, cursor movement, or other diagnostics where losing individual events does not erase an authoritative state change.

Keep terminal business facts exact even when verbose intermediate telemetry is sampled.

Sample deterministically#

Random decisions on every retry can retain the first attempt and drop the second, or vice versa. Use a stable key:

sample_bucket = hash(event_id) mod 10000
retain when sample_bucket < 1000

That produces a stable 10% sample when event_id is stable. If an event ID is unavailable, use a documented composite with enough entropy.

Choose the sampling unit to match the analysis:

• hash event_id for event-level estimates
• hash session_id to retain complete sessions
• hash user_id for user journeys
• hash account_id for account histories

Event-level sampling can destroy funnels because each step is retained independently. User-level sampling preserves complete user behavior but may be biased if missing identities differ from known users.

Include metadata:

{
  "event": "page_performance_observed",
  "event_id": "perf_019...",
  "sample_method": "cityhash64_event_id",
  "sample_rate": 0.1,
  "sample_policy_version": 2,
  "duration_ms": 842
}

Do not change the rate without versioning the policy.

Weight totals, not every statistic#

For a uniform event sample with inclusion probability p, an estimated event total is:

estimated total = observed events / p

If rates vary, weight each stratum by its own probability. Never multiply a distinct-user count by ten just because events were sampled at 10%; users with more events are more likely to appear.

Ratios require special care. If numerator and denominator are sampled under the same stable unit and policy, a ratio of weighted totals can be reasonable:

conversion rate
  = weighted converting eligible entities
  / weighted eligible entities

If only page views are sampled but conversions are exact, define the estimator explicitly and validate it against exact windows. Do not divide a sampled numerator by an exact but differently scoped denominator.

Choose exact or approximate distinct counts#

ClickHouse offers exact and approximate aggregation families. Use uniqExact when correctness and reproducibility matter more than memory:

SELECT
  toDate(toDateTime(timestamp)) AS day,
  uniqExact(JSONExtractString(json, 'account_id')) AS active_accounts
FROM product_events
WHERE JSONExtractString(json, 'event') = 'core_workflow_completed'
GROUP BY day
ORDER BY day

Approximate distinct functions can be appropriate for exploratory, very high-cardinality dashboards. Record the exact function and version in the metric contract, because different algorithms have different error and merge behavior.

Do not label an approximate result as exact. Compare it with uniqExact over representative settled windows before adoption, including low-volume segments and highly skewed entities.

Treat percentile algorithms as definitions#

Latency percentiles are also algorithm-dependent. Specify:

• population and exclusions
• event or entity weighting
• percentile function
• units
• minimum sample size
• treatment of timeouts and censored observations

An approximate p99 over successful requests alone can look healthy while timeouts rise. Keep failure rate beside latency and define whether timed-out requests receive a capped duration or remain a separate outcome.

Quantify uncertainty#

For a simple random sample proportion p̂ with n independent observations, a rough standard error is:

sqrt(p̂ × (1 - p̂) / n)

The familiar p̂ ± 1.96 × standard error interval is only an approximation. It becomes misleading with small samples, clustered observations, repeated events per user, weighted samples, or selection bias.

Use an interval method appropriate to the estimator, often in a statistical service or analysis notebook. GraphJSON can store and chart the resulting point estimate, lower bound, upper bound, method, and definition version.

{
  "event": "metric_estimate_calculated",
  "metric_key": "search_success_rate",
  "estimate": 0.742,
  "lower_bound": 0.719,
  "upper_bound": 0.764,
  "confidence_level": 0.95,
  "method": "cluster_bootstrap_by_session",
  "sample_policy_version": 2,
  "window_end": "2026-07-26T00:00:00Z"
}

GraphJSON does not calculate or certify these intervals for you.

Bias matters more than precision#

A narrow confidence interval does not fix biased collection. Common biases include:

• ad blockers and network failures
• opt-out and consent selection
• authenticated users being easier to identify
• bots overrepresented in traffic
• client crashes before an event is sent
• heavy users contributing many correlated events
• sampling rates changed by platform or region

Compare retained and eligible traffic by platform, region, plan, volume, and identity status. Record the eligible count outside the sample whenever possible.

Protect low-volume segments#

Suppress or label unstable results when:

• the eligible entity count is below a documented threshold
• the effective sample size is too small
• one account dominates the result
• an interval is too wide for the decision
• privacy rules prohibit reporting a small cell

Do not rank tiny segments by noisy conversion rates. A segment with one success out of one attempt is not necessarily the best performer.

Validate continuously#

Run periodic exact comparisons on bounded windows:

exact result
sampled or approximate result
absolute difference
relative difference
segment
policy version

Alert on missing sampling metadata, unexpected rate changes, estimator drift, and strata with no retained observations. Test boundary cases with metric SQL in CI.

Review checklist#

• Authoritative and rare events remain exact.
• Sampling unit and deterministic key match the analysis.
• Rate, method, and policy version travel with each sampled event.
• Weighting matches the inclusion probability and statistic.
• Approximate functions are named in the metric contract.
• Ratios use compatible numerator and denominator populations.
• Low-volume and privacy-sensitive cells have suppression rules.
• Uncertainty reflects clustering, weighting, and maturity where needed.
• Bias is investigated separately from random error.
• Approximate results are compared with exact settled windows.