Build a conversion funnel and A/B test

This guide instruments, validates, analyzes, and closes a controlled product experiment.

GraphJSON provides event storage, SQL, funnels, comparisons, and exports. It does not assign variants, calculate an authoritative sample size, or decide statistical significance for you.

Use an experiment service or deterministic application logic for assignment, then send immutable experiment facts to GraphJSON.

Write the protocol first#

Record before exposure begins:

Experiment: signup_copy_v2
Owner: Growth
Hypothesis: clearer value copy increases completed signup
Eligible entity: new anonymous visitor
Randomization unit: anonymous_id
Variants: control 50%, treatment 50%
Primary metric: signup completed within 7 days of first exposure
Minimum detectable effect: approved by experiment owner
Guardrails: signup errors, support contacts, 14-day activation
Start rule: instrumentation checks pass
Stop rule: planned sample/duration, plus safety stop
Decision rule: approved statistical method and practical threshold

Do not choose the primary metric after reading the results.

Choose the randomization unit#

Randomize the entity that receives one consistent experience:

Request-level randomization is usually wrong for a persistent interface. The same person can see both variants and contaminate downstream behavior.

Account-level randomization is safer when teammates interact or share the same configured product.

Define eligibility once#

Eligibility should be deterministic:

new visitor
AND country is supported
AND not employee/test traffic
AND first eligible time is inside experiment enrollment window

Do not include people who could never receive the treatment.

Persist:

• experiment
• randomization unit ID
• variant
• assigned time
• assignment algorithm version

Changing allocation does not justify reassigning already enrolled entities.

Separate assignment from exposure#

Assignment means an entity was allocated. Exposure means the entity actually received the experience.

Log assignment:

{
  "event": "experiment_assigned",
  "event_id": "exp_assign_01J...",
  "experiment": "signup_copy_v2",
  "variant": "treatment",
  "anonymous_id": "anon_42",
  "assignment_version": "murmur-v1",
  "schema_version": 1
}

Log the first real exposure:

{
  "event": "experiment_exposed",
  "event_id": "exp_expose_01J...",
  "experiment": "signup_copy_v2",
  "variant": "treatment",
  "anonymous_id": "anon_42",
  "surface": "signup_landing",
  "schema_version": 1
}

Emit exposure only when the variant-controlled surface is rendered or applied. Do not emit it merely because the application computed an assignment that was never seen.

Use one immutable first-exposure fact per experiment entity when that matches the analysis plan. Repeated exposure events can still be useful for frequency, but do not count them as additional participants.

Log outcome events independently#

The signup funnel:

landing_viewed → signup_started → signup_completed

Outcome events should remain valid product facts outside the experiment:

{
  "event": "signup_completed",
  "event_id": "evt_signup_01J...",
  "anonymous_id": "anon_42",
  "user_id": "usr_91",
  "schema_version": 1
}

You can attach experiment and variant for convenient debugging, but derive the authoritative analysis variant from the assignment or first-exposure fact. This prevents a later code path from stamping a different variant on the outcome.

For pre-signup experiments, preserve the anonymous-to-user link described in Identity stitching and account lifecycle.

Keep variant payloads immutable#

Store configuration version separately:

{
  "experiment": "signup_copy_v2",
  "variant": "treatment",
  "variant_payload_version": "copy-2026-07-26"
}

If the treatment changes materially after launch:

• start a new experiment, or
• create a new variant/configuration version and analyze separately

Do not keep the same label while changing the experience. “Treatment” must identify one reproducible condition.

Validate assignment#

Check that one randomization unit has one variant:

SELECT
  JSONExtractString(json, 'anonymous_id') AS unit_id,
  uniqExact(JSONExtractString(json, 'variant')) AS variants
FROM product_events
WHERE
  JSONExtractString(json, 'event') = 'experiment_assigned'
  AND JSONExtractString(json, 'experiment') = 'signup_copy_v2'
GROUP BY unit_id
HAVING variants > 1
LIMIT 100

The expected result is empty.

Also verify:

• empty unit IDs
• unknown variants
• duplicate assignment event IDs
• assignment before eligibility
• assignment after enrollment closed
• exposure before assignment
• outcome before exposure when the protocol requires exposure
• employee and test traffic

Check sample-ratio mismatch#

Compare observed assignments with the planned allocation:

SELECT
  JSONExtractString(json, 'variant') AS variant,
  uniqExact(JSONExtractString(json, 'anonymous_id')) AS assigned_units
FROM product_events
WHERE
  JSONExtractString(json, 'event') = 'experiment_assigned'
  AND JSONExtractString(json, 'experiment') = 'signup_copy_v2'
  AND JSONExtractString(json, 'anonymous_id') != ''
GROUP BY variant
ORDER BY variant

For a 50/50 test, a large unexplained imbalance can indicate:

• biased assignment
• one variant failing before event delivery
• eligibility differences
• duplicate or missing IDs
• a rollout configuration error

Use the preselected statistical sample-ratio-mismatch test outside GraphJSON. Do not decide that an imbalance is acceptable merely because the treatment looks better.

Assignment balance and exposure balance answer different questions. Inspect both.

Build a reach funnel#

This query counts distinct exposed units that reached each step:

WITH exposure AS (
  SELECT
    JSONExtractString(json, 'anonymous_id') AS unit_id,
    argMin(JSONExtractString(json, 'variant'), timestamp) AS variant,
    min(timestamp) AS exposed_at
  FROM product_events
  WHERE
    JSONExtractString(json, 'event') = 'experiment_exposed'
    AND JSONExtractString(json, 'experiment') = 'signup_copy_v2'
  GROUP BY unit_id
),
reach AS (
  SELECT
    exposure.unit_id,
    exposure.variant,
    max(JSONExtractString(events.json, 'event') = 'landing_viewed')
      AS landing,
    max(JSONExtractString(events.json, 'event') = 'signup_started')
      AS started,
    max(JSONExtractString(events.json, 'event') = 'signup_completed')
      AS completed
  FROM exposure
  LEFT JOIN product_events AS events
    ON exposure.unit_id =
      JSONExtractString(events.json, 'anonymous_id')
    AND events.timestamp >= exposure.exposed_at
    AND events.timestamp < exposure.exposed_at + 7 * 24 * 60 * 60
  GROUP BY exposure.unit_id, exposure.variant
)
SELECT variant, step, users
FROM (
  SELECT variant, '1. Landing viewed' AS step, sum(landing) AS users
  FROM reach GROUP BY variant
  UNION ALL
  SELECT variant, '2. Signup started' AS step, sum(started) AS users
  FROM reach GROUP BY variant
  UNION ALL
  SELECT variant, '3. Signup completed' AS step, sum(completed) AS users
  FROM reach GROUP BY variant
)
ORDER BY variant, step

This is still a reach funnel: a participant can reach steps out of order.

For strict ordering, use windowFunnel:

WITH exposed AS (
  SELECT
    JSONExtractString(json, 'anonymous_id') AS unit_id,
    argMin(JSONExtractString(json, 'variant'), timestamp) AS variant,
    min(timestamp) AS exposed_at
  FROM product_events
  WHERE
    JSONExtractString(json, 'event') = 'experiment_exposed'
    AND JSONExtractString(json, 'experiment') = 'signup_copy_v2'
  GROUP BY unit_id
),
progress AS (
  SELECT
    exposed.unit_id,
    exposed.variant,
    windowFunnel(7 * 24 * 60 * 60)(
      toDateTime(events.timestamp),
      JSONExtractString(events.json, 'event') = 'landing_viewed',
      JSONExtractString(events.json, 'event') = 'signup_started',
      JSONExtractString(events.json, 'event') = 'signup_completed'
    ) AS step_reached
  FROM exposed
  INNER JOIN product_events AS events
    ON exposed.unit_id =
      JSONExtractString(events.json, 'anonymous_id')
  WHERE
    events.timestamp >= exposed.exposed_at
    AND events.timestamp < exposed.exposed_at + 7 * 24 * 60 * 60
  GROUP BY exposed.unit_id, exposed.variant
)
SELECT
  variant,
  count() AS exposed_units,
  countIf(step_reached >= 3) AS completed_units,
  round(
    100 * completed_units / nullIf(exposed_units, 0),
    2
  ) AS completion_rate_pct
FROM progress
GROUP BY variant
ORDER BY variant

Validate ordered-funnel behavior against known test journeys, including repeated and out-of-order steps.

Use a settled analysis window#

If the outcome window is seven days, a participant exposed yesterday has not had the same opportunity as one exposed ten days ago.

For a final analysis:

• stop enrollment at the planned boundary
• wait for the outcome window to mature
• include only entities with a complete observation window
• continue monitoring late delivery

Do not classify incomplete participants as non-converters.

Define the primary outcome#

A trustworthy rate includes:

numerator: eligible exposed units that completed the outcome in 7 days
denominator: eligible exposed units with a complete 7-day window

Decide before launch:

• first exposure or assignment starts the clock
• repeated conversions count once or many times
• refunds or reversals negate conversion
• users who merge identities remain in which unit
• bot, employee, and test exclusions

Export one row per experimental unit or aggregated sufficient statistics to the approved statistical workflow.

Add guardrails#

Guardrails protect against a local win that harms the product:

Do not create twenty informal decision metrics. Predefine a small set and what would trigger a safety stop.

Plan sample size and power#

Before launch, choose:

• baseline conversion
• minimum effect worth acting on
• false-positive tolerance
• desired power
• allocation ratio
• expected eligible traffic
• maximum duration

Use an approved power calculation outside GraphJSON. Preserve its inputs with the protocol.

If the required sample would take months:

• test a larger product change
• choose a more frequent valid primary outcome
• run a qualitative or staged rollout
• accept that the experiment is underpowered

Do not lower standards after launch merely to produce a result.

Avoid uncontrolled sequential testing#

Repeatedly checking a conventional fixed-horizon significance test and stopping when it crosses a threshold inflates false positives.

Choose one:

• fixed sample or duration with no early efficacy stop
• a preplanned sequential method
• a Bayesian decision rule with documented priors and thresholds

GraphJSON can update descriptive charts continuously. A live chart is not permission to change the stopping rule continuously.

Safety stops are different: stop when a predefined harmful guardrail is violated.

Handle novelty and learning#

New interfaces can produce temporary curiosity or confusion.

Inspect:

• new versus returning users
• first exposure versus later use
• effect by week since exposure
• downstream retention

Do not call a short-lived click increase a durable product improvement.

Control multiple comparisons#

Every additional:

• primary metric
• variant
• subgroup
• time window
• post-hoc slice

creates another opportunity to find noise.

Predefine the primary comparison. Label other exploration as exploratory and confirm it in a future experiment.

Do not search segments for one impressive lift and present it as the original hypothesis.

Check interference and contamination#

One participant can affect another:

• teammates collaborate
• one marketplace side affects the other
• one customer controls shared configuration
• treatment changes network or queue load

When interference is likely, randomize at the shared account, workspace, team, or cluster level. Update the power plan for fewer independent units.

Check contamination:

• unit appears in multiple variants
• treatment configuration is visible to control
• support or sales manually changes the experience
• cached pages cross variant boundaries

Interpret practical significance#

A statistically credible effect can still be too small, costly, risky, or temporary to ship.

Decision record:

Observed effect and interval:
Primary metric result:
Guardrail results:
Sample-ratio check:
Instrumentation exceptions:
Practical value:
Known limitations:
Decision:
Owner:
Decision date:

Record “inconclusive” when the data cannot support a decision.

Roll out the winner safely#

An experiment result does not make production rollout risk-free:

1. preserve the winning configuration
2. roll out gradually
3. monitor the primary metric and guardrails
4. keep a rollback path
5. remove assignment code after the planned observation period
6. retain the decision record
7. deprecate experiment-specific fields deliberately

Do not immediately delete the experiment events needed to reproduce the decision.

Experiment checklist#

• Hypothesis, owner, and primary metric were written first.
• Eligibility and randomization unit are explicit.
• Assignment is sticky and versioned.
• Assignment and exposure are separate events.
• Outcome events remain valid product facts.
• One unit cannot appear in several variants.
• Sample-ratio mismatch is checked.
• Outcome windows mature before final analysis.
• Sample size and statistical method are preplanned.
• Peeking does not change the stopping rule.
• Guardrails and safety stops are predefined.
• Multiple comparisons and segments are controlled.
• Practical significance is part of the decision.
• Rollout is monitored separately from the test.

Continue with the Product metrics cookbook, Funnel SQL recipe, and Identity lifecycle guide.