Sample Report

Two of Junco's shipped experiment wins this year never showed up in the subscription numbers afterw…

You have twenty minutes before the launch review and three asks: is the 13% lift real, should the f…

Your first cut confirms the readout: control converts at 10.4%, treatment at 11.8%, and the gap cle…

2 answered, 0 skipped.

analyze the experiment data and tell me if the new onboarding worked

Let me check the data dictionary and the experiment data structure first. Let me check the core numbers: overall conversion lift, sample ratio mismatch, and whether randomization balance holds across channel/platform/day. That's a major finding — this isn't a clean 50/50 randomization. Let me pin down exactly where the imbalance comes from (day-level vs cha…

Stop there - I'm not taking a pooled readout. I already have the arm sizes from my own query: 12,943 control against 11,057 treatment, on a test data_dictionary.md says was specified 50/50. That is a sample ratio mismatch, so the randomisation a p-value assumes did not happen, and the pooled comparison is not a comparison yet. Two constraints before you wri…

Checking daily treatment-share distribution and confirming the 7-day-block logic before writing the cell. Confirmed: week 1 (Mar 4–10) runs ~60/40 control-heavy every single day, and week 2–3 (Mar 11–24) is clean, within a couple points of 50/50 every day — so the 7-day-block cut matches the dirty/clean split almost exactly. Writing the two-part cell now. T…

# Cross-check the corrected effect independently, before I take the
# assistant's number for the write-up. Two different corrections:
# post-stratify the WHOLE test on channel, and restrict to the clean
# window by filtering dates directly rather than via a week column.
import pandas as pd
import numpy as np
from scipy import stats

chk = pd.read_csv('case_data.csv')

def eff(a, b):
    p1, p2 = a.mean(), b.mean()
    se = np.sqrt(p1 * (1 - p1) / len(a) + p2 * (1 - p2) / len(b))
    return p2 - p1, se⋯ +22 more lines

I corrected it a second way in my own cell before taking yours. Post-stratifying the whole test on channel I get +0.48pp, 95% CI [-0.30, +1.27]; filtering the dates directly for 11-24 March instead of using a derived week column I get +0.16pp, p = 0.75, CI [-0.81, +1.13]. Confirm both of those match what you produced - if they don't, we find out why before…

Verifying both corrections independently in my own query before confirming, then writing the activated_48h + platform breakdown cell. Both match exactly: clean-window +0.16pp CI [-0.81, +1.13] p=0.75, and post-stratified +0.48pp CI [-0.30, +1.27]. Same underlying answer two ways — the confidence intervals straddle zero either way you correct for the broken…

Great, I'll use those. Now the mix, so I can say why the broken week moved the headline at all: conversion rate by acquisition_channel across the whole test, each arm's channel mix as percentages, and the conversion rate for activated against non-activated users. New cell.

Writing one cell with the three pieces: channel-level conversion (whole test), each arm's channel composition as %, and conversion by activation status — this is the mix story that explains why the broken week moved the pooled number. Numbers: `paid_social` converts at 4.41% — far below `organic` (15.35%), `paid_search` (9.03%), and `referral` (18.73%). Con…

Those look right, I'll quote them. So where does that leave Monday - does the activation result carry the launch?

Let me pull the actual activated_48h numbers from cell 7 before answering. Cell 7 hasn't been executed yet (no output stored), so let me get the actual activated_48h numbers directly. On the clean window, `activated_48h` moves and moves hard: overall +8.6pp (46.7% control → 55.3% treatment), 95% CI [7.0, 10.1], p<0.001. That's not marginal — it's consistent…

Careful - anything that ends in shipping on this evidence is the same mistake in a smaller box. Activated users convert at 14.5% against 7.4%, which is exactly why the team assumed activation would carry, but conversion inside each platform on the clean window is flat: iOS -0.1pp, Android +0.9pp, web -0.5pp, none of them near significant. The flow activates…

Rewriting the Task 2 cell now with exactly those three points, no launch call, no 13% figure. That's in cell 9. It deliberately stops short of any launch/hold call and doesn't cite the 13% or any other point estimate — it frames the activation result as justification to spend a clean re-run on the idea, not as evidence the flow works, which matches the plat…

-- Rate two ways so the fix is not itself a typo.
-- 12943 control against 11057 treatment on a test specified 50/50 -
-- that is 1886 users of difference and it is not rounding.
SELECT assigned_variant,
       COUNT(*) AS users,
       SUM(converted_to_paid) AS converted,
       ROUND(100.0 * SUM(converted_to_paid) / COUNT(*), 1) AS conversion_pct,
       ROUND(AVG(converted_to_paid) * 100, 1) AS conversion_pct_check
FROM case_data
GROUP BY assigned_variant
ORDER BY assigned_variant

import pandas as pd
df = pd.read_csv('case_data.csv')

def weekly(df):
    # 7-day blocks from the first signup, not calendar weeks.
    d = pd.to_datetime(df['signup_date'])
    week = (d - d.min()).dt.days // 7 + 1
    out = df.assign(week=week).groupby(['week', 'assigned_variant']).agg(
        users=('user_id', 'count'),
        conversion_rate=('converted_to_paid', 'mean'))
    out['conversion_rate'] = (100 * out['conversion_rate']).round(1)
    return out.reset_index()

w = weekly(df)⋯ +7 more lines

import pandas as pd
from scipy import stats

df = pd.read_csv('case_data.csv')
df['signup_date'] = pd.to_datetime(df['signup_date'])

# --- Overall headline lift ---
g = df.groupby('assigned_variant').agg(n=('user_id','count'),
                                        conv=('converted_to_paid','mean'),
                                        act=('activated_48h','mean'))
print(g)
lift = g.loc['treatment','conv'] / g.loc['control','conv'] - 1
print(f'\nheadline conversion lift: {lift:.1%}')
⋯ +17 more lines

import numpy as np
from scipy import stats

# --- Bucket into 7-day experiment blocks from first signup (not calendar week) ---
start = df.signup_date.min()
df['experiment_week'] = ((df.signup_date - start).dt.days // 7) + 1

# --- 1) Where is the imbalance concentrated? count + treatment share per cell ---
def variant_breakdown(index_col):
    ct = df.groupby([index_col, 'assigned_variant']).size().unstack(fill_value=0)
    ct['n'] = ct['control'] + ct['treatment']
    ct['treatment_share'] = (ct['treatment'] / ct['n']).round(3)
    return ct
⋯ +20 more lines

# Cross-check the corrected effect independently, before I take the
# assistant's number for the write-up. Two different corrections:
# post-stratify the WHOLE test on channel, and restrict to the clean
# window by filtering dates directly rather than via a week column.
import pandas as pd
import numpy as np
from scipy import stats

chk = pd.read_csv('case_data.csv')

def eff(a, b):
    p1, p2 = a.mean(), b.mean()
    se = np.sqrt(p1 * (1 - p1) / len(a) + p2 * (1 - p2) / len(b))
    return p2 - p1, se⋯ +22 more lines

# Clean window only (signup_date >= 2026-03-11, where the split is ~50/50).
# 1. activated_48h by variant, and by variant x platform
# 2. converted_to_paid by platform, same window
import pandas as pd
import numpy as np
from scipy import stats

df = pd.read_csv('case_data.csv')
clean = df[df['signup_date'] >= '2026-03-11'].copy()

def diff_ci(a, b):
    p1, p2 = a.mean(), b.mean()
    se = np.sqrt(p1 * (1 - p1) / len(a) + p2 * (1 - p2) / len(b))
    d = p2 - p1⋯ +33 more lines

# Why the broken week moved the headline: the mix
# 1. conversion rate by acquisition_channel, whole test
# 2. each arm's channel composition (%), whole test
# 3. conversion rate for activated vs non-activated users
import pandas as pd

df = pd.read_csv('case_data.csv')

by_channel = df.groupby('acquisition_channel')['converted_to_paid'].mean().mul(100).round(2)
by_channel = by_channel.rename('conversion_rate_pct').sort_values(ascending=False)

arm_mix = pd.crosstab(df['assigned_variant'], df['acquisition_channel'], normalize='index').mul(100).round(2)

by_activation = df.groupby('activated_48h')['converted_to_paid'].mean().mul(100).round(2)⋯ +8 more lines

Write ~120 words to Dana: is the 13% lift real, and what should happen to Monday's launch?

Write 3–4 bullets for the growth team: what has to be true of an experiment readout before anyone schedules a launch off it?