A/B test statistical significance: recheck the AI with a z-test

100 visits per variant, 8 conversions against 14. Ask an AI about those numbers and the verdict, "not significant", comes back right. The trouble is the numbers that come with it.

Asking the short way

Landing page A/B test results.Variant A: 100 visits, 8 conversionsVariant B: 100 visits, 14 conversionsIs B significantly better? Keep it short.

No, not significantly. A two-proportion z-test gives z ≈ 1.36 and p ≈ 0.18 (two-sided), well above 0.05. The 95% CI for the lift (+6 points) is roughly −3 to +15 points, so B could be worse or much better. To reliably detect a difference of this size (8% vs 14%) at 80% power, you'd need about 420 visits per variant.

The verdict holds. The required sample size does not: across four runs (two English, two Korean) it came out as "about 420", "roughly 350 to 400", "400 or more" and "400 to 450", with no working shown in any of them.

A prompt with counts and rules

Put raw counts in the { } slots. Send visits and conversions, not percentages, or there is nothing to recheck.

<data> below holds the raw counts from a landing page A/B test. Decide whether variant B has a statistically significantly higher conversion rate than variant A. Rules:- First name the test and write its formula. Say whether it is one-sided or two-sided. Use a significance level of 0.05 (two-sided).- Show every intermediate value: each variant's conversion rate, the pooled rate, the standard error, z and p. Four decimal places.- Write the conclusion at the very end, after all calculations, as "significant" or "not significant", with the reason in at most two sentences. Do not write any z or p value before you have calculated it.- If the sample size or expected counts make this test inappropriate, say so.- Do not guess what the data cannot tell you (whether visits are unique users, test duration, whether results were checked mid-test, etc.). List those separately under "Needs checking". <data>Variant A: {A visits} visits, {A conversions} conversionsVariant B: {B visits} visits, {B conversions} conversions</data>

The last rule follows Anthropic's guide to reducing hallucinations: give the model explicit permission to say it doesn't know. The conclusion-last rule started as "conclusion first"; in six runs of that version the model wrote the calculation first every time, so the instruction now matches that order.

Run it

--setting-sources "" stops Claude Code from loading your own settings and CLAUDE.md, which would otherwise change the language and tone of the answer.

$ claude -p --model claude-sonnet-5-5 --max-turns 1 --tools "" --setting-sources "" --strict-mcp-config < prompt.big.txt > out.big.txt

From the large-sample run (12,000 visits per variant), the calculation and the verdict:

- Conversion rate A: 540 / 12000 = **0.0450**- Conversion rate B: 612 / 12000 = **0.0510**…- Standard error: √(0.0480 × 0.9520 × (1/12000 + 1/12000)) = √(7.616 × 10⁻⁶) = **0.0028** (0.002760 before rounding)- z = 0.0060 / 0.002760 = **2.1741**- p (two-sided) = 2 × (1 − Φ(2.1741)) = **0.0297**…## Conclusion**Significant.** The two-sided p-value of 0.0297 is below 0.05 (z = 2.1741), so B's conversion rate (0.0510) is significantly different from A's (0.0450), in B's favor. …

The small set (100 per variant) gave z = 1.3560, p = 0.1751, "not significant". Both "Needs checking" lists asked about unique users, random assignment and peeking.

Recompute with statsmodels

Start again from the raw counts (statsmodels and scipy in a virtualenv). proportions_ztest defaults to a pooled, two-sided test, the same setup the prompt asked for.

#!/usr/bin/env python3"""usage: ztest.py conv_a n_a conv_b n_b"""import sysfrom statsmodels.stats.proportion import proportions_ztestfrom scipy.stats import fisher_exactxa, na, xb, nb = map(int, sys.argv[1:5])z, p = proportions_ztest([xb, xa], [nb, na])          # pooled, two-sidedpool = (xa + xb) / (na + nb)se = (pool * (1 - pool) * (1 / na + 1 / nb)) ** 0.5_, pf = fisher_exact([[xb, nb - xb], [xa, na - xa]])print(f"rate A {xa/na:.4f}  rate B {xb/nb:.4f}  pooled {pool:.4f}  se {se:.6f}")print(f"z {z:.4f}  p {p:.4f}  fisher p {pf:.4f}")

The third run is a set the AI never saw: the same 4.5% vs 5.1% rates, with only 2,000 visits per variant.

$ v/bin/python ztest.py 540 12000 612 12000rate A 0.0450  rate B 0.0510  pooled 0.0480  se 0.002760z 2.1741  p 0.0297  fisher p 0.0320$ v/bin/python ztest.py 8 100 14 100rate A 0.0800  rate B 0.1400  pooled 0.1100  se 0.044249z 1.3560  p 0.1751  fisher p 0.2582$ v/bin/python ztest.py 90 2000 102 2000rate A 0.0450  rate B 0.0510  pooled 0.0480  se 0.006760z 0.8876  p 0.3748  fisher p 0.4159

The AI's z and p matched to four decimals for both sets. The same 13% lift at 2,000 per variant gives p = 0.3748: "significant" is decided by sample size, not by the size of the lift.

Where it goes wrong

• Hand-checking from rounded values. Divide 0.0060 by the reported SE of 0.0028 and you get z = 2.1429. One Korean run warned about exactly this. Always recheck from the raw counts.
• Required sample size. The 350–450 figures in the short answers have no working behind them. NormalIndPower().solve_power(proportion_effectsize(0.14, 0.08), alpha=0.05, power=0.8) returns 419.3. Compute it yourself when you need it.
• When not to trust the verdict. If visits are sessions rather than users, if someone watched the dashboard and stopped on a good day, or if several variants were compared, the p-value itself is wrong. Matching numbers won't reveal that, so answer the "Needs checking" list before you report the result.