A/B Testing

Run experiments to find winning ad creatives and strategies using Bayesian statistics.

Why Bayesian Testing?

Scalegrowth uses Bayesian A/B testing instead of traditional frequentist (p-value) testing. Here is why:

Intuitive results: Instead of a p-value, you get "89% probability that Variant B is better." Easy to understand and act on.
No peeking penalty: You can check results anytime without inflating error rates.
Expected loss: Know exactly how much you would lose if you pick the wrong winner.
No fixed sample size: Tests can be called early if results are overwhelming, saving budget.

1. Choose what to test: Ad creative (image vs video), headline copy, CTA text, landing page, audience segment, or bidding strategy.
2. Set up variants: Control (your current version) vs Variant (the challenger). You can test up to 4 variants simultaneously.
3. Configure traffic split: Default is 50/50. Adjust if you want to reduce risk (80/20 gives the control more traffic).
4. Set minimum duration: Recommended: 7–14 days to capture day-of-week effects.
5. Choose primary metric: CPA, ROAS, CTR, or conversion rate. This is the metric used to determine the winner.
6. Launch: Traffic is automatically split between variants at the ad platform level.

Probability to Beat Control (PTB): The core metric. ≥95% is a strong signal to declare a winner. 80–94% is promising but consider waiting.
Expected Loss: If you pick this variant and it is actually worse, how much would you lose per conversion? Below $1.00 is very safe.
Confidence bar: Visual indicator. Green = strong (≥95%), Blue = promising (≥80%), Yellow = too early (≥60%), Gray = insufficient data.
Variant table: Shows impressions, clicks, conversions, spend, and CPA for each variant side-by-side.

We recommend declaring a winner when:

You can call a winner earlier if PTB is ≥99% and expected loss is near $0 — the data is overwhelming.