Question 1

How much traffic do I need to run a valid A/B test?

Accepted Answer

Use a sample size calculator (Evan Miller's, Optimizely's, or VWO's are all free) with your inputs: current conversion rate, minimum detectable effect (the smallest improvement you'd want to act on — typically 10–20% relative improvement), and confidence level (95% is standard). The output is the number of visitors needed per variant. Multiply by 2 for total visitors and divide by your monthly page traffic to get the test duration. If the duration comes out to more than 8 weeks, the test is impractical — seasonal effects and page changes will confound results. Under 4 weeks and you have a viable test. Between 4 and 8 weeks: proceed with awareness of the limitations.

Question 2

Does A/B testing affect SEO?

Accepted Answer

It can, if done incorrectly. Cloaking — showing Google a different page than users see — is a violation of Google's guidelines. Google specifically permits A/B testing where both variants are accessible to Googlebot, where you're not redirecting users based on user-agent, and where you're not attempting to hide the test from Google. Using JavaScript-based testing tools (Optimizely, VWO, Google Optimize replacements) that modify the page in the browser rather than serving a different URL to Google are generally safe. Server-side A/B tests that redirect users to different URLs should use the canonical tag to specify which URL is the preferred version for indexing, and set the test variant pages to noindex if they're being crawled.

Question 3

What's the best free A/B testing tool?

Accepted Answer

Google Optimize was the dominant free option until it was deprecated in 2023. Current free and low-cost alternatives include: Clarity Experiments (Microsoft Clarity's built-in testing tool, free), Growthbook (open-source, can self-host for free), and AB Tasty (has a free tier with limitations). For most low-traffic sites, these tools provide more capability than you'll use — the constraint is sample size, not tool capability. If your concern is statistical validity rather than tool access, focus first on whether your traffic volume supports the test before evaluating which tool to use for it.

Question 4

Should I stop a test early if one variant is clearly winning?

Accepted Answer

No — stopping early when one variant appears to be winning is one of the most common A/B testing mistakes. Early in a test, the conversion rate difference between variants fluctuates widely due to small sample sizes. What looks like a 40% win after 200 visitors may reduce to 5% (or reverse) by 2,000 visitors. The phenomenon is called "peeking" and it produces a systematic bias toward false positives — you stop when randomness produces a temporary win, rather than when the true underlying conversion rate is known with confidence. Run tests to their predetermined sample size. The only valid reason to stop early is if the losing variant is causing significant harm (a broken form, a catastrophic drop in revenue).

Question 5

How do I avoid confounding variables in sequential improvement?

Accepted Answer

You can't eliminate confounding entirely with sequential testing, but you can reduce it: avoid making changes during periods of known seasonal variation (holidays, industry-specific busy/slow seasons) unless you're comparing like-for-like periods year over year, keep a log of any external factors that might affect traffic quality during the measurement window (a new ad campaign, a PR mention, a competitor's price change), and prefer 4-week windows over shorter ones to average out day-of-week and short-term noise. If a result looks unusually large, check the log before crediting the change — a spike from an unrelated traffic source is more likely than an 80% lift from one headline tweak.

Question 6

At what traffic level does it make sense to switch from sequential testing to formal A/B tests?

Accepted Answer

Once a page consistently exceeds roughly 4,000-5,000 monthly visitors with a conversion rate in the low single digits, formal A/B tests become viable within a 4-8 week window for moderate effect sizes — that's the point where the calculator-derived sample size stops requiring unreasonably long test durations. Below that, sequential improvement remains the better-calibrated approach. It's also reasonable to run both approaches in parallel during the transition: formal A/B tests on your highest-traffic 1-2 pages, sequential improvement everywhere else, scaling up the formal-testing footprint as overall traffic grows.