How Large Should An Email A/B Test Sample Be?

There is no reliable rule such as “test 10% of the list” or “use 5,000 subscribers per version”. An email A/B test needs enough recipients per variation to distinguish a meaningful result from ordinary variation. How many that means depends chiefly on the baseline rate of the metric, the size of change worth detecting, and the standard of evidence you set before sending.

For a high-volume promotional campaign, a small test group may be enough to compare subject lines. For a low-volume lifecycle email where the outcome is a purchase, the required sample can be tens of thousands of recipients per version. In some cases, the honest answer is that the programme does not generate enough eligible volume for a conventional A/B test to answer the question.

Start with the decision, not the percentage of the list

An A/B test should support a specific decision: for example, whether to make a new subject line the default, use a different call to action in a renewal email, or retire a discount that does not produce incremental orders.

Write down four things before choosing the test size:

• The primary metric: the one outcome that will determine the winner.
• The baseline rate: the recent rate for that metric under the current version.
• The minimum detectable effect (MDE): the smallest improvement that would justify changing course.
• The statistical standard: usually a 95% confidence level and 80% power for a two-variant test.

These inputs are more useful than a generic list-size recommendation. Sample-size calculators for proportion metrics use the same essentials: baseline conversion rate, desired effect size, significance threshold and statistical power. Statsig’s sample-size calculator provides a practical explanation of these inputs, while Optimizely’s guide to sample-size calculations explains why the method must match the design and assumptions of the experiment.

What “sample size” means in an email test

In a two-variant campaign test, sample size normally means the number of recipients in each randomly assigned version. If the calculation says 4,000 recipients per version, the test needs 8,000 recipients in total before any remainder is sent.

That distinction matters when an email platform says “test 20% of your audience”. With a 50/50 A/B split, each version reaches only 10% of the total audience. A 100,000-recipient campaign therefore gives 10,000 recipients to A and 10,000 to B, with 80,000 held back for the winner. Whether that is enough depends on the metric and MDE.

Choose the denominator before the test begins. For click rate, many teams use unique clicks divided by delivered emails. In that case, calculate the required number of delivered emails, then allow for the expected non-delivery rate when deciding how many recipients to allocate. For a more conservative intent-to-treat analysis, use everybody randomised into each arm. Either approach can be valid; changing the denominator after seeing results is not.

The four inputs that determine the answer

1. Baseline rate

Use recent, comparable performance rather than a broad annual average. The click rate from a weekly promotional email may not describe a transactional reminder, and a campaign sent to recent buyers may not resemble one sent to a dormant segment.

Low-rate events require disproportionately large samples. A move from a 2% click rate to 2.4% is only a 0.4 percentage-point difference, even though it is a 20% relative lift. Detecting it reliably takes far more recipients than detecting a two-point change from 10% to 12%.

2. Minimum detectable effect

The MDE is the smallest change the test is designed to find. It is a commercial choice, not a statistical preference. Ask: “What improvement would be valuable enough to adopt, given implementation effort, margin, customer experience and possible downside?”

Be precise about the two ways of expressing change:

• Absolute lift: from 2% to 2.4% is an increase of 0.4 percentage points.
• Relative lift: the same move is a 20% increase relative to the 2% baseline.

Small MDEs demand large samples. Setting an unrealistically tiny MDE merely to find a winner can turn a short campaign test into an underpowered exercise that never reaches a useful conclusion. Conversely, choosing a very large MDE reduces the required sample but can miss smaller improvements that would still matter over a year.

3. Confidence level and statistical power

A 95% confidence threshold is a common starting point. In the usual fixed-horizon framework, it limits the risk of declaring a difference when there is no real difference. Power, often set at 80%, is the chance that the test will detect the chosen MDE if that effect truly exists.

Higher confidence and higher power both increase the sample requirement. They are worthwhile where the decision is costly or difficult to reverse, such as changing a core lifecycle journey or an offer strategy. For a low-risk creative experiment, the organisation may reasonably set different pre-agreed standards, but it should understand the increased chance of a false positive or missed effect.

4. The number of variations and the split

For a straightforward A/B test, a 50/50 split is statistically efficient. Unequal allocation is sometimes sensible when a variant carries material commercial or deliverability risk, but it generally needs more total recipients to deliver the same evidential strength.

Adding versions is not free. An A/B/C/D subject-line test spreads the available audience across four groups and increases the risk of finding an apparently strong result by chance. If volume is limited, test two purposeful alternatives, or run sequential campaigns with a clearly defined learning plan, rather than testing every copy idea at once.

Illustrative sample sizes for email metrics

The table below uses an approximate two-sample proportion calculation with an even 50/50 split, a two-sided 5% significance threshold and 80% power. Figures are recipients per version, before any allowance for non-delivery. They are planning estimates, not a substitute for a calculator configured to your actual analysis method.

Primary metric and baseline	Target result	Relative lift	Approximate recipients per version
Click rate: 2%	2.4%	20%	21,100
Click rate: 2%	3%	50%	3,800
Purchase conversion: 1%	1.3%	30%	19,800
Click rate: 10%	12%	20%	3,800
Click rate: 20%	22%	10%	6,500

The practical lesson is simple: the expected outcome rate and the absolute difference drive the sample much more than the total database size. A list of 50,000 people does not make a 1% conversion test easy if only a fraction is eligible for the message and the desired lift is modest.

Choose a primary metric that can support the decision

For many campaigns, unique click rate or a downstream conversion rate is more useful than open rate. Apple’s Mail Privacy Protection prevents senders from seeing whether recipients have opened messages in the usual way, so measured open rates can no longer be treated as a clean measure of human attention for audiences that use protected Mail. Apple’s Mail Privacy Protection guidance confirms that the feature prevents senders from seeing whether a message was opened.

That does not make subject-line testing pointless. It changes how carefully the result should be interpreted. If a platform selects an automatic winner using opens, regard it as a directional operational signal, especially where Apple Mail usage is significant. Where practical, use clicks, qualified sessions, completed registrations, orders or another event that is closer to the business purpose of the email.

Revenue deserves particular care. A few unusually large orders can make average revenue volatile, which means a simple conversion-rate sample-size calculation may not apply. Consider setting conversion as the primary test metric and revenue per delivered email as a supporting measure, or use an analysis designed for revenue distributions.

Do not stop when one version happens to be ahead

Checking results repeatedly and declaring victory at the first encouraging gap creates false winners. With a conventional fixed-horizon test, decide the required sample and evaluation window in advance, then assess the result when both have been reached.

Some platforms use sequential or always-valid statistical methods that are designed to evaluate results during the test. Those can be useful, but only if the platform’s winner logic, significance setting, conversion window and stopping rule are understood. Do not combine a fixed-sample calculator with informal daily peeking and assume the stated confidence level still applies.

Allow enough time for the event being measured. A subject-line test based on clicks may settle relatively quickly; a test based on purchases, renewals or trial conversion needs to remain open through the normal attribution window. If variant A is sent on a Friday morning and B on a Sunday evening, the comparison has also become a send-time test.

Randomisation and audience consistency matter as much as volume

A large sample cannot repair biased assignment. Randomise eligible recipients at the recipient level and keep the rest of the campaign conditions consistent: send window, suppression rules, frequency limits, offer availability, landing page, tracking and provider routing.

Where the audience contains materially different groups, distribute them evenly across variants. For example, a loyalty programme might balance recent purchasers, lapsed purchasers and new members within each version. If the test is only intended for Gmail recipients or a particular country, calculate the sample against that eligible sub-audience rather than the whole database.

Avoid overlapping tests that expose the same people to competing changes unless the design explicitly accounts for it. A customer receiving a changed subject line in one campaign and a changed offer in another may make the observed effect difficult to attribute.

Use deliverability as a guardrail, not a footnote

Most copy tests are low risk, but different content, cadence, sender identities or targeting criteria can affect more than engagement. Monitor delivery, bounces, unsubscribes, spam complaints and inbox placement alongside the primary metric. A version that creates a small click-rate gain while materially increasing complaints is not an uncomplicated winner.

This is especially important at Gmail. Google advises senders to keep Postmaster Tools spam rates below 0.1% and avoid ever reaching 0.3% or higher; it also notes that low-volume traffic may not produce dashboard data on every day. Google’s sender guidelines and its Postmaster Tools dashboard guidance are useful references when setting operational safeguards.

Complaint rates are normally too low to serve as a quick winner metric in ordinary A/B tests. Treat them as pre-agreed stop or review conditions, especially when testing audience expansion, frequency or a more aggressive promotional approach. Permission, accurate suppression and a straightforward unsubscribe route are prerequisites, not test variables.

What to do when the list is too small

If the required sample exceeds the eligible audience, do not declare a winner from a weak signal. Choose a method that matches the constraint:

• Test a larger, commercially meaningful change rather than a tiny wording difference.
• Run the experiment over several comparable sends, while preserving randomisation and accounting for campaign-to-campaign variation.
• Use a higher-frequency lifecycle flow, provided the test does not alter contact pressure or the recipient’s experience unfairly.
• Collect qualitative evidence, such as customer research or usability review, before committing scarce audience volume.
• Make the decision using expected value and risk, while recording that the evidence is directional rather than statistically conclusive.

For triggered emails, a persistent 50/50 recipient-level split is often more appropriate than sending a small test batch and a later winner. It lets enough eligible recipients accumulate naturally and avoids making time of day or week a hidden difference between versions.

A practical planning example

Suppose a retailer’s recent comparable campaigns produce a 2% unique click rate. The team would only change its standard creative if it could reasonably improve clicks by at least 50%, from 2% to 3%. At 95% confidence and 80% power, it needs roughly 3,800 delivered recipients per version: about 7,600 delivered test recipients in total.

If the expected delivery rate is 98%, the initial allocation should be slightly higher than 3,800 per version. The team should randomly split a comparable eligible audience, run both versions in the same send window, retain click rate as the primary metric, and wait for the pre-agreed reporting window. It should also compare unsubscribe and complaint signals before rolling the apparent winner out to the remainder.

Change the target from 3% to 2.4%, however, and the requirement rises to roughly 21,100 per version. That is not a failure of the test. It is the cost of trying to distinguish a much smaller improvement from normal noise.

Where deliverability analysis fits

If results are unexpectedly erratic, the issue may not be creative. Authentication failures, uneven delivery by mailbox provider, suppression errors, an unhealthy segment or an unstable sending pattern can distort the audience that actually sees each version. In that situation, it is better to resolve the measurement and delivery problem before drawing copy conclusions. Email Foundry’s work is relevant where a permission-based sender needs to assess those infrastructure, audience-quality or provider-level delivery factors; it is not a substitute for a properly designed experiment.

The right email A/B test sample is therefore the smallest one that can detect a decision-worthy effect with a pre-agreed level of confidence and power, while keeping the audience, timing and delivery conditions comparable. Start with that calculation. A percentage of the list can follow from it, not the other way around.

Frequently asked questions

Is 10% of an email list enough for an A/B test?

Sometimes, but it cannot be known from the percentage alone. Split that 10% between the versions, then compare the recipients per version with the sample required for your baseline metric and minimum detectable effect. A 10% test on a large list may be excessive for a high-rate click metric and far too small for a low-rate purchase metric.

Should an email A/B test use a 50/50 split?

Usually, yes. An even split gives the most efficient comparison for a standard two-variant test. Use an unequal split only when there is a clear reason to limit exposure to a potentially risky variant, and recalculate the total sample because unequal allocation generally requires more recipients.

Can I use open rate as the winner metric?

It can be used as a directional signal, particularly for subject-line tests, but it is less dependable than it once was because Mail Privacy Protection obscures whether some recipients actually opened a message. Clicks, qualified sessions or conversions are usually stronger winner metrics when they match the campaign objective.

How long should an email A/B test run?

Run it until the pre-calculated sample is reached and long enough for the chosen outcome to occur within its normal reporting window. Click tests can often be evaluated sooner than purchase or renewal tests. Keep both variants in comparable send conditions and avoid calling a winner simply because one is ahead early.

Sources and further reading

• Google email sender guidelines — Google
• Google Postmaster Tools dashboards — Google
• Use Mail Privacy Protection on iPhone — Apple
• A/B test sample size calculator — Statsig
• Sample size calculations for A/B tests and experiments — Optimizely
• How to A/B test an email campaign — Klaviyo