A/B Testing Email Content: A Practical Guide

Email A/B testing compares two versions of an email to learn whether a deliberate difference changes recipient behaviour. It can answer practical questions: does a benefit-led subject line earn more qualified visits than a curiosity-led one? Does a simpler product explanation lead to more trials? Does a text link outperform a large button for a particular audience?

The useful part is not declaring that version B “won”. It is building evidence about what helps a defined group of subscribers take a worthwhile next step, without creating more complaints, unsubscribes or delivery problems. A well-run test produces a decision that can be used again. A poorly run one produces a number that is easy to misread.

Start with a decision, not a vague wish to optimise

Before writing variants, state the decision the test will inform. “Improve engagement” is too broad. “Choose the opening proposition for the renewal reminder” is specific enough to test.

A practical hypothesis has four parts:

• Audience: who is included, such as active UK subscribers who viewed a product category in the last 90 days.
• Change: the single meaningful difference between A and B.
• Expected mechanism: why that difference might affect behaviour.
• Decision rule: the result that will determine what happens next.

For example: “For subscribers who joined from the pricing page, leading with a worked saving example rather than a feature summary will increase completed demo requests, because it makes the commercial benefit easier to assess. We will adopt it if it improves completed demo requests without an unacceptable increase in complaints or unsubscribes.”

This discipline matters because email metrics are connected but not interchangeable. A subject line influences whether someone notices or opens a message. The message body, offer, landing page and form then influence whether they act. Testing a redesigned email against an unchanged control may be sensible, but it cannot credibly tell you whether the improvement came from the headline, layout, imagery, call to action or offer.

Choose the metric that matches the question

Pick one primary metric before sending. It should sit as close as possible to the decision being made.

What you are testing	Usually stronger primary measure	Useful guardrails
Subject line or preheader	Unique click rate, or downstream conversion rate where volume permits	Spam complaints, unsubscribes, bounces
Opening proposition, copy or creative	Unique click rate to the intended destination	Landing-page conversion, complaints, unsubscribes
Call to action	Clicks on that call to action, then completion of the intended action	Other-link clicks, conversion quality
Offer or commercial proposition	Qualified conversion, revenue or retained value	Refunds, cancellations, complaints and margin where relevant
Information hierarchy or layout	Clicks to the priority content or task completion	Rendering quality, accessibility and unsubscribe rate

Open rate has limited value as a deciding metric, particularly for content tests. An open happens before the recipient sees the main copy or design. It is also an increasingly imperfect signal: Apple explains that Mail Privacy Protection can download remote content in the background regardless of whether the recipient engages with the email, preventing senders from reliably learning Mail activity. That can inflate or distort pixel-based open measurements. Apple’s explanation of Mail Privacy Protection is a useful reminder to treat opens as directional, rather than as proof of attention.

Clicks, completed forms, purchases and other first-party events are generally more actionable, although they still need context. A more aggressive subject line may lift clicks while bringing poorly qualified visitors to the site. That is why the primary result should be read alongside the outcome that matters to the business.

Keep a small set of guardrail metrics. A variant should not become the standard simply because it produces a little more revenue if it also prompts a material rise in spam complaints, unsubscribes or support contacts. The right tolerance will vary by programme and message type, so define it in advance instead of rationalising it after the result is known.

Test one meaningful variable at a time

A classic A/B test changes one variable while holding the rest of the email constant. In practice, “one variable” means one idea, rather than one word. You might compare two subject lines, two opening propositions, two calls to action or two content structures.

For a subject-line test, use the same From name, preheader, email body, audience and send conditions. For a body-copy test, use the same subject line, sender identity, send time, offer and destination page. That gives the result a clear interpretation.

There are exceptions. A full creative concept may involve a different hero image, headline, ordering and call to action. Treat that as a concept test, document it honestly, and resist claiming that one individual component caused the result. It can be an efficient way to choose between two genuine editorial directions, but it is not a fine-grained diagnostic exercise.

Testing several independent variables at once is multivariate testing. It requires substantially more recipients because every combination needs enough observations to produce a useful comparison. For most marketing programmes, sequential A/B tests are easier to run and explain.

Build comparable groups

Random allocation is what makes an A/B comparison persuasive. Assign eligible subscribers randomly to A or B, once, and ensure each person receives only one version. Most email platforms can do this, but check how their test and winner-selection settings work before relying on an automatic outcome.

Keep the audience definition stable. Do not send variant A to recent purchasers and variant B to lapsed subscribers, then attribute the difference entirely to copy. If important groups are likely to respond differently, such as customers versus prospects, test within those groups or report their results separately. Otherwise a variant can appear to win simply because it reached a more responsive mix of people.

Also avoid testing during an unusual period unless that period is the point of the test. A subject line that works during a flash sale, major product outage or Black Friday may not provide a reliable rule for routine messages. Operational changes can interfere too: a new segment definition, altered suppression rule, website incident or price change can make results hard to interpret.

Decide how much audience and time the test needs

There is no responsible universal sample size for email A/B testing. The necessary number depends on the usual rate of the event you care about, the smallest change worth acting on, the number of variants and how confident you need to be before making the decision.

A test of purchases will usually need more recipients than a test of clicks because purchases are less frequent. A small list may be adequate for choosing between two clearly different editorial approaches, but it may be incapable of distinguishing a small difference in conversion rate from ordinary variation.

Use a sample-size calculator or your testing platform’s statistical method before launch. Enter a realistic baseline rate and a minimum detectable effect: the smallest improvement that would justify changing future campaigns. If the programme cannot generate enough volume, treat the exercise as exploratory. Do not call a winner just because one version is ahead after a handful of clicks.

For a promotional campaign, a common approach is to send the variants to randomly selected test groups, wait for a pre-agreed evaluation window, then send the selected version to the remaining eligible audience. The trade-off is clear: a larger test gives more reliable evidence but leaves fewer people to receive the chosen version.

Let the window reflect normal recipient behaviour. A same-day retail offer may need a shorter window than a B2B webinar invitation that is commonly considered over several days. Do not keep checking the dashboard and stop as soon as the preferred version moves ahead. Repeatedly declaring a result at an attractive moment increases the chance of adopting random noise.

If a platform chooses winners automatically, understand its rules. For example, some tools allow choices such as opens, clicks or revenue, and can send a selected winner to the remaining audience after a set period. Their available variables, thresholds and reporting methods are product features, not universal testing standards. Mailchimp’s explanation of A/B tests illustrates the kinds of settings that should be checked, including the distinction between automatic and manual winner selection.

Content tests worth running

Subject line and preheader

Test the framing, not superficial punctuation. Compare a concrete benefit with a timely use case, or a direct announcement with a question that accurately reflects the email. Keep claims supportable and ensure the body fulfils the expectation created in the inbox. A subject line that earns an open by concealing the real purpose of the email may damage trust and increase negative feedback.

Preheaders deserve the same care. Many inboxes show them alongside the subject line, so an unchanged preheader can either support or undermine the test. If the subject is the variable, make the preheader neutral or write a matched preheader for each subject-line concept.

Opening proposition

The first screen of the email often determines whether the subscriber continues. Test whether a concise statement of value, a customer problem, a new product announcement or an editorial insight gives the clearest reason to continue. This is especially useful when the email has a single priority action.

Call to action and link placement

Compare the action being requested, the label used and where it appears. “View your options” and “Renew now” do not ask for the same level of commitment, so measure the conversion outcome rather than clicks alone. A softer label may attract more clicks but fewer completed renewals; that can still be useful information about the decision journey.

Message length and content hierarchy

Rather than asking whether “short” or “long” emails win in general, test whether a particular audience needs more information before taking the next step. A high-consideration service may benefit from proof, detail and answers to objections. A repeat-purchase reminder may work better with a clear prompt and little else.

Use accessible HTML and include a usable plain-text alternative where your sending system supports it. Check rendering in major clients before launch. A variant cannot fairly lose a content test because its button disappears in a common inbox or its key message is trapped in an image.

Deliverability is part of the experiment

Email content testing takes place within a permission-based sending programme, not in isolation from deliverability. A version that induces recipients to complain, ignore future messages or unsubscribe can reduce the value of a short-term uplift.

Use only people who have consented to receive the relevant type of message. Keep marketing and transactional traffic distinct, honour suppressions, and do not use an A/B test as a reason to reintroduce disengaged or previously unsubscribed contacts. The answer to an uncertain result is better test design, not a larger and less qualified audience.

For larger Gmail senders, Google requires SPF, DKIM and DMARC, alignment of the visible From domain with SPF or DKIM, and one-click unsubscribe for marketing and subscribed messages. Google also advises keeping reported spam rates below 0.10% and avoiding 0.30% or higher. Google’s sender guidelines set out the requirements and monitoring expectations. Yahoo’s sender guidance similarly calls for authentication, easy unsubscribe and low complaint rates for bulk senders. Yahoo’s sender best practices provides the current detail.

Those requirements do not mean every test needs a deliverability intervention. They do mean complaints, unsubscribes, hard bounces and delivery patterns belong in the test record. When a test is run during a period of reputation difficulty, inbox placement changes can alter measured engagement independently of content. In that situation, resolve or isolate the sending issue before drawing broad conclusions from clicks or conversions.

One-click unsubscribe is an email-header mechanism, distinct from the visible footer link. The relevant technical standard is RFC 8058. A visible unsubscribe and clear preference options remain good recipient experience, while the header supports mailbox-provider handling of marketing opt-outs.

Where a programme is recovering from deliverability problems, it can be prudent to limit tests to confirmed, recently engaged recipients and keep volume, cadence and sending infrastructure stable. Email Foundry’s work is relevant at that point: an authentication, audience and provider-level sending review can establish whether a content result is being obscured by a broader delivery problem. It should not be used to manufacture engagement or bypass mailbox-provider protections.

Read results with appropriate caution

When the test ends, review the primary metric first, then the guardrails and practical context. Ask:

• Was the difference large enough to matter commercially, not merely numerically?
• Did the result meet the statistical threshold or decision rule agreed before the send?
• Did complaint, unsubscribe or conversion-quality signals remain acceptable?
• Was there an operational event that could have distorted delivery or website conversion?
• Did the result hold across important audience groups, or is it confined to one segment?

A no-difference result is still useful. It may show that the tested distinction is not important enough to this audience to justify further creative effort. Record it, along with the audience, dates, variants, primary metric, sample size, evaluation window, result and decision. A simple testing log prevents teams from repeating the same inconclusive experiments and makes patterns visible over time.

Finally, validate important findings. If a new proposition appears to improve completed conversions, use it again in a later comparable campaign or lifecycle message. Repetition under slightly different circumstances is stronger evidence than one successful send.

What good email A/B testing looks like

Good testing is deliberately modest. It asks one clear question of a defined, permission-based audience; changes what is necessary to answer it; uses a measure connected to the real objective; and respects the signals recipients use to say no.

That approach improves more than campaign reporting. Over time, it helps teams understand which promises to make in the inbox, how much information people need, what actions feel appropriate and where relevance begins to turn into pressure. The result is a more useful email programme for subscribers and a more dependable source of commercial learning for the sender.

Frequently asked questions

What should I A/B test first in an email?

Start with the part of the message most likely to affect the decision you need to make. For a campaign that is being overlooked, test the subject line and preheader. For a campaign with opens but few meaningful visits, test the opening proposition, call to action or content hierarchy.

How many recipients do I need for an email A/B test?

There is no fixed number. Calculate it from your baseline conversion rate, the smallest improvement worth acting on, the number of variants and the confidence level required. Low-frequency outcomes such as purchases need more recipients than clicks.

Should open rate choose the winner?

Usually not for a content test. Opens happen before recipients see the body, and privacy features can make pixel-based opens unreliable. Use clicks, completed conversions or revenue where those measures fit the question, while monitoring complaints and unsubscribes as guardrails.

Can I test subject lines and email copy at the same time?

You can, but the result will compare two complete concepts rather than identify which change caused the outcome. If you need a clear learning about subject lines or copy, hold the other elements constant and test them separately.

Sources and further reading

• Email sender guidelines — Google
• Sender best practices — Yahoo Sender Hub
• Mail Privacy Protection & Privacy — Apple
• RFC 8058: Signaling One-Click Functionality for List Email Headers — RFC Editor / IETF
• About A/B Tests — Mailchimp