A/B Testing Email Subject Lines: A Practical Guide

A/B testing email subject lines sounds simple: send two versions, compare the results and use the better one. In practice, the useful part is not writing two lines. It is designing a fair comparison, choosing a measure that reflects the campaign’s purpose and resisting the temptation to draw big conclusions from a small, noisy result.

A subject line affects the decision to notice, trust and open an email. It cannot rescue an irrelevant send, an unrecognisable sender name or an offer that disappoints once opened. A sound test therefore helps you learn what a particular, permission-based audience responds to in a particular context. It does not uncover a universal formula for higher opens.

What a subject line A/B test can tell you

In a basic A/B test, a randomly selected part of one audience receives subject line A and another comparable part receives subject line B. The email body, sender identity, audience rules, delivery period and tracking should otherwise remain the same. Any meaningful difference in outcome is then more plausibly related to the subject line.

For a large campaign, the usual pattern is to send the alternatives to test groups first, wait for a pre-agreed period, then send the selected version to the remaining eligible subscribers. Email platforms differ in their settings, but this is a common workflow: for example, Mailchimp allows a subject-line test to be sent to a percentage of subscribed contacts, with a winner selected by open rate, click rate, revenue or manually before the remainder is sent. Mailchimp’s explanation of email A/B tests is a useful illustration of the mechanics.

For a smaller audience, a hold-out approach may not be worthwhile. Sending A to one random half and B to the other half can still produce a useful learning result, but there is no remaining audience to receive a ‘winner’. That is perfectly reasonable when the goal is to improve future campaigns rather than maximise the immediate send.

Start with one useful question

“Which subject line performs best?” is too broad. Begin with the reader decision you want to understand. A good test question has a clear contrast and a reason for existing.

For example:

• Does stating the practical benefit outperform announcing the content format?
• Does a product category in the subject line work better than a seasonal theme for recent purchasers?
• For subscribers who have attended a webinar, does naming the speaker improve qualified registrations compared with naming the topic?
• Does an explicit deadline prompt more purchases than a line focused on limited stock?

Those questions suggest an underlying hypothesis. For a webinar email, it might be: people who know the speaker will be more likely to open and register when their name appears in the subject line. Writing this down matters. It stops a team from declaring a winner after the fact for whichever result happens to look encouraging.

Keep the contrast focused. Compare:

• Benefit-led: “Make month-end reporting less manual”
• Format-led: “New guide to month-end reporting”

Do not compare a different offer, a different tone, a different sender name, an emoji and a different preheader all at once. If one version wins, you will not know why. Most platforms reflect this principle by treating subject line, From name, content and send time as separate test variables. Mailchimp’s test setup guidance, for example, recommends choosing a single variable type for an A/B test.

Choose the right success measure before sending

Open rate is the obvious measure for a subject-line test because the subject line is encountered before the email body. It can be directionally useful, especially when the alternatives are sent at the same time to random groups. It should not be treated as a precise record of human attention.

Open tracking typically depends on remote content such as a tracking pixel. Apple’s Mail Privacy Protection can privately download remote content in the background and prevents senders from seeing whether or when an email was opened. Apple’s Mail Privacy Protection guidance explains the limitation. Google also cautions that it does not track open rates and cannot verify third-party open-rate accuracy. Google’s sender guidelines make that point explicitly.

That does not mean subject-line testing is pointless. It means the decision rule should suit the campaign:

Campaign purpose	Primary measure	What to check alongside it
Editorial newsletter	Unique clicks or clicks to the lead article	Unsubscribes, complaints and longer-term reader activity
Product launch	Qualified product-page visits or purchases	Click rate, revenue per delivered email and refunds where relevant
Webinar or event	Completed registrations	Attendance quality, not merely registration volume
Content announcement	Unique clicks or meaningful on-site engagement	Open rate as a secondary, imperfect diagnostic

If the email content and call to action are identical, clicks and conversions give a stronger view of commercial or editorial value than opens alone. They are still not flawless: web analytics can be incomplete, recipients may convert later on another device, and small conversion counts can be volatile. The answer is not to chase a single perfect number; it is to define the main outcome in advance and read the other signals as safeguards.

For instance, a curiosity-led subject may create more reported opens yet attract fewer clicks, more unsubscribes or more complaints because the message did not meet the expectation it created. It should not be considered a meaningful winner.

Build a fair experiment

Use a genuine random split

Each subscriber should have a comparable chance of receiving either version. Avoid hand-picking one group of highly active readers for A and a broader group for B. Engagement level, customer status, country, device mix and inbox provider can all influence results independently of the subject line.

If the campaign must be segmented, run the same random allocation within each important segment. A retailer might randomise separately among recent purchasers, lapsed purchasers and newsletter-only subscribers. This avoids a result being driven by one group receiving a disproportionate share of a variation.

Do not send both alternatives to the same person. Apart from irritating the recipient, the second email is influenced by the first exposure and cannot be treated as independent evidence.

Keep everything else constant

For a subject-line test, hold constant the From name, From address, preheader, creative, offer, send time, audience eligibility and suppression rules. The preheader deserves particular attention: in many inboxes it is displayed beside or below the subject line and can strengthen, weaken or contradict it.

If you want to test the preheader, make that a separate experiment. If you want to learn whether a personal sender name is more trusted than the brand name, test that separately too. The distinction is more than methodological tidiness; it is what makes the learning reusable.

Use enough recipients to make a decision worth trusting

There is no responsible universal minimum list size. The amount of data needed depends on the baseline behaviour, the size of the difference you care about and the chosen metric. A test in which each variation produces thousands of clicks has a different level of certainty from one in which each produces a handful.

Before launch, decide what difference would actually change your action. A tiny apparent uplift is often not enough to justify treating a phrasing preference as settled. If the audience is small, send the test to everybody, regard the result as a signal rather than proof, and repeat the same underlying question across comparable future sends.

Do not keep checking the dashboard and stop the test at the moment one line pulls ahead. Early results can swing sharply. Set the test audience, winner measure and observation window in advance. A short sale with an expiring offer may need a short window; a B2B newsletter or considered purchase may accrue useful clicks over several days. Platform defaults are conveniences, not evidence that four hours, for example, is right for every programme.

Write variations that teach you something

The strongest tests are often modest. They compare two honest ways of framing the same value, rather than trying to make one version dramatically louder.

Benefit versus topic

A: “Reduce failed-payment churn this quarter”
B: “New guide: failed-payment recovery”

This tests whether the audience responds more to the business outcome or the resource itself.

Specificity versus broad relevance

A: “Three changes to your October inventory plan”
B: “Plan inventory with more confidence”

This tests whether a concrete promise appeals more than a broader, less committal benefit. It works only if the email genuinely contains the three changes promised.

Known entity versus topic

A: “How Priya Shah handles complex renewals”
B: “A better way to handle complex renewals”

This is useful only where the named person is recognisable and relevant to the segment receiving the message.

Urgency versus value

A: “Registration closes tomorrow”
B: “Join the session on practical retention analysis”

Both can be truthful, but they answer different motivations. If urgency wins only at the cost of increased complaints or unsubscribes, the programme may be training subscribers to expect pressure rather than value.

Personalisation can also be tested, but only when the data is accurate and the personal reference is genuinely useful. Inserting a first name is rarely a meaningful hypothesis by itself. Referencing a product someone owns, an event they attended or a preference they selected may be more relevant, provided the use of that data is expected and appropriate.

Do not mistake curiosity for quality

A subject line should accurately represent the sender and the message. Google’s guidance is clear that subject lines, headers and display names should not be misleading, including fake “Re:” or “Fwd:” prefixes where the message is neither a reply nor a forward. Google’s sender guidelines also advise against deceptive message content and sender information.

That is both a compliance and a relationship issue. “Your account needs attention” may lift attention in the inbox, but it is inappropriate for a routine promotion. “Final hours” should only be used when the deadline is real and understandable. A recipient who feels misled has an easy alternative to opening the next email: unsubscribe, ignore it or report it as spam.

There is no reliable list of forbidden “spam words” that turns a weak subject line into a deliverability risk by itself. Mailbox providers evaluate broader patterns: authentication, recipient feedback, sending practices and message quality among them. For subscribed promotional mail, Google advises senders to keep reported spam rates below 0.1% and avoid ever reaching 0.3%, while also requiring appropriate bulk senders to authenticate mail and support one-click unsubscribe. Google’s sender-guidelines FAQ sets out the current details.

That is why A/B-test reporting should include delivered rate, hard bounces, unsubscribes, spam complaints where available, clicks and conversions alongside the headline measure. The subject line that produces a slightly lower open rate but attracts more qualified clicks and fewer negative signals may be the better choice.

Read the result without overclaiming

Once the window closes, ask four questions:

1. Was one version ahead on the primary measure by enough to matter operationally?
2. Did supporting measures point in the same direction, or expose a trade-off?
3. Was the result concentrated in a particular segment, inbox-provider group or device audience?
4. What does the result suggest about the audience’s motivation, rather than merely its response to a string of words?

Record the campaign type, audience definition, send date and time, variants, preheader, From name, allocation, metric, window and result. Also record contextual factors such as a public holiday, product launch, price change or unusually active news cycle. Without this record, a subject-line ‘library’ soon becomes a collection of slogans detached from the conditions in which they worked.

Be wary of broad rules such as “questions always win” or “shorter always wins”. A question can be compelling because it exposes a familiar problem, not because it ends in a question mark. A short line may work because the audience already understands the offer; a longer line may work because specificity reduces uncertainty. Test the principle in relevant future campaigns before making it a house rule.

Subject-line testing during deliverability work

If inbox placement or complaint levels are already deteriorating, a subject-line experiment should not become a way to squeeze more activity from an overmailed or poorly targeted audience. First establish that the sending programme is permission-based, the audience is appropriately suppressed and the mail is authenticated and correctly categorised. Testing a more enticing line against an unengaged or unsuitable cohort can create a misleading short-term lift while worsening recipient feedback.

In that situation, the useful question may be narrower: which accurate framing earns engagement from recent, likely-human engagers without increasing complaints? This is where deliverability monitoring and contact-pressure controls matter alongside creative testing. Email Foundry’s audit and managed recovery work can be relevant when a sender needs to separate an email-content question from an underlying audience, authentication, infrastructure or provider-reputation problem.

The aim is not to find the most aggressive subject line. It is to make a clear promise to people who asked to hear from you, then keep that promise in the email itself. Well-run A/B tests make that promise more relevant over time.

Frequently asked questions

Should subject line A/B tests use open rate or click rate?

Use the measure that best matches the campaign’s purpose. Open rate can be a directional indicator for a subject-line test, but it is affected by privacy protections and image loading. For campaigns intended to drive action, unique clicks, registrations, purchases or other meaningful outcomes are usually better primary measures. Review unsubscribes and complaints as guardrails.

How many subject lines should I test at once?

Two is usually enough for a clear A/B test. A third variation can be useful with a sufficiently large audience, but every additional version divides the available data and makes results harder to interpret. Test one underlying idea at a time.

Can I change the preheader when testing subject lines?

You can, but then you are testing the inbox presentation as a combination rather than the subject line alone. Keep the preheader unchanged if the aim is to isolate the subject line. Test alternative preheaders separately when that is the question you need answered.

How long should I wait before choosing a subject line winner?

Choose the observation window before sending, based on how quickly the audience normally acts and how long the offer remains valid. Avoid selecting a winner as soon as a dashboard changes. A fast retail promotion and a B2B newsletter may need very different windows.

Sources and further reading

• Email sender guidelines — Google
• Email sender guidelines FAQ — Google
• Use Mail Privacy Protection on iCloud.com — Apple
• About A/B tests — Mailchimp
• Create an A/B test — Mailchimp
• RFC 8058: Signaling one-click functionality for list email headers — IETF