Open Rate Vs Click Rate For Choosing An A/B Winner

Choosing an email A/B-test winner sounds simple: send version A to one group, version B to another, then keep the version with the better number. The difficult part is deciding which number deserves to decide the outcome.

Open rate and click rate answer different questions. An open rate estimates whether recipients reached and displayed the email. A click rate shows whether delivered recipients took an observable action inside it. Neither metric is universally right, and both can be misleading if they are disconnected from the change being tested.

The practical rule is this: choose the winner metric before you send, based on the decision the test is meant to inform. If you are changing the subject line, sender name or preheader, opens may be a useful directional measure. If you are changing the offer, message, creative, call to action or email layout, click rate is usually the more meaningful choice. Where dependable conversion or revenue data is available, that outcome should normally take precedence over either.

Start with what each metric actually measures

Most email platforms calculate open rate as unique opens divided by successfully delivered messages. Tracking normally relies on a tiny invisible image, often called a tracking pixel, being requested when the message is displayed. Click rate is commonly the proportion of delivered messages that generated at least one tracked click. Check the definitions in your own platform: some reports distinguish unique recipients who clicked from total clicks, and some label click-through rate differently. Mailchimp, for example, defines both its email open rate and click rate against delivered emails. Its metric definitions are a useful example of the distinction.

Metric	Question it is closest to answering	Most appropriate for	Main limitation
Open rate	Did the recipient’s mail client display the message?	Subject line, preheader, From name and, with care, send-time tests	Pixel-based tracking is incomplete and can be artificially triggered by privacy or security systems
Click rate	Did a delivered recipient click a tracked link?	Offer, email copy, creative, CTA, link placement and content-layout tests	A click is interest, not necessarily a purchase, lead, booking or other business result
Click-to-open rate	Among recorded opens, how many recorded a click?	Diagnosing whether the email body converts attention into action	Its denominator inherits the weaknesses of open tracking, so it is rarely the best primary winner metric
Conversion or revenue per delivered email	Did the campaign create the desired downstream outcome?	Commercial, lead-generation or product-activation campaigns with reliable attribution	Needs sufficient volume, sound tracking and enough time for recipients to act

These measures describe a journey rather than competing versions of the same thing. A recipient must first receive the email, notice it in the inbox and open it before they can click. They may then click but abandon the landing page, or buy without clicking because they return later through another channel. Treating any one stage as the whole outcome creates avoidable false winners.

Why open rate is no longer a clean winner metric

Open tracking was always an estimate. It misses people who read an email with images blocked, while counting people who merely cause the tracking image to load. It is now less reliable as a measure of deliberate human attention because some mail clients and security tools fetch remote images or inspect links automatically.

Apple’s Mail Privacy Protection is the best-known example. When it is enabled, Apple says remote content is privately downloaded in the background rather than when a recipient views a message, and senders cannot see whether the message was opened. Apple’s explanation of Mail Privacy Protection makes clear why a recorded open cannot always be read as an intentional read.

This does not mean open data is useless. It remains a helpful directional signal when comparing two versions sent at the same time to genuinely random, similar groups. If version B records a materially better open rate and subsequent clicks and conversions do not deteriorate, it may well have improved inbox-entry appeal. The point is that the result is less precise than the label suggests, particularly where a large share of the audience uses Apple Mail.

For that reason, avoid letting a small open-rate difference automatically decide a high-value campaign. A result such as 31.8% versus 32.2% may reflect normal variation, privacy-driven image loading or a difference in the mix of mailbox apps, rather than a real preference. Look at the number of delivered messages, the scale of the gap, the uncertainty around it and the downstream behaviour before declaring a lesson learned.

When open rate is the right primary metric

Use open rate as the primary metric when the variant changes the reason somebody would choose to enter the email, rather than what they encounter inside it. Typical examples include:

• a subject-line test, such as an explicit offer versus a curiosity-led message;
• a preheader test where the body is otherwise identical;
• a recognisable brand sender name versus an individual sender name;
• a send-time experiment intended to understand when the audience is more likely to notice the message.

Even then, treat click and conversion performance as guardrails. Imagine that “Last chance: 50% off” wins on opens, but the more urgent wording produces fewer clicks, more unsubscribes or more complaints. It may have attracted attention from people who then felt misled, pressured or disappointed by the email. That is not a durable win.

A better decision statement would be: “Choose the subject line with the strongest qualified inbox-entry performance, provided it does not reduce click rate or worsen negative signals beyond an agreed limit.” The test is still open-led, but it does not reward a subject line that merely overpromises.

This is also why subject-line testing should not change the email body at the same time. If version A has a different subject line and a stronger offer inside the email, an uplift in clicks cannot tell you which change caused it. Keep the body, audience, sending identity, links and send time constant where possible.

When click rate should choose the winner

Click rate should normally decide when the test changes the email itself: the lead message, offer framing, product selection, length of copy, imagery, CTA wording, CTA position, template or number of links. The recipient must encounter those elements after opening, so open rate cannot assess whether they worked.

Email platforms make the same practical distinction. Mailchimp advises using click rate rather than open rate when testing content, because opens occur before a subscriber sees the content. Its A/B-test guidance is a concise expression of the principle.

Consider a retailer testing two versions of a campaign:

• Version A: a grid of eight products and three equally prominent calls to action.
• Version B: one seasonal collection, a short benefit-led introduction and one primary button.

The same subject line gives both versions a fair chance to be opened. The useful question is not which version generated more pixel events, but which persuaded more delivered recipients to visit the collection. Click rate is therefore the sensible primary measure. Review the distribution of clicks too: if B’s main CTA performs well but its unsubscribe link or preference-centre link accounts for an unusual share of activity, the headline click rate needs interpretation.

Unique click rate is generally more useful than total clicks for selecting a winner. One person clicking four product links is valuable behavioural detail, but it should not count as four people responding to the campaign. Where your platform only reports total clicks prominently, find out whether unique-recipient reporting is available before automating a decision.

Click-to-open rate is a diagnostic, not usually the prize

Click-to-open rate, sometimes called CTOR, divides unique clickers by unique openers. It can help answer a narrow creative question: once the platform recorded an open, which version was more effective at turning that attention into a click?

It is useful for diagnosis after the test. If a campaign has a healthy open rate but a weak click-to-open rate, the mismatch may sit in the offer, creative, message hierarchy or CTA. If it has a weak open rate but a strong click-to-open rate, the email body may be compelling to the people who reach it, while the inbox-facing elements need work.

It is a poor default winner metric for two reasons. First, it ignores reach: an email can appear successful among openers while producing fewer clicks per delivered recipient. Secondly, the denominator is affected by the same open-tracking distortions discussed above. A version should not win simply because it reached fewer people but happened to generate a flattering ratio among a noisy subset.

For commercial emails, use the closest reliable business outcome

When the aim is a sale, booking, trial start, donation or qualified lead, the ideal winner metric is the outcome itself, measured fairly across both groups. Revenue per delivered email is often more useful than total revenue because it accounts for unequal delivery volumes. For lead generation, a better measure may be qualified leads per delivered email rather than form completions if many submissions are low quality.

There are qualifications. Conversion data can arrive slowly, attribution can be incomplete, and a small test may not generate enough outcomes for a defensible decision. In those cases, clicks can serve as the short-term primary signal, with conversion reviewed later. Do not retrofit the success criterion after seeing the data because one version happened to win on the metric you prefer today.

It is also worth separating a campaign’s immediate purpose from its broader commercial value. A product-launch email may be designed to drive a product-page visit, so click rate is appropriate. A renewal reminder may seek completed renewals, so completed renewal is better. A content newsletter may seek sustained reader interest, for which repeat clicks, future engagement and subscription retention may matter more than a single email’s click rate.

How to set up a decision that survives scrutiny

A robust test is less about finding an impressive percentage than controlling the comparison. Before launch, document the variable, audience, primary metric, guardrail metrics, sample split and decision time.

Test one meaningful change

Changing a subject line, hero image, offer and CTA together may help identify a better whole email, but it does not explain why it was better. That is acceptable when the only decision is which complete version to deploy. It is not a learning test. For reusable insight, isolate one variable or make clear that the test compares packages rather than individual elements.

Randomise comparable recipients

Versions should be assigned randomly within a relevant audience. Do not send A to last month’s active purchasers and B to a broad, ageing list, then attribute the difference to copy. If the campaign serves materially different groups, such as new subscribers and long-standing customers, either stratify the test or analyse those groups separately.

Exclude recipients who should not receive promotional email in the first place: unsubscribed contacts, hard bounces, known complaints, people under a frequency cap and any segment suppressed for a valid consent or customer-service reason. A/B testing is not a justification for sending a questionable audience two variants of the same message.

Use a realistic observation window

Opens, clicks and purchases do not all mature at the same speed. A flash sale may justify a shorter click window than a considered B2B purchase or an annual subscription. Decide the window from normal recipient behaviour, rather than choosing the earliest point at which one version pulls ahead. Some platforms offer automatic winners after a set period, but automation is only as sensible as the metric, sample and wait time behind it.

Do not confuse a numerical lead with a reliable difference

Small samples create volatile rates. If one version is clicked by 12 of 500 delivered recipients and another by 15, the apparent gap may be chance. Use your platform’s statistical-confidence calculation if it explains its method clearly, or use a suitable two-proportion significance test or experiment-analysis tool. More importantly, decide what size of improvement would be worth acting on. A statistically detectable uplift that adds negligible business value may not justify a permanent creative change.

Review negative signals alongside the winner metric

Track unsubscribes, spam complaints, bounces and, where available, placement or reputation signals. A test that boosts short-term clicks by using misleading urgency or increasing message pressure can damage permission and future performance. Google’s sender guidance requires bulk senders to keep unwanted mail low and make unsubscribing easy; its Postmaster Tools provide daily spam-rate data for eligible senders. Google’s sender-guidelines FAQ explains those expectations and the role of Postmaster Tools.

Yahoo likewise advises senders to use its complaint feedback loop to suppress recipients who report mail as spam, and emphasises relevance, opt-in and prompt handling of unsubscribes. Yahoo’s Complaint Feedback Loop documentation describes how these reports can support list hygiene. These are not just compliance tasks: excluding disengaged or complaining recipients helps ensure that a test measures audience preference rather than avoidable deliverability damage.

A practical hierarchy for choosing the winner

Use this hierarchy when setting up an A/B test:

1. If reliable downstream outcomes are available, use them. Choose conversion, qualified lead rate or revenue per delivered email when that is the campaign’s actual purpose.
2. If the body of the email is changing, use unique click rate. It is the clearest immediate measure of whether the changed content motivated action.
3. If only inbox-facing elements are changing, consider open rate. Use it as a directional primary metric, but inspect clicks, conversions and negative signals before rolling the result out.
4. Use click-to-open rate to interpret performance, not to overrule reach.

This hierarchy is deliberately more useful than a blanket “clicks beat opens” rule. A content test cannot be won by opens, but a subject-line test cannot be fully understood from clicks alone. The key is matching the measurement to the mechanism being changed.

Where deliverability fits into the decision

An email test assumes both variants have a comparable opportunity to reach the inbox. That assumption can fail when the sending programme has unresolved authentication, reputation, volume-pattern or audience-quality problems. A sudden dip in opens and clicks may be a placement issue, not proof that the new copy is weaker.

Before relying heavily on campaign tests, make sure sending domains are properly authenticated, suppression processes work, complaint feedback is acted on and promotional volume is appropriate for the engaged audience. This is particularly important for high-volume programmes, where mailbox-provider requirements and reputation signals can affect the visibility of both variants. Email Foundry’s role, where needed, is to audit and stabilise those delivery conditions so that performance tests are less likely to be measuring infrastructure or audience problems instead of creative choices.

Once that foundation is in place, open and click data become far more useful. Pick the metric that reflects the change, make the decision rule explicit before launch, and retain the test record. Over time, that creates a body of evidence about what your own permission-based audience values, which is considerably more useful than chasing a generic benchmark.

Frequently asked questions

Should I always choose click rate over open rate for email A/B tests?

No. Use click rate when the email content, offer, layout or CTA changes. Open rate can be appropriate when testing subject lines, preheaders or sender names, although it should be interpreted cautiously because open tracking is affected by privacy features and automated image loading.

What is a good sample size for an email A/B test?

There is no universal minimum. It depends on your baseline rate, the smallest improvement worth detecting and the confidence level you need. A low click-rate campaign usually needs a larger sample than an open-rate test. Avoid deciding from a handful of clicks simply because one version has a higher percentage.

Can a subject line win on opens but lose on clicks?

Yes. A subject line can generate curiosity or urgency that increases opens without attracting the right readers or accurately setting expectations. Keep clicks, conversions, unsubscribes and complaints as guardrails when using opens to select a subject-line winner.

Is click-to-open rate better than click rate?

Usually not as the main winner metric. Click-to-open rate is useful for diagnosing whether the email body motivated recorded openers, but click rate measures action across all delivered messages and is less likely to reward a version that reached fewer recipients.

Sources and further reading

• Create an A/B Test — Mailchimp
• All the Custom Report Metrics — Mailchimp
• Protect email privacy in Mail on Mac — Apple Support
• Email sender guidelines FAQ — Google
• Complaint Feedback Loop — Yahoo Sender Hub