How to Test a Nurture Email: One Change, Real Outcome
How to test a nurture email honestly: change one thing, judge it on the outcome that matters, respect small samples, and branch on strong signals, not opens.
To test a nurture email, change one variable at a time and judge the result on the destination of the sequence, not on a proxy such as opens. Be honest about sample size too, because a result read off a few dozen sends is a reasoned guess and not an experiment.
Testing is the part of email work that looks most scientific and is most often done carelessly. A team swaps two subject lines, sees a difference in a number and declares a winner. This post is about doing it so that the answer means something, and about the related question of when a sequence should branch.
How to test a nurture email: change one thing
The rule is old and still ignored. If you change the subject line, the opening paragraph and the call to action together, a difference in results tells you that something worked and nothing about what. You have learned nothing you can reuse.
So pick a single variable. Write down what you are changing and what you expect it to do. Keep everything else identical. That discipline is slow, and it is also the only route to a conclusion you can trust. A string of small, clean answers builds understanding. One large muddled test builds a story.
The single-variable habit is easier when each email already has one job and one action, as set out in how to write a nurture email. An email with six links and four purposes is almost impossible to test, because there is no single thing for the change to act on.
Judge it on the destination, not the proxy
This is the heart of the matter. A test needs a measure, and the temptation is to choose the one that is easiest to read. Opens are the easiest of all. They are also the least trustworthy, because some mail applications open messages automatically on the reader's behalf and privacy features hide whether a human ever looked.
The standard is the destination you named for the sequence: a booked call, a started trial, a reply, a deal. A subject line that wins on opens and loses on booked calls has lost. It attracted attention that did not turn into anything, and you would have been better off with the quieter version.
| Measure used to judge a test | Verdict | Reason |
|---|---|---|
| Opens | Weak | Partly automated and partly hidden |
| Clicks on the one action | Useful, with care | A readable sign of what the reader wanted, when the email has one action |
| Replies | Strong | Still mean what they appear to mean |
| The destination (calls, trials, deals) | The one that counts | It is what the sequence exists to produce |
If a test cannot be judged on the destination yet, because too few people have reached it, say so. Do not fall back on opens simply because they arrive faster. How to define the destination as a rate over a fixed window, and how to credit deals fairly, is explained in how to measure email nurture by deals.
Be honest about sample size
A result read off a few dozen sends is a reasoned guess, not an experiment. Small numbers wobble. One person replying or not replying can swing a rate of a few dozen, and a pattern you see may simply be luck.
That does not mean small lists should not test. It means they should describe their findings accurately. "This version seemed to do better, and we will keep watching" is an honest sentence. "We proved version B wins" is not. Most early lists are too small for a clean answer, and saying so protects you from locking in a change that was only noise.
- Treat a thin result as a lead to follow, not a rule.
- Say how many sends are behind a result whenever you report it.
- Do not present a guess as a finding, even to yourself.
What a clean test looks like on paper
Before running anything, fill in four lines. They take a few minutes and they catch most of the mistakes described above.
| Line | What to write |
|---|---|
| The one change | The single variable that differs between the two versions |
| The measure | The destination, or the nearest honest step towards it |
| The win rule | What result will count as a win, written before you look |
| The caveat | How many sends are behind it and what that means for confidence |
If any line is hard to fill in, the test is not ready. That is useful information, and it costs far less to learn it on paper than after you have changed a live sequence.
Decide before you look
As with attribution, the safest habit is to decide the rule for judging the test before the results arrive. Write down which measure decides it and what would count as a win. Otherwise you will find, as people reliably do, that whichever number favours your preferred version turns out to be the important one.
The discipline is boring and protective. It means that when the result arrives, it is the result, not an argument.
Branching is a kind of test, and it has its own rule
A sequence can adapt as well as be tested, and a little branching is worth having. The rule is to branch on strong signals only, and never on opens.
| What the reader did | Reasonable response |
|---|---|
| Clicked the pricing link | Move them towards a conversation, or out of the sequence and into a salesperson's hands |
| Clicked nothing in four emails | Try a different approach, or close the sequence gracefully |
| Replied | Release them so a person can answer |
| Opened an email | Do nothing different, because the signal is noise |
Branch on clicks, replies, tags and actions. A sequence that branches on opens is making decisions on noise, and the more elaborate the branching, the more decisions rest on a signal that may not describe a human at all.
Keep the shape simple
The temptation is to build a branching flowchart with dozens of paths. Resist it. A short, linear sequence with a few well-chosen exits outperforms an elaborate one that nobody can maintain, and the elaborate one is usually wrong in ways nobody notices. Every extra branch is another thing to test and another thing to keep true. The build principles behind this are in how to build an email nurture sequence.
Test by subtraction as well as variation
Not every test compares two versions. Sometimes the most valuable experiment is asking what happens if an email is removed. Read the sequence step by step: for each email, look at how many received it, how many clicked its action and how many unsubscribed. An email that sheds readers is telling you something about its content or timing. An email that produces nothing is spending attention and earning nothing.
Most sequences improve more from deleting their weakest email than from adding a new one. That makes subtraction one of the cheapest and most reliable tests you have, and it needs no clever design. It needs only the willingness to cut.
Where the list allows, a related test is to compare people who received the sequence with people who did not. It is the most direct way to check whether the sequence added anything, though a small list will not give a clean result, and you should say that plainly.
Keep a record, and recheck what you learned
Write down each test, what changed, how it was judged and what you concluded. Without a record, the same idea gets tested twice and the same lesson gets forgotten. Put the findings into the monthly review so that they actually change the emails.
Remember also that findings age. The offer changes, the proof ages, the objection shifts and links break. A conclusion from last year may not describe your readers now, so it is worth rereading every email as the recipient would, on a schedule. If a test reveals that the sequence works, the next question is how its leads reach a person, which is the subject of how to hand a nurtured lead to sales.
A testing checklist
- The test changes one thing.
- It is judged on the destination, not on opens.
- The rule for deciding was written before the results arrived.
- I have been honest about how many sends are behind the result.
- Any branch depends on a click, reply, tag or action, never on an open.
- I know what happens to a reader who clicks the main action.
Frequently asked questions
What should I test first?
Whatever single thing you believe is holding back the destination. Change only that, and keep everything else the same.
Why not judge a subject line test on opens?
Because opens are unreliable, and because a version that wins on opens can still lose on the outcome that matters, such as booked calls.
Is a small list too small to test?
No, but treat the result as a reasoned guess and say so. A few dozen sends cannot support a firm conclusion.
Can I branch on whether someone opened?
No. Branch on clicks, replies, tags and actions. Branching on opens means deciding on noise.
The full method, with the checklists, is in The Email Nurture Playbook, a free 34-page playbook.