How to Measure a Newsletter by Replies, Not Opens
How to measure a newsletter by replies: count replies, conversations, deals and revenue in order, and treat opens and clicks as hygiene signals, not results.
To measure a newsletter by replies, count what readers do in order: replies, the conversations they become, the deals those open and the revenue that follows. Opens and clicks still have a use, but as hygiene, not as the result.
Opens tell you that an email arrived. Replies tell you that a person is ready to talk. Only one of those is a business outcome. Yet most tools put the open rate at the centre of the screen, and many founders judge their newsletter by it, watch it for a few weeks and quit when it refuses to move. A better scoreboard is built on what readers do.
How to measure a newsletter by replies and what follows them
The playbook sets out four measures, in an order that follows the path from reader to revenue. Each one is a count you can make by hand, with no special software, if you keep the right records.
| Step | What you count | The question it answers |
|---|---|---|
| Replies | How many readers answered, and how many of those were about a real problem | Is the issue touching something real? |
| Conversations | How many replies became a call or a meeting | Are replies turning into talks? |
| Deals | How many conversations entered your pipeline, and how many closed | Are talks becoming business? |
| Revenue | What those deals were worth, and how much of your business had the newsletter in its history | What is the newsletter worth? |
The order matters. Replies come first because they are the earliest sign that an issue worked, and they are within your control. Revenue comes last because it takes time and depends on much more than the newsletter. Reading the four together shows where the path narrows.
Why opens mislead
The playbook calls the open rate the least reliable number on the dashboard, and gives the reason. Modern mail clients and privacy features can register an open that no human performed, or hide one that did. A number built on that cannot carry decisions.
It is also the number most likely to make you quit or optimise the wrong thing. A falling open rate in a small list can reflect nothing but the way mail apps behave, while a founder reads it as rejection. The alternative is not to ignore opens but to put them in their place. A sudden drop in opens is an early warning about deliverability. A click tells you which link interested people. Both are useful, as signals about health and interest, and neither is the result.
What counts as a reply that matters
Not every reply is equal, and the count should reflect that. The playbook says that many replies are thanks, some are questions and a few describe a problem from someone who is, without saying so, looking for help.
- Thanks. Pleasant and a sign the issue landed. Answer it, and note it, but it is not a conversation yet.
- Questions. A sign of engagement and a source of future issues. Answer them, since the question might be one that other readers share.
- A described problem. The reply the newsletter exists to produce. Record it and respond to the substance.
Keeping these separate is why the playbook's first measure has two parts: how many readers answered and how many of those replies were about a real problem. A newsletter with modest opens and a steady trickle of serious replies is working. One with impressive opens and no conversations is not.
Record the conversations where your pipeline lives
Counting only works if there is something to count. A reply that becomes a call and then a proposal is the newsletter working, but if that history lives only in your inbox, nobody will ever know it happened. The playbook asks you to log the reader, the conversation and any deal that follows where you track the rest of your pipeline, with the newsletter recorded as where it started.
It is unglamorous, and it is the only way the newsletter's real return becomes visible. The playbook adds a warning: an invisible channel is the first one cut. That is the reason this record is on the per-issue checklist as well as the monthly one: every reply that described a real problem is recorded, and any reply that turned into a call or a deal is recorded with the newsletter as its starting point.
Ask, because attribution is always incomplete
Some influence will never be traceable: the reader who clicked nothing for a year and then typed your name into a search bar. You can close the gap the simplest way, by asking new customers how they first heard of you and recording the answer.
The pattern across those answers is often the clearest evidence of what the newsletter is doing. It also fits the deeper point in the growth advice: the source must stay attached to the person. How subscribers are labelled at the door is covered in how to grow a newsletter list from your own site, and the record you build there is what lets you trace a deal back to its first touch later.
A monthly review that fits an hour
The playbook sets out a monthly review that runs outcomes first. It can be done with a spreadsheet and a calm half hour.
- Count replies this month, and how many were about a real problem.
- Count the conversations, calls and meetings those replies became.
- Check which deals in the pipeline started with, or were touched by, the newsletter.
- Ask new customers how they first heard of you, and record the answers.
- Note which issues produced the most replies and look for a pattern.
- Choose one thing to change next month, and only one.
The final step guards against overreaction. The checklist adds a line to keep in view: do not let a quiet month talk you into changing the cadence or stopping. A quiet month is a data point, not a verdict, which is another reason to hold the schedule described in how to set a newsletter cadence.
Using the replies to improve the issues
The reply channel is also a research panel. Ask a direct question and the replies tell you, in customers' own vocabulary, what they are struggling with. That vocabulary goes straight into your positioning, your topic list and your product decisions.
So the measure has a second use beyond scoring. When you note which issues produced the most replies, you are finding what your readers care about. Plan more of it. This is a cheaper form of learning than most surveys, since the reader volunteered the answer in their own words.
It starts with the sign-off. The reason to end every issue with a direct, easy question is that the question produces the replies you will count. The shape of that last line is explained in how to structure a newsletter issue.
Keeping the hygiene signals in view
Do not discard the dashboard. Three signals are worth a regular glance, because they warn you about the health of the list rather than the success of the business.
| Signal | What it warns you about |
|---|---|
| A sudden drop in opens | A possible deliverability problem |
| Bounces and spam complaints rising | The list or the sending is in trouble |
| Clicks on particular links | Which topics interest people |
Treat these as a smoke alarm. They tell you when to look, but they do not tell you whether the newsletter is paying.
Reading a month of numbers
Numbers only help if you know what to do with them. The four measures work as a funnel, and the point at which the count falls away tells you where to look.
| If this is low | Look at |
|---|---|
| Replies | The closing question, and whether the idea touches a real problem |
| Replies are fine but conversations are few | How promptly and personally you answer, and whether you suggest a call |
| Conversations are fine but deals are few | Whether the readers who reply are the people you can serve |
| Deals exist but revenue is unclear | Whether the newsletter is recorded as the starting point |
This turns the scoreboard into a diagnosis. Instead of asking whether the newsletter works, you ask which link in the chain is weak, which is a question you can act on.
Patience in the numbers
A newsletter is slow at the start, and the early issues go to a list small enough to feel pointless. The measures will be small. That is not a reason to change the plan. The playbook's monthly checklist is explicit: do not let a quiet month talk you into changing the cadence or stopping.
Look for direction rather than size. Are replies slowly more serious? Are more of them becoming calls? A small number moving the right way is a healthier sign than a large number that never turns into a conversation.
Frequently asked questions
What if I get very few replies?
Check that each issue ends with a direct, easy question and that the idea touches a real problem for your named reader. A small list will produce few replies, and a handful of serious ones is a good sign.
Should I ignore the open rate completely?
No. Use it as an early warning about deliverability, but do not judge the newsletter by it. It can register opens no human performed.
How do I count a newsletter's revenue if people do not click?
Record the conversations and deals that began with a reply, and ask new customers how they first heard of you. Some influence is untraceable, which is why asking matters.
Do I need special tools to measure this?
No. The counts can be made by hand if the conversations are recorded in one place. The discipline is in the record, not the software.
The full method, with the checklists, is in The Founder's Newsletter Playbook, a free 32-page playbook.