Home/Blog/Scorecards/Find Dead Questions in a Scorecard Befor…
Scorecards

Find Dead Questions in a Scorecard Before They Cost You

How to find dead questions in a scorecard: read each question's answer spread, flag any with one dominant option, then cut or rewrite it to sharpen the bands.

Cover of the FIKR Space guide: Find Dead Questions in a Scorecard Before They Cost You

To find dead questions in a scorecard, read the answer distribution for each question and flag any where a single option holds eighty per cent or more of the answers. Such a question separates nobody, so it spends the respondent's attention and returns no information.

A scorecard exists to separate people. Quality in that sense is discrimination, and no tool computes it for you. You are given distributions and you have to read them by eye, so you need to know what a healthy one looks like and what a dead one looks like. This post is the question-level half of calibration. The pillar-level and band-level half is in how to calibrate a scorecard.

How to find dead questions in a scorecard: read the shape

For each question, look at how respondents spread across its options. Think of it as a small bar chart, one bar per option. The shape tells you whether the question is doing its job.

ShapeWhat it looks likeVerdict
DeadOne bar holds eighty per cent or moreSeparates nobody. Cut it or rewrite it
WorkingSpread across at least three options, with the extremes populatedThe question is doing its job
BimodalTwo groups, at opposite endsStrongly discriminating, so keep it

The target is spread across at least three options with the extremes populated. If hardly anybody picks the lowest option or hardly anybody picks the highest, the question has not found the edges of your population. If almost everyone picks the same middle option, the question is telling you nothing you did not already know.

Why a dead question costs more than it seems

A question that nearly everybody answers the same way is not harmless padding. It does three kinds of damage.

  • It spends attention. Every screen costs the respondent effort, and a screen that returns no information is effort wasted.
  • It lengthens the instrument. A longer instrument loses more people before they finish.
  • It adds noise. A question that does not separate people still contributes to every pillar score, so it blurs the bands built from them.

This is why cutting or rewriting a dead question is the highest-leverage edit a designer ever makes. It shortens the instrument, which raises completion, and it removes noise, which sharpens every band. Few other changes help on both counts at once.

Free guide, 39 pages: The Scorecard Playbook. Send me the guide.

Why questions die

Once you spot a dead question, the useful step is to diagnose the cause. A handful of patterns account for most of them.

CauseWhat it looks likeLikely fix
The question signals its own right answerNearly everyone picks the flattering optionReword so no option looks like the correct one
Options are not genuinely distinctRespondents cannot tell the options apart, so they pile onto oneRewrite the options to cover the real range
It asks about opinions or intentionsEveryone says what sounds sensibleAsk about behaviour and what exists
It is there for completenessThe topic matters but the answer never variesCut it
Rating scale left at its starting pointA pile of answers at the middleCheck for the midpoint effect below

The common thread is that every question should plausibly split the population. If you cannot imagine two reasonable people answering it differently, it should not be in the instrument.

The midpoint trap

A rating scale deserves a special look. A slider always has a position, so it typically starts at the midpoint and counts as answered before the respondent has touched it. That means a pile of answers in the middle can be real indifference, or it can be people who moved on without engaging. The two are indistinguishable in the data, because the unmoved answer scores the same as a deliberate middle one.

When a rating question looks dead, consider whether it is a weak question or an artefact of how the control behaves. Keep that in mind when you decide whether to cut it. The wider consequences of choosing question types are covered in how to weight scorecard questions.

Read each question against its own scale

Questions in the same instrument can sit on different ladders. A chart that assumes every question runs the same way can mislead badly. In one reported case a distribution chart assumed a scale of nought to one hundred and bucketed every answer from an instrument built on a short run of whole numbers into the bottom bucket. It showed a catastrophe that was not there.

So read each question against its own option points, using your own wording for the options. The shape you are judging is the shape across the options a respondent could actually pick, not across some assumed scale.

Mind the population you are reading

Two charts that appear to describe the same respondents can disagree legitimately. In-progress respondents usually count towards per-question statistics but not towards completed-only statistics such as the band distribution. Before you compare two charts, check which people each one includes. That habit prevents false alarms, and it prevents you from cutting a question because of a count that was never about the people you thought.

When to read, and how to act

You cannot judge discrimination on paper. A scorecard that looks plausible in a document needs real respondents before it can be fixed. The suggested rhythm is to read the distributions after the first fifty or so completions, and then monthly. Early on, the numbers are small and shapes are noisy, so treat the first read as a prompt to look harder, not as a verdict.

  1. List every question with a dominant option. Anything at eighty per cent or more goes on the list.
  2. Decide for each: cut or rewrite. Cut if it exists only for completeness. Rewrite if the topic matters and the wording is the problem.
  3. Check the effect on the pillar. A pillar must still have enough questions that do real work.
  4. Batch the edits and date them. Scores are written once and never recomputed, so each edit forks the instrument.
  5. Read the distributions again once enough new completions have arrived.

The batching step matters more than it sounds. If you change a question's options on a live instrument, you also change the scale for everyone who came before, and a trend drawn across the edit is meaningless. Question text can be changed safely, but option text cannot, because translations and in-flight respondents are tied to it.

FIKR Score, diagnostics that qualify: turn your audience questions into a quiz that finds buyers. See FIKR Score.

Look beyond the distribution

A dead question is one signal. Two other things deserve a glance when you review the instrument.

  • Per-question drop-off. Where does the questionnaire lose people? A question that many abandon at may be too hard, too intrusive or badly placed.
  • Question type and rhythm. Single-select questions move on by themselves, while multi-select, rating and text questions wait for a press. Changing a question's type changes the physical rhythm of the whole assessment.

The asking experience is a lever on completion in its own right, and it is discussed in how to design a scorecard email gate.

Three questions to ask of every flagged question

Once a question is on your list, resist the urge to delete it straight away. A short interview with the question usually settles the decision.

  1. What would I learn if people answered differently? If the honest reply is nothing, cut it. The question is there for completeness, not for use.
  2. Is the sameness real or is it my wording? If the audience truly behaves the same way, the question is dead for this audience. If the options make one answer look obviously right, the fault is the writing, and a rewrite may bring it back to life.
  3. Which pillar does it sit in? If the question is one of only a few in a pillar, cutting it may leave the pillar thin. In that case replace it with a harder question that forces separation, so the pillar still does its work.

What a healthy instrument looks like

You are not trying to make every distribution perfectly even. You are trying to make sure that each question could plausibly place one respondent above another. In a healthy instrument most questions show spread, a few are sharply split, and none holds the great majority on one option. When that is true, the pillar scores spread out, the bands fill and the advice can differ from person to person.

The reverse is a warning sign at the level above. If many questions are dead, the pillars they belong to will bunch up and the band distribution will pile into one tier. Question-level reading is where you find the cause. The consequences show up in the pillar spread and the bands, and that is where to go next once the dead questions are handled. If the bands themselves look wrong, the arithmetic of the lines is in how to set scorecard band boundaries.

Free guide, 39 pages: The Scorecard Playbook. Send me the guide.

A dead-question checklist

  • I have a distribution for every question.
  • I have flagged anything with eighty per cent or more on one option.
  • I know why each flagged question died, and I am prepared to cut it.
  • I have checked which population each chart describes.
  • I will batch and date my changes.
FIKR Score, diagnostics that qualify: turn your audience questions into a quiz that finds buyers. See FIKR Score.

Frequently asked questions

Is one dominant option always a problem?

It is a strong warning, especially at eighty per cent or more. Check that your respondents are not simply a uniform group, and consider whether the question flatters or signals an answer.

Should I cut or rewrite a dead question?

Cut it if it is there for completeness. Rewrite it if the topic matters and the wording or options are the cause.

How many responses do I need before judging?

A sensible first read comes after about fifty completions. Early shapes are rough, so keep reading monthly.

Will cutting questions hurt the scorecard?

Usually it helps. A shorter instrument raises completion, and removing noise sharpens the bands, provided each pillar keeps questions that genuinely separate people.

The full method, with the checklists, is in The Scorecard Playbook, a free 39-page playbook.

LG

Luis Goncalves

// Founder & CEO at FIKR Space

Three-time founder. Built and exited Evolution4All before this. Now building FIKR Space — the operating infrastructure underneath every innovation ecosystem (startups, accelerators, governments, investors). Lisbon-based, works global.

Claim your founding member spot.

Every founding member is hand-reviewed. Tell us who you are and how you want to co-create — we get back inside 48 hours.

Role
Job Title
Country
Cookies

We use cookies.

Essential ones keep the site working. The rest only run if you say so. No dark patterns, no pre-ticked boxes.Privacy policy

Co-create with us. It's about building it together.Co-Create With Us