Survey design: write questions people can answer

A useful questionnaire asks people questions they can understand and answer, then produces evidence for a defined decision. Good wording is only part of the design. The population, recall period, answer options, question order, routing and pretest all affect what the responses mean.

Begin with the information you need rather than a list of agreeable statements. “I support the new system” does not tell you whether someone can find a record, has working access or completed the task yesterday. Attitude, expectation, reported behaviour and observed performance should remain distinct.

Consider a hypothetical maintenance organisation introducing a shared key register. The manager needs to decide which practical conditions to resolve before a pilot expands. The short questionnaire developed below is an original example; it is not a validated readiness scale or evidence from real employees.

Choose the decision and the measure

The maintenance team already has a training-completion list. It wants to know whether employees can use the register during ordinary shifts and where the process creates difficulty. Those questions extend beyond whether a course was attended.

Use change readiness to clarify what the proposed change requires. Then write a research design connecting the questionnaire to a decision. In this example, findings about access should lead to an access check; unclear task categories should lead to clarification. Neither should automatically trigger another motivational presentation.

The Government Analysis Function’s questionnaire guidance recommends gathering data requirements and considering respondent burden, clear language, inclusive design and testing. Translate each requirement into a question only when a respondent can supply the information. If the system already records access failures reliably, examine that evidence rather than asking everyone to recreate it from memory.

Specify the intended unit. Are you collecting one account per employee, one recent key issue or one shift? A questionnaire can include more than one, but the analysis must not treat them as interchangeable.

Replace agreement with a recent task

Start with what someone did in a defined period. “Do you use the register regularly?” gives respondents an undefined frequency and an undefined task. The night colleague who issues one key a week and the dispatcher who handles twenty a day could both answer yes.

An original alternative is: “During your last five completed shifts, did you need to issue a maintenance key?” The team would pretest whether five shifts is a period people can recall and whether “issue” has a shared meaning. Offer no, yes and a suitable cannot-recall route instead of forcing everyone into the task.

For those who answer yes, ask about the most recent case: “On that occasion, were you able to open the key register at the workstation you were using?” This measures a reported event. It does not establish that access works at every workstation or that the employee followed the full process correctly.

AAPOR’s survey best practices recommend one concept per question, understandable language and avoidance of wording that pushes an answer. They also warn about socially desirable responses and a tendency to agree. A positive slogan followed by an agreement scale can therefore be a poor diagnostic starting point.

When you need to know why a task was difficult, leave room for an explanation or follow up with customer-style interviews. A questionnaire containing every possible diagnostic conversation can become burdensome and still miss the relevant circumstance.

Ask one thing and make the terms concrete

“The new register is easy to use and helps us work efficiently” contains two judgements. An employee might find it understandable but slow, or fast once a specialist helps. One answer cannot tell the team which statement is being assessed.

Split the concepts and name the task. Instead of “Is support adequate?”, ask whether the person knew whom to contact during the recent issue and, separately, whether help was available when needed. Those answers can point to different remedies.

Avoid assumptions in the question. “Why did you ignore the register?” assumes the behaviour and assigns intention. Ask which route was used and what affected the choice. The guide to ADKAR distinguishes understanding, desire, knowledge and ability; use those distinctions to resist calling every difficulty an attitude problem.

Specify vocabulary respondents use. The organisation may call one screen the asset-location module while workers call it the key list. Test the term rather than deciding that the official name must be clear. Define a necessary technical term briefly in the question if local testing shows that respondents need it.

Avoid unnecessary absolutes such as always and never. An employee who encountered one exceptional outage may struggle to answer a broad statement about being able to access the register. A defined recent event is often easier to interpret.

Build answer options that fit the question

The scale should match the judgement. A difficulty question needs difficulty options; a frequency question needs frequency options; a question about whether something happened may need a yes/no route. Do not reuse an agreement scale simply because the survey platform supplies it.

For the recent register task, an original difficulty scale could use very difficult, somewhat difficult, neither easy nor difficult, somewhat easy and very easy. Whether a midpoint is useful depends on the question and the respondent’s possible experience. Keep “did not attempt the task” separate from neutral: it describes a different situation.

Use balanced labels and a consistent direction where it helps respondents. Avoid mixing “strongly agree” with “satisfied” in one question. For counts, use non-overlapping options and state the time period. Zero, one, two to four and five or more can be interpretable categories if the research need justifies that granularity.

Do not invent precision the respondent cannot provide. Asking how many minutes the task took during the past month can produce guesses. A system timestamp or direct observation might supply a more suitable measurement, depending on the decision.

An open response is valuable for unexpected circumstances, but plan how to analyse it. “Anything else?” should not become the sole route through which an essential shift’s problem can be discovered.

Draft a short route through the questionnaire

This table is a proposed design for the fictional maintenance pilot. It shows the information each question is meant to produce, rather than pretending to be a finished instrument.

Question purpose	Original draft question	Route or interpretation
Relevant experience	During your last five completed shifts, did you need to issue a maintenance key?	No or cannot recall: skip recent-task questions; retain these as separate answers
Working access	On the most recent occasion, could you open the register at your workstation?	Separate no from did not attempt
Task clarity	How easy or difficult was identifying the correct key record?	Ask only if the person attempted to identify a record; unavailable access is a separate finding
Support	On that occasion, did you need help from another person?	If yes, ask how help was obtained
Unexpected condition	What, if anything, made that issue difficult?	Optional explanation; examine before coding

A clear introduction states the purpose, expected time, voluntary arrangements and how the information will be used. Only promise confidentiality that the organisation can deliver. In a small team, role and shift information may identify someone even without a name.

Use stakeholder mapping to check whose task and language the draft reflects. An employee issuing keys, a contractor receiving one and a coordinator correcting records may need different routes. Do not require all three to answer a generic “system use” item as though their work were identical.

Check order, layout and access

Start with understandable questions relevant to the respondent. Put linked task questions together and delay sensitive background questions unless they are needed for routing. An opening paragraph celebrating the pilot may influence how people interpret later difficulty questions.

Check whether previous questions supply an answer to later ones. Asking a list of suspected faults before an overall judgement can make those faults especially salient. Order should serve the research purpose, and its possible influence should be considered during testing.

Keep the route usable on the actual devices and in any alternative mode. A table that works on a desktop may be difficult on a phone or with a screen reader. If a paper version is offered, make skip instructions understandable without relying on hidden software logic.

Provide an accessible completion route during a workable part of the shift. Selecting a suitable sample also means considering who can encounter and complete the invitation. A questionnaire sent only to personal email addresses may miss people who rarely use them at work.

The working conditions discussed in psychological safety matter when the topic includes admitting uncertainty or reporting a problem. Good wording cannot neutralise a manager who later uses responses to punish employees.

Pretest the whole route

A pretest investigates how people interpret a question, recall information and choose an answer. Proofreading with colleagues is useful but insufficient. People who know what the researcher intended may silently repair ambiguous wording.

CDC’s cognitive-interviewing guidance describes probes about comprehension, recall, judgement and response. Use these to learn what the question does for a participant, rather than asking only whether they like it.

In the hypothetical pretest, ask a worker what “issue a key” includes. They might include passing a key to another colleague after reception has recorded it. That interpretation differs from the manager’s intended transaction. Revise the definition or the question before combining those responses.

Include people with no recent task, different shifts and relevant access needs. Ask them to complete the entire route, including introduction, skips and submission. Note the version, problem and revision, then test the affected route again.

A small technical pilot can then check invitations, data capture, missing values and routing at more realistic scale. Cognitive and technical checks contribute different evidence. Neither, alone, establishes that a new questionnaire is a valid psychological measure.

Plan interpretation before sending invitations

Keep the questionnaire version and analysis plan with the invitation record. Define which responses form each denominator. Employees without a relevant task should not be counted as neutral about its difficulty, and missing responses should not be coded as no difficulty.

For formal readiness measurement, Shea and colleagues’ ORIC study illustrates the separate work of developing and examining a theory-based instrument. Its evidence does not transfer automatically to locally rewritten questions. Our key-register questions are task diagnostics, not ORIC or a new validated score.

Report the distributions relevant to the decision, including missing groups. If one essential shift lacks access, a positive organisation-wide average does not resolve that condition. Link reported barriers to practical checks instead of ranking employees by willingness.

After review, explain what happened because of the feedback. The maintenance manager might commission a workstation check and clarify record ownership. Those are proposed actions; no improvement is claimed. A survey earns trust through an accountable response as well as careful wording.

Evidence and limitations

Self-reports can be affected by memory, interpretation, social pressure and who chooses to respond. A clear question can still be answered inaccurately. A large response count cannot repair an inaccessible invitation or a category that combines different tasks.

The sources provide questionnaire practice guidance and an example of instrument research. They do not validate this short workflow or show that favourable answers predict adoption. The government design page is under review; its general design principles are used with explicit local testing.

Keep the instrument proportionate to its purpose. If the team needs an explanation, conduct a conversation. If it needs observed performance, arrange an appropriate task check. If it needs a population estimate, design recruitment and analysis accordingly. A questionnaire is useful when its answers have a clear and honest job.