Plan a small experiment: hypothesis, measure and decision rule

A useful small experiment starts with a consequential assumption and ends with a bounded decision. Write what you expect to observe, with whom and under which conditions; choose a method that can expose a contrary result; and agree in advance what different findings would justify. Record possible harms and when the team would pause.

“Launch a pilot and see what happens” leaves too much room for interpreting any outcome as success. A defined test makes the intended learning, limits and next commitment visible. It can support progress without pretending to validate an entire service or business.

Consider a fictional garden centre exploring a refillable bird-feed container. It wants to learn whether customers can return an empty container and recover their deposit through the proposed counter process. Before buying a larger dispensing installation, the team plans a limited live trial. The participants, quantities, results and thresholds below are invented teaching examples.

Choose the assumption that affects the next commitment

Begin with assumption mapping, not a preferred test format. The garden centre’s idea depends on demand for refills, timely container return, a workable refund process, acceptable staff effort and a viable supply arrangement. One trial cannot answer all of those equally well.

The next commitment is a larger operational trial. A failed return-and-refund process would make that commitment difficult to justify. The team therefore focuses on that transaction before testing signage, adding a membership scheme or promoting the service more widely.

Describe how the assumption connects to the proposition. A container that is economical only after many reuse cycles requires evidence about actual reuse, not merely an enthusiastic answer to “Would you choose less packaging?” A Business Model Canvas can reveal which cost, channel and relationship assumptions depend on the behaviour.

Keep the original business question in view. If the real question is whether the deposit increases returns relative to another arrangement, the team needs a comparison design. If it is whether the counter process is feasible at all, a small trial may be an appropriate first step. Our research-design guide explains why these questions require different evidence.

Write a hypothesis that could be contradicted

Replace “customers like reusable containers” with an observable statement. The team proposes: “Among 20 trial customers who expect to empty a small container within 28 days, at least 14 will return it within that period and complete the stated refund process without staff prompting beyond the agreed standard instruction.”

Strategyzer’s hypothesis guidance emphasises statements that are testable, precise and discrete. Here, the statement specifies the participants, behaviour, time and assistance boundary. It does not bundle return behaviour with profitability or environmental benefit.

Define the terms before starting. “Return” means that the actual container reaches the agreed counter, not that a customer says they intend to return it. “Without extra prompting” excludes an adviser explaining a missing step during the transaction. A routine greeting is not extra assistance; a staff member supplying the answer is.

The 14-of-20 threshold is a fictional operational judgment about whether the team can consider a next trial. It is not a research-backed target or a sample-size calculation. Twenty participants would not make a reliable population estimate simply because the prediction includes a percentage.

Write a plausible contrary result as well. If many customers return containers but need help understanding the refund, the problem may be the process rather than unwillingness to reuse. If few return during the period, the trial needs to examine eligibility, consumption timing and inconvenience before assigning a cause.

Decide whether the design can answer the question

Business teams often use “experiment” broadly for a prototype, rehearsal or pilot. Formal causal experiments require a defensible comparison and appropriate allocation and analysis. A single live process can reveal difficulties without establishing what caused a change in behaviour.

The Magenta Book’s analytical-methods annex describes randomised trials, sufficient statistical power and conditions such as avoiding effects on the control group. Randomisation improves the basis for comparison; it does not guarantee perfectly balanced small groups or remove missing-data and implementation problems.

Our garden-centre trial has no control group. It examines return-and-refund feasibility under one stated arrangement. The team therefore cannot claim that the deposit caused the returns or that returns would be lower without it.

Choose a prototype or a live MVP according to the question and consequences. A rehearsal with empty containers can expose a confusing step. A live offer can observe an actual return decision, but it creates real obligations to participants. The trial must deliver the refund and support it promises.

Write the test before collecting the observations

Strategyzer’s Test Card makes hypothesis, test, measurement and threshold explicit. The worksheet below is an original expanded record for this hypothetical case, including conditions and guardrails.

Field	Proposed trial entry	Why it matters
Decision	Whether to prepare a larger operational trial	Limits what a promising result authorises
Participants	20 consenting customers expecting to empty a small container within 28 days	Makes the opportunity to return plausible
Offer	One container size, one stated deposit and one return counter	Defines what is being tested
Observation	Timely returns, extra assistance, refund errors and staff time	Keeps different outcomes separate
Continue rule	At least 14 timely unassisted returns, with guardrails satisfied	States the local threshold in advance
Revise or investigate	Returns occur but assistance or timing undermines the rule	Distinguishes design problems from absent demand
Pause rule	Refund mismatch, unsafe container condition or unmanageable counter disruption	Gives staff authority to protect participants

Store the protocol with a version and date. If the team changes instructions midway, preserve the change and separate observations from each version. A living process is sensible; silently treating every version as the same test is not.

Name the decision owner and the person responsible for monitoring. Somebody must be available to resolve an exception while customers are present. An experiment record is incomplete if it describes observation carefully but leaves service responsibility vague.

Recruit people who can expose the assumption

Choose participants because their situation is relevant. Include plausible differences in transport, availability and need for assistance rather than recruiting only enthusiastic staff acquaintances. Strategyzer’s guidance on strong experiments connects a clear hypothesis with defined participants and a suitable test artefact.

The trial’s requirement to empty the container within 28 days narrows its applicability. It makes an early operational check manageable, but excludes occasional users whose containers last much longer. Keep that exclusion in the findings. A later trial must examine whether the process works for them too.

Our sampling guide helps distinguish relevant qualitative recruitment from claims of representativeness. If the garden centre recruits volunteers at one counter, it should not describe their return rate as the behaviour of all local shoppers.

Explain what is being trialled, what participation entails, how records are used and how questions or withdrawal are handled. Collect only the details needed to link a container with its refund and observation period. Separate participation in research from access to the promised service.

Use interviews about a recent transaction to explore an unexpected observation. Ask what happened rather than suggesting an explanation such as “Was the deposit too small?” A plausible account still needs to be considered alongside the process record.

Keep conditions and measurement consistent enough to interpret

Record each issued container, the end of its 28-day observation period, whether it returned, when it returned and what assistance occurred. Keep the denominator at the issued eligible cohort. Do not calculate success only among people who came back.

If the counter changes staff or gets unusually busy, record that condition. If a reminder is added later, treat it as a changed intervention. Staff may naturally try to help a trial succeed; that assistance is relevant evidence about what ordinary delivery would require.

NIST’s guidance on randomised block designs explains addressing nuisance factors through blocking and randomisation. For a later comparison, factors such as day or counter conditions may matter. Comparing one option on a quiet weekday with another on a busy Saturday is not a fair allocation plan. Seek suitable design help when causal estimates matter.

For this feasibility trial, keep a simple event record and distinguish a missing record from an observed non-return. A participant whose observation period has not ended is pending. A complete period with no recorded return is a non-return under the protocol, subject to checking the reliability of the record.

Give the trial guardrails and a workable fallback

Track the main behaviour alongside refund accuracy, damaged containers, extra staff time and inconvenience to other customers. These measures are different from the hypothesis. A promising return count cannot compensate for mishandling deposits.

Rehearse exceptions before offering the trial. What happens if a token is missing, the counter cannot confirm the refund, or a container is damaged? Staff need a named escalation route and an agreed fallback that honours the customer commitment.

A short premortem can expose candidate failure pathways before the trial. It cannot certify the offer as safe or compliant; the responsible people still need to check its real requirements.

Make stopping practical. State who can pause new participation, how current participants will be supported and what conditions require investigation before restarting. Keep the scope small enough that staff can carry out these actions.

Check the result against the rule and the guardrails

Suppose the fictional completed cohort has 12 timely unassisted returns, three timely returns needing extra help, two late returns and three non-returns. These mutually exclusive categories total 20. The timely-return count is 15, but the planned timely-unassisted criterion is met by only 12.

Reporting “75% success” would silently change the measure. The pre-agreed continuation threshold of 14 unassisted timely returns has not been met. It is also wrong to label the entire proposition a failure: the assistance records may identify a fixable refund instruction.

Suppose one returned lid is damaged. The team separates it, checks what this means for storage and reuse, and follows the pause rule if its condition is unsafe. The count is still recorded, but passing a numeric threshold would not overrule the guardrail.

IHI’s PDSA guidance connects planning, doing, studying the consequences and choosing modifications, while keeping a record for each test. The useful discipline is to compare the result with the prediction and preserve what changed. It does not make this garden-centre trial a healthcare intervention or a causal evaluation.

Fund the next question and preserve the limits

The fictional team decides to revise the refund instruction, rehearse it and then consider another bounded trial. It does not buy the larger installation on the strength of the 15-return headline. Use staged innovation funding to match expenditure to the unanswered question.

The pivot-or-persevere decision process helps distinguish refining an instruction, changing the collection arrangement and abandoning the proposition. A missed threshold alone does not identify which choice is justified.

Keep the experiment’s limitations attached to its conclusion: one garden centre, a selected cohort, one container size, a short period and no counterfactual comparison. Evidence about a feasible refund does not establish profitability, repeated reuse or total environmental effects. The strongest result is a clear account of what the trial changed in the team’s decision and what remains to be learned.