Survey data is increasingly used as grounds for management decisions: programmes are adjusted, budgets reallocated and agencies assessed on the strength of it. Yet the margin of sampling error — the parameter that determines which differences between figures are worth discussing at all — is rarely mentioned in management documents.
What “±3.1%” actually means
The margin of error describes the interval within which the true population value lies at a given probability. If two regions return 47% and 50% with an error of ±3.1%, their confidence intervals overlap — there is no basis for claiming that one region scores higher.
The problem compounds when working with sub-samples. A national sample of 1,500 respondents gives acceptable precision for the country as a whole, but split across 17 regions each cell holds fewer than a hundred observations and the error rises to ±10% or more. Regional rankings built on such data largely reflect random variation.
Three questions to ask of any report
- What is the margin of error not for the sample overall, but for the subgroup the conclusion is drawn from?
- How many observations remain in the cell after all filters — region, age and status applied together?
- Has the methodology been preserved from the previous wave, or did the question wording, mode or sample structure change?
If the contractor cannot state the error for the subgroup you care about within a minute, that figure was never calculated — and a decision cannot rest on it.
The practical conclusion is simple: the requirement to state the margin of error for each analytical breakdown belongs in the terms of reference, not at the contractor's discretion. It does not complicate the work — it makes the result usable.