What Does “Statistically Significant” Really Mean?

Few phrases sound more final than “statistically significant.” In everyday language, significant means important. In statistical practice, it has a narrower meaning. Confusing those meanings turns a calculation into a verdict it was never designed to deliver.

Start with the question a p-value asks

A p-value is calculated under a statistical model that includes a null hypothesis—often no difference or no association. Loosely stated, it describes how incompatible the observed data, or more extreme data, are with that model. A small value says the data would be relatively unusual under the specified assumptions.

It does not tell us the probability that the null hypothesis is true. It does not tell us the probability that the result happened “by chance.” It does not measure the size or practical importance of an effect. Those common translations reverse or enlarge what the calculation actually says.

Where the 0.05 threshold came in

Many fields developed a convention: call p < 0.05 statistically significant and p ≥ 0.05 not significant. The boundary can help standardize a decision when a rule genuinely must be set. Trouble begins when a continuous measure is treated as a cliff. Results with p = 0.049 and p = 0.051 are not meaningfully opposites.

The American Statistical Association has emphasized that scientific conclusions and decisions should not rest only on whether a p-value crosses a threshold. Context, design quality, measurement, evidence from other studies and the costs of errors all matter.

Significance is not effect size

With a very large sample, a tiny and practically irrelevant difference can produce a small p-value. With a small sample, a potentially important effect may not cross 0.05 because the estimate is imprecise. “Not significant” therefore does not mean “no effect.” It may mean the study could not distinguish among several possibilities.

Read the interval, not only the label

Suppose a study estimates an improvement of 3 units with an interval from 1 to 5. Another estimates 3 units with an interval from −4 to 10. The central estimates match, but the second leaves much more uncertainty. Reporting only whether each crosses a threshold hides that difference.

Intervals also help readers compare the evidence with a meaningful benchmark. If an effect smaller than 2 units would not matter in practice, an interval concentrated between 0 and 1 tells a different story from one spanning 0 to 12—even if both are described as “not significant.”

Why design still comes first

A precise analysis of biased data can produce a precise wrong answer. Statistical calculations do not repair unmeasured confounding, selective sampling, unreliable instruments, missing outcomes or a comparison that fails to answer the research question. Before reading the p-value, ask how observations were generated.

Randomization, blinding, prespecified outcomes, appropriate controls and transparent handling of missing data protect different parts of the study. Statistical significance is meaningful only inside that design.

The multiple-testing problem

If researchers test many outcomes, time points, subgroups and models, some p-values can fall below 0.05 through random variation even when no corresponding effect exists. Imagine testing 100 independent null relationships at a 0.05 threshold. Under simplified assumptions, about five could cross the line on average.

Researchers can address this by specifying primary questions in advance, adjusting for multiple comparisons, separating exploratory from confirmatory work and validating patterns in new data. Readers should ask how many analytical routes were available, not only which result was reported.

Selective reporting and the literature

Statistically significant findings have historically been easier to publish than null or ambiguous results. This can make the published literature look more decisive than the full research record. Trial registration, study registration, results reporting and access to protocols help reveal what was planned and what changed.

A systematic review may detect signs of missing small studies, but no statistical method can perfectly reconstruct unseen evidence. Publication practices are part of statistical interpretation.

A better way to report a result

1. Name the design and comparison. Explain how the data were produced.
2. Give the effect in understandable units. Include absolute quantities where possible.
3. Show uncertainty. Report an interval and explain what range matters.
4. State the analysis status. Was it planned, exploratory or one of many tests?
5. Test robustness. Do reasonable alternative assumptions change the conclusion?
6. Use the wider evidence. Compare with replications, systematic reviews and plausible mechanisms.

How to read “not significant”

Ask whether the estimate is close to zero and precise, or merely uncertain. A well-powered study with a narrow interval around a negligible effect can provide evidence against a meaningful benefit. A small study with a wide interval may be compatible with benefit, harm or little difference. Both can produce p ≥ 0.05; they do not communicate the same knowledge.

Connect the number to the claim

Our guide to Correlation vs. Causation explains why an association does not become causal by crossing 0.05. How to Read a Scientific Study shows where to find outcomes, figures and limitations. For the larger framework, see How Scientific Discovery Works and How to Evaluate New Scientific Discoveries.