Key points
- A p-value is the probability of seeing a result at least as extreme as the one observed if the drug truly had no effect.
- Trials commonly treat p < 0.05 as statistically significant, but that threshold is a convention, not a law of nature.
- A p-value is not the probability that the drug works, and it says nothing about how large the benefit is.
- Confidence intervals show the range of effect sizes consistent with the data and are often more informative than the p-value alone.
- Testing many endpoints or subgroups increases the chance of a false positive, which is why trials prespecify how they will control for multiple comparisons.
Nearly every clinical trial announcement includes a phrase like "statistically significant (p < 0.05)." It is one of the most important ideas in clinical research and one of the most misunderstood. Statistical significance answers a narrow question: could this result plausibly be a fluke? It does not answer whether a drug is worth taking.
The problem significance solves
People vary. If you give a drug to one group and a placebo to another, the groups will almost never have identical outcomes, even if the drug does nothing at all. Some difference will appear by chance. Statistical testing asks whether the difference observed is larger than chance alone would usually produce.
What a p-value means
The p-value is the probability of observing a result at least as extreme as the one seen, assuming the drug has no real effect. That assumption is called the null hypothesis.
- A small p-value (for example, 0.01) means the observed result would be unusual if the drug did nothing.
- A large p-value (for example, 0.40) means the result is quite compatible with no effect.
By convention, many trials treat p < 0.05 as the threshold for statistical significance. That corresponds to accepting about a 1-in-20 chance of declaring an effect when none exists, a false positive. The threshold is a convention, and regulators do not apply it mechanically.
What a p-value does not mean
In 2016 the American Statistical Association issued a formal statement on p-values to correct common misreadings. Among its principles:
- A p-value is not the probability that the hypothesis is true. A p-value of 0.03 does not mean there is a 97% chance the drug works.
- Conclusions should not rest only on whether a p-value crosses a threshold. A result with p = 0.049 is not meaningfully different from one with p = 0.051.
- Statistical significance does not measure the size or importance of an effect. A very large trial can produce a tiny p-value for a benefit too small to matter to patients.
Effect size and confidence intervals
Because the p-value says nothing about magnitude, the effect size matters at least as much. How much did the drug reduce the risk, or improve the score?
A confidence interval shows the range of effect sizes compatible with the data. For example:
The drug reduced the risk of hospitalization by 20% (95% confidence interval: 8% to 30%).
This tells you that the data are consistent with a benefit as small as 8% or as large as 30%. If the interval had run from −5% to 40%, it would include no benefit, or even harm, and the result would not be statistically significant at the conventional level.
| Scenario | p-value | What it tells you |
|---|---|---|
| Large effect, small trial | Possibly not significant | Promising, but too uncertain; more data needed |
| Small effect, huge trial | Highly significant | Real, but may be too small to matter clinically |
| Large effect, adequately sized trial | Significant | The strongest kind of evidence |
The multiple-comparisons problem
Every additional statistical test is another chance for a false positive. A trial that tests 20 independent outcomes at the 0.05 level would be expected to produce about one "significant" result by chance alone, even for an ineffective drug.
International trial guidelines (ICH E9) and FDA guidance on multiple endpoints require trials to deal with this in advance. Common approaches include:
- choosing a single primary endpoint,
- testing secondary endpoints in a prespecified hierarchy and stopping at the first failure, and
- adjusting the significance threshold when several comparisons are made.
This is why a significant result in a subgroup or a secondary endpoint, found after the primary endpoint failed, is treated with skepticism. See What Is a Primary Clinical Trial Endpoint?
Interim analyses
Some trials look at the data before they finish, for example to stop early if a drug is clearly working or clearly harmful. Each look is another chance for a false positive, so trials that plan interim analyses use stricter thresholds at each look to keep the overall error rate controlled.
Clinical significance
A result can be statistically significant without being clinically significant, meaning large enough to make a real difference to patients. Researchers and regulators judge clinical significance by asking whether the size of the benefit is meaningful for the disease, and whether it outweighs the drug's risks and burdens.
Reading significance claims in trial news
- Was the result significant on the prespecified primary endpoint?
- How big was the effect, and what was the confidence interval?
- Is "trend toward significance" being used to describe a result that was not significant? That phrase often describes a failed test.
- How many endpoints or subgroups were tested? A single significant result among many tests is weak evidence.
- Has the result been replicated? One significant trial is less convincing than two. See What Is an Independent Clinical Trial Replication?
Sources
- Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose. The American Statistician. 2016;70(2):129–133.Peer-reviewed
- International Council for Harmonisation. E9: Statistical Principles for Clinical Trials.Regulator
- U.S. Food and Drug Administration. Guidance: Multiple Endpoints in Clinical Trials.Regulator
- 21 CFR 314.126: Adequate and well-controlled studies.Statute / regulation
This article is for education. It is not medical advice and not investment advice. Figures are drawn from the sources above and describe historical averages, not the prospects of any particular drug. See our editorial standards and disclaimer.