Drug development

What Is an Independent Clinical Trial Replication?

A replication is a separate trial that tests the same question and finds a consistent result. Independent replication is one of the strongest protections against results that turn out to be flukes.

Key points

  • A replication is a separate study, with different patients and often different investigators, that tests the same question and reaches a consistent result.
  • The FDA has traditionally looked for at least two adequate and well-controlled trials, which builds replication into the approval standard.
  • In some cases the FDA accepts a single trial plus confirmatory evidence instead of a second trial.
  • In a study of 45 highly cited clinical studies that reported an effective treatment, 16% were later contradicted and another 16% found stronger effects than later research.
  • Preclinical research also has a replication problem: one company reported confirming the findings of only 6 of 53 landmark cancer studies.

A single positive clinical trial is encouraging. Two independent positive trials are far more convincing. Replication, getting a consistent answer from a separate study, is one of the main ways science separates real effects from chance findings, and it is built into how the FDA judges whether a drug works.

What counts as a replication

A replication is a new study that tests the same hypothesis as an earlier one and finds a consistent result. It is most informative when it is independent, meaning it differs from the original in ways that could have biased the first result:

  • Different participants, so the result does not depend on one group of patients.
  • Different sites and investigators, so it does not depend on one team's methods or judgments.
  • Ideally, different analysts or sponsors, so it does not depend on one organization's choices.

A replication does not need identical numbers. It needs a result consistent with the original: an effect in the same direction, of broadly similar size, on the same endpoint.

Replication in the FDA's approval standard

U.S. law requires "substantial evidence" of effectiveness from adequate and well-controlled investigations. The FDA has traditionally interpreted this as at least two adequate and well-controlled trials, each convincing on its own. That requirement is a replication requirement: the second trial checks that the first was not a fluke.

The FDA's 2019 draft guidance on demonstrating substantial evidence of effectiveness explains that the agency can sometimes rely on one adequate and well-controlled trial plus confirmatory evidence. Confirmatory evidence can include:

  • strong mechanistic or pharmacological evidence,
  • results from a closely related indication,
  • data from a related drug in the same class, or
  • natural-history data in rare diseases where a second trial is impractical.

This flexibility matters most for rare diseases and serious conditions, where running two large trials may be infeasible or unethical.

Why first results are often too optimistic

Several forces make an initial positive result likely to overstate the truth:

  • Chance. Small studies produce variable results. When many are run, some will look impressive by luck.
  • Selection. Positive results are more likely to be published, publicized, and followed up, so the findings people hear about first skew positive.
  • Flexibility in analysis. Early studies are more likely to choose among endpoints, subgroups, and analysis methods after seeing the data.
  • Optimistic conditions. The first trial is often run at experienced centers in carefully selected patients.

The evidence

In a 2005 study in JAMA, John Ioannidis followed 45 highly cited clinical studies that reported an effective intervention. When later, comparable or larger studies were done:

Outcome Share of studies
Replicated 44%
Remained largely unchallenged 24%
Later found weaker effects than originally reported 16%
Contradicted 16%

Source: Ioannidis, JAMA (2005).

Five of the six highly cited non-randomized studies were later contradicted or found to have overstated their effects, compared with 9 of 39 randomized trials. Randomization makes results considerably more likely to hold up.

Replication starts before the clinic

Replication problems begin in the laboratory. In a 2012 commentary in Nature, scientists from the biotechnology company Amgen reported trying to confirm the findings of 53 "landmark" preclinical cancer studies before pursuing them. They succeeded in only 6. Drugs built on preclinical findings that do not replicate are likely to fail in clinical trials, often expensively, in Phase 2.

Replication versus a confirmatory trial

The terms overlap but are not identical:

  • A replication asks whether a finding holds up in a new sample.
  • A confirmatory trial is a trial designed with a prespecified hypothesis and analysis to test a result suggested by earlier, exploratory work. A confirmatory Phase 3 trial testing a Phase 2 finding is a replication of that finding under stricter conditions.

Accelerated approvals depend on confirmatory trials. A drug approved on a surrogate endpoint must confirm its clinical benefit afterward.

How Spectral Genomics treats replication

On drug profiles, we mark replication status explicitly:

  • Not established: a single positive trial, or results only from uncontrolled studies.
  • Partially established: a consistent result in a second trial that differed in important ways, such as a different population or endpoint.
  • Established: consistent results in at least two adequate and well-controlled trials, or one trial with strong confirmatory evidence.

We also distinguish replication by independent investigators from additional trials run by the same sponsor.

Sources

  1. U.S. Food and Drug Administration. Draft Guidance: Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products (2019).Regulator
  2. Ioannidis JPA. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005;294(2):218–228.Peer-reviewed
  3. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012;483:531–533.Peer-reviewed
  4. 21 CFR 314.126: Adequate and well-controlled studies.Statute / regulation

This article is for education. It is not medical advice and not investment advice. Figures are drawn from the sources above and describe historical averages, not the prospects of any particular drug. See our editorial standards and disclaimer.