We don’t know whether most medical treatments work, and we know even less about whether they cause harm – new study

17/06/2022 - The Conversation

Only one in 20 medical treatments have high-quality evidence to support their benefits, according to a recent study. The study also found that harms of treatments are measured much more rarely (a third as much) as benefits.

Patients and doctors – and anyone who pays for them – need to know that medical treatments are safe and effective, but it’s an open secret in the medical field that not all treatments, including ones that are commonly used, are safe and effective. For example, antiarrhythmic drugs were widely prescribed in the belief that they would reduce heart attack deaths until a clinical trial found that they actually increased the risk of death.

In another example, putting infants to sleep on their stomach was recommended based on expert opinion that babies would be less likely to choke on their vomit until large studies found that stomach sleeping increased the risk of sudden infant death syndrome.

So how big is this problem?

In the early 2000s, researchers estimated that between quarter and a half of treatments are supported by high-quality evidence. But these estimates are now out of date and used old methods (such as researcher opinion) to determine whether the evidence was high quality or not. More recently, in 2020, a more rigorous estimate was published and found that only 10% of medical treatments were based on high-quality evidence. However, this estimate was based on a small sample of 151 studies.

Meanwhile, some continue to insist that most treatments must work. How else can we explain that we live ten years longer than our great-grandparents? Yet the extension in lifespan is explicable at least partly by public health measures such as clean water, better nutrition and restrictions on smoking.

A more accurate picture

To resolve the controversy about the proportion of treatments that are based on good evidence, an international team of researchers from the UK (University of Oxford), US, Switzerland and Greece conducted a large study of 1,567 healthcare treatments. The sample included all treatments tested in Cochrane Reviews between 2008 and 2021. Cochrane Reviews are rigorous studies that amalgamate all available relevant evidence about treatments. They are often referenced in national and international healthcare guidelines.

The year 2008 was chosen as the cut-off because that was when Cochrane Reviews incorporated a system called grading quality of evidence and strength of recommendations (Grade) to rate how trustworthy the evidence is. Unlike the earlier estimates that often relied on opinions, Grade is more widely accepted and is used by over 100 organisations around the world. Using Grade results in a quality rating of high, moderate, low or very low.

The study revealed that 95% of treatments do not have high-quality evidence to support their benefits. Worse, the harms are reported in only about 33% of Cochrane Reviews.

It is particularly worrying that the harms of healthcare interventions are rarely quantified. For a doctor or patient to decide whether to use a treatment, they need to know whether the benefits outweigh the harms. If the harms are inadequately measured, an “informed choice” is not possible.

A potential limitation of the study is that Grade might be too strict. Doctors and patients may be happy to use treatments whose benefits are not supported by high-quality evidence as long as they are supported by moderate-quality evidence. Even if this is right, the study found that less than half of treatments are supported by high or moderate-quality evidence.

Patients with ailments for which there are no effective treatments may be willing to try treatments that are not yet even supported by low-quality evidence. The study should not be used to constrain these patients’ choices.

Also, the sample may not have been representative. In theory, treatments tested in recent Cochrane Reviews may be less effective or based on lower-quality evidence than older treatments. However, given the rigour of Cochrane Reviews, this seems unlikely.

In practice, doctors can use “off-label” treatments which are less likely to have been studied in Cochrane Reviews and generally have lower-quality evidence to support them. Despite these potential limitations, the study still showed that most treatments are not supported by high-quality evidence.

Doctors, patients and those who pay for them may wish to focus on treatments whose benefits and safety are established by high-quality evidence. Research funding should be allocated to generating high-quality evidence for treatments that are widely used but not yet supported by high-quality evidence about their benefits and harms.

Finally, potential harms should be measured with the same rigour as potential benefits. The evidence-based medicine community is correct to continue calling for higher-quality research, and also justified in their scepticism that high-quality evidence for medical treatments is common or even improving.