Randomized trials of invasive cardiovascular interventions that include a placebo control: a systematic review and meta-analysis

Abstract

Aims

The difference in the benefit of invasive cardiovascular interventions compared with placebo controls has not been analysed systematically.

Methods and results

MEDLINE and Web of Science were searched through 29 March 2020. Randomized, placebo-controlled trials of invasive cardiovascular interventions (including catheter-based interventions and pacemaker-like devices) investigating predefined primary outcomes were included. Standardized mean differences (SMD) and odds ratios were calculated for continuous and dichotomous outcomes, respectively. Meta-regression analyses were performed to assess whether estimates of treatment effects were associated with methodological characteristics of trials. Thirty trials, including 4102 patients, were analysed. The overall risk of bias was judged to be low in only 43% of the trials. Ten trials (33%) demonstrated statistically significant superiority of invasive interventions over placebo controls for the respective predefined primary outcomes. In almost half of the 16 trials investigating continuous predefined primary outcomes, the SMD between the active and placebo procedure indicated a small (n = 4) to moderate (n = 3) treatment effect of active treatment over placebo. In contrast, one trial indicated a small treatment effect in favour of the placebo procedure. In the remaining trials, there was no relevant treatment effect of active treatment over placebo. In trials with a protocol-mandated stable and symmetrical use of co-interventions, the superiority of active procedures vs. invasive placebo procedures was significantly larger as compared with trials with frequent or unbalanced changes in co-interventions (P for interaction 0.027).

Conclusions

The additional treatment effect of invasive cardiovascular interventions compared with placebo controls was small in most trials.

Open in new tabDownload slide

Graphical Abstract

Open in new tabDownload slide

This paper has been added by Guest Editor John Cleland, University of Glasgow, Glasgow.

See page 2569 for the editorial comment on this article (doi: 10.1093/eurheartj/ehaa558)

Introduction

In cardiovascular medicine, progress in science and technology have remarkably reduced the number of deaths from cardiovascular disease.¹ Much of this was related to the development and use of invasive interventions and surgical procedures.¹ Objective testing of new treatments starts with preclinical and first-in-man observational studies, which are ideally followed by randomized placebo-controlled trials. However, only a few trials investigating the efficacy and safety of invasive cardiovascular interventions used placebo controls.² In contrast to placebo pills, placebo procedures are invasive and are thought to be associated with a higher degree of complexity, including ethical concerns of performing a procedure conferring an immediate risk of adverse events and potential harm without potential benefit to the patient.³ As medical devices have received more public attention due to safety and efficacy issues in recent years,⁴ the US Food and Drug Administration (FDA) has called for placebo-controlled trial designs, whenever ethical and feasible.⁵ Therefore, we systematically analysed the comparative efficacy and safety of active cardiovascular interventions and placebo controls.

Methods

Search strategy and definitions

This systematic review and meta-analysis follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.⁶ We searched MEDLINE and Web of Science for patient- and outcome assessor-blinded, randomized, placebo-controlled trials of invasive cardiovascular interventions. An invasive intervention was defined as a procedure during which a device was percutaneously or surgically inserted into the body and significantly modified the target-tissue. Trials using an invasive route only to administer medications (e.g. the intracoronary application of antithrombotic drugs or stem cells) or investigating cardiac resynchronization therapy/implanted cardioverter defibrillators were not considered. A placebo procedure was defined as a non-therapeutic, invasive procedure intended to mimic the active treatment as closely as possible without having a therapeutic effect beyond the placebo effect. All records through 29 March 2020 were considered, without language restrictions. Animal and paediatric studies were excluded. Detailed search terms are outlined in the Supplementary material online, Methods 1. Current clinical practice guidelines, reference lists of original articles, and review articles were hand-searched to identify further eligible trials that might have been missed using the search terms. Three reviewers (L.L., S.E., and S.S.S.) screened all abstracts independently for eligibility. Full-text articles were reviewed in duplicate by two reviewers (L.L. and S.S.S.). In the case of disagreement, a third reviewer (F.M.) was consulted, and disagreements were resolved by consensus.

Data extraction

Details on interventions, methods, patients’ characteristics, length of follow-up, outcomes, and adverse events were extracted for the active treatment and the placebo procedure group. The data underlying this article will be shared on reasonable request to the corresponding author. Although all primary efficacy and safety outcomes were extracted, only one predefined primary outcome per trial was included in the main analysis [for calculation of standardized mean differences (SMD) or odds ratios (OR)]. If a trial assessed more than one primary outcome, three reviewers (L.L., S.S.S., and F.M.) independently chose the outcome most relevant for the specific disease condition. As objective observer-reported outcomes are thought to be less prone to placebo effects than private phenomena, observer-reported outcomes were preferred, when both outcomes were available.⁷ Objective outcomes included biological measures such as blood pressure and the documentation of survival or events, whereas private phenomena were defined as subjective outcomes that were assessable by the patient only (e.g. the frequency of angina pectoris or quality of life).⁷ Results of intention-to-treat analyses were given precedence to prevent attrition bias.⁸ In crossover trials, results from the first phase were given precedence if reported separately. If data were missing, the trials’ corresponding authors were contacted.

Risk of bias assessment

Risk of bias was assessed according to the revised Cochrane risk of bias tool for randomized trials (RoB2).⁸ Two researchers (L.L. and S.E.) reviewed the publications and used the templates for randomized parallel-group and randomized crossover trials.⁸

Statistical analysis

Owing to the heterogeneity of the study populations, patients’ conditions, primary outcomes, and interventions performed, it was considered inappropriate to conduct a meta-analysis for all placebo-controlled trials. However, SMD and OR with the corresponding 95% confidence intervals were calculated for continuous (e.g. blood pressure) and dichotomous outcomes [e.g. major adverse cardiovascular events (MACE)], respectively. We standardized estimates so that positive SMDs indicated a benefit of the active intervention over placebo. Standardized mean differences were calculated by dividing the between-group difference in mean changes between baseline and follow-up by the pooled standard deviation of changes, with approximations used, as previously described.⁹ The magnitude of the SMDs was interpreted as originally suggested, with an SMD of 0.20 tentatively considered to be small, 0.50 moderate, and 0.80 large.¹⁰ We performed univariable subgroup analyses accompanied by random-effects meta-regression to test for an interaction between treatment effects and the following methodological characteristics: catheter-based intervention, pre-randomization run-in period, cut-off used to define minimal disease severity, concealment of allocation, blinding of patients, blinding of interventionalists, stable and symmetrical use of co-interventions, blinding of outcome assessors, assessment of objective outcomes, and intention-to-treat analysis (see Supplementary material online, Methods 2 for definitions of criteria for methodological quality). For these analyses, SMDs were converted to OR, as previously described.^{11  ,  12} A two-tailed P-value <0.05 was considered statistically significant. Statistical analyses were performed using STATA version 15 (StataCorp LLC, College Station, TX, USA) and GraphPad Prism version 8.2.1 (GraphPad Software, La Jolla, CA, USA).

Results

Description of trials

The search strategy identified 547 publications after duplicates were removed. Of these, 30 (5%) trials with a total of 4102 participants (median size of 97.5 patients) were eligible for the systematic review (Figure 1). Table 1 and Supplementary material online, Table S1 depict the key features of the included trials. This analysis included trials from August 2000 until March 2020. In these trials, patients were treated for angina pectoris or coronary artery disease (n = 10), hypertension (n = 9), vasovagal syncope (n = 4), chronic heart failure (n = 3), patent foramen oval suspected of causing migraine (n = 2), dysfunctional dialysis access grafts (n = 1), and carotid sinus hypersensitivity (n = 1) (Take home figure). Twenty-seven trials (90%) used a parallel-group design while three trials (10%) were designed as crossover trials.

Risk of bias

Figure 2 depicts a summary of the risk of bias, with details on the rationales for judgments provided in the Supplementary material online, Results. The overall risk of bias was judged to be low in 13 trials (43%).^{16  ,  24  ,  28  ,  30–32  ,  34  ,  36  ,  41  ,  42} There were some concerns and a high risk of bias in 14 (47%) and 3 (10%) trials, respectively. The interventionalist and outcome assessors were adequately blinded in only 10 (33%) and 25 (83%) trials, respectively. The underlying reasons for judging three trials to be at high risk of bias were the pooling of data of an unblinded pilot phase and the randomized trial phase,²⁰ insufficient blinding of outcome assessors due to device-induced artefacts seen during the echocardiographic assessment of the primary outcome⁴⁰ and some concerns for multiple domains of bias.²⁵

Primary outcomes

In total, the included trials evaluated 35 predefined primary outcomes, including 15 dichotomous (e.g. MACE) and 20 continuous (e.g. change in blood pressure) outcomes. Four trials assessed coprimary outcomes.^{16  ,  21  ,  22  ,  36} In 10 trials (33%), the null hypotheses were rejected for all predefined primary outcomes.^{18  ,  23  ,  27  ,  30  ,  36–40  ,  42} Of the 30 outcomes included in the main analysis, the majority of the trials assessed objective outcomes (n = 26, e.g. change in blood pressure or MACE) while four trials used patient-reported outcomes (private phenomena, e.g. the severity of angina pectoris). Active treatment demonstrated significant superiority over placebo procedures in 8/26 (31%) and 2/4 (50%) of the objective and patient-reported outcomes, respectively.

In 5/16 (31%) trials assessing a continuous primary outcome, the active treatment was significantly superior to placebo (Figure 3). The SMD between the active and placebo procedure was at most small (SMD 0.2–0.5, n = 5) to moderate (SMD 0.5–0.8, n = 3). Figure 4 presents OR for the trials reporting dichotomous primary outcomes. Of these, 4/14 (29%) trials showed the superiority of active treatment over placebo for their primary outcome. One trial had not observed any primary outcome events.³³

Subgroup analyses by methodological characteristics of trials

Figure 5 shows the results of subgroup analyses according to the methodological characteristics of trials (see Supplementary material online, Table S2). In trials with a protocol-mandated stable and symmetrical use of co-interventions, the average difference in effects between active and placebo procedures was larger compared to trials with frequent and/or unbalanced changes in co-interventions (P for interaction 0.027). For the remaining methodological characteristics, there were only minor variations between subgroups of trials.

Safety outcomes

Twenty-six trials (87%) provided data on predefined safety outcomes (Supplementary material online, Table S3). Of these, six trials (23%) reported no adverse events or safety outcomes in the active treatment or placebo group, whereas nine trials (35%) reported more adverse events following placebo procedure than active treatment but did not report further statistical analyses.^{32  ,  37}

Discussion

In total, 30 placebo-controlled trials analysed 35 predefined primary outcomes. In only 10 trials, the null hypothesis was rejected for all predefined primary outcomes. In trials assessing continuous primary outcomes, the effectiveness, defined as the SMD between the active and placebo procedure, was at most small to moderate. More than two-thirds of the trials evaluating dichotomous outcomes failed to show the superiority of active over placebo treatment. Taken together, this suggests that the therapeutic efficacy of some active experimental treatments in interventional cardiology may be smaller than generally assumed. Subgroup analyses, according to methodological characteristics, indicated that average treatment effects were larger in trials with stable and symmetrical use of co-interventions.

A placebo is generally defined as a substance or treatment of no intended therapeutic value, although it can exhibit relevant effects.² The placebo effect refers to a clinical benefit attributable to the interaction of the patient with the healthcare system and underlies complex psychological and neurobiological mechanisms.⁴³ There are strong indications that there is not one mechanism of the placebo effect, but many, depending on the physiological system involved, the medical condition and its severity as well as the placebo’s nature.⁴³ The appropriate placebo control in interventional cardiology is thought to be an invasive placebo, mimicking the interventions of the active treatment closely. It has indeed been suggested that the method of treatment delivery and the invasiveness of the placebo procedure correlates with its effectiveness.^{44  ,  45} In a meta-analysis investigating studies of migraine prophylaxis, placebo surgery was associated with higher responder rates (58%) than placebo acupuncture (38%) or oral pharmacological placebos (22%), respectively.⁴⁴ Similarly, the response to placebo pills in the treatment of Parkinson’s disease was minimal, whereas the response to placebo surgery large.⁴⁵

In the recent ORBITA trial, patients with stable angina and angiographically significant non-occluded lesions ≥70% were randomly allocated to percutaneous coronary intervention (PCI) with a drug-eluting stent or a placebo procedure (coronary catheterization while the patient was sedated).³⁴ In contradiction to randomized controlled trials which indicated that subjective outcomes, such as the severity of angina, was reduced following PCI compared with optimal medical therapy alone,^{46  ,  47} there was no difference in the primary efficacy outcome of exercise duration at 6 months, an objective but patient-dependent outcome, between patients who underwent PCI compared with a placebo procedure.³⁴ Several potential reasons have been discussed, which include methodological issues, including insufficient power due to a higher than expected standard deviation and a smaller than expected treatment effect. Importantly, the commonly observed symptomatic improvement from PCI may be encouraged by the potential power of placebo procedures.² The selection of the primary outcome appears essential, as patient-reported outcomes, in general, were more susceptible to placebo effects compared with observer-reported outcomes.⁴⁸ In this analysis, the null hypothesis for patient-reported (e.g. angina pectoris) and objective observer-reported outcomes (e.g. change in blood pressure) was rejected in 50% and 31% of the trials, respectively.

Blood pressure, as an objective primary efficacy outcome, was assessed in seven placebo-controlled trials investigating device-based therapies for hypertension. Early uncontrolled trials and registries of device-based therapies for hypertension documented large falls in blood pressure.⁴⁹ In the SYMPLICITY HTN-3 trial, the first placebo-controlled trial investigating renal denervation in patients with severe, resistant hypertension, blood pressure was reduced following active and placebo treatment without significant between-group differences.²⁸ The same observation was documented in a trial investigating baroreflex activation therapy in severe, resistant hypertension.²⁶ Several potential factors, such as inadequate patient selection, changes in antihypertensive medications after randomization, lifestyle changes, and variation in adherence to medication may have contributed to the significant blood pressure drop following placebo treatment.⁵⁰ Recently published trials, however, which minimized numerous issues identified in prior trials, provided proof of principle for the efficacy of renal denervation in both the presence and absence of antihypertensive medications.^{3  ,  38  ,  39  ,  42} Interestingly, in these well-controlled, rigorously executed trials no significant placebo effect on blood pressure was noticed, indicating that the use of placebo procedures per se does not eliminate other sources of bias that could result in both an underestimation and overestimation of treatment effects. In our analysis, subgroup analyses according to the methodological characteristics suggested that the treatment effect was largest in trials with stable and symmetrical use of co-interventions in active and placebo groups.

Objective testing of new devices and technologies in randomized trials with an appropriate control intervention is desirable wherever feasible. The fact that half of the trials eligible for the present analysis were published in 2015 or later indicates a growing awareness of the need to minimize the risk of bias in interventional cardiology. The use of a placebo procedure, however, adds complexity. Not only the patients’ blinding but also blinding of interventionalists and outcome assessors are necessary as incomplete blinding may introduce performance and detection bias.^{51  ,  52} Especially when investigating invasive interventions, it can be challenging to assure successful patient’s and physician’s blinding. Blinding of outcome assessors and treating physicians is important to reduce bias arising due to asymmetrical co-interventions (deviations from intended intervention) and bias in measuring the outcome.⁸ In five trials (17%) included herein, it remained unclear whether or not the outcome assessors were adequately blinded.^{20–23  ,  40} In a placebo-controlled trial assessing the effects of chronic vagal nerve stimulation for the treatment of heart failure with reduced ejection fraction, the majority of patients in the active treatment group (77%) correctly guessed their randomization assignment due to tickling sensations and other signs and symptoms during titration to the highest comfortable output. Patient’s blinding is in particular important when investigating private phenomena, whereas objective observer-reported outcomes are thought to be less prone to placebo effects.⁷ Blinding success is often underreported or, if reported, unsatisfactorily low.⁵³

The advantages of patient- and outcome assessor-blinded, placebo-controlled trials in reducing bias and investigating the specific treatment effect of invasive interventions over and above placebo effects are undeniable but exposing patients to potential risk by subjecting them to an invasive placebo procedure can raise ethical concerns. Therefore, the potential benefit of interventions has to be carefully weighed against the risks of a convincing placebo procedure, and there must be a sense of equipoise between active intervention and placebo control due to conflicting or weak evidence on the effectiveness of the intervention. Safety concerns are frequently raised as an argument against the use of invasive placebo procedures. For many of the trials included in this analysis, the risk of adverse events following placebo was relatively low, thus not necessarily supporting ethical arguments against placebo interventions. Taken together, this indicates that the benefits of including a placebo group need to be carefully weighed against the risk on a case-by-case basis.

Limitations

Some limitations of our analysis need to be acknowledged. First, assessing the risk of bias was problematic, as, especially in older trials, methods for blinding were only vaguely described. Second, we cannot exclude that regression to the mean might have contributed to the change of an outcome of interest. However, as we only included randomized controlled trials, regression to the mean is assumed to be equally distributed between active and placebo groups.^{44  ,  54} Third, it is likely that the failure to reject the null hypothesis in some trials was due to a lack of power to detect minimal clinically important differences between groups, particularly if the effect size used for sample size considerations was unrealistically large. Fourth, even though there are more than 200 randomized trials, which compared placebo against non-interventional controls, that did not receive a placebo,⁴⁸ none of the trials included in our analysis used a non-interventional control; hence, the true magnitude of the placebo effect is impossible to estimate.

Conclusion

The SMD between active and placebo procedures was at most small to moderate, which underlines the influence of non-specific effects on trial outcomes and an overestimation of the clinical efficacy of interventions in many circumstances. For most trials, the risk of adverse events following placebo was relatively low. Finally, our analysis suggests that treatment effects were larger in trials with protocol-mandated stable and symmetrical use of co-interventions after randomization, which highlights the significance of diligent planning and execution of placebo-controlled trials.

Conflict of interest: S.E. has received scientific support and speaker honoraria from Medtronic and ReCor Medical. S.S.S. has received speaker honoraria from Pfizer. W.W. has received a research grant and honoraria from MicoPort. T.F.L. has received educational and research grants from Abbott/St. Jude, Biotronik, and Medtronic. E.R.E. has active research programmes and grant support from Boston Scientific, Cardiatis, Edwards Life Sciences, Medtronic and is a founder of BioRest and Panther therapeutics. E.R.E. is funded in part by a grant from the US National Institutes of Health (R01 GM 49039). M.B. has received honoraria for lectures and scientific advice from Abbott, Astra Zeneca, Boehringer Ingelheim, Medtronic, Servier, and Vifor. P.J. serves as unpaid member of the steering group of trials funded by Astra Zeneca, Biotronik, Biosensors, St. Jude Medical, and The Medicines Company, has received research grants to the institution from Astra Zeneca, Biotronik, Biosensors International, Eli Lilly, and The Medicines Company, and honoraria to the institution for participation in advisory boards from Amgen, but has not received personal payments by any pharmaceutical company or device manufacturer. F.M. is supported by Deutsche Gesellschaft für Kardiologie (DGK) and Deutsche Forschungsgemeinschaft (SFB TRR219) and has received scientific support and speaker honoraria from Bayer, Boehringer Ingelheim, Medtronic, and ReCor Medical. All other authors declared no conflict of interest.

References