Same evidence different recommendations: a methodological assessment of transatlantic guidelines for the management of valvular heart disease Free

Summary

OBJECTIVES

The aim of this study was to identify methodological variations leading to varied recommendations between the American College of Cardiology (ACC)/American Heart Association (AHA) and the European Society of Cardiology (ESC)/European Association for Cardio-Thoracic Surgery (EACTS) valvular heart disease guidelines and to suggest foundational steps towards standardizing guideline development.

METHODS

An in-depth analysis was conducted to evaluate the methodologies used in developing the transatlantic guidelines for managing valvular heart disease. The evaluation was benchmarked against the standards proposed by the Institute of Medicine.

RESULTS

Substantial discrepancies were noted in the methodologies utilized in development processes, including Writing Committee composition, evidence evaluation, conflict of interest management and voting processes. Furthermore, despite their mutual differences, both methodologies demonstrate notable deviations from the Institute of Medicine standards in several essential areas, including literature review and evidence grading. These dual variances likely influenced divergent treatment recommendations. For example, the ESC/EACTS recommends transcatheter edge-to-edge repair for patients with chronic severe mitral regurgitation ineligible for mitral valve surgery, while the ACC/AHA recommends transcatheter edge-to-edge repair based on anatomy, regardless of surgical risk. ESC/EACTS guidelines recommend a mechanical aortic prosthesis for patients under 60, while ACC/AHA guidelines recommend it for patients under 50. Notably, the ACC/AHA and ESC/EACTS guidelines have differing age cut-offs for surgical over transcatheter aortic valve replacement (<65 and <75 years, respectively).

CONCLUSIONS

Variations in methodologies for developing clinical practice guidelines have resulted in different treatment recommendations that may significantly impact global practice patterns. Standardization of essential processes is vital to increase the uniformity and credibility of clinical practice guidelines, ultimately improving healthcare quality, reducing variability and enhancing trust in modern medicine.

Open in new tabDownload slide

INTRODUCTION

Valvular heart diseases (VHDs) present a major healthcare challenge globally as they substantially elevate the risk of mortality and severe morbidity, thereby contributing significantly to a decline in quality of life in patients at all ages [1–3]. An ageing population, enhanced accessibility and advancements in diagnostic methods, along with emerging technologies, are continuously evolving the landscape of VHD treatments, thereby increasing the burden on hospitals and healthcare systems. Clinical practice guidelines (CPGs) are developed to establish the best standards of care, provide assistance during shared treatment decision-making, guide healthcare policies and enhance quality. However, a lack of unanimity often results in differing recommendations for screening, prevention or treatment [4–6].

The most frequently cited and discussed VHD guidelines come from North America and Europe. In December 2020, the American College of Cardiology (ACC) and American Heart Association (AHA) jointly published the comprehensive guideline on the management of VHD [7]. This guideline provides extensive recommendations on diagnosis, risk assessment and treatment. Similarly, the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS) released their VHD guideline in August 2021, covering the similar aspects of patient care [8]. This narrative review evaluates the primary differences in methodologies that may led to discrepancies in the recommendations between these 2 CPGs. It also contrasts them with the standards set forth by the Institute of Medicine (IOM) and calls for the harmonization of fundamental methodological standards to improve the consistency and trustworthiness of the guidelines.

STANDARDS FOR DEVELOPING CLINICAL PRACTICE GUIDELINES

The development of CPGs is currently marked by a diversity of methodologies, resulting in absence of universally accepted standards. This results in variations in literature review, rigour of evidence appraisal, and the ways conflicts of interest (COIs) are managed. Numerous healthcare agencies and non-governmental organizations propose sets of standards to assess, appraise, and grade evidence to inform treatment recommendations [9–11]. The IOM, now known as the National Academy of Medicine as one of 3 academies that make up the National Academies of Sciences, Engineering, and Medicine in the United States, has established the most respected and thorough methodology for developing CPGs, as detailed in their seminal publication, 'Clinical Practice Guidelines We Can Trust’ [9]. Among others, they propose 8 essential attributes for creating trustworthy CPG: (i) transparency, (ii) managing COIs, (iii) involving diverse developers, (iv) comprehensive systematic literature review, (v) building on evidence foundations and rating strength of evidence, (vi) clearly articulating recommendations, (vii) external review and (viii) regular updates (see Table 1 for more details). Even though the implementation of these processes might appear to be straightforward, a study analysing a random sample revealed that most CPGs archived in the AHRQ National Guideline Clearinghouse fail to meet basic academic standards [12]. Resistance to adopting IOM standards is prevalent, with many organizations citing increased costs and publication delays that do not correspond with a perceived increase in value [12]. The debate continues as to whether all IOM criteria should be universally applied to each proposed recommendation, or if some degree of flexibility is permitted. Finally, a concerning issue is the frequency of updates, with CPGs becoming outdated within 5 years of publication [13], and updates occurring infrequently. This has resulted in multiple FDA alerts over the years [12], indicating a critical need for more timely guideline revisions.

Given that many developers overlook these basic methodological standards [14], substantial shifts without in-depth academic discussion seem unlikely. Despite some criticisms [15], the IOM guidance remains the ‘gold standard’ and enables CPG developers striving to create reliable resources that may enhance healthcare quality and safety. With no direct European counterpart to the IOM Methodology, numerous European associations, including EACTS, have embraced these principled standards within their guideline methodology.

HOW CONTEMPORARY CLINICAL PRACTICE GUIDELINES FOR VHD MANAGEMENT ARE DEVELOPED

American College of Cardiology/American Heart Association

In 2010, the ACC/AHA published a Methodology to streamline the guideline development process [16]. Key characteristics, as well as differences and similarities between the ACC/AHA, IOM and ESC/EACTS standards for guideline development, are provided in Table 1. The VHD Writing Committee consisted of 11 clinical and interventional cardiologists, 2 cardiac surgeons and 1 anaesthesiologist. All recommendations underwent a formal voting process, and each recommendation was considered approved if it received a simple majority of votes (50% + 1) for members eligible to vote [16]. Writing Committee members with COIs abstained from voting on a particular recommendation. Since 2019, the ACC and AHA have enhanced their COIs policy to ensure that no more than 50% of Writing Committee including Co-Chairs, have relevant COIs starting 1 year prior to the initiation of project continuing until the publication. Systematic literature reviews were conducted solely based on the explicit request made by the authors when the question was deemed of critical clinical importance [7]. Cost-value considerations were used for limited treatment modalities.

European Society of Cardiology/European Association for Cardio-Thoracic Surgery

The Methodology for joint ESC and EACTS CPG is determined on a project-by-project basis, drawing from similar key methods outlined in the separate methodologies released by EACTS in 2015 and ESC in 2017 (Table 1) [17, 18]. The Writing Committee comprised two-thirds of ESC representatives (both clinical and interventional cardiologists) and one-third of EACTS representatives [8]. The voting threshold was established at 75% of eligible Writing Committee members. Co-Chairs were allowed to have relationships with the industry as long as the total income did not exceed 10 000 euros yearly. However, there were no specific restrictions regarding the income of other Writing Committee members, except for prohibitions on their employment in the industry. The guideline had a methodology group, consisting of 2 delegates from ESC and 2 from EACTS, which was formed to assist in evidence collection and synthesis. This group primarily focused on the treatment of aortic stenosis (AS) and secondary mitral regurgitation (MR). An economic evaluation was not discussed, as these evaluations can vary significantly from country to country in Europe.

Class of recommendation and level of evidence

Both ACC/AHA and ESC/EACTS CPGs utilize a similar method for providing treatment guidance, employing different classes of recommendations (CORs) and levels of evidence (LOEs) (Table 2). The COR denotes the strength of the recommendation and certainty surrounding estimates, while the LOE reflects the quality of evidence supporting the recommendation. There are notable differences between the 2 methodologies (Table 2). Minor differences include: (i) suggested phrases for describing different CORs; (ii) the distinction between not recommended treatments that are harmful versus those that simply bring no benefits; and (iii) variations in whether LOE C recommendations are based on limited data or expert opinions only (Table 2).

COMPARISON OF THE 2020 AMERICAN COLLEGE OF CARDIOLOGY/AMERICAN HEART ASSOCIATION AND 2021 EUROPEAN SOCIETY OF CARDIOLOGY/EUROPEAN ASSOCIATION FOR CARDIO-THORACIC SURGERY GUIDELINES

Numerous review articles delve into the consistencies and discrepancies between these 2 guidelines from a clinical perspective, shedding light on existing gaps and proposing future research directions to address them [19–21]. In light of previously published comprehensive assessments, we have narrowed our focus to the management of patients with AS and MR, aiming to further analyse and acknowledge these issues from both clinical and methodological perspectives.

The Management of Patients with Aortic Stenosis

The ACC/AHA guideline features 37 recommendations for diagnosis, follow-up, medical therapy and the timing and choice of intervention for AS. In comparison, the ESC/EACTS guideline includes 19 recommendations, mainly concerning intervention indications. The distribution of COR is fairly similar in both guidelines: class I (53% for ESC/EACTS vs 52% for ACC/AHA), class II (42% vs 43%) and class III (5% vs 5%). However, there are significant differences in the LOE supporting the recommendations. The ESC/EACTS recommendations rely mostly on LOE C (Expert Opinion) (63%), while LOE C is assigned to only 15% of the ACC/AHA recommendations. Furthermore, only 1 recommendation (5%) in the ESC/EACTS guideline is supported by LOE A, compared to 9 recommendations (24%) in the ACC/AHA guideline. Although both guidelines largely align in their approach to diagnosing AS and determining the timing for intervention, their differing recommendations regarding the mode of intervention carry substantial implications for patients, healthcare providers and payers (for more details, see Table 3).

The Management of Patients with Mitral Regurgitation

The ACC/AHA guideline includes 29 recommendations for evaluation, treatment and intervention in patients with acute and chronic primary and secondary MR. On the other hand, ESC/EACTS guideline provides 12 recommendations, primarily on intervention indications. The distribution of COR markedly differs between the guidelines: class I (42% for ESC/EACTS vs 52% for ACC/AHA), class II (58% vs 42%) and class III (0% vs 7%). However, the LOEs supporting these recommendations are quite similar. Most recommendations in both guidelines are supported by LOE B (75% for ESC/EACTS vs 79% for ACC/AHA), with LOE C supporting 25% and 18% of recommendations, respectively. Generally, the ESC/EACTS guidelines align with the ACC/AHA recommendations for intervention in primary MR. However, there are several notable differences concerning MR intervention modalities that could significantly impact practice patterns:

1. Both ACC/AHA and ESC/EACTS guidelines recommend transcatheter edge-to-edge repair (TEER) for severely symptomatic patients with primary severe MR and high surgical risk (COR IIa, LOE B), using the same LOE B but different CORs (IIa and IIb, respectively) with a clinically noticeable implication.

2. The ESC/EACTS guideline recommends concomitant surgery for patients with severe secondary MR undergoing coronary artery bypass grafting or other cardiac surgery (COR I, LOE B). However, the ACC/AHA guideline rates this recommendation as COR IIa (LOE B), suggesting mitral valve surgery should not be the standard of care for all cases of secondary MR when coronary artery bypass grafting is indicated.

3. The ESC/EACTS updated guideline recommends TEER (COR IIa, LOE B) for patients not suitable for surgery, as determined by the heart team, and who met the COAPT-like criteria indicating a higher likelihood of treatment response. In contrast, the ACC/AHA guideline recommends that TEER is reasonable for patients with suitable anatomy, as defined on TEE and LVEF between 20% and 50%, regardless of surgical risk (COR IIa, LOE B).

WHY DO THESE DISCREPANCIES IN TREATMENT RECOMMENDATIONS HAPPEN

Discrepancies in treatment recommendations frequently originate from a variety of seemingly minor, yet significant ‘errors’ incorporated within the development process. Within the analysed guidelines, distinct characteristics like Writing Committee diversity, evidence evaluation, and voting mechanisms collectively contribute to the observed discrepancies. It is noteworthy that both groups opted to include age cut-offs in their recommendations, even though these thresholds were not part of the inclusion criteria for any of the referenced clinical trials used to support the given LOEs. This approach could result in inconsistencies when groups attempt to define clinically relevant and suitable age cut-offs, possibly overlooking the necessary scientific rigour that should substantiate treatment recommendations. Moreover, differences in the composition of the Writing Committees might have significantly influenced the formation of divergent recommendations. In the ACC/AHA guidelines, representatives from a single specialty could develop recommendations, easily overruling objections from other specialties. Conversely, the ESC/EACTS guidelines mandated that consensus and compromise be achieved among all specialty representatives within the group. This contrast in procedural approach likely contributed to opportunities for systematic errors and introduced subjectivity in developing the recommendations.

In the case of discrepancies in the treatment of MR, the underlying cause appears more evident and tied to differing interpretations of the scientific evidence. The ESC/EACTS group noting a high risk of bias in the most robust clinical trial, leading them to provide conditional treatment recommendations for TEER exclusively for non-surgically eligible candidates. Conversely, the ACC/AHA group found the trial results to be of utmost relevance and consequently expanded their recommendations for TEER to a broader patient population. The varying perspectives on the robustness of evidence required to support decision-making have significant worldwide implications for clinical practice patterns. However, the reasons behind these divergent opinions on the same evidence remain largely unclear.

PROPOSAL FOR PROCESS STANDARDIZATION AND IMPROVEMENTS

Several aspects warrant attention to standardize and improve the quality and reliability of the guideline development process. It is crucial that this process remains consistent, follows a systematic approach, and adheres to universal methodologies that align with the highest standards. We suggest several key refinements based on the IOM’s fundamental characteristics for developing credible CPGs internationally.

First, the Writing Committee should be multidisciplinary and balanced to ensure a broad representation of opinions, not only from those involved in primary hospital care but also from experts involved in the continuum of patient care beyond the index procedures. Moreover, it is essential to clearly distinguish between the evidence review committee and the clinical content experts and patient representatives, who provide their insights based on personal disease experiences and clinical practice. Second, factors impacting the quality of evidence require rigorous consideration [22, 23]. Fundamental research limitations that can compromise the quality of evidence should lead to downgrade of the COR, while strong, consistent trial results should trigger an upgrade. Third, a transparent and standardized benefit−risk assessment should influence the strength of recommendation. Fourth, the proposed recommendations should avoid using patient characteristics not directly informed by pivotal trials. Fifth, a zero-tolerance COI policy for Co-Chairs and vast majority of Writing Committee members should be enforced across all guidelines. To minimize the risk of unconscious bias, principal investigators of guideline-relevant trials should not participate in the guideline development; instead, they should serve as advisors during the external validation process. Sixth, guidelines must comply with reporting standards, including a summary of the systematic review, a summary of findings and evidence grading, vote count and any disagreement within Writing Committee. Additionally, the voting threshold should be set high to ensure that critical decisions aren't determined by a simple majority. Seventh, the involvement of representative patient organizations, with the aim of empowering patients about their healthcare choices, is essential. Finally, an external independent agency should certify guidelines prior to the publication and local application. This certification process could operate on regional or country levels, acknowledging and upholding the value and credibility of the guidelines across different healthcare systems.

Along with refining standards, it is imperative that leaders of representative associations work together to strengthen international collaboration in standardizing methodologies for the process improvements of clinical practice documents. This harmonization will aim to reduce inconsistencies in treatment recommendations, thereby lowering associated costs, enhancing comprehensiveness, and shortening the intervals between updates.

CONCLUSIONS

CPGs are invaluable tools for guiding healthcare decisions, enhancing quality and improving patient outcomes. Yet, significant discrepancies between the guidelines developed by prominent organizations underscore the challenges in producing consistent and reproducible evidence-based recommendations. These differences highlight a concerning trend where guideline developers do not fully comply even with the basic principles for their development, which are fundamental for creating sound and methodologically rigorous guidelines. This situation underscores a broader issue within the healthcare community, the absence of consensus on minimum universally accepted standards for creating practice guidelines. The growing importance of this issue is evident as healthcare quality measures and reimbursement systems are likely to be tied to recommendations from ‘trustworthy’ practice guidelines in the near future, despite their advisory nature and the fact that they are not designed to encompass all medical conditions.

Recognizing these challenges is a vital first step towards addressing them. To effectively tackle this issue, collaborative efforts are needed among the academic medical community, professional societies, government organizations, payers, patient representatives and other stakeholders to find a long-term solution. Such an effort will aim to minimize the likelihood of essential health decisions varying based on geographic location and differing interpretations of the same evidence, which could lead to disparate recommendations and cause confusion and doubts about the credibility of CPGs.

ACKNOWLEDGEMENTS

We extend our gratitude to the INTEGRITTY Group for sharing their expertise with the methodology for developing clinical practice guidelines and for their extensive assistance with data interpretation.

FUNDING

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Conflict of interest: Dr Sanjay Kaul reported receiving personal fees from Boehringer-Ingelheim, NovoNordisk, Bayer, GSK, Abbott Consulting and AstraZeneca outside the submitted work; and holding equity interest in Johnson & Johnson. The authors have reported that they have no relationships relevant to the contents of this paper to disclose.

DATA AVAILABILITY

The data will be shared on reasonable request to the corresponding author.

Author contributions

Milan Milojevic: Conceptualization; Data curation; Methodology; Writing—original draft. Miguel Sousa-Uva: Conceptualization; Investigation; Validation; Writing—original draft. Mateo Marin-Cuartas: Project administration; Validation; Writing—review & editing. Sanjay Kaul: Validation; Writing—review & editing. Aleksandar Nikolic: Validation; Writing—review & editing. John Mandrola: Validation; Writing—review & editing. J. Rafael Sádaba: Validation; Writing—review & editing. Patrick O. Myers: Visualization; Writing—review & editing;

Reviewer information

European Journal of Cardio-Thoracic Surgery thanks Krishna Khargi, Carlos A. Mestres, Alexander Wahba and the other, anonymous reviewers for their contribution to the peer review process of this article.

REFERENCES

ABBREVIATIONS

ABBREVIATIONS
 
• ACC

  American College of Cardiology

 
• AHA

  American Heart Association

 
• AS

  Aortic stenosis

 
• CORs

  Classes of recommendations

 
• COIs

  Conflicts of interest

 
• EACTS

  European Association for Cardio-Thoracic Surgery

 
• ESC

  European Society of Cardiology

 
• IOM

  Institute of Medicine

 
• LOEs

  Levels of evidence

 
• MR

  Mitral regurgitation

 
• TEER

  Transcatheter edge-to-edge repair

 
• VHDs

  Valvular heart diseases