Critical Appraisal of Paralyzed Veterans of America Guidelines in Spinal Cord Injury: An International Collaborative Study Using the Appraisal of Guidelines for Research and Evaluation II Instrument (AGREE II) Free

ABSTRACT

Introduction

Spinal cord injuries (SCI) in military personnel, veterans, and others require an evidence-based, multidisciplinary approach to their care. This appraisal used the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument to evaluate the methodological quality of clinical guidelines for the management of SCI published by the Paralyzed Veterans of America (PVA) organization.

Materials and Methods

We searched clinical guidelines on SCI published by PVA until December 2019. Four appraisers across three international centers independently evaluated the quality of eligible clinical guidelines using AGREE II. Mean AGREE II scores for each domain were calculated. In higher quality domains, scores for individual items were analyzed.

Results

A total of 12 guidelines published by PVA on SCI were assessed. Mean scores for all six domains were as follows: Scope and Purpose (78.8%), Stakeholder Involvement (63.7%), Rigor of Development (68.4%), Clarity of Presentation (80.1%), Applicability (53.0%), and Editorial Independence (28.5%). The mean score for the overall quality of all PVA guidelines was 71.9% (95% CI: 69.7–74.1). No guideline was assessed as “not recommended” by any appraiser. Overall quality was significantly associated with year of publication (r_s = 0.754, P = 0.0046). Overall agreement among appraisers was excellent (intraclass correlation coefficients for each guideline ranged from 0.96 to 0.99).

Conclusions

PVA guidelines for the management of SCI demonstrated acceptable or good quality across most domains. We recommend the use of PVA guidelines for the assessment and treatment of SCI and related disorders. The quality of PVA guidelines for the management of SCI have improved over time.

INTRODUCTION

Spinal cord injuries (SCIs) are common events in modern warfare. In 2013, a study of military casualties in Iraq and Afghanistan found that the proportion of casualties with spinal injuries reached 11.1% and, among combat-specific personnel, 14%.¹ In the general population, up to half a million people suffer traumatic SCIs each year.² Global incidence of SCI varies from 8.0 to 246.0 cases per million inhabitants per year. The global prevalence varies from 236.0 to 1,298.0 per million inhabitants.³ Etiology varies, with “violence” responsible for a higher proportion of SCI in developing countries and where gun-crime or warfare is prevalent.³ Injury level, extent of spinal cord injury and further injuries and patient physiology and comorbidities will affect the extent of neurological dysfunction, respiratory, cardiovascular, autonomic, psychological, and other organ system complications. Therefore, a multidisciplinary approach to the management of SCI in both military and civilian patients is required.

The Paralyzed Veterans of America (PVA) organization was formed in 1946 and is “dedicated solely for the benefit and representation of serving personnel and veterans with spinal cord injury or disease.”⁴ In order to diagnose and treat patients with spinal cord injury according to evidence-based principles, the PVA established the Consortium for Spinal Cord Medicine in 1994, representing 19 professional, insurance, or consumer stakeholder organizations. With the support of invited experts, this consortium has published a series of clinical guidelines on the management of SCI and the prevention and management of associated complications.⁵ PVA guidelines have generally included the following

• Preface, acknowledgements, panel members, and contributors/expert reviewers

• Summary of recommendations

• Guideline development process and methodology

• Evidence for recommendation and recommendations for future research

• Appendices, references, and index.

It is fundamental in the development of clinical guidelines that rigorous methodological strategies are used to assess and reduce the potential for bias.^6,7 There are several assessment tools developed to evaluate the methodological quality of clinical guidelines.⁸ Among them, the Appraisal of Guidelines for Research and Evaluation II (AGREE II) instrument is widely used and recommended in the World Health Organization Handbook for Guideline Development.^9,10 Published in 2003 by a group of international guideline developers and researchers (the AGREE Collaboration), AGREE II benchmarks methodological strategies for the development of new guidelines and clarifies which information should be included and how to report it.¹¹

The AGREE II tool has been used to evaluate guidelines for orthopedics, neurological injury, and rehabilitation from brain trauma,^12–14 but there is no relevant research on the AGREE II tool to evaluate clinical practice guidelines for SCI.

The aim of this study was to assess the quality of PVA-derived clinical practice guidelines on SCI and related pathologies according to AGREE II criteria.

MATERIAL AND METHODS

Guideline Identification

The search for PVA guidelines was carried out by two researchers (First author and Second author) independently on the organization’s website (https://pva.org/) until end of 2019. The search keywords were “spinal cord injury” and “guideline”. A keyword search and manual search were performed consecutively. The guidelines retrieval process followed the PRISMA flow algorithm.¹⁵ Inclusion criteria were: (i) literature relates to spinal cord injury or SCI-related complications; (ii) literature meets the guidelines standard of the National Guidelines Clearinghouse.¹⁶

AGREE II Instrument

The AGREE II tool contains 23 items in six domains and two overall assessment items. The six domains are: Scope and Purpose (three items), Stakeholder Involvement (three items), Rigour of Development (eight items), Clarity of Presentation (three items), Applicability (four items), and Editorial Independence (two items). Each item is rated on a 7-point scale (1—strongly disagree to 7—strongly agree). The two overall assessment items comprise: item (1) “rate the overall quality of the guideline” scored on a 7-point scale’ and item (2) “would recommend the guideline for use” scored as one of three responses “yes,” “yes with modifications,” and “no.”

Evaluation of the Guidelines

Each guideline was assessed by a panel of four appraisers (First author, Second author, Third author/Fourth author, and Sixth author), all of whom were orthopedic surgeons and either native English speakers or able to evaluate medical English literature proficiently. A minimum of four appraisers rated each guideline; reliability of the AGREE II tool was previously found to be greater with four assessors than with two or three.¹⁷ The appraisers each completed the online training tools required by AGREE II before guideline assessment.¹¹ No communication between appraisers occurred during the rating process. Data analysis was performed after completing the evaluation of all PVA guidelines.

Statistical Analysis

The calculation method to obtain scores for each domain is to summate the scores of all items in the domain and record this as a percentage of the maximum possible score for the domain.¹¹ Mean values (and 95% CIs) for all raters were calculated. Although domain scores can be used to compare different guidelines and to help determine whether guidelines should be recommended, the AGREE II tool does not set a minimum domain score, nor does it define the boundary criteria for identifying the quality of the guidelines. The aforementioned decisions are made by the user. According to existing research reports, the domain score criteria we used were: <40% very low quality, 40%–59% low quality, 60%–79% acceptable quality, and ≥80% good quality.^18,19

Scores obtained for individual items within domains were calculated using the same method as for domain scores. Since the purpose of the AGREE scale is to emphasize and encourage best practice, we took the view that domains that fail to reach the “acceptable” or “good” quality threshold should not be the focus of critique. Consequently, although all PVA domain scores are presented, item scores are only displayed and discussed for domains that reach “acceptable” or “good” quality.

Appraiser scores for item 1 of the overall assessment section were used to calculate an overall score for each guideline by the same method as for domain scores. Correlation between the overall scores and guideline publication date was calculated using Spearman’s test. Interrater reliability of domain scores was assessed using intraclass correlation (ICC) coefficient. Based on the 95% CI of the ICC estimate, ICC values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 are indicative of poor, moderate, good, and excellent reliability, respectively.²⁰

RESULTS

A total of 12 guidelines fulfilled the inclusion criteria. These were numbered in alphabetical order of the guide file name and evaluated by four appraisers (Table I). The scores of each domain in the 12 guidelines after evaluation by AGREE II criteria are shown in Table II. All 12 PVA guidelines are categorized as having “acceptable” or “good” quality in the following domains: “Scope and Purpose,” “Rigour of Development,” and “Clarity of Presentation”; 10 also achieved “acceptable” or “good” quality in the “Stakeholder Involvement” domain; only 2 PVA guidelines achieve this in “Applicability” and “Editorial Independence” domains.

Mean domain score is highest for “Clarity of Presentation” (80.1%), followed by “Scope and Purpose” (78.8%), “Rigour of Development” (68.4%), and “Stakeholder Involvement” (63.7%). Mean domain scores for “Applicability” (53.0%) and “Editorial Independence” (28.5%) are lower (Table II).

Four domains exceeded the threshold for “acceptable” quality (mean score of 60%)

“Scope and Purpose” domain

Items 1–3: “The overall objective(s) of the guideline is (are) specifically described,” “The health question(s) covered by the guideline is (are) specifically described,” and “The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described.” Information relating to these items is usually found in narrative form written by the Chair of the Consortium Steering Group in the preface. Additional information may be found in the introduction and the headings of sections describing evidence for recommendations.

“Stakeholder involvement” domain

Items 4–6: “The guideline development group includes individuals from all relevant professional groups,” “The views and preferences of the target population (patients, public, etc.) have been sought,” and “The target users of the guideline are clearly defined.” Information may be found among panel members and contributors. Target users are often identified in the preface.

“Rigour of development” domain

Items 7–12: “Systematic methods were used to search for evidence,” “The criteria for selecting the evidence are clearly described,” “The strengths and limitations of the body of evidence are clearly described,” “The methods for formulating the recommendations are clearly described,” “The health benefits, side effects, and risks have been considered in formulating the recommendations,” and “There is an explicit link between the recommendations and the supporting evidence.” This information can be found in the guideline development section and methodology section.

Items 13 and 14: “The guideline has been externally reviewed by experts prior to its publication” and “A procedure for updating the guideline is provided.” If present, information may be found among the list of expert reviewers and in the preface.

“Clarity of presentation” domain

Items 15–17: “The recommendations are specific and unambiguous,” “The different options for management of the condition or health issue are clearly presented,” and “Key recommendations are easily identifiable.” This information is found in the summary of recommendations and in the sections describing evidence for recommendations.

The quality of evaluation of three items in these four domains fell below the 60% threshold for acceptability; item 5 “the views and preferences of the target population (patients, public, etc.) have been sought,” item 13 “the guideline has been externally reviewed by experts prior to its publication,” and item 14 “a procedure for updating the guideline is provided.”

The overall scores of all PVA guidelines are above the “acceptable” level, with a mean score of 73.2% (95% CI: 70.4%–75.9%) (Table II). Overall scores are positively correlated with the date of publication (Spearmans’s r_s = 0.754, P = .0046).

ICC values for each PVA guideline ranged from 0.96 to 0.99 (Table III), denoting excellent interrater reliability among appraisers.

DISCUSSION

The rate of SCI among military personnel is approximately double the highest incidence found in national population estimates at 430–560 per million person-years. Overall, up to 8% of soldiers wounded in modern wars had an SCI, and the incidence appears to be increasing; 1.0%–1.2% in the Korean, Vietnam, and Gulf wars (1950–1990) and 11.1% in the Afghan and Iraq conflicts (2000–2010). Spine-related disability is currently responsible for over 14% of medical discharge from U.S. Military service.²¹

The PVA represents not only the interests of U.S. Military servicemen and veterans but also seeks to “set the standard in Clinical Practice Guidelines for healthcare practitioners at all levels and all those who care for individuals living with Spinal Cord Injury and Disease.”⁴ In support of these goals, the PVA developed and published 12 clinical practice guidelines on the diagnosis and management of SCI and related complications. Yaman et al.¹² evaluated the quality of guidelines published by the North American Spine Society; however, their clinical assessment mainly focused on degenerative diseases and spinal surgery. Other studies^13,14 have evaluated the guidelines for the rehabilitation of brain trauma and nerve injury. Ours is the first study to use the AGREE II instrument to assess the quality of PVA clinical practice guidelines on SCI.

Reliability

ICC of overall guideline scores in our study is between 0.96 and 0.99, confirming excellent interrater reliability, comparing well with the original psychometric analysis of the scale.¹⁷

Domains Reaching Acceptability Threshold

We found the highest mean score for the domain “clarity of presentation,” with mean scores of a further three domains (“scope and purpose,” “stakeholder involvement,” and “rigour of development”) also exceeding the 60% threshold. The members of the PVA Consortium for Spinal Cord Medicine include health professionals and methodologists in addition to multiple invited experts appropriate to each guideline who have “made developing guidelines their mission.”²² Within the domain “stakeholder involvement” the item “the guideline development group includes individuals from all relevant professional groups” scored highly (83.2%). Widening participation in guideline development increases both the range of specialist opinion and credibility of the guideline.²³ Items 5 and 14 failed to reach acceptability threshold in any of the guidelines; finding patients willing and able to provide input into guideline production has been found to be difficult elsewhere, for example, the UK National Institute for Health and Care Excellence’s Public Involvement Program often fails to garner patient responses to new guidelines. Updating is important due to occasional rapid changes in the field. PVA guidelines have clearly been regularly updated, but the procedure for doing so was difficult to identify for assessors.

Domains Not Reaching Acceptability Threshold

Two domains “applicability” and “editorial independence” scored below 60% in all but one guideline. The applicability of a guideline is key to its success but may be highly dependent on structural factors within health systems and, therefore, requires separate and special consideration. Some guideline developers have added audit tools, including extractable spreadsheets to assist clinicians with the assessment of impact and outcomes. The domain “editorial independence” was added to the AGREE II scale as a new construct in 2010. The two most recent PVA guidelines published since 2016 achieved improved scores in this domain. Overall guideline evaluation scores correlate with the year of publication, suggesting a process of continuous PVA guideline audit and quality improvement.

Overall Score

Regarding overall evaluation, the AGREE II manual does not describe how to perform quantitative scoring.¹² Previous studies have applied domain score calculation methods to calculate mean scores for the item “rate the overall quality of this guideline” without reference to the item “recommendations of the guideline for use.”^23,24 In others, assessors have scored this based on the average rating given to the six domains.^25,26 In our study, we gave no instructions to reviewers about scoring overall recommendations. Overall scores for all PVA guidelines exceeded the 60% threshold for acceptability, and two most recent guidelines were close to 80%, the highest quality threshold. No guideline was “not recommended” by any assessor.

Limitations

Our study has several limitations. First, AGREE II remains a subjective evaluation tool. Second, the method for calculating a consensus-derived overall guideline score using AGREE II is established neither by developer instructions nor by precedence in the literature. Consequently, the method we chose may be considered arbitrary. However ICC for the overall score of each PVA guideline in our study is between 0.96 and 0.99, which demonstrates that the evaluations of the four appraisers were highly reliable.

Summary

Our consensus is that the PVA guidelines for the diagnosis and management of SCI-related conditions should be recommended for clinical practice as all PVA guidelines demonstrated either acceptable or good methodological quality. Ongoing quality improvement of PVA guidelines was demonstrated over time.

ACKNOWLEDGMENT

None declared.

FUNDING

None declared.

CONFLICT OF INTEREST STATEMENT

None declared.

REFERENCES

Author notes