What matters for people with brain cancer? Selecting clinical quality indicators for an Australian Brain Cancer Registry

Abstract

Background

The goal of a clinical quality registry is to deliver immediate gains in survival and quality of life by delivering timely feedback to practitioners, thereby ensuring every patient receives the best existing treatment. We are developing an Australian Brain Cancer Registry (ABCR) to identify, describe, and measure the impact of the variation and gaps in brain cancer care from the time of diagnosis to the end of life.

Methods

To determine a set of clinical quality indicators (CQIs) for the ABCR, a database and internet search were used to identify relevant guidelines, which were then assessed for quality using the AGREE II Global Rating Scale. Potential indicators were extracted from 21 clinical guidelines, ranked using a modified Delphi process completed in 2 rounds by a panel of experts and other stakeholders, and refined by a multidisciplinary Working Group.

Results

Nineteen key quality reporting domains were chosen, specified by 57 CQIs detailing the specific inclusion and outcome characteristics to be reported.

Conclusion

The selected CQIs will form the basis for the ABCR, provide a framework for achievable data collection, and specify best practices for patients and health care providers, with a view to improving care for brain cancer patients. To our knowledge, the systematic and comprehensive approach we have taken is a world first in selecting the reporting specifications for a brain cancer clinical registry.

No new effective treatment has been developed for childhood and adult brain cancer for decades.¹ The brain cancer health burden remains disproportionately high compared to many other cancers.² We have unpublished data showing that in Australia, there is considerable variation in care of brain cancer patients³ and anecdotally this also occurs internationally. Optimizing currently available treatments is likely to improve outcomes and survivorship for patients with primary brain cancer.

To determine treatment patterns that are associated with best patient outcomes, an Australian Brain Cancer Registry (ABCR) will be developed to identify, describe and measure the impact of clinical variation. This project is led by Brain Cancer Biobanking Australia (BCBA)⁴ with a multidisciplinary working group representing consumers, neurosurgery, neuro-oncology, cancer epidemiology, and health informatics. The ABCR will collect data on key variables including pathology, treatment, and outcomes. In the long term, the registry will enable registry trials and data linkage to biospecimens for translational research.

The purpose of this current activity was to identify a set of clinical quality indicators (CQIs) as part of the foundation stage of the ABCR development. A CQI measures the clinical management and/or outcome of care, and can be used to assess, compare and determine the potential to improve care.⁵ This study systematically selected a set of CQIs using the Delphi approach with a panel of experts and other stakeholders invited to reach a consensus.⁶ The final set of CQIs identified in this study will measure the quality of care across the entire trajectory of care for people with brain cancer.

Materials and Methods

Selection of Preliminary Set of CQIs

In consultation with Medical Librarians, a comprehensive database search was conducted on PubMed, Embase, CINAHL, and Cochrane Library in May 2019 to identify literature to extract CQIs. A full list of search terms is available in Supplementary Material 1. The database search for clinical guidelines and systematic reviews published between 2009 and 2019 yielded a total of 4552 publications. After removing duplicates (n = 1481), the remaining literature was filtered by title and abstract. Because a large number of systematic reviews were identified, the first focus was on clinical guidelines. If a certain topic was not well covered, systematic reviews and other literature were sought. An internet search was also conducted to identify clinical guidelines published as gray literature. To reflect contemporary evidence-based clinical practice, we restricted the results to clinical guidelines published since 2015.

A PICAR statement⁷ was used to form the research question and to define eligibility criteria for synthesis of brain cancer clinical guidelines. Table 1 outlines the PICAR statement for clinical guidelines related to brain cancer care for this study.

A total of 52 brain cancer-specific clinical guidelines were identified since 2015. These were screened by full-text and 28 clinical guidelines were excluded, mostly because they were procedural specifications rather than clinical guidelines, or not restricted to brain cancer. As the initial search recognized a lack of pediatric guidelines, additional searches were conducted by identifying and adding pediatric-specific subject headings and keywords (eg, child, adolescent, pediatrics) verified by a Medical Librarian. However, this process did not result in any additional clinical guidelines specific to pediatrics.

A quality appraisal was conducted for the remaining 24 clinical guidelines using the AGREE II Global Rating Scale (AGREE II GRS) instrument.⁸ A consensus definition was agreed upon for each element of the AGREE II GRS to improve the scoring process and to aid decision making and a priori it was decided to include guidelines with a mean score above 50%. Two reviewers independently carried out a quality appraisal, and 3 guidelines were excluded for poor quality, resulting in 21 clinical guidelines^9–29 for use in establishing a preliminary set of CQIs. The search processes are summarized in Figure 1.

A total of 712 evidence-based recommendations were extracted. These recommendations were then categorized and pooled under predefined brain cancer management stages (1) Diagnosis, (2) Surgery, (3) Radiotherapy, (4) Chemotherapy, and (5) Other care (eg, patient-reported outcomes, QoL, rehabilitation, palliative care). Synonymous or overlapping CQIs were collapsed into a single CQI to create a preliminary comprehensive set of CQIs. Each indicator was summarized and presented as an indicator index including details of the source(s) from which the indicator was derived, the proposed numerator and denominator of the indicator measurement, as well as the grade of evidence and recommendation. One example is included in Table 2.

The pooled CQIs were first circulated to the BCBA CQI Working Group and two roundtable discussions were held to determine the preliminary set of CQIs. During the second meeting, the CQIs were further refined and expanded, resulting in a total of 91 preliminary set of CQIs (Figure 2), including sub-indicators where similar recommendations are presented together. For example, Item S6 (Table 2) was subdivided into separate indicators for low- and high-grade glioma and ependymoma. An additional resource³⁰ was identified, which addressed time to radiotherapy and this was included. The full preliminary set of indicators, including terms and definitions, was circulated for the Delphi process and is available in Supplementary Material 2.

Delphi Process

The Delphi component of the study received ethical clearance from Sydney Local Health District Human Research Ethics Committee, New South Wales, Australia (X20-0287 and 2020/ETH01323). Medical professional societies and brain cancer consumer organizations were approached and requested to invite their members to participate in the Delphi process as brain cancer clinical experts and consumers. An invitation email outlined the project and explained that participating and completing the questionnaire was considered informed consent. A two-round modified Delphi was conducted using a survey function of an online data capturing system (REDCap).

Panelists were asked to use a Likert scale ranging from 1 (not important) to 9 (very important) to rank the importance of the CQIs. They were asked to take into consideration the literature from which the indicator was derived, as well as the strength of evidence and the level of the recommendation. The option “unable to comment” was provided in case panelists felt that they had inadequate knowledge or experience to rate a proposed indicator. Feasibility was not measured in the Delphi rounds to ensure that panelists focused on the CQIs that are most important for brain cancer quality of care regardless of the feasibility of collecting the indicator. Panelists were given 2 weeks to complete each round of the Delphi questionnaire.

Outcome Measurement

The Delphi results were analyzed after each round. The median importance and disagreement index (DI) which represents the variation in expert ratings, were calculated using the “RAND/UCLA Appropriateness Method” inter-percentile range adjusted for symmetry (IPRAS).³¹ A large variance suggests a high level of disagreement; a variance of 0 represents a complete consensus.³² CQIs with the median rating of ≥7 and DI <1 were classified as “necessary.” All CQIs with DI ≥1 in round 1 were subject to repeat voting in round 2. Following the round 2 questionnaire, results were analyzed and synthesized to present to the Working Group to further define the CQIs. Data analysis was conducted using Microsoft Excel.

Results

Delphi Panel Characteristics

Six professional/consumer societies were approached to send an invitation letter to their members. A total of 73 people completed an expression of interest form. In round 1 of the Delphi process, 52 panelists completed the questionnaire. Two panelists did not submit their questionnaire but only missed 1 and 3 questions; these responses were reclassified as “unable to comment” and these individuals were included in the analysis. The total number of panelists in round 1 was therefore 54. Round 2 was completed by 49 panelists. The panel included representatives from all Australian states and territories and participation reflected the population distribution of Australia. One overseas-based respondent, a member of an Australian-based professional society, participated in both Delphi rounds. The majority of respondents (89%-92%) were from metropolitan areas. The panel characteristics are presented in Table 3.

Delphi Results

Round 1 Delphi panelists were presented with a total of 91 CQIs which comprised 18 Diagnosis, 15 Surgery, 9 Radiotherapy, 11 Chemotherapy, and 38 Other Care categories. Round 1 Delphi resulted in the consensus of 54 CQIs as “necessary.” The remaining 37 CQIs were reconsidered in round 2 and comprised 6 Diagnosis, 7 Surgery, 2 Radiotherapy, 5 Chemotherapy, and 15 Other Care categories. Round 2 Delphi resulted in the consensus of 18 CQIs as “necessary.” In total, 72 CQIs reached a consensus after 2 rounds of the Delphi process.

Final Set of CQIs

After the second Delphi round, the BCBA working group reconvened to review the results. The consensus CQIs were grouped into domains and mapped across the brain cancer patient journey. All CQIs that achieved a median Likert score of 9 were included, then further consensus items that achieved a median Likert score of 8 were added to cover the entire patient journey and patient population (pediatric and adult). One item was duplicated (Referral to a multidisciplinary team for decision making of management and treatment) to apply both on diagnosis and on recurrence. The final list included 19 key quality reporting domains. Within each domain-specific CQIs defined particular inclusion (denominator) and outcome (nominator) characteristics to be measured, with a total of 57 CQIs to be included (Table 4).

Discussion

The registry working group convened by BCBA has established a list of core quality domains and CQIs for reporting on the quality of care that patients with brain tumors receive from the time of diagnosis to the end of life. These CQIs have been selected by a robust process interrogating current evidence and polling the opinions of clinical experts and other stakeholders. The importance of the selected CQIs can therefore be justified to government, funders, treating institutions, health practitioners, and the public.

Our call for volunteers to help select CQIs resulted in more than 70 expressions of interest and more than 50 participants completed the Delphi questionnaires. This was an appropriate number of participants as Delphi literature³³ suggests that consensus can be achieved with as little as 3-9 participants. Moreover, our participants were relatively evenly spread across the spectrum of brain tumor treating medical specialists and brain tumor patients and carers and included representatives of all major geographical regions in Australia. The Delphi method has previously been successfully applied to establish CQIs in the management of other types of cancer.^34–42 However, the panel is typically comprised of health professionals only, and inclusion of patients and carers is rare despite the increasingly recognized importance of patient-public involvement in improving health care quality.^43,44 The involvement of this representative community will provide the CQIs and the ABCR with face validity and public acceptability.

The project to establish core CQIs encountered a number of challenges. Firstly, there was a large volume of evidence regarding appropriate brain tumor management, compounded by the multiple different pathological diagnoses included under the “brain cancer” umbrella. This difficulty was overcome by taking advantage of existing reviews and limiting our scope to current guidelines rather than primary research publications. An added advantage of this approach is that the final ABCR CQIs have been both recommended by other prominent organizations and chosen by representatives of the Australian community.

The Delphi process is a well-described methodology for achieving consensus in circumstances where there is insufficient evidence for decision making by other means. We modified the Delphi process by beginning with evidence-based recommendations. One disadvantage of this was that a very high proportion of the proposed CQIs were seen to be “necessary” by the Delphi participants. As a consequence, the selection process needed to be adjusted from our predetermined cut-offs. This was achieved by selecting the CQIs with the highest median Likert ranking and the lowest spread of votes until an appropriate number and spread (treatment stages and specialty domains) of CQIs were obtained.

To our knowledge, the systematic and comprehensive approach we have taken is a world first in selecting reporting specifications in a neuro-oncology setting. Of the existing international brain tumor registries, most report on incidence, survival, and some demographics^45–47 or have ad hoc reporting based on individual research projects.^48–50 Sweden,⁵¹ Denmark,⁵² and Japan⁵³ report treatment patterns and some patient experience outcomes but it is unclear how these reporting items were selected, and many are generic cancer-related indicators. The selection of brain cancer-specific indicators is particularly important as many of the challenges for brain tumor patients are unique to brain cancer, and the consistent application of quality care may be especially important for managing rare cancers.

Determination of the core CQIs to be collected and reported by a registry is the foundation step in our project to establish an ABCR. The next step is to determine data collection processes and pilot data collection to confirm the feasibility of each of the CQIs. Many of the necessary data points are already routinely collected in some form in one or more administrative datasets and thus could be automatically electronically collected for the registry, in accordance with the Australian “National Strategy For Clinical Quality Registries.” ⁵⁴

Given the lack of existing data pertaining to brain cancer treatment pathways and their relation to outcomes, CQI data collected in the ABCR project will contribute to understanding the variation and gaps in care and will generate practice-changing evidence on the use of appropriate and effective care.

Acknowledgments

We thank Delphi participants (panelists): A/Prof Rosalind Jeffree, A/Prof Craig Gedye, A/Prof Eng-Siew Koh, Dr Hao-Wen Sim, Dr Ganessan Kichenadasse, Ms Linda Powell, Dr Ganes Pranavan, Dr Marion Mateos, Ms Leah Bloomfield, Ms Elisabeth Wightman, Mr Raymond Taylor, Ms Diane Lear, Ms Dianne Legge, Mrs Christina Boys, A/Prof Rosemary Harrup, A/Prof Andrew Cole, Mr Pedro Casas, A/Prof Helen Wheeler, Prof Jennifer Philip, Mr Peter Radic, Dr Andrew Gogos, Dr Marketa Skala, Dr Yingda Li, Dr Saeed Kohan, Dr Vanessa Perotti, Ms Maiken Ueland, Mr Paul Malouf, A/Prof Robert Smee, Dr Carolina Sandler, Ms Kelly Skelton, Ms Marion Corbett, Dr Dionee Liefman, A/Prof Sharon La Fontaine, Prof Anna Nowak, Prof Mark Rosenthal, Dr Seckin Akgul, Dr Elizabeth Ahern, Dr Bradford Moffat, Mr Flavio Nelli, Dr Sonia Brownsett, Dr Yi Chieh Lim, Dr Catherine Bettington, Clinical A/Prof Verity Ahern, Mrs Diana Andrew, Dr Arian Lasocki, Ms Amanda Griffin, Dr James Laban, A/Prof Michael Rodriguez, Dr Kimberley Docking, Dr Laveniya Satgunaseelan, Clinical Prof Richard Chye, Dr Juliet Lokan, Ms Robyn Leonard, Prof Hui Gan.

Funding

BCBA received grants from Australian Research Data Commons (ARDC) and the Brain Cancer Collective Ltd (BCC) to complete this work.

Conflict of interest statement. None declared.

Authorship statement. All authors have read and approved the manuscript.

References