Prioritizing Measures That Matter Within a Person-Centered Oncology Learning Health System Open Access

Abstract

For more than a decade, learning health systems (LHS) have been promoted as a means to improve health-care quality and health outcomes and narrow the divide between research and practice (1). LHS can optimize care processes, facilitate patient and clinician engagement in research, evaluate comprehensive data from existing sources, use data to support real-time decision making, and embed clinical trials into care delivery (2-7).

Rapid growth in oncology LHS has been supported by advances in the ability to harness data from multiple sources to quickly understand the impact of interventions on health and well-being of people with cancer, identify opportunities for improvement in care processes, and support public reporting (8-14). Despite this growth and the recognized need for rigorous measurement of LHS impact (15,16), a gap remains in identifying specific measures that can evaluate the longitudinal impact of an oncology LHS within a health-care system.

Although numerous performance indicators exist, measurement sets often narrowly focus on impact of treatment on clinical outcomes (14), subpopulations of people living with cancer (17), or single domains of quality (18,19). The onus has fallen on individual systems to identify key indicators of an effective LHS based on their focus of interest.

The clinical value compass (20) offers a framework for measuring a balanced set of critical process and outcome indicators that have the greatest importance to multiple stakeholders in evaluating health-care quality and value. The objective of our study was to use a multistakeholder process to define key domains and prioritize measures within each domain to create a clinical value compass for an emerging person-centered oncology LHS.

Methods

We conducted a 3-round modified Delphi process (21,22) to identify a balanced measure set to evaluate the progress and impact of a person-centered oncology LHS. The modified Delphi process was conducted between July and September 2020, with oversight by a 19-member Data, Measurement, and Scholarship Workgroup representing diverse stakeholders within the health system (researchers, people living with cancer, clinicians, leaders).

Setting

This work occurred in a National Cancer Institute–designated comprehensive cancer center in northern New England, serving a largely rural catchment area. The cancer center treats approximately 32 000 patients annually and includes 17 interdisciplinary clinical oncology groups serving discrete populations.

At the time of this undertaking, the cancer center and its larger health system and affiliated health services research organization were preparing to launch The Promise Partnership Oncology LHS to advance the health system’s strategic plan. The Promise Partnership LHS has 3 primary aims: promoting continuous improvement of health service quality, advancing cancer research, and supporting cancer care with and for people living with cancer. It is designed to optimize care processes by fostering mutually beneficial partnerships between people living with cancer, their families and support networks, and health professionals. Initial projects are designed to enhance care experiences and joy and fulfillment in work among health professionals.

Recruitment

We used purposive sampling to identify a multidisciplinary group of stakeholders to participate in the modified Delphi process with the intent of achieving input from 6 stakeholder groups (Table 1). Potential participants were selected from the health system, cancer center, health services research organization, people living with cancer and family members, and external partners. No participation incentives were provided.

Domain and Measure Identification

The initial measurement framework was informed by the condition-agnostic clinical value compass domains (20,23) and complementary measurement domains that reflect the LHS ecosystem (16,24). Seven domains were identified a priori: 1) clinical outcomes; 2) functional health outcomes; 3) experience of care; 4) cost, resource use, and financial indicators; 5) team well-being and joy in work; 6) learning culture and community; and 7) scholarly engagement and productivity. An eighth domain was added following round 1 of the Delphi process, reflecting diversity, equity, inclusion, and belonging.

Three rounds of blinded voting occurred in July, August, and September 2020 via online ballots (Qualtrics, www.qualtrics.com). First-round ballots included an extensive list of subdomains within each domain. Second- and third-round voting explored, then narrowed, candidate measures. The research team provided each participant with comprehensive supplementary materials to support voting decisions and was available by phone or e-mail to answer questions. Each round of voting was followed by a 50-minute virtual video meeting to discuss voting results, creating opportunities for participants to advocate for lower-ranked measures and contribute alternate measures that may have been overlooked. Discussions were recorded with participants’ knowledge and consent. We used descriptive statistics to analyze survey results using SPSS (version 26.0).

Ballot Content

Ballot 1 (see the Supplementary Materials, available online) prioritized measurement subdomains within 7 LHS domains. Fifty-three subdomains were presented based on a prototype value compass for serious illness care (23), the quadruple aim (25), the Institute for Healthcare Improvement’s Whole System Measures (26), and a targeted review of measurement within each domain. Participants were asked to rank importance of subdomains on a 1-9 scale (1 = extremely unimportant; 9 = extremely important). To account for potential ceiling effects in Likert ratings, participants were asked to identify the 2 most important subdomains within each domain. Ballot 1 was distributed via email and available for 7 days. Median time of completion was 17 minutes.

Ballot 2 (see the Supplementary Materials, available online) identified and ranked measures within each subdomain that advanced from Ballot 1. Subdomains advanced to round 2 based on a combination of mean score, number of respondents categorizing the subdomain as a priority, and prioritization during facilitated discussion. The following sources were consulted to identify potentially relevant measures for each subdomain: National Quality Forum (27), Consumer Assessment of Health Plans Study surveys (28), National Committee for Quality Assurance (29), Lown Institute Hospitals Index (30), US News and World Report cancer-specific metrics (31), Commission on Cancer Accreditation Measures (32), Quality Oncology Practice Initiative (33), Institute for Healthcare Improvement Whole System Measures white paper (26), New Hampshire State Cancer Registry (34), and Measures of Person-Centered Coordinated Care (35). Measures were also identified through targeted literature reviews, input from the panel, and measures available from the cancer center or health system. Candidate measure inclusion criteria included being valid, reliable, sensitive to change, feasible to measure, and aligned with values and priorities of the LHS. In domains with limited valid measures, conceptual definitions of measurement areas were included. Respondents were asked to rank a revised list of subdomains according to importance, then rate measures using a 1-5 Likert scale (1 = not at all important; 5 = extremely important). Ballot 2 was distributed via email and available for 7 days. Median time of completion was 48 minutes.

Ballot 3 (see the Supplementary Materials, available online) identified recommendations for the final set of measures. Measures that advanced to Ballot 3 had an average rating score greater than 4, were identified by more than 50% of Delphi panel members as a top priority, or were promoted during Delphi panel discussions. Ballot 3 asked respondents to rank measures within each domain. Ballot 3 was distributed via email and available for 6 days. Median time of completion was 21 minutes. An additional discussion session was held with Delphi participants living with cancer and family members to discuss Ballot 3 measures.

Recommendations for the final measurement set were derived from the proportion of respondents ranking a measure in the top 3 measures within a domain. The final discussion session focused on improving the selected measure set. Synthesis of discussion informed a set of draft recommendations presented to cancer center and health system leaders.

Results

Participants

Three-quarters (76%) of individuals invited to the measurement panel agreed to participate (77 of 103). Individuals represented 6 stakeholder groups: people living with cancer and family members, clinicians and clinical staff, health-care system or cancer center leaders, quality improvement leaders, clinician and nonclinician researchers, and policy makers or funders. Ballots were completed by 94% or more of participants in each round of voting (Table 1). Discussion group attendance ranged from 64% to 77%.

Identification of Domains, Subdomains, and Measures

Round 1 voting and discussion (Supplementary Table 1, available online) resulted in 8 domains and 13 subdomains that drove selection of candidate measures for Ballot 2 (Box 1). In Round 2, 36 of 82 measures (44%) received a score of 4 or higher from two-thirds or more of participants (Supplementary Table 2, available online). Round 3 voting and discussion ranked the importance of 50 measures. Figure 1 shows prioritization of measures by domain (detailed ratings in Supplementary Table 3, available online).

Figure 1.

Prioritization of measures, according to ranking during round 3 voting. Green shading represents proportion of respondents ranking an item as first, second, or third most important within a domain. Gray shading represents proportion of respondents ranking an item as fourth through last within a domain. *Included within final recommendations. Brackets denote measures combined in final recommendation set. ^aScreening for breast, cervical, and/or colorectal cancer and/or tobacco use and cessation interventions. ^bPopulation health research includes health equity and disparity research, health promotion and disease prevention research, social determinants of health research, community health needs assessment, or community engaged research. CMS = Centers for Medicare & Medicaid Services; CoC = Commission on Cancer; COST-FACIT = COST-Functional Assessment of Chronic Illness Therapy; DEI = diversity, equity, and inclusion; ECOG = Eastern Cooperative Oncology Group; PRO-CTCAE = Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events; PROMIS = Patient Reported Outcomes Measurement Information System; QI = quality improvement; SDoH: social determinants of health.

Open in new tabDownload slide

Final Delphi panel recommendations for the value compass included 22 measures in 8 domains. Recommendations were presented to 1) the LHS data, measurement, and scholarship workgroup; and 2) senior leaders of the cancer center and health system for review and approval. These discussions resulted in 3 modifications: replacement of the claims-based total cost of care with the patient-focused economic analysis (due to availability of local data), inclusion of voluntary staff turnover (due to availability of historical data), and removal of the clinical improvement measure (due to lack of a field-defined variable within the electronic health record [EHR]).

Figure 2 depicts a value compass comprising the final set of recommended measures to support the Promise Partnership Oncology LHS. Each domain is referenced by its “point” on the compass. Cardinal points (north, south, east, and west) represent factors associated with the patient and caregiver, and ordinal points (northeast, northwest, southeast, southwest) represent the newly emerging context and ecosystem of the LHS. Each domain includes 2 or 3 measures, except experience of care, which includes 6 measures (3 perception measures; 3 process of care measures). Measures are derived from clinical data within the EHR (n = 3), patient- or caregiver-reported data that can be embedded within the EHR or experience surveys (n = 9), administrative data (n = 6), clinician or staff data (n = 2), and registry or claims data (n = 2). Descriptions of measures, data sources, and reporting strategy are shown in Table 2.

Figure 2.

An oncology learning health system value compass. AD = advance directives; ECOG = Eastern Cooperative Oncology Group; POLST: physicians orders for life sustaining treatment; PROMIS = Patient Reported Outcomes Measurement Information System.

Open in new tabDownload slide

Four domains and 14 measures are associated with the patient and caregiver. “Functional health and quality of life” includes measures reflecting overall well-being of people living with cancer and their caregivers. The Patient Reported Outcomes Measurement Information System (PROMIS) Global-10 (36) provides a global composite of health-related quality of life; Distress Thermometer (37) identifies distress among people with cancer and caregivers, and Modified Caregiver Strain Index (38) identifies impact of serious illness on caregivers. “Experience of care” includes shared decision making [collaboRATE (39)], coordination of care, and the PROMIS Self-efficacy Scale for Managing Symptoms (40) as well as access to care, documentation of advance care planning, and avoidance of aggressive end-of-life care (27). “Cost and resource use” assesses financial hardship because of illness (41,42), avoidance or delay in accessing treatment because of cost (43), and total cost of care measured through billing codes and encounter data (44). “Clinical outcomes” reflect clinician-rated functional status of the patient [ECOG Performance Status (45)] and population-level survival by cancer type and stage.

Four domains and 8 measures are associated with the LHS context and ecosystem. “Team well-being and joy in work” measures align with system-level data collection, including the Well-Being Index (46) and voluntary turnover rate among staff. “Diversity, equity, inclusion, and belonging” assesses the proportion of patients screened for social determinants of health with follow-up documented in the EHR and inclusivity among patients and workforce. “Learning culture and community” includes Improvement Readiness (47), which assesses ability of the work environment to support quality improvement, and a composite score of Commission on Cancer quality of care indicators (32). Finally, “scholarly engagement and productivity” assesses institutional commitment and support for research and academic productivity inclusive of research and teaching.

Discussion

We identified a comprehensive and balanced set of 22 measures in 8 domains to guide the development and evaluation of a person-centered oncology LHS. The LHS value compass includes measures that matter to a diverse stakeholder group and is weighted toward experiences of care. This codesigned set of measures was developed to support evaluation, improvement, and scholarship within the developing person-centered oncology LHS, allowing data to be turned into information that answers important questions and can guide future actions. This measure set has relevance for studying care from multiple perspectives, including clinical care, health-care quality and value improvement, system performance monitoring and improvement, and population health. Measures complement and expand data regularly reported to leadership and those within CancerLinQ and state cancer registries (8–10,34).

The person-centered oncology LHS value compass facilitates a deliberate approach to measurement and reporting, feeding data back to multiple levels of the system (eg, clinical program level, cancer center system level, and health system level), and evaluating whether a balanced set of longitudinal measures can help drive practice-level and system-level improvement. The measure set includes leading and lagging indicators of performance for improvement interventions and the larger clinical enterprise. Leading indicators provide information on factors or processes important to achieving desired results (eg, Well-Being Index), and lagging indicators measure current performance (eg, voluntary staff turnover).

Our work aligns with principles for designing a learning measurement system, including scanning the landscape of existing measurement and implementation efforts, engaging key stakeholders from diverse sectors, and developing criteria for measure selection to guide care delivery, improvement, and science (48). Our prioritization of person-centered measures aligns with the coproduction LHS model (4,49) and recent work by others to develop indicators of effective person-centered oncology care (18,19). Our measure set improves on existing measure sets because of its codesigned origins, which resulted in a balanced set of measures associated with the patient and caregiver, alongside measures associated with the LHS ecosystem.

The traditional clinical value compass includes 4 domains: functional outcomes, patient experience, cost and resource use, and clinical outcomes. Functional outcomes offer opportunities to gauge the effectiveness of health interventions on patient and caregiver well-being and inform future interventions. Patient experience measures support our ability to align care with what matters most to people living with cancer and their caregivers. Cost and resource use measures emphasize financial toxicity to the patient and caregiver while enabling the health system to better understand value and variation in costs. Finally, clinical outcomes provide indicators of effectiveness of cancer care, with ECOG providing a means to stratify other analyses.

The codesign process generated a novel evolution of the clinical value compass, adding 4 priority areas for measurement. Team well-being and joy in work provides short- and long-term views of employee well-being. Diversity, equity, inclusion, and belonging measures support better understanding of factors that influence access, outcomes, and disparities in care and align with the evolving concept of the quintuple aim (50). Learning culture and community measures reflect our ability to support continuous improvement and innovation within the LHS while delivering guideline-driven care. Finally, scholarly engagement and productivity aim to measure constructs associated with the institution’s support of scholarship and associated scholarly activities.

This study has both strengths and limitations. Our Delphi panel comprised a large and diverse sample of stakeholders within the cancer community and beyond who participated actively in the work. Response rates to online ballots were consistently high (>94%), and discussion groups included an average of 72% of participants. Although we view this as a strength, the large size of the group presented some challenges for ensuring that all voices were heard. To address this, discussions were conducted with smaller subgroups, which allowed time for all individuals to speak. Because this multigroup discussion strategy limited the ability for panel members to participate in every discussion, we distributed recordings and transcripts of chats to all participants after each round of discussion. A strength and key feature of our Delphi process was inclusion of 7 people living with cancer or family members on the panel. To enhance engagement and psychological safety, we conducted a series of additional email interactions and a supplemental online video meeting with this group to invite and address questions and support candid conversations. We believe this led to more balanced participation in group discussions and to a stronger set of measures. Our panel included a higher proportion of individuals in clinical and research leadership roles and lower representation of staff nurses and other members of interdisciplinary teams that support cancer care delivery. This may have influenced measures selected and may affect uptake and use of measures. To support local acceptance and implementation of measures, membership in the Delphi panel was weighted toward our local institution; more than 10% of panel members represented external institutions to support generalizability. Our measure set was larger than planned, risking loss of focus on what contributes most to a high-performing oncology LHS. We believe, however, that addition of an eighth domain (diversity, equity, inclusion, and belonging) and the higher proportion of experience of care measures represent a strength of our measure set, prioritizing previously underappreciated features of a person-centered oncology LHS.

Measures are being deployed in a phased implementation process, with widespread deployment of some measures and small-scale testing of other measures. For example, at the health system level, we have incorporated collaboRATE (a measure of shared decision making) into the Consumer Assessment of Health Plans Study Clinician and Group Surveys and oncology postvisit experience surveys and have deployed employee surveys across the cancer center to capture Well-Being Index and Improvement Readiness. At a smaller scale, we have developed EHR capability to capture advance care planning and serious illness conversations and are pilot testing electronic collection of the Distress Thermometer and social determinants of health screening in several locations.

The resulting data and reporting infrastructure will support regular feedback of performance trends across the identified measure set using an audit and feedback approach (51) and will provide trends to target improvements in different domains. We are codesigning report prototypes, which highlight primary outcomes and allow users to drill down to access detailed data. A subset of Delphi panel participants is engaged in deploying measures and assessing performance of measures, with iterative adaptations over time.

Our person-centered oncology LHS value compass was codesigned by a diverse group of stakeholders. It will contribute to a deliberate approach to both measurement and reporting, feeding data back to multiple levels of the system and evaluating whether a balanced set of longitudinal measures can help drive practice-level and system-level improvements in health outcomes and experience, health-care quality, the well-being of health-care teams, and enhanced person-centered clinical and health services research and scholarship.

Funding

This work was supported by the Gordon and Betty Moore Foundation (Grant #7485); the Robert Wood Johnson Foundation (Grant #75925); the National Cancer Institute at the National Institutes of Health (Grant P30 CA023108); and The Couch Fund at The Dartmouth Institute for Health Policy and Clinical Practice.

Notes

Role of the funder: The funders had no role in the study design, collection, analysis, or interpretation of data; in the writing of the report; or in the decision to submit this work for publication.

Disclosures: Steven Leach is on the Medical Advisory Board of Nybo Therapeutics and is cofounder and Chair of the Scientific Advisory Board for Episteme Prognostics. Eugene Nelson holds stock in Quality Data Management, Inc. which provides information services on patient experience of care and data on quality of care.

Author contributions: Conceptualization: EN, KK, AT, MBF, AVC, AMK. Data curation: AVC and AMK. Formal analysis: AVC and AMK. Funding acquisition: SR, AT, SL, MBF, MMH, KK, EN. Investigation: AVC and AMK. Methodology: EN, AMK, AVC, KD, EM, AH, MBF, AT, KK. Project administration: AMK and MMH. Supervision: EN. Visualization: AVC. Writing—original draft: AVC, AMK, KK, AT, EN. Writing—review and editing: AVC, AMK, KK, KD, SL, MBF, EM, MMH, AH, EN, SR, AT. All authors read and approved the final manuscript.

Acknowledgments: The authors wish to acknowledge the contributions of The Promise Partnership Learning Health System data, measurement, and scholarship working group for their support in guiding this work and the members of the Promise Partnership Delphi Panel, including: Albert Mulley, Amber Barnato, Amelia Cullinan, Andrew Williams, Ann Bradley, Anna Tosteson, Anne Holmes, Anne Ireland, Brant Oliver, Brock Christensen, Carol Majewski, Carolyn Kerrigan, Catherine Reed, Cathy Morrow, Corey Siegel, Daniel Jantzen, David Finley, Elissa Malcolm, Elizabeth Bengtson, Elizabeth McGrath, Elizabeth Stedina, Ellen Flaherty, Elliott Fisher, Eric Henderson, Erick Lansigan, Evan Benjamin, Gabriel Brooks, Garret Wasp, George Blike, Ira Byock, Janet Haines, Jenn Alford-Teaster, Jenna Schiffelbein, Jennifer Snide, Joanna Leyenaar, Jocelyn Chertoff, Joga Ivatury, Johanna Beliveau, John Sweetenham, Judith Rees, Julia Dalphin, Julie Kim, Karen Clements, Kathryn Kirkland, Kenneth Meehan, Konstantin Dragnev, Kris Bowen, Lawrence Dacey, Linton Evans, Malavika Govindan, Marcus Thygeson, Martha Goodrich, Mary Chamberlin, MaryAnn Stump, Matthew Mackwood, Matthew Wilson, Meredith Sorensen, Michael Calderwood, Paul Barr, Polly Campion, Ralph Jean-Mary, Rian M. Hasson, Sai Cherala, Sally Kraft, Samuel Casella, Samuel Shields, Sandra Wong, Shoshana Hort, Stephanie Tomlin, Stephen Liu, Stephen LeBlanc, Steven Leach, Susan DiStasio, Susan Reeves, Virginia Reed, Wendy Wells, Whitney Hammond, and Yolanda Sanchez.

Disclaimers: Not applicable.

Prior presentations: Not applicable.

Data Availability

The dataset is a summary of the data collected over the 3 rounds of the Delphi process. It is included in the Supplementary Materials (available online).

References