https://orcid.org/0000-0002-4729-0282
Abstract
The IBD Qorus Collaborative aims to reduce variation and increase the value of care for the adult inflammatory bowel disease (IBD) community. To evaluate the success of the collaborative, we aimed to develop a balanced set of outcome measures that reflect a multistakeholder view of value in IBD care. To achieve this, we used the Clinical Value Compass framework and engaged a mixed-stakeholder group to conduct a modified Delphi process. The end result was a 10-measure set to assess the value of IBD care.
The modified Delphi process included 3 iterative rounds of blinded voting and interactive webinar-style discussion. We recruited 18 participants for the Delphi panel, including clinicians, researchers, patients, Crohn’s & Colitis Foundation staff, and payers. Participants first identified constructs to measure, then identified the tools to measure those constructs. A literature review and environmental scan of current measures in 4 domains were performed, and relevant measures were proposed for discussion and voting in each domain. Throughout the process, participants were invited to contribute additional measures.
The modified Delphi process led to selection of 10 value measures across 4 domains: (1) patient experience; (2) functional status; (3) clinical status; and (4) health care costs and utilization. We have successfully completed a 3-stage modified Delphi process to develop a balanced set of value measures for adult IBD care. The value measure set expands upon prior efforts that have established quality measures for IBD care by adding cost and experience of care elements. This work positions IBD Qorus to better assess, study, improve, and demonstrate value at individual, system, and population levels and will inform and empower related research, improvement, and implementation efforts.
Introduction
Inflammatory bowel disease (IBD), including Crohn’s disease and ulcerative colitis, is a costly, complex, and chronic inflammatory disease of the gastrointestinal tract that affects 3 million adults in the United States and significantly affects health-related quality of life.1, 2 Despite many evidence-based practice guidelines for the management of IBD,3 significant variation exists in care processes, delivery, and outcomes across practitioners, health systems, and geographic areas.3 This variation could contribute to underutilization or overutilization of health services and poor health outcomes for people with IBD, including increased morbidity and mortality.3
Rapid collection and reporting of quality measures drives rapid improvement in the context of a learning health system, where data in a shared information environment come to life. In 2016, Nelson et. al described a registry-enabled learning health system,4 wherein patient-reported outcomes and clinical data are combined and fed back to the patient and the clinician (health care provider), centering the patient’s concerns and goals in the clinical encounter. These data are shared with a large collaborative network to accelerate learning and exchange of ideas. The data are also fed into a research registry to drive population-level research and standards of care improvement.
Learning health systems utilizing learning collaborative structures5 have been developed to improve population health outcomes in many complex, chronic health conditions, including rheumatology (RA),6 oncology,7 multiple sclerosis (MS),8 and cystic fibrosis (CF).9 National multicenter learning health systems have been developed in IBD care, including the ImproveCareNow consortium (for pediatric IBD care)10 and the Crohn’s & Colitis Foundation IBD Qorus Collaborative (for adult IBD care),2 both of which utilize quality measures to inform improvement efforts in IBD care.
Quality measures can include clinician-documented health outcomes (clinical and functional health status), patient-reported outcomes (including patient-reported outcome measures [PROMs] and patient-reported experience measures [PREMS]), and process performance measures. As described previously, these measures can be incorporated in learning health systems11 to feed information forward from people with IBD to members of their health care team. This feedforward loop of quality measures better enables the care system to predict emerging or urgent health needs, facilitate more effective partnerships between clinicians and people living with IBD, and help shared decision-making at the point of care.11 The quality measures can also be used to feedback population-level comparative performance reports to IBD care centers and professionals engaged in research and quality improvement.11
Performance measures for IBD care have been developed by the American Gastroenterological Association and utilized in the CMS Physician Quality Reporting System.12 In addition, Melmed and colleagues (2013) used the RAND appropriateness method to develop a set of priority quality improvement measures for IBD care.3 These measures have been employed to measure IBD care quality and outcomes and to inform collaborative-wide quality improvement initiatives aimed at reducing unwarranted variation in care and optimizing population health outcomes for people with IBD.
It is important to note that IBD treatment, including expensive biologics used to manage inflammation, generates considerable financial burden on people with IBD, health care systems, and payers.3 As a result, an increasing emphasis on cost and health care utilization is emerging in IBD and influencing a shift from quality-based to value-based health care models, which include both quality and cost/utilization measures.2 In response to this emerging value focus, we have endeavored to build upon the previous development of IBD quality measures that focus predominantly on clinical and functional health indicators to include cost/utilization and broader patient experience of care domains,13 with the understanding that value = quality/cost.14 These measures can then be used within the context of a learning health system to assess and improve outcomes of IBD care and the associated costs.
MATERIALS AND METHODS
We used the conceptual model of the clinical value compass framework and the general methodology of a modified Delphi process approach, both described later on. The Delphi panel composition was critical to this work in that it included a diverse national panel of key informant stakeholders, who were led through a formal priority voting process to facilitate our selection of value measures for IBD care.
Conceptual Model
Our development method follows a “concept-forward” design process described by Oliver and colleagues,15 and the whole system measures approach pioneered by the Institute for Health care Improvement (IHI).4 Concept-forward design principles argue for the development of condition, population, and context specific measures starting with the identification of key measurement domains followed by the identification and operationalization of measures within the domains.
We used Nelson’s Clinical Value Compass (CVC) as the conceptual model for our measurement domains.14 The CVC is named to reflect its similarity in layout to a compass used for navigation with 4 cardinal points: north—functional status; south—costs and utilization; east—clinical status; and west—patient experience. The CVC was developed for clinical teams and/or organizations to monitor and improve the value of health care services, defined as improvement in clinical status, functional status, and patient experience in relationship to health care expenditures and service utilization.16 The design of the compass is rooted in the understanding that the process of measurement should be intertwined with the process of care delivery so that front-line providers are involved in both managing the patient and measuring the process and related outcomes and costs.14
The CVC has been used to measure surgical value in the Spine Patient Outcomes Research Trial (SPORT),17 in the Multiple Sclerosis Continuous Quality Improvement (MS-CQI) Research Collaborative,8 and in the Cystic Fibrosis Foundation Learning and Leadership Collaborative.18 The CVC assumes that patient outcomes—health status, patient experience, and costs—are applicable to any patient population.15 We adapted the CVC for the IBD population to include the following 4 core value measurement domains: (1) personal and functional well-being; (2) pathophysiology and biology (clinical outcomes); (3) experience of care; and (4) health care costs and health care services utilization.
General Methodology
We used a modified Delphi consensus method to identify a small and balanced set of measures within the 4 cardinal points of the CVC, from a large set of candidate measures. The Delphi method, popularized by the RAND Corporation, offers an approach to reach consensus under conditions of disagreement on best measure sets and generally involves ballots.19 The Delphi method emphasizes that all voices are given equal opportunity to be heard and strives for consensus building.20 We opted to use a modified Delphi process, adding webinar-based discussion after each round of blinded voting, in consideration of practicality, timing, and feasibility. Both qualitative and quantitative information were presented to the group during discussion to bring forth all voices.
We used this modified Delphi process to identify a small set of measures (aiming for a total of 10–12 measures), which in sum could measure the value of IBD care based on the performance of the IBD Qorus Collaborative. Concurrently, it could inform strategy and evaluation at the clinical site and Crohn’s & Colitis Foundation (Foundation) levels. Our modified Delphi process began with an evidence synthesis and development of measurement domains (using the CVC model), proceeded through 3 discussion and balloting cycles to develop subdomains and candidate measures, and ended with a parsimonious set of 10 value measures.
The development of the IBD Qorus Value Compass utilized 4 main inputs4:
1) A literature review was performed by the project team to identify existing measure sets that quantify system performance and/or reduce measurement burden, including review of academic and gray literature. Experts in the field were consulted to identify additional measures.
2) The development of subdomains using the CVC framework to support identification of condition-specific candidate measures. Subdomains are categories nested within a larger global domain that focus on different aspects of a domain and cumulatively describe performance within the domain.
3) A modified Delphi19 method in which stakeholders participated in 3 iterative rounds of voting, debate, discussion, and consensus building.
4) Additional consultations (cognitive feedback testing21 with researchers, payers, patients, Foundation staff, and clinicians to provide guidance in identifying and further refining the measures).
Recruitment of Key Informant Stakeholders
The RAND/UCLA Delphi method recommends the use of a heterogenous multidisciplinary group in this process.19, 20 In collaboration with the Foundation and the lead investigators of IBD Qorus, we recruited participants for the modified Delphi process with the intent of achieving balanced input from clinicians, payers, measurement and academic experts, Foundation staff, and people living with IBD. In August 2018, we invited 30 diverse stakeholder key informants, with an anticipated 50% acceptance rate. Eighteen invitees accepted the invitation and formed the Delphi panel, consisting of 4 people with IBD (3 of whom were enrolled in the IBD Qorus program), 4 with insurance/payer expertise, 3 IBD researchers, 5 IBD physicians, and 2 Crohn’s & Colitis Foundation staff. This group (see Table 1) committed to participate in 3 rounds of Delphi voting and 3 subsequent 50-minute discussions, as outlined in the Development of Value Measures for the IBD Qorus Learning Health System Fact Sheet.
Key Informants
| Members of the IBD Qorus Delphi Panel | |
|---|---|
| People with IBD: | Researchers/Foundation: |
| Jeffrey Bank, MD | Caren Heller, MD, MBA |
| Jessica Caron, MHCDS | Mike Kappelman, MD |
| Anya Platt | Sid Singh, MD |
| Dan Sharp | Alandra Weaver, MPH |
| Payers: | Physicians: |
| Michael Belman, MD | Chris Almario, MD |
| John Murphy, MD | Ann Flynn, MD |
| MaryAnn Stump, RN, MBA, COHQ | Erica Heagy, NP |
| Nels Marcus Thygeson, MD | Gil Melmed, MD |
| Wei Ying, MBA | Samir Shah, MD |
| Members of the IBD Qorus Delphi Panel | |
|---|---|
| People with IBD: | Researchers/Foundation: |
| Jeffrey Bank, MD | Caren Heller, MD, MBA |
| Jessica Caron, MHCDS | Mike Kappelman, MD |
| Anya Platt | Sid Singh, MD |
| Dan Sharp | Alandra Weaver, MPH |
| Payers: | Physicians: |
| Michael Belman, MD | Chris Almario, MD |
| John Murphy, MD | Ann Flynn, MD |
| MaryAnn Stump, RN, MBA, COHQ | Erica Heagy, NP |
| Nels Marcus Thygeson, MD | Gil Melmed, MD |
| Wei Ying, MBA | Samir Shah, MD |
Key Informants
| Members of the IBD Qorus Delphi Panel | |
|---|---|
| People with IBD: | Researchers/Foundation: |
| Jeffrey Bank, MD | Caren Heller, MD, MBA |
| Jessica Caron, MHCDS | Mike Kappelman, MD |
| Anya Platt | Sid Singh, MD |
| Dan Sharp | Alandra Weaver, MPH |
| Payers: | Physicians: |
| Michael Belman, MD | Chris Almario, MD |
| John Murphy, MD | Ann Flynn, MD |
| MaryAnn Stump, RN, MBA, COHQ | Erica Heagy, NP |
| Nels Marcus Thygeson, MD | Gil Melmed, MD |
| Wei Ying, MBA | Samir Shah, MD |
| Members of the IBD Qorus Delphi Panel | |
|---|---|
| People with IBD: | Researchers/Foundation: |
| Jeffrey Bank, MD | Caren Heller, MD, MBA |
| Jessica Caron, MHCDS | Mike Kappelman, MD |
| Anya Platt | Sid Singh, MD |
| Dan Sharp | Alandra Weaver, MPH |
| Payers: | Physicians: |
| Michael Belman, MD | Chris Almario, MD |
| John Murphy, MD | Ann Flynn, MD |
| MaryAnn Stump, RN, MBA, COHQ | Erica Heagy, NP |
| Nels Marcus Thygeson, MD | Gil Melmed, MD |
| Wei Ying, MBA | Samir Shah, MD |
Guiding Principles
The IHI Whole System Measures approach provided primary guidance for measures selection (Martin, Nelson, Rakover & Chase, 2016):
• Balance: A measure set must address each of the 4 domains: functional status, clinical status, experience of care, and costs and utilization, yielding a patient-centered whole person perspective generated from the cumulative measurement set that in aggregate can provide system and collaborative balanced measurement views of performance. In addition, a measure set should balance the current needs of the IBD Qorus Collaborative and its future direction, allowing for cross-comparability between projects.
• Parsimony: To maintain a systems perspective, a small set of measures is required. If there are too few measures, significant dimensions may be overlooked; if there are too many, the measures cease to have targeted utility in providing strategic guidance to improvers and system leaders and can be overwhelming.
• Alignment: The IBD Qorus Value compass builds on the system performance measures developed in 2017 by the collaborative and considers additional measure sets.
• Usefulness: The measures need to be useful to the Foundation and participating Qorus sites/clinicians to drive improved performance. The measures—as a set—must signal stability, improvement, or decline and must be sensitive enough to detect changes in short time intervals to inform strategy and improvement in a timely fashion. To facilitate uptake and use, immediate usefulness was a consideration.
• Consensus: In the final selection of balanced measures, experience was leveraged from a diverse group of stakeholders including patients, clinicians, payers, Foundation staff, and health services researchers through a modified Delphi process. This modified process included virtual meetings where stakeholders were given the opportunity to defend certain positions or lobby for concepts that were not yet brought forth. Gustafson et al22 and Graefe & Armstrong23 suggest that the modified Delphi process, with opportunity for discussion and debate, may be superior to the original developed by Dalkey and Helmer.24
• Adaptability: Measures must work in the present but also need to account for the ever-changing health care landscape. To that end, the Foundation will need to periodically revisit the IBD Qorus Value Compass to ensure the measures serve the intended purpose. As conditions change, some measures may become less viable or relevant, requiring removal or replacement, and others may need to come to the forefront.
National Quality Forum guidelines were also utilized as selection criteria25. All measures must be:
Important to measure and report, to keep focus on priority areas, where the evidence is highest that measurement can have a positive impact on health care quality; specifically, focused on priority IBD areas previously established by Qorus leadership and where evidence is highest that measurement in this area of the Qorus Collaborative can have a positive impact on IBD-related health care quality.
Scientifically acceptable so that the measure, when implemented, will produce consistent (reliable) and credible (valid) results about the quality of care.
Useable and relevant, to ensure that intended users—patients, purchasers, clinicians, and policy makers specializing in and concerned with IBD care—can understand the results of the measure and are likely to find them useful for quality improvement and decision-making.
Feasible to collect, with data that can be readily available for measurement and retrievable without undue financial or time burden.26
These general guiding principles, in conjunction with the NQF guidelines, were used in initial selection of measure sets for the ballots, in addition to guiding inclusion and exclusion criteria for measures subsequently suggested by field experts and panel members.
Conduct of the Delphi Process (Fig. 1)
Three rounds of blinded voting via online ballots (Qualtrics Online Survey Software, www.qualtrics.com) occurred in September, October, and November 2018. Participants were asked to rate each potential subdomain or measure based on its importance as part of a balanced, parsimonious set of measures for the Qorus Collaborative. In rounds 2 and 3, participants were provided the blinded results of the prior round, including both numerical ratings for each subdomain or measure and qualitative comments. Each round of voting was followed by a 50-minute virtual meeting to discuss the voting results and offer participants an opportunity to contribute alternative measures, make arguments to support their measure selections, and elaborate on responses. These discussions were recorded with the participants’ knowledge and consent.
Modified Delphi process.
Two lead facilitators (BJO and EN) conducted the series of 3 modified Delphi process discussion and voting sessions, aiming to allow all stakeholders to suggest measures and provide feedback from the perspective of their constituency group. After each meeting, The Dartmouth Institute for Health Policy and Clinical Practice (TDI) disseminated a recording of the discussion to all participants. Each round resulted in the exclusion of domains/measures with limited support (ie, those with low importance ratings) and the addition of measures suggested by participants, ultimately resulting in a final set of 10 measures covering the 4 domains of the Clinical Value Compass.
Ballot 1 was designed to select 2 or 3 subdomains within each of the 4 domains of the CVC. The ballots were composed of subdomains with conceptual definitions within each of the 4 domains. Respondents were asked to rate each item using a 9-point scale ranging from 1 (extremely unimportant) to 9 (extremely important) to examine the magnitude of importance. Respondents then were asked “If you could only choose 2 of the subdomains above, which would you choose?” and were given a dropdown list of the subdomains they had rated. Ballot 1 was distributed via email to the expert panel on September 4, 2018, and closed on September 13, 2018, allowing 9 days to review and respond. Each participant was sent a unique link to the electronic ballot to track completion; on September 11, 2018, those who had not yet responded were sent a reminder email to prompt completion. Ballot 1 response rate was 100%.
Ballot 2 was designed to identify measures within each of the subdomains selected based on ballot 1. Measures were identified through literature reviews and input from experts in the panel and Qorus Leadership. Respondents were asked to rank measures within the identified subdomains. Ballot 2 was distributed via email to the expert panel on October 9, 2018, and closed on October 14, 2018, allowing 5 days in which to review and respond. Each participant was sent a unique link to track completion; on October 13, 2018, those who had not yet responded were sent a reminder email to prompt completion. Ballot 2 response rate was 100%.
Ballot 3 was designed to define the final set of measures. It included a narrowed and refined measure set, which was arrived at through panel discussions, ballot free-response feedback, and consultation with experts. Once again, respondents were asked to rank measures within the subdomains. Ballot 3 was distributed via email to the expert panel on November 1, 2018, and closed on November 8, 2018, allowing 7 days in which to review and respond. Each participant was sent a unique link to track completion; on November 6, 2018, those who had not yet responded were sent a reminder email to prompt completion. The ballot response rate was 100%.
Analysis Plan
Voting analyses had 2 purposes: (1) to describe voting results and (2) to focus and stimulate discussion during the key informant sessions. Subdomain analyses described magnitude of preference (ranking) for options under consideration and also prioritized the top 2 options, providing the key informant group and the modified Delphi process coordinating team at TDI with 2 different perspectives to inform the iterative specification and optimization of the value measure set. Ballot responses were analyzed using descriptive statistics including mean rankings, standard deviations, interquartile ranges, and percentage rankings.
ETHICAL CONSIDERATIONS
This work was determined not to be human subjects research after a thorough review of the requirements set forth by the Dartmouth College Committee for Protection of Human Subjects (CPHS).
RESULTS
Key Informant Session Participation
Virtual meeting attendance for the 3 key informant sessions was 94%, 72%, and 67%, respectively. Participants who were not in attendance notified the facilitator in advance of their absence and, if necessary, provided feedback.
Clinical Status Domain Development (Fig. 2)
For the selection of subdomains for the clinical status domain, 6 subdomains were considered: IBD-related disease activity, IBD-related complications, symptoms, signs, improvement of symptoms, and IBD-related surgery. After the first round of voting, the IBD-related disease activity and IBD-related complication domains were selected. Existing tools to measure disease activity and state were examined and discussed during 2 additional rounds of voting. The group was particularly challenged to come to consensus on a definition of IBD-related complications. Different definitions were considered, including (1) the International Consortium for Health Outcomes Measurement (ICHOM) definition27; (2) a self-defined construct that included anemia, malnutrition, and colorectal cancer; and (3) a patient-defined complication collected through open text report. Open text reports by patients would not have been feasible to analyze with the volume of data that the group anticipated collecting over time. Enumerating a limited set of complications, such as from a picklist, also did not meet the criteria for the measures to adapt with the population and placed the onus of diagnosis on the patient, which clinicians did not find appropriate. Therefore, the ICHOM definition was considered for the voting process.
Clinical status domain development.
Over the course of 3 rounds of voting and discussion, consensus was ultimately achieved to include steroid-free remission via patient-reported outcomes 2 and 3.28, 29 Patient-reported outcome (PRO) 2 and patient-reported outcome (PRO) 3 are composite scores used to measure symptom severity based on stool frequency, rectal bleeding, abdominal pain, and perceived wellbeing. Consensus was also achieved for the Physician Global Assessment, which is a brief physician-reported measure at the time of an encounter capturing the physician’s perception of disease activity on that day. These final clinical status measures can be tracked easily over time, require minimal time or burden to report, and build a balanced view of disease activity history for the patient and clinician.
Functional Health Status (Fig. 3)
Patient functional health status underwent a considerable transformation beginning with an expansive set of initial candidate measures for the 4 subdomains identified. It was then narrowed iteratively to a final consensus with 2 priority measures encompassing all 4 subdomains. A significant transition between rounds 2 and 3 and in the final panel discussion after round 3 occurred for this domain. The group came to consensus that all subdomains and measurement criteria were captured in 2 measures: (1) the patient-reported outcomes measurement information system (PROMIS) Global 1030 and (2) IBD-Control.31 This “collapsed” the subdomains developed earlier in the modified Delphi process. The result aligned with the guiding principle of parsimony, minimizing the burden of measurement on respondents and researchers.
Functional health status domain development.
Health Care Experience (Fig. 4)
Patient health care experience subdomains chosen were “meeting patient goals and expectations,” “access to care,” and “patient self-efficacy.” Consensus was achieved for the following measures: collaboRATE, Primary Care Medical Home (PCMH) Consumer Assessment of Health Care Providers and Systems (CAHPS),32 and PROMIS Self Efficacy for Managing Symptoms.33 These selections reflect prioritization of 3 elements: (1) shared decision-making (collaboRATE); (2) system-level experience of care and satisfaction; and (3) individual-level perceived coping ability in the context of a chronic disease with an uncertain trajectory (self-efficacy). This reflects a shared recognition of the importance of provider-patient communication and partnership in IBD clinical care (shared decision-making) and the importance of building self-management skills, resilience, and health confidence in people with IBD as part of the care process. The addition of CAHPS measures reflects a recognition of system-level effects on patient experience that span beyond the individual care encounter, expanding the scope of value assessment in this domain beyond the traditional patient-clinician dyad. One concern with the CAHPS measure is that it increases the number of post-visit surveys that the patient is required to complete, which could negatively impact measurement yield. Finally, it is interesting to note that the final measures set for this domain emphasizes satisfaction and a patient-centered assessment of illness and care experience both in the clinical setting and (in the case of self-efficacy) in the lives of people living and coping with IBD in their daily lives.
Healthcare experience domain development.
Health Care Cost and Utilization (Fig. 5)
Cost and utilization development presented the challenge of understanding financial burden per capita in a way that was reliable, actionable, and efficient to report. There was considerable debate concerning options including claims data, imputing costs based on averages, or patient report. All stakeholders had strong feelings about burden and feasibility. A great deal of cross-learning took place in the rich discussion rounds for this topic, as those with insurance experience advocated for relationship-building with insurance companies to access claims data across conditions, whereas clinical partners advocated for patient report. Participants with IBD shared concerns about reporting on personal financial constraints unless it could be connected to beneficial personalized action. Researchers advocated for using an imputed method as done with the cost measure in the SPORT trial.17
Costs and utilization domain development.
After the ballot and discussion of round 3, the group came to consensus on 2 patient-reported measures of health care utilization and 1 (to be determined) measure of cost-related care avoidance. Through subsequent discussion with stakeholders, a measure used to quantify financial barriers to follow-up care after acute myocardial infarction was also explored as a guide.34 Our process did not reach final consensus on a cost-related care avoidance measure. Additional development work is underway in this area.
Final Synthesis of IBD Qorus Value Measures
After completion of the modified Delphi progress, the final set of value measures was organized by domain to create the final IBD Qorus Value Compass (Fig. 6). Each value domain (clinical status, functional status, health care experience, and health care cost and utilization) includes 2 to 3 specific measures, comprising a final set of 10 IBD Qorus value measures (Fig. 6).
Final IBD value measure recommendations.
Discussion
We used a modified Delphi process and the Clinical Value Compass conceptual framework to develop a parsimonious set of high priority value measures for adult IBD care for planned use in improvement, implementation, innovation, research, and strategic planning for the IBD Qorus learning health system. This measure set builds upon prior work establishing quality measures for adult IBD care (which predominantly focuses on clinical and functional health measures) by adding cost/utilization and experience of care domains to more broadly facilitate capability for measuring value in IBD care.
Strengths of the Development Process
We completed a rigorous modified Delphi process to develop a parsimonious, balanced, and actionable value measurement set for IBD Qorus over the course of 3 months with the participation of a diverse panel of key IBD Qorus stakeholders. The use of a concept-forward design helped to avoid measurement selection bias by first defining the most important domains to measure before choosing the measures to assess each domain. The use of a “mixed methods” qualitative discussion and quantitative balloting process gave the panel multiple lenses to view the process as it progressed, and the facilitators invited all perspectives to be heard throughout the entire process. The guiding principles of feasibility, alignment, and adaptability emerged as priority criteria, suggesting that our process did not lose sight of the primary stakeholders who will benefit from the development of these new value measures—patients and their IBD care team. This contributed to consensus in choosing some measures that were already being collected by IBD Qorus, including the Physician Global Assessment, the collaboRATE measure, and emergency department visits in the past 6 months. These measures were feasible and immediately useful. Similarly, the inclusion of the PROMIS measures in 2 of our 4 value domains aligned well with data collection tools the Crohn’s & Colitis Foundation is currently using in other mission-related projects.
Limitations
We used a modified Delphi process instead of the original Delphi approach in this work, which introduced some design-based limitations. The initial selection of subdomains by a smaller set of stakeholders, which preceded the key informant sessions, could have introduced a selection bias. This could have influenced the remainder of the process by constraining the initial conceptual domains. We feel that these limitations are outweighed by the advantages introduced when using a modified Delphi approach—feasibility, timeliness, and practicality. The expectations of this effort included a tight timeline, a practical (rather than research) context, and robust engagement from a diverse group of key informants over multiple months.
We also selectively excluded a key stakeholder group from the Delphi process. In an effort to minimize bias, pharmaceutical partners were not included as key informants, with the understanding that this would exclude the potential benefit of this stakeholder group. Additionally, the key informants group was convened to select value measures for adult IBD care that would be used within the IBD Qorus collaborative. Further study would be needed to determine whether the results of this process are applicable to the majority of IBD care systems.
A final potential limitation is attrition. We received a 100% ballot return in all 3 rounds of voting. However, we observed a sequential drop in stakeholder webinar meeting participation. This dropped from 94% in discussion 1, to 72% in discussion 2, and 67% in discussion 3. We did not observe dropout of categories of participants (ie, broad stakeholder representativeness was preserved throughout), which limits the potential biasing impact. Attrition is commonly observed in similar efforts, and the greatest drop in participation was seen between sessions 1 and 2. Participation then held relatively unchanged between sessions 2 and 3. The facilitators worked to mitigate this limitation in a number of ways. First, we were notified of absences in advance of the sessions and sought the absentees’ recommendations to bring forward during the upcoming session discussion, thereby including as many voices as possible even if not present for the virtual discussion. Second, all key informants received balloting results and meeting summaries in an effort to maximize transparency of the process. Participants were encouraged to send follow-up comments via email.
CONCLUSION: DEMONSTRATING VALUE IN IBD AND OTHER CHRONIC ILLNESS LEARNING HEALTH SYSTEMS
Value measures employed systematically in a networked context, such as the IBD Qorus learning health system, can help the IBD community answer these fundamental questions: (1) Are outcomes for patients with IBD improving? (2) Is the experience of receiving care as good as it should be to meet patient preferences, needs, and expectations? (3) Are the costs of care and utilization of care rising or falling over time? and (4) For what types of patients, in which practices, and with what treatment regimens and what patterns of care are value outcomes best? Value measures can also provide rapid and ongoing feedback in visual, longitudinal, and meaningful data displays. Clinicians, patients, researchers, and payers can use this information to determine whether improvement efforts are paying off by generating quantifiably better results for people with IBD.
This effort represents the first documented effort in the IBD community to establish a set of priority value indicators for adult IBD care that can be applied and studied in the IBD Qorus Collaborative and in other IBD settings. It is unique in that these value measures were designed specifically for use in an established and growing learning health system focused on serving people with a costly, complex, chronic condition (IBD). Other organized learning health system efforts employ performance measures, but few have undertaken formalized efforts to develop specific quality and value measures to use in collaborative-level learning health systems with over 30 medical practices spread across the United States.
When equipped with appropriate measures, learning health systems can enable learning at 3 levels: (1) treatments and services that result in highest value for subsets of individuals and under specific conditions; (2) system-level learning about how to best apply evidence-based practices to obtain high value population health outcomes; and (3) national collaborative-level learning to inform the development of care pathways, personalized medicine, and predictive analytics to better understand the emerging health needs of populations, to monitor for accountability, and to more intelligently inform continuous quality improvement efforts.35, 36 It is significant to note that through the results of this effort, patient-reported outcomes (PROs), including PROMs (eg, patient-reported steroid-free remission) and PREMs (eg, collaboRATE) play a substantive role in the voice of value measurement in the IBD learning health system. This approach departs from traditional clinically dominated measurement approaches and generates a much stronger contribution from people with IBD in defining what value is in IBD care.
It is significant to note that the design of IBD Qorus was inspired, in part, by Lindblad’s Swedish Rheumatology Quality Registry, a national learning health system for rheumatology (RA), which demonstrated national-level improvements in outcomes in a collaborative for 62 RA clinics in Sweden.6 Now, through the development of IBD value measures and the ongoing efforts of IBD Qorus, the adult IBD care community is positioned to assess, research, improve, and demonstrate value at system- and population-levels in a similar fashion. As the IBD Qorus Collaborative continues to expand, it will be able to pursue the improvement of system-, regional-, and national-level value outcomes using these new value measures for IBD in the United States. This approach provides a prototype that could be adapted for other chronic condition learning health systems seeking to develop value-based care and value measurement approaches in contexts such as oncology or end-of-life care.
Supported by: This work was supported by program and development grant funding provided to the Dartmouth Institute from 2018–2020 by the Crohn’s & Colitis Foundation #3201638.
Conflicts of Interest: SAW and CH are employees of Crohn’s and Colitis Foundation. GM is a consultant for Abbvie, Boehringer-Ingelheim, Bristol-Meyers-Squibb/Celgene, Entasis, Janssen, Medtronic, Pfizer, Samsung Bioepis, Takeda, Techlab. Dr. Melmed and receives research funding from Pfizer, and Crohn’s & Colitis Foundation.
CS is a consultant/advisory board member for Abbvie, Amgen, BMS, Lilly, Janssen, Pfizer, Prometheus, Takeda; speaker for CME activities for Abbvie, Celgene, Janssen, Pfizer, and Takeda; and receives grant support from Abbvie, Janssen, Pfizer, and Takeda.
EN owns stock in QDM, Inc., a company that provides patient experience data and information services to health care systems. All other authors have no conflicts to disclose.
Acknowledgments
The authors acknowledge the significant contributions of the IBD Qorus Value Measurement Delphi Panel for lending their time and expertise to these recommendations. The authors also acknowledge the ongoing contributions of our patient partners, without whom this work would not be possible. IBD Qorus is made possible in part by the support of AbbVie, AMAG Pharmaceuticals, Eli Lilly, Helmsley Charitable Trust, Janssen Biotech, Inc., Luitpold Pharmaceuticals, Inc., Nephroceuticals LLC, Nestle Health Sciences, Pfizer, Inc., Takeda Pharmaceuticals U.S.A., Inc., and UCB.
References
