https://orcid.org/0000-0002-2772-2316
Abstract
Digital collection of patient-reported outcome measures [PROMs] is largely unexplored as a basis for follow-up for patients with ulcerative colitis [UC]. Our aim was to develop a model to predict the likelihood of escalation of therapy or intervention at an outpatient appointment that may be used to rationalize follow-up.
TrueColours-IBD is a web-based, real-time, remote monitoring software that allows longitudinal collection of ePROMs. Data for prediction modelling were derived from a Development Cohort, guided by the TRIPOD statement. Logistic regression modelling used ten candidate items to predict escalation of therapy or intervention. An Escalation of Therapy or Intervention [ETI] calculator was developed, and applied in a Validation Cohort at the same centre.
The Development Cohort [n = 66] was recruited in 2016 and followed for 6 months [208 appointments]. From ten items, four significant predictors of ETI were identified: SCCAI, IBD Control-8, faecal calprotectin, and platelets. For practicality, a model with only SCCAI and IBD Control-8, both entered remotely by the patient, without the need for faecal calprotectin or blood tests was selected. Between 2018 and 2020, a Validation Cohort of 538 patients [1188 appointments] was examined. A 5% threshold on the ETI calculator correctly identified 343/388 [88%] escalations and 274/484 [57%] non-escalations.
A calculator based on digital, patient-entered data on symptoms and quality of life can predict whether a patient with UC requires escalation of therapy or intervention at an outpatient appointment. This may be used to streamline outpatient appointments for patients with UC.
1. Introduction
Conventional management of ulcerative colitis [UC] pays scant attention to fluctuating symptoms. In the UK, most patients rely on outpatient-based treatment initiated by a physician at scheduled appointments, generally every 3–12 months, with demand almost universally exceeding capacity.
A significant proportion of outpatient clinic capacity is taken up by patients who are well, which consequently causes a delay for patients who require more urgent attention.1–3 The Covid pandemic dramatically affected conventional outpatient services, contributing further to the dilemma of effectively managing patient cohorts according to need. This presented an opportunity to introduce alternative methods of care provision, with the aim of more efficient patient triaging.4
Symptom and quality of life [QoL] data, if entered by patients and captured electronically, have the potential to determine the need for an outpatient appointment.5,6 Patient-reported outcome measures [PROMs] are a key metric in UC,7 including symptom-based disease activity scores and QoL measures. QoL assessment evaluates social and emotional well-being and provides a patient perspective, which is a crucial component of medical decision-making.8 The international STRIDE consensus and its updates propose that improvement in QoL is the ultimate patient-reported outcome.7,9,10
Using PROMs as the basis for outpatient follow-up of chronic disease is largely unexplored.6 An exception is the Danish AmbuFlex,11 a generic system for nine diagnostic groups, that uses electronic or paper-based disease-specific PROMs data collection. By 2015, AmbuFlex had been implemented in Denmark in 15 outpatient clinics. One aim was to optimize healthcare utilization, using a traffic light system to identify those needing clinical attention, based on disease-specific algorithms derived from PROMs data. In epilepsy clinics, over 70% of outpatients were referred for PROMs data collection, generating 8256 patient-responses, with almost half [48%] requiring no outpatient appointment. Clinicians, as well as patients, reported high satisfaction with AmbuFlex. A 15-year trend analysis indicated that by 2019, AmbuFlex was used in 47 diagnostic groups covering 16 000 patients and 94 departments throughout Denmark.12 While digital patient-reported data have been explored as part of routine care for inflammatory bowel disease [IBD] in the Netherlands,2 Australia,13 and the USA,14 no such system has been introduced into routine care for IBD in the UK.
Our aim was to develop a model that could be used to predict the likelihood of escalation of therapy or intervention at an outpatient clinic appointment for patients with UC, based on regular digital data collection of PROMs.
2. Materials and Methods
2.1. Data collection
TrueColours-IBD is a web-based, real-time, remote monitoring software that allows longitudinal collection of ePROMs.15 The platform functions through e-mail prompts linked to validated questionnaires, with results held on a secure [Oxford Health] server. The questionnaires assess symptoms, QoL, and a range of PROMs derived from the International Consortium of Health Outcome Measurements [ICHOM] IBD Standard Set16 which reflect long-term outcomes that are most important to patients with IBD. Both patients and clinicians can see the longitudinal results, displayed on colour-coded [Red–Amber–Green] graphs. The software also allows date stamped pathology [blood, faecal calprotectin, endoscopy, and histopathology] results to be entered. All patients who consented to the Gastrointestinal Illness in Oxford: prospective cohort for outcomes, treatment, predictors and biobanking REC Ref:16/YH/0247 are offered the opportunity to register with the programme. All data analysed were de-identified.
2.2. PROMs
For UC, symptoms were recorded through the Simple Clinical Colitis Activity Index [SCCAI]17 following daily or weekly email prompts, and QoL through IBD Control-818 after fortnightly prompts [Supplementary Tables S1 and S2]. Feasibility testing of the TC-IBD platform indicated these intervals for symptom and QoL reporting were acceptable to patients. The SCCAI was chosen because it includes symptoms important to patients [urgency of defecation and nocturnal bowel movements] that other indices exclude. It does not require a physician’s global assessment and therefore can be readily completed by patients. It has been prospectively compared to other non-invasive indices and is among the best for validity, reliability, responsiveness, and feasibility.19,20 IBD Control-8 was selected because it is fast to complete, internally reliable, reproducible, and acceptable to patients, with validity confirmed against more complex QoL questionnaires, disease activity assessment, and Physician Global Assessment.18 IBD Control has been adopted by ICHOM and the UK IBD Registry.16 TrueColours-IBD also collects PROMs data derived from the ICHOM IBD outcomes every 3 months, but these did not form part of the analysis.21
2.3. Development Cohort
In 2016, TrueColours-IBD was piloted for patients with UC attending the John Radcliffe Hospital, Oxford. This pilot group formed the Development Cohort—66 patients, monitored for 6 months. Inclusion criteria were patients aged 18–65 years with a known diagnosis of UC of any extent or severity. Patients who had undergone colectomy and had a pouch or stoma were excluded owing to the lack of validated indices. Daily symptoms, fortnightly QoL, monthly blood tests, and faecal calprotectin [using IBDoc®] results were captured by the TrueColours-IBD platform.
2.4. Demographics
Demographic data [including age, gender, diagnosis, disease distribution, disease activity, educational status] were collected electronically at the time of TrueColours-IBD registration via a single questionnaire, derived from the ICHOM IBD Standard Set.16
2.5. Outcome measurement
For this prediction model, the outcome to be predicted was escalation of therapy or intervention resulting from an outpatient clinic appointment, defined as commencement of or any increase in therapy, including topical rectal therapies, oral mesalazine, oral prednisolone, oral budesonide, oral thiopurines, intravenous hydrocortisone, biologic medication, hospitalization, need for dietetic or surgical consultation, need for referral to another IBD-associated specialist team [rheumatology, ophthalmology, or dermatology], or intervention such as radiological investigation or an endoscopic procedure. Conversely, de-escalation of therapy was defined as cessation or any decrease in the above measures. The decision to alter therapy was made by the physician [IBD Consultant, Fellow or Gastroenterology Registrar] seeing the patient at the IBD clinic, independently of data from TrueColours-IBD. Phone calls to the IBD advice line, emails, or consultation with IBD nurses were excluded from the analysis.
2.6. Identification of escalation status by an independent reviewer
Each patient’s hospital medical file was retrospectively examined by a physician who was independent of clinical decision-making and who had no access to, or experience with, TrueColours-IBD. Each outpatient clinic appointment was documented by date and need for escalation, or de-escalation of therapy or intervention. Changes in therapy during inpatient admissions were excluded from the analysis. The identification of an escalation status was unambiguous and clearly stated in the patients’ clinical notes, necessitating only one reviewer.
2.7. Predictors
Dates of each patient’s outpatient clinic appointment were cross-matched with the corresponding TrueColours-IBD electronic record, enabling all variables collected by TrueColours-IBD on those dates to be retrieved. Variables are referred to as ‘predictors’. If predictors listed below were not available, they were coded as ‘missing’. The proximity of data collection to a patient’s outpatient clinic appointment for prediction modelling was judged by clinical relevance.
SCCAI [range 0–19, where a higher score indicates higher disease activity] within 5 days before the outpatient clinic appointment
IBD Control-8 [range 0–16, where a higher score indicates better QoL] within 2 weeks before the outpatient clinic appointment
IBDoc® faecal calprotectin [Fcal] within 4 weeks before the outpatient clinic appointment
Blood tests within 2 weeks before the outpatient clinic appointment: haemoglobin [Hb, g/dL], white cell count [WCC, ×109/L], platelet count [Plt, ×109/L], C-reactive protein [CRP, mg/L], albumin [Alb, g/L], transferrin saturation [%], and ferritin [µg/L].
Data were merged so that the date of the outpatient clinic appointment, outcome [escalation of therapy or intervention vs no escalation of therapy or intervention], and predictor data were tabulated.
2.8. Sample size
Effective sample size in prediction studies is based on the number of events, rather than the number of participants. Sample size estimates to develop multivariable models are based on the ratio of the number of outcome events [in this case, escalation vs non-escalation of therapy or intervention] to the number of items examined, referred to as events per variable [EPV].22 A minimum EPV of five is needed to avoid overfitting.23 Analysis included all ten potential items described above.
2.9. Missing data
Missing data were assumed to be missing at random and handled through multiple imputation. The function ‘aregImpute’ in the R software was used for analysis.24
2.10. Prediction model development
Data for training and testing the prediction model were derived from the Development Cohort, guided by the TRIPOD statement which acts as a checklist for robust prediction model development.25 Logistic regression was used to predict escalation of therapy by finding the model that best describes the relationship between the outcome of interest [in this case, escalation vs no escalation of therapy or intervention] and independent items [in this case, SCCAI, IBD Control-8, Fcal, Hb, WCC, Plt, CRP, Alb, transferrin saturation, and ferritin]. The outcome was defined as either escalation of therapy or intervention [coded as one] or no escalation of therapy or intervention [coded as zero]. Selection of items for inclusion in the model was based on backwards elimination, using the Akaike information criterion [AIC].26 Logistic regression was used to generate the coefficients of a formula to predict a logit transformation of the probability of the outcome in question. For practical reasons, other models, including those that only required patient-reported outcomes with no need for blood tests, were explored—SCCAI alone, IBD Control-8 alone, SCCAI plus IBD Control-8, and SCCAI plus IBD Control-8 plus Fcal.
2.11. Model performance
Performance of each model was assessed in terms of calibration and discrimination. Calibration reflects the agreement between the predictions from the model and what was actually observed. Discrimination describes how well the prediction model separates those with and without the event [in this case, escalation of therapy or intervention] and was calculated using c-statistic.
2.12. Escalation of therapy or intervention nomogram
A nomogram [graphical representation of the mathematical regression formula] was developed [see Figure 1].
ETI nomogram for SCCAI and IBD Control-8 prediction model. Instructions for use: Plot SCCAI total score on the SCCAI line and draw a line UPWARDS to the points scale and record value; Plot IBD Control-8 total score on IBD Control-8 line and draw a line UPWARDS to the points scale and record value; Add all points together; Mark this value on the Total Points line; Draw a line DOWNWARDS to the Probability of escalation line.
2.13. Escalation of Therapy or Intervention calculator
To simplify the scoring system, a tabulated system (known as the Escalation of Therapy or Intervention [ETI] calculator) was developed by converting regression coefficients to easy-to-sum integers, for each predictor used in the final model. The total score [sum of the integers] relates to the outcome probability of escalation of therapy or intervention chosen at a corresponding outpatient clinic appointment [see Figure 2].
![UC Escalation of Therapy or Intervention [ETI] calculator for SCCAI and IBD Control-8. Instructions for use: For each box, circle the score and corresponding points; Add up the results of the two points to get total points; Correspond total points with the probability of escalation. Working example: If a patient had a SCCAI of 4 and an IBD Control-8 of 12, they score 29 plus 6 points = 35 total points, corresponding to a probability of escalation of therapy or intervention of approximately 25%.](https://oup.silverchair-cdn.com/oup/backfile/Content_public/Journal/ecco-jcc/17/11/10.1093_ecco-jcc_jjad099/1/m_jjad099_fig2.jpeg?Expires=1749034214&Signature=F-TJaPqpS8j1kD2lxzLpdWytSNKdI7ssSnP9-SaaeMFNUcN~qVuO85z0sEX4G2-maX0hwqmNC6gkOEWmpJjW8u6JWSl4~ZEdQou7wkcvehgFAfyGQ3d~B3vlDa1~lZ8b48MKOXyAZ8wVntQFdvpG8KX5wAWy7LD5Gcdt5Its7wCCNxXv1RvTgrs9-OTREejTvTbXOF0szRuGmXnkHO9FssqJq3D5aKSLnRBgqNvGqAbdqJzLhCAPY8sAGV8CZ~ViuWqYVmee3bk0lNQtDXmQZNcjX~A5MvNsLwpCPKuhUael6ngoBQxLSL4Bv4YR0LlTL4X6Zvd-kNE0D1yIFjDOmQ__&Key-Pair-Id=APKAIE5G5CRDK6RD3PGA)
UC Escalation of Therapy or Intervention [ETI] calculator for SCCAI and IBD Control-8. Instructions for use: For each box, circle the score and corresponding points; Add up the results of the two points to get total points; Correspond total points with the probability of escalation. Working example: If a patient had a SCCAI of 4 and an IBD Control-8 of 12, they score 29 plus 6 points = 35 total points, corresponding to a probability of escalation of therapy or intervention of approximately 25%.
2.14. Validation Cohort
From June 2018, TrueColours-IBD was extended to all patients with UC attending IBD clinics [referred to as the Validation Cohort]. Patients with prior colectomy were excluded. Data collection in the Validation Cohort included SCCAI [daily or weekly, depending on patient preference], and IBD Control-8 fortnightly. Demographics were recorded in the same way as in the Development Cohort. Taking place between June 2018 and March 2020, a total of 538 patients’ outpatient clinic appointments were included in the Validation Cohort.
To evaluate the potential impact of using the UC ETI calculator in the real world, dates of each patient’s outpatient clinic appointment in the Validation Cohort were cross-matched with the corresponding TrueColours-IBD electronic record. Adequate entry of SCCAI [within 5 days of outpatient clinic appointment] and IBD Control-8 [within 2 weeks] was required to qualify for application of the ETI calculator. Letters from each clinic appointment were assessed by an independent clinician for any escalation decisions. The accuracy of the ETI calculator was then tested in patients whose ETI score indicated a ≤5% probability of escalation of therapy [a total ETI score of ≤13, Figure 2]. A low threshold was chosen to ensure that as few as possible escalations would be missed. Sensitivity, specificity, positive predictive value [PPV], and negative predictive value [NPV] were calculated. The records of patients who scored ≤5% probability of escalation yet received an escalation of therapy or intervention were carefully examined and escalations documented. De-escalations were scrutinized in patients who scored within the ≤5% threshold.
3. Results
3.1. Demographics
Clinical details of patients in the Development and Validation Cohorts are shown in Table 1. There were no significant differences between the two groups.
Demographics of Development and Validation Cohorts for the Escalation of Therapy or Intervention prediction model for patients with ulcerative colitis.
| Demographic and clinical data | Development Cohort [n = 66] | Validation Cohort [n = 538] |
|---|---|---|
| Gender, n [%] | ||
| Male | 29 [44%] | 242 [45%] |
| Female | 37 [56%] | 296 [55%] |
| Median age, years [IQR] | 40 [32, 49] | 41 [30, 47] |
| Median disease duration, years [IQR] | 5 [1, 11] | 7 [1, 10] |
| Disease extent, Montreal, n [%] | ||
| E1 | 14 [21%] | 127 [24%] |
| E2 | 27 [41%] | 194 [36%] |
| E3 | 25 [38%] | 217 [40%] |
| Current biologic use, n [%] | 28 [42%] | 170 [32%] |
| Disease activity at entry, SCCAI, n [%] | ||
| Remission | 25 [38%] | 319 [59%] |
| Mild | 28 [42%] | 138 [26%] |
| Moderate | 12 [18%] | 74 [14%] |
| Severe | 1 [2%] | 3 [1%] |
| Unknown | 0 [0%] | 4 [1%] |
| Education level, n [%] | ||
| None | 3 [5%] | 16 [3%] |
| Primary | 3 [5%] | 4 [1%] |
| Secondary | 19 [28%] | 168 [31%] |
| Tertiary | 41 [62%] | 328 [61%] |
| Unknown | 0 [0%] | 22 [4%] |
| Demographic and clinical data | Development Cohort [n = 66] | Validation Cohort [n = 538] |
|---|---|---|
| Gender, n [%] | ||
| Male | 29 [44%] | 242 [45%] |
| Female | 37 [56%] | 296 [55%] |
| Median age, years [IQR] | 40 [32, 49] | 41 [30, 47] |
| Median disease duration, years [IQR] | 5 [1, 11] | 7 [1, 10] |
| Disease extent, Montreal, n [%] | ||
| E1 | 14 [21%] | 127 [24%] |
| E2 | 27 [41%] | 194 [36%] |
| E3 | 25 [38%] | 217 [40%] |
| Current biologic use, n [%] | 28 [42%] | 170 [32%] |
| Disease activity at entry, SCCAI, n [%] | ||
| Remission | 25 [38%] | 319 [59%] |
| Mild | 28 [42%] | 138 [26%] |
| Moderate | 12 [18%] | 74 [14%] |
| Severe | 1 [2%] | 3 [1%] |
| Unknown | 0 [0%] | 4 [1%] |
| Education level, n [%] | ||
| None | 3 [5%] | 16 [3%] |
| Primary | 3 [5%] | 4 [1%] |
| Secondary | 19 [28%] | 168 [31%] |
| Tertiary | 41 [62%] | 328 [61%] |
| Unknown | 0 [0%] | 22 [4%] |
IQR, interquartile range; Montreal classification: E1 = proctitis, E2 = left-sided colitis, E3 = extensive colitis; SCCAI, Simple Clinical Colitis Activity Index: remission [0–2], mild [3–5], moderate [6–11], severe [≥12].
Demographics of Development and Validation Cohorts for the Escalation of Therapy or Intervention prediction model for patients with ulcerative colitis.
| Demographic and clinical data | Development Cohort [n = 66] | Validation Cohort [n = 538] |
|---|---|---|
| Gender, n [%] | ||
| Male | 29 [44%] | 242 [45%] |
| Female | 37 [56%] | 296 [55%] |
| Median age, years [IQR] | 40 [32, 49] | 41 [30, 47] |
| Median disease duration, years [IQR] | 5 [1, 11] | 7 [1, 10] |
| Disease extent, Montreal, n [%] | ||
| E1 | 14 [21%] | 127 [24%] |
| E2 | 27 [41%] | 194 [36%] |
| E3 | 25 [38%] | 217 [40%] |
| Current biologic use, n [%] | 28 [42%] | 170 [32%] |
| Disease activity at entry, SCCAI, n [%] | ||
| Remission | 25 [38%] | 319 [59%] |
| Mild | 28 [42%] | 138 [26%] |
| Moderate | 12 [18%] | 74 [14%] |
| Severe | 1 [2%] | 3 [1%] |
| Unknown | 0 [0%] | 4 [1%] |
| Education level, n [%] | ||
| None | 3 [5%] | 16 [3%] |
| Primary | 3 [5%] | 4 [1%] |
| Secondary | 19 [28%] | 168 [31%] |
| Tertiary | 41 [62%] | 328 [61%] |
| Unknown | 0 [0%] | 22 [4%] |
| Demographic and clinical data | Development Cohort [n = 66] | Validation Cohort [n = 538] |
|---|---|---|
| Gender, n [%] | ||
| Male | 29 [44%] | 242 [45%] |
| Female | 37 [56%] | 296 [55%] |
| Median age, years [IQR] | 40 [32, 49] | 41 [30, 47] |
| Median disease duration, years [IQR] | 5 [1, 11] | 7 [1, 10] |
| Disease extent, Montreal, n [%] | ||
| E1 | 14 [21%] | 127 [24%] |
| E2 | 27 [41%] | 194 [36%] |
| E3 | 25 [38%] | 217 [40%] |
| Current biologic use, n [%] | 28 [42%] | 170 [32%] |
| Disease activity at entry, SCCAI, n [%] | ||
| Remission | 25 [38%] | 319 [59%] |
| Mild | 28 [42%] | 138 [26%] |
| Moderate | 12 [18%] | 74 [14%] |
| Severe | 1 [2%] | 3 [1%] |
| Unknown | 0 [0%] | 4 [1%] |
| Education level, n [%] | ||
| None | 3 [5%] | 16 [3%] |
| Primary | 3 [5%] | 4 [1%] |
| Secondary | 19 [28%] | 168 [31%] |
| Tertiary | 41 [62%] | 328 [61%] |
| Unknown | 0 [0%] | 22 [4%] |
IQR, interquartile range; Montreal classification: E1 = proctitis, E2 = left-sided colitis, E3 = extensive colitis; SCCAI, Simple Clinical Colitis Activity Index: remission [0–2], mild [3–5], moderate [6–11], severe [≥12].
3.2. Prediction model development [Development Cohort]
In the Development Cohort, the median (interquartile range [IQR]) number of outpatient clinic appointments per patient was three [IQR 1, 4], with 208 clinic appointments in 66 patients over 6 months. Independent review found that 62/208 of these appointments resulted in escalation of therapy or intervention. Ten predictor items were used to develop the prediction model. See Supplementary Table S3 for median [IQR] values, odds ratios, and percentage of missing data for each predictor item. The EPV was 6.2, indicating sufficient sample size. When all ten predictors were included in the backwards elimination model using the AIC as the pre-specified stopping rule, four significant predictors of escalation or therapy or intervention were identified: SCCAI, IBD Control-8, Fcal, and Plt. The best performing model was SCCAI plus IBD Control-8 plus Fcal plus Plt. Calibration, discrimination, and bias-corrected c-statistics of all models are summarized in Table 2.
Summary of prediction models with corresponding bias-corrected c-statistics.
| Model | Calibration Intercept, slope | Discrimination c-statistic | Bias-corrected c-statistic |
|---|---|---|---|
| IBD Control-8 alone | 0.00 [−0.38 to 0.38] 1.00 [0.74 to 1.26] | 0.90 [0.84 to 0.94] | 0.86 |
| SCCAI alone | 0.00 [−0.41 to 0.41] 1.00 [0.72 to 1.28] | 0.86 [0.78 to 0.91] | 0.90 |
| SCCAI plus IBD Control-8 | 0.00 [−0.42 to 0.42] 1.00 [0.73 to 1.27] | 0.90 [0.84 to 0.94] | 0.90 |
| SCCAI plus IBD Control-8 plus Fcal | 0.01 [−0.47 to 0.48] 1.09 [0.78 to 1.40] | 0.95 [0.91 to 0.97] | 0.94 |
| SCCAI plus IBD Control-8 plus Fcal plus Plt | 0.02 [−0.53 to 0.48] 1.12 [0.80 to 1.45] | 0.96 [0.92 to 0.98] | 0.94 |
| Model | Calibration Intercept, slope | Discrimination c-statistic | Bias-corrected c-statistic |
|---|---|---|---|
| IBD Control-8 alone | 0.00 [−0.38 to 0.38] 1.00 [0.74 to 1.26] | 0.90 [0.84 to 0.94] | 0.86 |
| SCCAI alone | 0.00 [−0.41 to 0.41] 1.00 [0.72 to 1.28] | 0.86 [0.78 to 0.91] | 0.90 |
| SCCAI plus IBD Control-8 | 0.00 [−0.42 to 0.42] 1.00 [0.73 to 1.27] | 0.90 [0.84 to 0.94] | 0.90 |
| SCCAI plus IBD Control-8 plus Fcal | 0.01 [−0.47 to 0.48] 1.09 [0.78 to 1.40] | 0.95 [0.91 to 0.97] | 0.94 |
| SCCAI plus IBD Control-8 plus Fcal plus Plt | 0.02 [−0.53 to 0.48] 1.12 [0.80 to 1.45] | 0.96 [0.92 to 0.98] | 0.94 |
SCCAI = Simple Clinical Colitis Activity Index, Fcal = IBDoc® Faecal Calprotectin, Plt = platelets.
Summary of prediction models with corresponding bias-corrected c-statistics.
| Model | Calibration Intercept, slope | Discrimination c-statistic | Bias-corrected c-statistic |
|---|---|---|---|
| IBD Control-8 alone | 0.00 [−0.38 to 0.38] 1.00 [0.74 to 1.26] | 0.90 [0.84 to 0.94] | 0.86 |
| SCCAI alone | 0.00 [−0.41 to 0.41] 1.00 [0.72 to 1.28] | 0.86 [0.78 to 0.91] | 0.90 |
| SCCAI plus IBD Control-8 | 0.00 [−0.42 to 0.42] 1.00 [0.73 to 1.27] | 0.90 [0.84 to 0.94] | 0.90 |
| SCCAI plus IBD Control-8 plus Fcal | 0.01 [−0.47 to 0.48] 1.09 [0.78 to 1.40] | 0.95 [0.91 to 0.97] | 0.94 |
| SCCAI plus IBD Control-8 plus Fcal plus Plt | 0.02 [−0.53 to 0.48] 1.12 [0.80 to 1.45] | 0.96 [0.92 to 0.98] | 0.94 |
| Model | Calibration Intercept, slope | Discrimination c-statistic | Bias-corrected c-statistic |
|---|---|---|---|
| IBD Control-8 alone | 0.00 [−0.38 to 0.38] 1.00 [0.74 to 1.26] | 0.90 [0.84 to 0.94] | 0.86 |
| SCCAI alone | 0.00 [−0.41 to 0.41] 1.00 [0.72 to 1.28] | 0.86 [0.78 to 0.91] | 0.90 |
| SCCAI plus IBD Control-8 | 0.00 [−0.42 to 0.42] 1.00 [0.73 to 1.27] | 0.90 [0.84 to 0.94] | 0.90 |
| SCCAI plus IBD Control-8 plus Fcal | 0.01 [−0.47 to 0.48] 1.09 [0.78 to 1.40] | 0.95 [0.91 to 0.97] | 0.94 |
| SCCAI plus IBD Control-8 plus Fcal plus Plt | 0.02 [−0.53 to 0.48] 1.12 [0.80 to 1.45] | 0.96 [0.92 to 0.98] | 0.94 |
SCCAI = Simple Clinical Colitis Activity Index, Fcal = IBDoc® Faecal Calprotectin, Plt = platelets.
All models performed well. For practical reasons, it was decided that a model capturing both symptoms and QoL that could be entered remotely by the patient without the need for faecal calprotectin or blood tests was the most pragmatic and therefore the optimal choice. Despite the bias-corrected c statistic suggesting that a single predictor model using SCCAI might have worked as well, a two-component model including QoL data via IBD Control-8 was preferred because it was able to capture a more holistic view of a patient as opposed to symptoms alone. No model can be expected to have 100% predictive value, so the 4% loss of discrimination by excluding biological data [Fcal and Plts] was considered acceptable for clinical purposes.
3.3. The ETI nomogram
The nomogram [Figure 1] allows precise data to be entered to calculate the likelihood of treatment escalation or intervention in an individual patient.
3.4. The ETI calculator
The final model for the ETI calculator for UC is shown in Figure 2.
3.5. Application of the ETI calculator in the Validation Cohort
Between June 2018 and March 2020, 538 patients with UC [Table 1] had a total of 1188 outpatient appointments; TrueColours-IBD data were available for 956/1188: 484/956 [51%] no escalation, 388/956 escalation [40%], and 84/956 de-escalation [9%].
3.6. Escalations
Using a probability of escalation of therapy or intervention threshold of ≤5% on the ETI calculator [Figure 2], 343/388 [88%] escalations were correctly identified as requiring escalation, whilst 274/484 [57%] non-escalations were correctly identified as not requiring escalation [Table 3].
Escalation of therapy or intervention using a threshold of ≤5%.
| ETI calculator probability | Escalation of therapy or intervention | No escalation of therapy or intervention |
|---|---|---|
| Predicted probability ≤5% [ETI total points ≤13] | 45 | 274 |
| Predicted probability >5% [ETI total points >13] | 343 | 210 |
| ETI calculator probability | Escalation of therapy or intervention | No escalation of therapy or intervention |
|---|---|---|
| Predicted probability ≤5% [ETI total points ≤13] | 45 | 274 |
| Predicted probability >5% [ETI total points >13] | 343 | 210 |
Escalation of therapy or intervention using a threshold of ≤5%.
| ETI calculator probability | Escalation of therapy or intervention | No escalation of therapy or intervention |
|---|---|---|
| Predicted probability ≤5% [ETI total points ≤13] | 45 | 274 |
| Predicted probability >5% [ETI total points >13] | 343 | 210 |
| ETI calculator probability | Escalation of therapy or intervention | No escalation of therapy or intervention |
|---|---|---|
| Predicted probability ≤5% [ETI total points ≤13] | 45 | 274 |
| Predicted probability >5% [ETI total points >13] | 343 | 210 |
The corresponding values of sensitivity, specificity, PPV, and NPV of escalation of therapy or intervention and their corresponding 95% confidence intervals are summarized in Supplementary Table S4. The area under the receiver operating characteristic curve [AUC ROC] is displayed in Figure 3.
ROC curve of the ETI calculator.
Of the 45/388 [12%] patients who had treatment escalated or an intervention planned when their ETI calculator probability of escalation was ≤5%, five related to topical 5-aminosalicylic acid [5-ASA] commencement, six to oral 5-ASA commencement, seven to immunosuppressant commencement [five azathioprine and two methotrexate], six to biologic commencement, two to biologic dose increase, five to biologic switch, three to iron infusion, four to dermatology/rheumatology referral, and seven to endoscopy referral. In total, 16/45[36%] patients had been taking prednisolone or budesonide within 1 month of the ETI assessment so may have had their ETI score artificially lowered, most commonly in the immunosuppressant commencement group [6/7] followed by biologic commencement [4/6], endoscopy referral [3/7], oral 5-ASA commencement [2/6], and biologic switch [1/5].
Of the 84 patients in total who had de-escalation of therapy, 43/84 [52%] of these patients fell within the ≤5% threshold of escalation. Six pertained to topical 5-ASA frequency reduction, two to topical 5-ASA cessation, 13 to oral 5-ASA dose reduction, three to oral 5-ASA cessation, three to azathioprine dose reduction, 14 to immunosuppressant cessation [13 azathioprine and one methotrexate], and two to biologic dose reduction.
4. Discussion
The current study shows that PROMs can be integrated into decision-making and could be used to rationalize outpatient clinic follow-up of patients with UC. Many PROMs for IBD have been used in research, but this is the first time that they have been considered as part of an interventional tool that could help manage routine follow-up for patients with UC in the National Health Service. The current findings demonstrate the potential efficacy of the TrueColours-IBD PROMs and ETI calculator model in effectively and safely managing large patient cohorts based on clinical need. These assertions are supported by the fact that when applying the ETI calculator in the Validation Cohort, a large pool of patients whose ETI scores were assessed fell within the ≤5% probability of escalation bracket [n = 319], with only 45 [14%] of these patients in fact requiring an escalation of therapy or intervention and the remaining 274 [86%] either receiving no escalation [n = 231] or a de-escalation [n = 43] of therapy. The accuracy of the ETI calculator in correctly identifying those patients needing an escalation of treatment or intervention is highlighted by the strong sensitivity and NPV statistics. These statistics are bolstered further by taking into account those patients whose ETI score may have been artificially lowered by prednisolone or budesonide use in the month prior to ETI assessment.
Patient perception of TrueColours-IBD with UC is strongly positive,27 but this is the first time that the tool has been tested with a view to outpatient resource utilization in IBD. The Ambuflex PROMs system has shown promise in this respect, finding that 48% of 8256 PROM responses indicated no further contact with the clinic was required in patients with epilepsy.11 Meanwhile, specifically in an IBD context, the myIBDcoach research team found that significantly fewer outpatient visits were required in patients randomized to the myIBDcoach monitoring group than a standard care group.2 The current findings extend these approaches by virtue of establishing thresholds for appointment decision-making through mathematical modelling, based solely on validated indices designed for patients with UC. Crucially, in both Ambuflex and myIBDCoach, threshold values for decision-making were predefined and thus susceptible to subjectivity on behalf of the clinicians defining them, unlike the likelihood of escalation thresholds of the ETI calculator.
The current study has potential implications for practice, the most notable being to assess the efficacy of the ETI calculator in light of the changing landscape for outpatient care following on from the SARS COV-2 pandemic. The need for active monitoring and intervention for patients via digital PROMs has been recognized in the care of COVID-19 patients,28 but has not translated to other disease groups where traditional outpatient follow-up protocols have been interrupted, such as for patients with UC. For example, in light of reduced hospital footfall, concerns of suboptimal clinical management of cancer patients have been raised.29 With tools such as TrueColours-IBD, supplemented by the ETI calculator, these concerns may be alleviated by strategic allocation of limited outpatient resources to those patients most in need. Future research must therefore determine whether the ETI calculator can be as effective a model of resource allocation in practice as it has indicated in the current retrospective analysis.
However, there are limitations to this study. First, the data collated come from a cohort of patients who are predominantly highly educated. IBD cohorts from other regions are likely to differ, which may affect the efficacy and impact of PROM-guided care decisions. External validation by means of applying the ETI calculator to an IBD patient cohort outside of Oxford is therefore required. Furthermore, the ETI calculator was only formulated for patients with UC, since TC-IBD was originally developed for this patient group to establish the value of longitudinal monitoring in a disease with validated clinical, endoscopic, and histological indices. Application of the calculator for patients with Crohn’s disease is in progress. There was also an unintended disparity in the number of outpatient appointments per person between the development [208/66] and validation [1188/538] cohorts, although clinical demographics were similar [Table 1]. This reflected a change in clinical practice, since the policy during the Development Cohort [2016] was to follow-up patients on biological therapy every 2 months, while this had lapsed by the time of the Validation Cohort [2018–2020] to a clinically orientated follow-up interval.
When applying the ETI calculator in the ‘Validation Cohort’, a small number of patients had de-escalations of therapy which would have been missed if a clinic appointment was deferred at the ≤5% ETI threshold. Reassuringly, over 50% of these potential missed de-escalations involved 5-ASA treatment rather than medications higher up the therapeutic chain. Our primary goal was to develop a tool for detecting escalation of therapy or intervention, since this has most clinical relevance and it is those appointments that should be prioritized. Timing of an appointment is usually less critical for patients who have no change in therapy or de-escalation. The tool is not intended for unlimited application to prioritize appointments; in practice, a limit to the number of times the calculator is used can be set, or the guidance over-ridden. Safety netting in the form of guaranteed 18- to 24-monthly outpatient follow-up regardless of one’s ETI score could ensure that these de-escalations are addressed. Finally, given the observational nature of the study, subsequent assessment of the safety of outpatient appointment deferral using the ETI calculator in real-time will be necessary with regard to subsequent disease recurrence or hospitalization. Nevertheless, there is also an opportunity to evaluate the arbitrary intervals between outpatient appointments that differ between doctors.
This study demonstrates that systematic PROMs collection and analysis may improve clinic resource allocation in IBD management. Future research needs to relate PROMs assessment to other treat-to-target goals in IBD care such as those in the STRIDE and SPIRIT consensus,7,9,10 the development of intelligent decision-support systems, and health economics analyses. This will ultimately influence whether stakeholders endorse PROMs assessment and its potential to underpin more efficient models of care.
Conference Presentation
European Crohn’s and Colitis Organisation [ECCO] conference 2019, 2020 and 2021.
Funding
This work was supported by the National Institute for Health Research [NIHR] Oxford Biomedical Research Centre [BRC], Oxford Academic Health Science Network [AHSN], the Norman Collisson Foundation, AbbVie, Buhlmann Laboratories, Janssen, Lilly, Pfizer, and Takeda.
Conflict of Interest
ST has received grants from AbbVie, Janssen, Lilly, Pfizer, Takeda, and the Norman Collisson Foundation; has received consulting fees from Atlantic, AstraZeneca, BMS, Cosmo, Ferring, GSK, Janssen, Lilly, Mestag, Pfizer, Sensyne, and Takeda; has received payment for lectures from BMS, Ferring, Lilly, Pfizer, and Takeda; has received payment for expert testimony from Takeda; has received support for attending meetings and/or travel from Lilly and Takeda; and has participated on a Data Safety Monitoring Board or Advisory Board for BMS, Janssen, Galapagos, Lilly, Pfizer, Sanofi, and Takeda. TC has received consulting fees from Sermo Inc, Techspert, IQVIA, and M3 Global Research; has received payment or honoraria for lectures, presentations, speakers bureaus, manuscript writing, or educational events from Janssen and Tillotts Pharma; and has received support for attending meetings and/or travel from Dr Falk Pharma. LW has received royalties from Springer Publishers and received honoraria from Takeda UK, Takeda Europe, Ferring Singapore, and AbbVie NZ. AK has received grants from GSK and the NIHR. GC has received a grant from Cancer Research UK; and is a Statistical Editor for the British Medical Journal, an Associate Editor for Statistics in Medicine, Research Integrity and Peer Review, on the Editorial Board of the Journal of Clinical Oncology and co-founded and is Editor-in-Chief of the journal Diagnostic & Prognostic Research. AW has received grants from AbbVie, Buhlmann Laboratories, Ferring, Galapagos Janssen, Lilly, Pfizer, Taked,a and the Norman Collisson Foundation; has received consulting fees from AstraZeneca, BMS, Ferring, Janssen, Lilly, Pfizer, and Takeda; has received payment for lectures from Abbvie, BMS, Falk, Ferring, Lilly, Pfizer, and Takeda; has received support for attending meetings and/or travel from Janssen; and has participated on an Advisory Board for Abbie and Janssen. All other authors declare no competing interests.
Acknowledgements
We acknowledge support from the National Institute for Health Research [NIHR] Oxford Biomedical Research Centre [BRC], Oxford Academic Health Sciences Network [AHSN], the Norman Collisson Foundation, AbbVie, Buhlmann Laboratories, Janssen, Lilly, Pfizer, and Takeda. We are grateful to our patients, nursing and specialist colleagues in the Translational Gastroenterology Unit, Nuffield Department of Experimental Medicine, Oxford. We acknowledge the contribution of Oxford IBD Cohort Investigators: Adam Bailey, Beth Bird-Lieberman, Oliver Brain, Barbara Braden, Jeremy Cobbold, Emma Culver, James East, Alessandra Geremia, Lucy Howarth, Paul Klenerman, Simon Leedham, Rebecca Palmer, Fiona Powrie, Astor Rodrigues, Jack Satsangi, Alison Simmons, and Peter Sullivan.
Author Contributions
AW, GC, and ST conceived the study. AW, LM, TPC, and ST wrote the first draft of the manuscript. AW, GC, and AK completed the statistical analysis. LM, JW, RK, AT, MH, and DMS oversaw and implemented data collection. AW, LM, and ST have verified the underlying data. AW, LM, TC, and RK contributed to the revision. AW, LM, TPC, ST, RK, GC, KS and AK edited the first draft of the manuscript. All authors had full access to all the study data and contributed to the interpretation of the data and editing of the final manuscript. All authors have seen and approved the final text and accept responsibility to submit for publication.
Data Availability
De-identified data sets and outpatient appointment outcome coding are available upon individual reasonable request to the corresponding author following publication. Requests will be reviewed on a case-by-case basis and subsequent investigator support will not be provided. Enquiring investigators pledge not to attempt to re-identify individuals or share the data with a third party.
