Using datasets to ascertain the generalizability of clinical cohorts: the example of the European QUALity study on the treatment of advanced chronic kidney disease

Abstract

Background

Cohort studies are among the most robust of observational studies but have issues with external validity. This study assesses threats to external validity (generalizability) in the European QUALity (EQUAL) study, a cohort study of people >65 years of age with Stage 4/5 chronic kidney disease.

Methods

Patients meeting the EQUAL inclusion criteria were identified in The Health Improvement Network database and stratified into those attending renal units, a secondary care cohort (SCC) and a not primary care cohort (PCC). Survival, progression to renal replacement therapy (RRT) and hospitalization were compared.

Results

The analysis included 250, 633 and 2464 patients in EQUAL, PCC and SCC. EQUAL had a higher proportion of men compared with PCC and SCC (60.0% versus 34.8% versus 51.4%). Increasing age ≥85 years {odds ratio [OR] 0.25 [95% confidence interval (CI) 0.15–0.40]} and comorbidity [Charlson Comorbidity Index ≥4, OR 0.69 (95% CI 0.52–0.91)] were associated with non-participation in EQUAL. EQUAL had a higher proportion of patients starting RRT at 1 year compared with SCC (8.1% versus 2.1%; P < 0.001). Patients in the PCC and SCC had increased risk of hospitalization [incidence rate ratio 1.76 (95% CI 1.27–2.47) and 2.13 (95% CI 1.59–2.86)] and mortality at 1 year [hazard ratio 3.48 (95% CI 2.1–5.7) and 1.7 (95% CI 1.1–2.7)] compared with EQUAL.

Conclusions

This study provides evidence of how participants in a cohort study can differ from the broader population of patients, which is essential when considering external validity and application to local practice.

INTRODUCTION

The rigour of study design (internal validity) has tended to be considered of greater importance by funding bodies, researchers and journals than the extent to which the results of a study can be generalized to other situations and other people (external validity/generalizability) [1, 2]. This emphasis on internal versus external validity could explain the delay or even failure to translate research findings into healthcare improvement [3, 4]. Therefore researchers need to pay attention to external validity—also known as generalizability—when designing studies [5].

Threats to external validity can occur at several steps in study design and conduct. Centre selection can influence the case mix and introduce bias [6–8], and within centres, recruitment can threaten generalizability, with frailer patients and those with linguistic barriers less likely to take part in studies [9, 10]. Considering recruitment bias specifically, findings may not apply to the patient population that experiences the most morbidity and mortality, especially with absolute risk reduction and numbers needed to treat [11].

This study aimed to quantify threats to generalizability in the UK arm of the European QUALity (EQUAL) study, a high-quality international prospective observational cohort study of treatment in older people with advanced chronic kidney disease (CKD), by looking at baseline characteristics and outcomes of recruited participants and comparing these with non-participating patients meeting the EQUAL age and kidney function criteria in primary and secondary care [12].

MATERIALS AND METHODS

This was a prospective cohort study.

The EQUAL cohort

EQUAL is an international prospective cohort study of patients ≥65 years of age attending a nephrology clinic with an incident estimated glomerular filtration rate (eGFR) <20 mL/min/1.73 m² [12]. The EQUAL cohort in this analysis included the first 250 patients recruited into the study in the UK between 30 May 2013 and 22 October 2014, with 12 months follow-up completed by 22 September 2015, censored for death and loss to follow-up.

The Health Improvement Network cohorts

A general practice (GP) dataset called The Health Improvement Network (THIN) was used to identify patients registered in GPs who met the EQUAL inclusion criteria: age ≥65 years and incident (first) eGFR ≤20 mL/min/1.73 m² after their 65th birthday. THIN has been demonstrated to be generalizable to the UK regarding demographics and the crude prevalence of major conditions and has been validated for use in epidemiologic studies of CKD [13–15]. THIN holds longitudinal anonymized patient records for 588 of the 9458 GPs across the UK with >12 million patients, with 157 of these GPs linked with hospital episode statistics (HES).

A ‘primary care’ cohort (PCC) was defined as a representative sample of eligible cases from the THIN database not attending nephrology services. A ‘secondary care’ cohort (SCC) was defined as a representative sample of eligible cases from the THIN database attending a nephrology clinic. These two cohorts are mutually exclusive and together should provide a representative sample of all patients in primary and secondary care meeting the EQUAL age and eGFR eligibility criteria.

The start and end dates for identifying patients in THIN meeting EQUAL inclusion criteria in the PCC and SCC were 1 April 2007 (1 year after the introduction of mandatory eGFR reporting in the UK) and 31 December 2012, respectively. Between the two dates, patients would have had an incident eGFR ≤20 mL/min/1.73 m² and had the outcome of interest [progression to end-stage kidney disease (ESKD), hospitalization or death]. All patients had the opportunity for 12 months follow-up from the date they entered the study, censoring for death and loss to follow-up.

Further details relating to inclusion and exclusion criteria to identify suitable PCC and SCC patients in THIN are shown in Supplementary data, Figure S1.

Index date and laboratory data in each cohort

Laboratory data included the eGFR. The index eGFR in EQUAL was defined as the first decrease in eGFR to ≤20 mL/min/1.73 m² within the 6 months before the baseline visit (30 May 2013–22 October 2014), regardless of subsequent eGFRs. Other laboratory data included creatinine, albumin: creatinine ratio (ACR), haemoglobin, calcium, phosphate, parathyroid hormone, albumin and blood pressure (BP). In EQUAL, they were captured in the case record forms (CRFs) at their baseline visit up to 6 months + 6 weeks (222 days) after the index eGFR.

In the THIN database (additional health data codes), the index eGFR was defined as the first decrease in eGFR to ≤20 mL/min/1.73 m² after 1 April 2007. Other laboratory data for the PCC and SCC were recorded in a similar window as the blood tests recorded at baseline for patients within EQUAL. Units of measurement for the laboratory data were harmonized to those that were used in the EQUAL study.

Comorbidities and other study measurements

The Charlson Comorbidity Index (CCI) was used to quantify comorbidity [16, 17]. The CCI was captured in EQUAL in CRFs. In THIN, the comorbidity accrued up until the index date was used to calculate the CCI using the Read code to Charlson weight mapping previously validated by Khan et al. [18]. CCI was used to provide a summary of comorbidity for each patient and to adjust for comorbidity burden in the regression models. Given the skewed distribution of CCI in the study population, for adjustment in the Cox regression models, CCI was grouped into three categories (2–3, 4–5 and ≥6), in keeping with other publications [19, 20].

Other study measurements included the Townsend socio-economic deprivation score and patient medication (antihypertensives, lipid lowering and anticoagulants/antiplatelets). The Townsend score and urban classification for the PCC and SCC were included in the THIN dataset, and for the EQUAL cohort, this was mapped to the patient’s postcode using the subset of postcodes in the THIN dataset. Medication history in EQUAL was captured in the CRFs at the baseline visit. In the THIN database, this was determined using the relevant British National Formulary (BNF) codes, with prescriptions issued in the 28 days before the index date included.

Outcomes

Outcomes included patient survival at the 1-year post-index date, hospitalization, progression to ESKD and renal replacement therapy (RRT). By shifting the start of survival time as below for patients in SCC, the risk of immortal time bias and survival bias was mitigated [21]. Supplementary data, Figure S2 illustrates the fix used to negate the risk of immortal time bias. The start of survival time for patients in the PCC was the index date. For patients in the SCC under the care of a nephrologist at the time they became eligible (eGFR ≤20 mL/min/1.73 m²), the average time spent by EQUAL patients from the index date to the first study visit (116 days) was added to the index date. For patients in the SCC who were referred to a nephrologist after they became eligible, 6 weeks was added to the referral date (the date referred to a nephrologist) in addition to the average time spent by EQUAL patients from the index date to the first study visit (116 days).

Within EQUAL, hospitalizations between one study visit and the next were recorded retrospectively within the CRFs. To calculate the burden of hospitalizations in the PCC and SCC, the dataset was restricted to patients attending GPs in THIN linked to the HES.

Patients who commenced RRT (dialysis or transplantation) in the 12 months after reaching an eGFR ≤20 mL/min/1.73 m² were identified using the appropriate codes within THIN (HES). In EQUAL, the RRT modality and date of the first dialysis were captured in the CRFs.

Statistical analyses

Summary statistics were produced using frequencies and proportions for categorical variables and means, standard deviations, medians and ranges for numeric variables. The three cohorts were compared using the chi-square test for categorical data, one-way analysis of variance for normally distributed numeric data and the Mann–Whitney test for skewed numeric data.

A logistic regression model was used to identify variables that were associated with being in the EQUAL cohort and to determine if the patients in EQUAL differed from a broader population of eligible patients (SCC). We considered in the models the code 1 as participating in EQUAL and 0 as not participating in EQUAL (SCC). Univariable logistic regression models were run for each of the following explanatory variables of known clinical importance: age, gender, deprivation, urban classification, individual comorbidity, CCI, haemoglobin, albumin, BP and drug count [22].

A Cox proportional hazards regression model was used to compare all-cause mortality at the 1-year post-index date for patients in the PCC, SCC and EQUAL cohorts [23]. Confounders were chosen based on a priori knowledge of aetiological importance [18, 22, 24] and included the sociodemographic index age as 5-year age bands, sex, Townsend score and rurality, laboratory variables (haemoglobin, albumin, systolic BP and diastolic BP) and CCI. The variables were sequentially added into the model, sociodemographics followed by laboratory variables and comorbidity, so we could examine the effect of adjustment for our main exposure. As the THIN database had a greater proportion of patients living in the most affluent areas, Townsend was retained in all the multivariable models. The final models were checked for the assumption of proportionality. Confounders that were not proportional were included as a time-varying covariate (TVC).

Given the overdispersed count of hospitalizations in the three cohorts, a negative binomial regression model was used to model the number of hospitalizations with adjustment of variables as in the other regression models [25]. A hospital-free risk period was calculated for the patients in each of the three cohorts, based on the number of days during follow-up that a patient was out of the hospital (and therefore at risk of hospital admission) and the number of admissions to the hospital that were recorded during follow-up. All the models were also adjusted for the hospital-free period at risk. The output of the model was reported as an incidence rate ratio (risk of hospitalization in exposed divided by risk of hospitalization in the unexposed).

All regression analyses were restricted to patients, with 100% completeness for all variables. All analyses were performed using Stata version 13.1 (StataCorp, College Station, TX, USA).

RESULTS

There were 633 patients in the PCC, 2464 patients in the SCC and 250 patients in the EQUAL cohort. The baseline characteristics of the patients in the three cohorts are shown in Table 1. Patients in the PCC and SCC were, on average, 10 years and 3 years older than patients in the EQUAL study, respectively. There was a greater proportion of male participants in the EQUAL study (60.0%) compared with the PCC (34.8%) and SCC (51.4%).

There was a greater proportion of patients in the EQUAL study in the most deprived Townsend quintile (28.4%) compared with the PCC and SCC (11.2 and 13.6%). EQUAL participants were also more likely to be living in an urban postcode (86.4%) than patients in the PCC and SCC (72.4 and 80.3%, respectively). The range of CCI in the SCC was greater when compared with the PCC and EQUAL cohorts.

Although the overall medication burden was similar between the three cohorts, the EQUAL cohort had a greater proportion of patients on antihypertensives, lipid-lowering drugs and thromboembolic/antiplatelet drugs when compared with the SCC and PCC (Supplementary data, Table S1).

The absolute mean values of laboratory variables and BP readings were clinically similar between the three cohorts. However, there was a clinically relevant difference in ACR, with the patients in the EQUAL cohort having an ACR 2 and 8 times that of the SCC and PCC, respectively (Supplementary data, Table S2). The greater difference between the EQUAL and PCC compared with the difference between the EQUAL and SCC could potentially reflect referral to secondary care and ESKD progression.

Variables associated with participation/non-participation in EQUAL

Patients participating in EQUAL were compared with the SCC of presumed non-participants in EQUAL to explore variables that are associated with being in one cohort versus the other (Table 2). Increasing age was associated with non-participation in EQUAL, with patients ≥85 years of age having 75% reduced odds of participating. Women had 29% reduced odds of participating, and patients in the Townsend quintiles 4 and 5 had 1.6- and 3.0-fold increased odds of participating when compared with the least-deprived patients (Townsend quintile 1). An increasing comorbidity burden was also associated with non-participation in EQUAL: patients with a CCI of 4–5 and ≥6 were 30% less likely to participate compared with those with a CCI <4. Patients who were less likely to take part in EQUAL included those with heart disease (47% reduced odds), peripheral vascular disease (PVD; 42% reduced odds) and rheumatological disease (69% reduced odds). Patients with current or a history of cancer had 40% increased odds of participating.

Outcomes

Figure 1 shows the unadjusted mortality at 1 year for the three cohorts. The EQUAL cohort had a greater proportion of patients alive at 1 year (90.7%) compared with the SCC (85.0%) and PCC (69.6%) (log-rank <0.001).

Table 3 shows the output of the unadjusted and adjusted multivariable Cox regression models comparing all-cause mortality at the 1-year post-index date for patients in the PCC, SCC and EQUAL cohorts. In the unadjusted model, compared with EQUAL, the unadjusted hazard ratio (HR) of all-cause mortality was 1.7 [95% confidence interval (CI) 1.1–2.7; P = 0.02] and 3.5 (95% CI 2.1–5.7; P ≤ 0.001) in the SCC and PCC, respectively. In multivariable model 3, the HR decreased moderately upon adjustment for sociodemographics, laboratory variables and comorbidity.

Supplementary data, Table S3 shows the output of the unadjusted and adjusted negative binomial regression models comparing the number of hospitalizations at the 1-year post-index date for patients in the PCC, SCC and EQUAL cohorts. Patients in PCC [incidence rate ratio (IRR) 1.76 (95% CI 1.27–2.47)] and SCC [IRR 2.13 (95% CI 1.59–2.86)] had nearly more than twice the rate of hospital admissions compared with patients in the EQUAL cohort.

EQUAL had a higher proportion of patients starting RRT in the 1-year follow-up period after reaching an eGFR ≤20 mL/min/1.73 m² compared with those in the SCC (8.1% versus 2.1%; P < 0.001). There were no patients who started RRT in the PCC in this 1-year follow-up period.

DISCUSSION

This study examined whether patients participating in EQUAL were similar to ‘real-world’ patients with an eGFR ≤20 mL/min/1.73 m² regarding baseline characteristics, survival and hospitalization. Patients in EQUAL were more likely to be younger, male and from an urban setting compared with the PCC and SCC patients. EQUAL patients were also less likely to have cardiovascular, peripheral vascular and rheumatic diseases. EQUAL patients were more likely to start RRT and had a greater probability of being alive at 1 year compared with PCC and SCC patients. The overall better health of EQUAL patients meant that they were less likely to be admitted to hospital for illnesses.

There were decreasing odds of participation in EQUAL for every 5-year age band increase. It has been recognized that patients recruited into a study may differ from the target population and be younger and healthier than referred and registry patients [26, 27]. This is a common problem in research, with a middle-aged group of patients more likely to be enrolled in studies and patients at the extremes of ages (youngest and the oldest groups) less likely to participate [28]. Hence the study sample is less likely to include the elderly [29, 30], who have a higher burden of comorbidity and therefore higher expected mortality [31]. Such patients may also differ from younger participants regarding treatment effects. The implications of this are that ‘evidence-based’ research findings based on younger patients are applied to elderly patients with comorbidities through clinical practice guidelines [32]. Health research should therefore be conducted in the populations most affected by high disease prevalence [33]. Solutions such as liberal inclusion criteria, improved communication, reducing respondent burden, provision of travel support and data collection at home may facilitate the participation of older people in research [34, 35]. Unfortunately, despite these measures, as older patients increase as a proportion of the population, those who agree to participate in randomized controlled trials and observational studies may be less representative of the population.

Women were less likely to be represented in EQUAL in the UK, with only 40.0% of participants in EQUAL being women, compared with 48.6 and 65.2% in the SCC and PCC, respectively. A probable explanation for a lower proportion of women in the EQUAL cohort could be due to slow progression rates in women [36]. The slower progression rate means that there will be a smaller cohort of women reaching an incident eGFR of ≤20 mL/min/1.73 m² or commencing RRT. The variation in gender seen in EQUAL can be explained by the variation in the incidence of CKD among men and women, with a higher incidence of CKD in women but a lower incidence of progression to ESKD requiring RRT [37]. A large European registry study by Antlanger et al. [38] assessed sex-specific differences in RRT incidence and prevalence using data from nine countries and showed that the incidence and prevalence rates were consistently higher in men than women. The recruitment of women in research studies is an essential issue for researchers. Medical research results cannot be extrapolated between genders, as the pathophysiological process varies. For example, cardiovascular disease and some of the cancers are affected by hormones. As a result, much of our understanding of illnesses and its treatments are based on research conducted disproportionately with men [39]. Alternatively, women are no more likely than men to decline to participate in studies but are merely underrepresented in target populations [40].

In the univariable logistic regression model, higher comorbidity was associated with lower odds of participation in EQUAL. The findings of this study are consistent with prior reports in other study designs showing that patients participating in trials have better survival not only on account of being healthier, but perhaps also reflecting the better medical oversight [41–43].

There was a greater proportion of EQUAL participants starting RRT compared with SCC patients. The potential explanation for this finding could be that they represented a cohort of patients who had a quicker rate of progression of their kidney disease and therefore formed a cohort of patients who were chosen to be studied. This is necessarily not a limitation of EQUAL, but the results cannot be generalized to all patients with an eGFR <20 mL/min/1.73 m².

Patients in the PCC and SCC had nearly twice the rate of hospital admissions compared with patients in EQUAL. EQUAL hospitalization data came from the nurse-collected CRFs, whereas the THIN data came from the HES linkage. It could be that the HES linkage identified more hospital admissions. An alternate explanation for this finding could be attributed to the source of the hospitalization data.

In observational studies, the classification errors, selection bias and uncontrolled confounders and the uncertainty introduced by these types of biases are seldom quantified. When designing a study, incorporating a comparison between the experimental and eligible study population at the same time would enhance understanding of the generalizability of future studies. This was done in the North American Atherosclerosis Risk in Communities study, where generalizability was examined by nesting study patients in communities covered by broad surveillance [44]. Alternatively, embedding trials/studies within chronic disease registries will allow generalizability to be ascertained. The International Society of Nephrology International Network of CKD cohort studies initiative, which includes 12 prospective cohort studies and two registries covering 21 countries, will play a significant role in understanding the generalizability of current and future CKD research [45]. Accrual to clinical trials is an initiative created to improve the efficiency of clinical trials by effectively identifying eligible participants in the recruitment stage of a study and therefore might play a crucial role in improving the generalizability at the recruitment stage of future studies [46]. Finally, using statistical techniques such as probabilistic sensitivity analysis [Episens model (st0138)] in the analysis stage may help to quantify the effect of bias and thus researchers can report results that take into account the systematic errors and hence avoid overstating their certainty about the effect under study [47, 48].

The strengths of this study are the use of routinely collected generalizable GP data (THIN) to understand the generalizability of an observational cohort study [14]. This has not necessitated the recruiting of patients who have declined to participate in a study and overcomes the complex ethical issues of re-approaching patients who have already refused to take part in a study. In the era of ‘big data’, research using routinely collected data offers more significant potential and has underpinned research in recent years [49]. The strengths of GP data are that they are population-based and are derived from a representative subset of the population [14, 50].

There were several limitations to this study. Identification of the appropriate comparison control group was crucial to an inference of the study, as any observational design will always be limited by unmeasured confounding [51]. Although this study did not directly assess the generalizability of EQUAL data by understanding the differences between EQUAL agreed and EQUAL declined patients, routinely collected data has shown the differences in EQUAL patients and patients in secondary care meeting the same eligibility criteria. There is also the potential for multiple biases as a result of differences in data capture methods between THIN and EQUAL and resultant misclassification of the THIN subjects.

This article provides empirical evidence concerning how participants in a carefully conducted observational cohort study differ from the broader population of patients that they are intended to represent. Older and sicker patients were less likely to be recruited into EQUAL in the UK, and this was supported by follow-up data on health outcomes, with patients in EQUAL more likely to be hospitalized and alive at 12 months. This selection pattern is likely to be found in most observational studies of chronic diseases. These issues can be overcome by designing observational studies to be embedded within disease registries or by using novel statistical techniques in the analysis.

SUPPLEMENTARY DATA

Supplementary data are available at ndt online.

ACKNOWLEDGEMENTS

The authors would like to thank all the patients and health professionals participating in the EQUAL study. We would also like to thank the local investigators. Funding was received from the European Renal Association–European Dialysis and Transplant Association (ERA-EDTA); Svenska L€akares€allskapet (SLS-248981, SLS-503991); the Stockholm County Council (20140020), Njurfonden (Sweden); the Italian Society of Nephrology; the Dutch Kidney Foundation (SB 142); the Young Investigators grant in Germany and the National Institute for Health Research (NIHR) in the UK.

CONFLICT OF INTEREST STATEMENT

C.W. reports grants from Sanofi; personal fees from Sanofi, Takeda, Chiesi, Amicus and Idorsia; grants from Idorsia and Boehringer-Ingelheim; personal fees from Lilly, Merck Sharp & Dohme, Mundipharma, Glaxo Smith Kline, Boehringer-Ingelheim, AstraZeneca, Bayer, Reata, Akebia and Triceda, outside the present work. M.E. reports personal fees from Astellas, AstraZeneca and Vifor Pharma and non-financial support from Baxter Healthcare, outside the submitted work. K.J.J. reports grants from ERA-EDTA, during the conduct of the study. F.J.C. reports grants from the NIHR and Kidney Research UK and personal fees from Baxter, outside the submitted work. The rest of the authors have no conflicts of interest to report.

REFERENCES