Measuring patients’ experience with renal services in the UK: development and validation of the Kidney PREM Open Access

ABSTRACT

Background

Patient experience is a recognized aspect of quality of care for people with chronic kidney disease (CKD), but current patient-reported experience measures (PREMs) only focus on dialysis care. We developed and validated the Kidney PREM to assess patients’ experience with renal services in secondary care for any CKD stage or treatment (transplant, haemodialysis and peritoneal dialysis).

Methods

We developed the Kidney PREM in two phases, informed by a multidisciplinary expert group to ensure face validity. We organized three national data collections (2016–8) to investigate item response profiles and to conduct exploratory and confirmatory analyses to assess internal consistency. We also explored content validity in cognitive interviews and evaluated test–retest reliability. Finally, we developed the Kidney PREM Short Form for more frequent measurement of patient experience to inform local service improvements.

Results

We analysed 32 959 responses across data collections, with the 2018 collection covering all 71 UK renal centres. The Kidney PREM final version consisted of 38 items grouped into 13 themes, all pertaining to one underlying dimension reflecting the construct of ‘patient experience’ with high internal consistency (Cronbach's α = 0.94). The Kidney PREM Short Form consisted of 15 items across the same 13 themes.

Conclusions

The Kidney PREM supports the collection of reliable information on patient experience that people with CKD consider relevant, regardless of CKD stage or treatment modality. Kidney PREM data have the potential to guide local and national initiatives to improve patients’ experiences with renal services in the UK and other countries.

Graphical Abstract

Open in new tabDownload slide

INTRODUCTION

Patient-reported experience has become a recognized aspect of quality of care for people with chronic kidney disease (CKD) [1, 2]. Experience refers to how people perceive their interactions with healthcare professionals and facilities [1, 3]. Measuring kidney patient experience as part of performance monitoring and audit schemes supports patient-centred approaches to evaluating and improving the quality of renal services [3–6]. In other clinical areas, patient experience has been linked to the quality of care processes and to outcomes [7, 8]. In CKD, an enhanced patient experience is associated with improved treatment adherence and outcomes [9–11].

Robustly measuring patient experience requires validated questionnaires to elicit feedback on topics such as communication with the healthcare team, information provision, social and emotional support and coordination of care [2, 3]. Reviews [2, 12] identified two validated, dialysis-specific patient-reported experience measures (PREMs): the Consumer Assessment of Healthcare Providers and Systems (CAHPS) questionnaire for in-centre haemodialysis (HD) [13, 14] and the Consumer Quality Index for in-centre HD and peritoneal dialysis (PD) [15, 16]. In addition, Sanabria-Arenas et al. [17] reported the development and validation of a Scale for Evaluation of Haemodialysis Patients’ Satisfaction (ESUR-HD).

Currently no PREMs have been developed and validated in other CKD populations, such as people living with a kidney transplant or those with advanced CKD not yet requiring kidney replacement therapy (KRT). Hence renal services lack the means to assess patient experience across the full range of treatment modalities. Although centres may have their own instruments to measure experience within specific services, the UK Kidney Association (UKKA) and Kidney Care UK commissioned the development of an instrument that could measure patient experience across entire service provisions, aiming to support evaluations of renal services and inform national and local quality improvement initiatives.

This study aimed to develop and validate the Kidney PREM, an instrument to measure patients’ experiences with renal services in secondary care for any CKD stage or treatment modality.

MATERIALS AND METHODS

Figure 1 illustrates our phased approach in developing the Kidney PREM (available in colour as online supplementary material). We consulted with a multidisciplinary expert group throughout to ensure face validity and inform item selection. The group included 13 patient and carer representatives, three health professionals, three methodologists and one policy maker; all were involved in a national programme promoting person-centred approaches to kidney care [18, 19].

Data collection across phases

In this section we describe the collection of data across all phases of Kidney PREM development. Data were integral to the evolution of the instrument, with results of each Kidney PREM version used to inform the next. We organized three annual data collections (2016–8). Each took 2 months and used the most recent Kidney PREM version then available. The UK Renal Registry (UKRR), part of the UKKA, distributed paper copies to all renal centres, focusing on England in 2016 and extending to Wales, Northern Ireland and Scotland from 2017. All CKD patients were attending outpatient clinics in secondary care settings and had responded to the Kidney PREM following an invitation from their secondary care provider. They comprised a heterogenous group, including those living with advanced CKD (CKD stages 3B, 4 and 5), those with less advanced disease requiring management of conditions such as glomerulonephritis, connective tissue disease, vasculitis and other immune-mediated diseases, and those with progressive conditions such as diabetic nephropathy and adult polycystic kidney disease. Patient guidance stated that the Kidney PREM was anonymous and would take ∼15 min to complete, assisted, if required, by a friend or family member. From 2017, the PREM was also available online, translated into Welsh (facilitated by the Welsh Renal Network) and Gujarati and Urdu [20]. Paper copies returned to the UKRR were scanned into electronic format and merged with online responses into one dataset. Responses with two or fewer items completed were rejected as scanning errors.

Phase 1: development of the Kidney PREM preliminary version

The initial version (i.e. preliminary version 1; Supplementary data, Table S1) consisted of 43 candidate items across 13 themes, selected with reference to other validated and non-validated PREMS [13, 15, 21–24] identified through an informal review of published and grey literature. The expert group commented on relevance, completeness, wording and response scales. We evaluated the initial version's item response profile and internal consistency using the 2016 data and content validity through cognitive interviews. Test–retest reliability was evaluated using a Welsh dataset collected for that purpose. Findings from phase 1 informed changes to the preliminary version 1, resulting in the Kidney PREM preliminary version 2.

Item response profile and exploratory factor analysis. We evaluated each item's response profile by calculating its mean, median and the percent missing and ‘not applicable’ values. To assess internal consistency, we conducted an exploratory factor analysis with Varimax rotation. Eigenvalues and scree plots were used to determine the number of underlying dimensions of patient experience. We considered the following widely accepted [25] fit statistics and thresholds to identify items whose removal might improve internal consistency: factor loading (<0.4), uniqueness (<0.4), item-test and item-rest correlation (<0.5), average interitem covariance (<0.15) and any increase in the dimension's Cronbach's α after item removal. Candidate items for removal were discussed during the cognitive interviews.

Test–retest reliability. To evaluate the reliability of preliminary version 1 over time, we conducted a test–retest exercise with patients from five renal centres in Wales who did not take part in the 2016 data collection. Three weeks after completing a first Kidney PREM, participants were asked to complete a second, together with a questionnaire to identify significant changes in health, treatment or service delivery since initial completion (Supplementary data, Table S2). For each dimension identified in exploratory factor analysis, we calculated test and retest mean scores and the correlation between them, excluding responses with >20% of items missing. Dimensions with correlation coefficients (r) <0.7, indicating poor reliability [26], were explored in the cognitive interviews.

Cognitive interviews. We conducted cognitive interviews to further improve content validity [27]. We invited 13 patients from four geographically convenient renal centres, using purposive sampling to achieve variation in age, ethnicity and modality [28]. The interview guide included all Kidney PREM items, focusing on those with high proportions of missing or ‘not applicable’ responses and those with suboptimal fit statistics in the exploratory factor analysis. We asked participants to respond to each item while verbalizing their thoughts, with the researcher using concurrent and immediate retrospective probing to explore potential problems with item structure and presentation (e.g. negative wording and framing) [29]. We audio recorded and transcribed all interviews before coding and analysing the transcripts to identify general, as well as item- and response-specific issues.

Phase 2: development and validation of the Kidney PREM final version

Phase 2 aimed at improving the internal consistency of preliminary version 2 by reducing the number of items, using the 2017 data. This resulted in the Kidney PREM final version, for which we confirmed internal consistency using the 2018 data.

Item response profile, exploratory factor analysis and interitem correlation. To identify candidate items for removal and revision, we analysed items’ response profiles and conducted an exploratory factor analysis to evaluate their fit statistics. We used an approach similar to phase 1, adding high item mean and low standard deviation (SD; indicating limited response variation) and discrimination (>4.5; indicating participants’ tendency to disproportionately use the scale's upper response options) as criteria for nominating items for removal. For each theme we also investigated interitem correlations, considering an r >0.85 to indicate item redundancy [25]. We presented suggestions for item removal to the expert group, who selected items for inclusion in the Kidney PREM final version, aiming to retain at least one item per theme to preserve face validity.

Confirmatory factor analysis and sensitivity analyses. To confirm the robustness and internal consistency of item selection, we conducted a confirmatory factor analysis with the 2017 data. To evaluate model fit, we primarily considered the comparative fit index (CFI) as the most informative for large sample sizes [30]. Secondary model fit statistics included χ², the root mean square error of approximation, the standardized root mean square residual and item's standardized expected parameter change (SEPC; indicating the magnitude of change if relaxing the constraint of variance). We compared a model where items were not allowed to covary with models where, within a theme, any pair of items with a modification index (MI) >100 was allowed to covary.

As some items only applied to certain patient groups (i.e. related to needling, blood tests and transport), we undertook a series of sensitivity analyses to ensure that group selection did not influence the instrument's internal consistency. For this, we repeated the confirmatory factor analysis while excluding items not applying to all participants.

Confirming internal consistency in a new dataset. We conducted a principal component analysis using the 2018 data to confirm the stability of the final version's internal consistency [30]. We used Cronbach's α as a global statistic of internal consistency, further investigating individual item statistics if α was <0.7.

Phase 3: development of the Kidney PREM Short Form

We developed a shorter version of the Kidney PREM to facilitate quicker and more frequent measurement to inform local service improvements. Items for the short form were selected from items in the final version of the Kidney PREM. To identify representative items in each theme in the final version we selected those with significant covariance based on the MI and SEPC model fit statistics. We then assessed the fit of five models with different representative questions per theme, comparing them with the fit of the model resulting from the phase 2 confirmatory factor analysis. Using this information, the expert group selected the model that represented an optimal balance between model fit and face validity.

RESULTS

Participants in the annual data collections

Table 1 shows the characteristics of participants in the three annual data collections, who together contributed 32 959 responses. Coverage of centres was 39/52 in England in 2016, 56/57 in England and Wales in 2017 and all 71 UK centres in 2018. The majority of participants were 56–74 years of age (centre range 38.4–43.9%), with good representation of those ≥75 years (centre range 25.9–28.8%); male (centre range 52.1–53.6%); and white (centre range 74.2–74.6%), with 15.3–16.5% being of minority ethnic heritage. In 2017 and 2018, <1% of valid responses were received through translated questionnaires. Most participants (centre range 76.4–81.5%) were on some form of KRT; only 6.0% were not in 2016, increasing to 15.2% (2017) and 12.0% (2018).

Preliminary versions of the Kidney PREM (phase 1)

Item response profile, factor structure and test–retest reliability. The item response profile of the preliminary version 1 showed that most items (35/42) had means of at least 3 on a scale of 1–4 with SDs <1. This implied that participants tended to use the upper two (of four) response options, which constrained response variance. Only items 34 and 37 had median scores of 2. Both were framed negatively, perhaps indicating that their direction confused participants. A total of 16 items had a ‘not applicable’ option. Around 10% of participants selected this option for 11 of these items. In most cases this suggested that items were relevant only for some patient groups (e.g. transport). In others, it may have indicated that participants did not know how to respond. Negatively framed items, those with high ‘not applicable’ responses and those with >10% missing, were selected for further exploration in the cognitive interviews. Supplementary data, Table S3 contains the response profiles and fit statistics for all items.

Exploratory factor analysis suggested one underlying factor (i.e. dimension) reflecting the construct of ‘patient experience’ with high internal consistency (Cronbach's = α, 0.904), encompassing 13 themes. Three additional factors had eigenvalues >1, but these were >10 points smaller than the eigenvalue for the first factor (Supplementary data, Table S4).

Table 1 shows the characteristics of the 296 test–retest participants. The mean total scores across all 42 items were 139.4 (SD 19.3) and 140.7 (SD 20.1) for the test and retest, respectively (r = 0.79), and 8.8 (SD 1.4) and 9.0 (SD 1.3) (r = 0.72) for the overall experience item. This implied high test–retest reliability. Reliability remained high in participants reporting a significant change or event between the test and retest compared with those who did not (Supplementary data, Tables S2 and S5) and where correlation was lower, the question (1e) related more closely to patient experience than to an event unrelated to experience (Supplementary data, Table S6).

Content validity and changes from Kidney PREM preliminary version 1 to 2. From four centres, 12 patients took part in cognitive interviews. Ages ranged from 21 to >75 years. Three were of minority ethnicity. Seven were on in-centre or home HD and five on other types of KRT or attending low clearance clinics. Supplementary data, Table S7 presents detailed findings from the cognitive interviews.

Table 2 summarizes the main changes to the Kidney PREM preliminary version 1 resulting from phase 1. These included response scales increased from 4 to 7 points with labels only for extreme response options (addressing constrained response variance), ‘not applicable’ and ‘don't know’ added as response options for all items, negative items (34 and 37) were reverse scored so they aligned with other items (higher scores implying better experience), HD vascular access cannulation added as a theme, 1 item removed and 8 added and 24 items revised to improve clarity. Some items with suboptimal response profiles and fit statistics (e.g. transport) were retained because patients considered them essential to the experience. This resulted in the Kidney PREM preliminary version 2 (Supplementary data, Table S8), consisting of 50 items across 13 themes.

Kidney PREM final version (phase 2)

Items response profile and factor structure. The response profile and fit statistics of most items in the preliminary version indicated no major issues. As in phase 1, some had high proportions of ‘not applicable’ or missing responses (e.g. transport and parking) because they did not apply to all patient groups. This affected their fit statistics. All had factor loadings ≥0.4 (Supplementary data, Table S9), were considered important by the expert group and were retained to preserve face validity.

Exploratory factor analysis confirmed the single ‘patient experience’ dimension, which again had high internal consistency (Cronbach's α = 0.977) and accounted for 78% of the observed variance.

In seven themes (e.g. diet, fluid, patient information, communication, etc.), there was a high correlation between items (Supplementary data, Table S10). Overall, we removed 11 and reworded 2 items (Table 3). This led to the Kidney PREM final version, consisting of 38 items in 13 themes and 1 overall experience question. The instrument is freely available at https://ukkidney.org/sites/renal.org/files/KQuIP/PREM%20paper_11.20.pdf, with further information on analysis and use at https://ukkidney.org/kidney-patient-reported-experience-measure. A copy of the Kidney PREM is also included in Supplementary data, Table S2.

Confirmation of internal consistency. Confirmatory factor analysis of the final version showed a moderate model fit if items were not allowed to covary (CFI 0.753). Relaxing this constraint improved model fit substantially (CFI >0.9). Sensitivity analysis confirmed that model fit was not affected by the removal of items or themes that did not apply to all patient groups, with CFIs all remaining >0.9 if items were allowed to covary (Supplementary data, Table S11).

Table 4 displays response profiles and fit statistics of all items in the final version resulting from the principle component analysis conducted using the 2018 data. The analysis confirmed high internal consistency (Cronbach's α = 0.94) with no need for further change.

Kidney PREM Short Form (phase 3)

We assessed four alternative versions of the Kidney PREM Short Form. All had an excellent model fit with CFIs >0.9 (Supplementary data, Table S12). We selected the model with the strongest face validity in consultation with the expert group (Table 5).

DISCUSSION

Summary of findings

We developed the Kidney PREM using a phased approach involving 32 959 responses across three annual data collections, informed by cognitive interviews and a multidisciplinary expert group to ensure content and face validity. The instrument is suitable for assessing the experiences of people with CKD within secondary care renal services, regardless of disease stage and treatment modality. It consists of 38 items, contributing to a single underlying dimension reflecting the construct of ‘patient experience’ with high internal consistency. Items are grouped into 13 themes and rated on a 7-point scale, with higher scores indicating better experience. The Kidney PREM Short Form consists of 15 items, across the same 13 themes, allowing quicker and more frequent measurement to inform local service improvement.

Relation to other studies

The Kidney PREM differs substantially from other established PREM measures used in the renal context [13, 15, 17] despite both CAHPS [13] and the CQ Index [15] informing its development. These instruments were developed in different healthcare systems, have different target populations and specify involvement with different professional staff groups. The Kidney PREM refers to the whole spectrum of experience of CKD patients, regardless of modality, in the context of the whole renal team, across themes including communication, information provision, scheduling and tests. The PREM has a single underlying dimension, with all items pertaining to the construct of ‘patient experience’. This is in contrast with the CAHPS and CQ Index, which divide patient experience across 3 and 10 dimensions, respectively. Hence the Kidney PREM provides an integral measure of patient experience compared with the more fragmented versions offered by the CAHPS and CQ Index.

Strengths and limitations

The Kidney PREM was validated using almost 33 000 responses across all UK renal centres, making it the largest study internationally in terms of participating patients and centres. It enables assessing experiences of all patients using renal services in secondary care, capturing the broadest possible range of experience within one instrument. Service users have contributed to its development from the start.

It has not been validated for non-English speakers, a limitation most pertinent for South Asian patients and perhaps for those of Eastern European and Somali origin. Although our data had good representation of people of Black and other minority ethnicities, the PREM's cross-cultural validity remains to be assessed. The absence of information on non-responders is a further limitation, leaving patient-level response rates unclear. In addition, underrepresented groups, including those with low literacy levels and/or poor health, may be poorly represented or even missing altogether. Exploring non-response patterns will be important to address escalating health inequalities.

Some treatment modalities were underrepresented, with a significantly higher proportion of people on HD responding than expected based on UKRR data. As data are only recorded for people receiving KRT, we cannot determine the true patient population composition. However, Kidney PREM collection for 2020 [31] was entirely online; this resulted in a proportional increase in participation by people with a kidney transplant, albeit still underrepresented. All treatment groups had a sufficiently large number of responses such that differences between modalities could be explored during almost all phases of Kidney PREM development. However, differences by treatment modality appeared marginal, especially in compared with differences between domains and treating centres. This strongly suggests that the Kidney PREM measures patients’ experiences of the service rather than the characteristics of the patient. The test–retest reliability statistics were similar among patients who did or did not report a significant event between tests, largely irrespective of event type. This provided further evidence that the Kidney PREM measures experiences of the service rather than people's experiences of their condition or circumstances.

The Kidney PREM is prone, similar to other scales of this kind, to a positive response bias, often due to the need to project a socially acceptable face or a social desirability bias [32]. We addressed this phenomenon to our best ability by modifying the response anchors to discourage extreme response options and eliminating items where positive response bias was extreme, indicating limited discrimination. Although most Kidney PREM responses on the 7-point scale still ended up in the range of 5–7, the mean scores for treatment centres and domains did vary, and in some cases were <5. There was also a small but significant number of patients who provided negative responses in the range of 1–3. This gives confidence that the ceiling effect is at least partly accounted for. Data collected during the coronavirus disease 2019 pandemic gave additional confidence that the scale is sensitive to change [31], providing evidence that scale scores are capable of discrimination.

Implications for practice

The Kidney PREM was developed as a driver for change. It is supported by UK professional and patient organisations and deployed by the renal community to inform person-centred care [33]. Kidney PREM data are available to units as a national report with analysis at the centre level, with local data available to unit clinical directors (via an interactive portal since 2019). Analysis by theme and patient characteristics can facilitate initiatives relevant to local priorities.

Kidney PREM data have also been used to compare patient experiences across UK renal centres. While our study has shown some evidence of positive response bias (i.e. clustering of responses towards the scale's higher end), subsequent analyses have demonstrated sufficient response variance to reveal significant differences between centres and across some patient characteristics [34]. Of the 13 Kidney PREM themes, transport, vascular access needling and shared decision making have been consistently identified as areas where patient experience varies unwarrantedly within and between centres. These are areas where patient experience in the UK could be improved.

Although it was developed in the UK, the Kidney PREM is suitable, with limited adaptation, for use in other countries with similar health delivery systems. Kidney PREM themes were selected to represent key service delivery indicators with relevance beyond local and national contexts. The instrument can be adapted without substantially affecting its measurement properties as long as key meaning is retained, e.g. by aligning wording of items to local terminology or by removing items not applicable to a particular healthcare context. Our sensitivity analysis showed a limited impact on internal consistency of such changes. Using the Kidney PREM may be more challenging in settings where healthcare delivery has a different cultural context and where access to healthcare is limited.

Future research

The Kidney PREM Short Form is currently used to inform local service improvement initiatives. To be useful in research, future studies should confirm its internal consistency in an external dataset. This would also allow assessment of its convergent validity by comparing short form scores with Kidney PREM data collected as part of the annual national audit.

To be useful in guiding kidney service improvement, the Kidney PREM (full and short form) must be responsive to change. Analysis has shown evidence that the Kidney PREM is able to detect differences between renal centres [34]. Results from 2020 [31], the year of the global pandemic, provide further evidence of change between years and between groups of patients. However, further research is needed to establish the link between changes in service delivery and changes in Kidney PREM scores.

CONCLUSION

We developed and validated a stable instrument to robustly measure patient experience in people receiving secondary care for CKD, regardless of disease stage and treatment modality. The full instrument has been adopted by the UKKA to collect reliable patient experience data as part of the national audit of renal services. Kidney PREM data can be used to guide local and national initiatives to improve patients’ experiences of renal services in the UK and elsewhere. Ultimately this will contribute to improving CKD care and outcomes.

ACKNOWLEDGEMENTS

We would like to thank all the kidney patients and renal centres who participated in this study. In particular, we acknowledge the members of the multidisciplinary expert group who provided guidance and valuable advice throughout the development process.

FUNDING

Development and validation of the Kidney PREM was supported by funding from Kidney Care UK, with initial support for development from the Renal Association and NHS England.

AUTHORS’ CONTRIBUTIONS

S.V.D.V., D.W., C.C., R.F., R.G., K.F. and J.H. were responsible for the study design. C.C., R.G., J.H. and S.V.D.V. were responsible for PPI. N.H., A.B. and D.W. were responsible for analysis of data. N.H., A.B., K.F., D.W., J.H. and S.V.D.V. were responsible for interpretation of data/results. S.V.D.V., A.B., B.R., K.F., S.S., D.W. and J.H. drafted the manuscript. All authors were responsible for editing and approval of final of the manuscript.

DATA AVAILABILITY STATEMENT

Data are held by the UKRR. To apply to access Registry data see https://ukkidney.org/audit-research/how-access-data.

ETHICS APPROVAL

The study involved secondary analysis of anonymous UKRR Kidney PREM data. The UKRR has NHS Health Research Authority research ethics approval (16/NE/0042).

REFERENCES

Author notes