Abstract

Objective

The aim of this study was to develop and refine indices to measure organization and care pathway-level quality management systems in Australian hospitals.

Design

A questionnaire survey and audit tools were derived from instruments validated as part of the Deepening Our Understanding of Quality improvement in Europe (DUQuE) study, adapted for Australian hospitals through expert opinion. Statistical processes were used to explore the factor structure, reliability and non-redundancy and descriptive statistics of the scales.

Setting

Thirty-two large Australian public hospitals.

Participants

Audit of quality management processes at organization-level and care pathway processes at department level for three patient conditions (acute myocardial infarction (AMI), hip fracture and stroke) and senior quality manager, at each of the 32 participating hospitals.

Main Outcome Measure(s)

The degree of quality management evident at organization and care pathway levels.

Results

Analysis yielded seven quality systems and strategies scales. The three hospital-level measures were: the Quality Management Systems Index (QMSI), the Quality Management Compliance Index (QMCI) and the Clinical Quality Implementation Index (CQII). The four department-level measures were: Specialised Expertise and Responsibility (SER), Evidence-Based Organisation of Pathways (EBOP), Patient Safety Strategies (PSS) and Clinical Review (CR). For QMCI, and for seven out of eight subscales in QMSI, adequate internal consistency (Cronbach’s |$\alpha$| >0.8) was achieved. For CQII, lack of variation and ceiling effects in the data resulted in very low internal consistency scores, but items were retained for theoretical reasons. Internal consistency was high for CR (Cronbach’s |$\alpha$| 0.74–0.88 across the three conditions), and this was supported by all item-total correlations exceeding the desired threshold. For EBOP, Cronbach’s |$\alpha$| was acceptable for hip fracture (0.80) and stroke (0.76), but only moderate for AMI (0.52). PSS and SER scales were retained for theoretical reasons, although internal consistencies were only moderate (SER) to poor (PSS).

Conclusions

The Deepening our Understanding of Quality in Australia (DUQuA) organization and department scales can be used by Australian hospital managers to assess and measure improvement in quality management at organization and department levels within their hospitals and are readily modifiable for other health systems depending on their needs.

Introduction

The ‘Deepening our Understanding of Quality in Australia’ (DUQuA) study [1] was a 5-year Australia-wide, multi-level, multi-million dollar cross-sectional study aiming to identify how quality management systems (QMS), leadership and culture in Australian hospitals are related to care delivery and patient outcomes for acute myocardial infarction (AMI), stroke and hip fracture. Based on the original, landmark European Union-funded ‘Deepening our Understanding of Quality improvement in Europe’ (DUQuE) study [2], the rationale for undertaking DUQuA included potential for comparison with the large European sample of 188 hospitals across seven countries, in addition to in-depth understanding of quality management in Australian acute settings. Evidence- or consensus-based measurement tools were designed or modified and then utilized to collect quantitative data on QMS at hospital and care pathway levels, department-level safety culture and leadership amongst clinicians, clinical treatment processes, patient outcomes and patient perceptions of safety. Collection methods included paper-based and electronic surveys, medical record reviews, external audits and accessing national datasets from the National Stroke Foundation registry and the Australian Institute of Health and Welfare. Linear and multi-level modelling was applied to the datasets to identify relationships between quality management, care delivery and patient outcomes. Findings have the potential to influence decision-making and improvements in quality and safety in Australian and international hospitals.

The aim of this study was to develop and validate indices to measure organization and care pathway-level QMS in Australian hospitals. The DUQuA measurement tools were derived from the DUQuE tools and, where possible, the DUQuE terminology retained. For example, the definition of QMS mirrors that of DUQuE: ‘a set of interacting activities, methods and procedures used to monitor, control and improve the quality of care’ [3]. The organization and department scales described are the same measures used by DUQuE for organization-level QMS, modified for the Australian context as described later in our ‘Method’ section. The three organization-level measures are the Quality Management Systems Index (QMSI), the Quality Management Compliance Index (QMCI) and the Clinical Quality Implementation Index (CQII). The four measures of quality management activities at department level focus on Specialised Expertise and Responsibility (SER), Evidence-Based Organisation of Pathways (EBOP), Patient Safety Strategies (PSS) and Clinical Review (CR).

This article presents information about the structure, reliability and non-redundancy and descriptive statistics of the QMSI, QMCI and CQII organization scales and the SER, EBOP, PSS and CR department scales. Relations between the organization and department scales and other outcomes are reported elsewhere in this Supplement [4].

Method

Development of the organization scales

The three organization-level measures from DUQuE (QMSI, QMCI and CQII) were modified by the DUQuA team to reflect the evolution of our condition-specific indicators and to ensure relevance to the Australian context, in consultation with national quality assessment experts, such as the Australian Council on Healthcare Standards. QMSI quantifies the managerial aspects of quality management that might influence the implementation of quality systems in hospitals. QMSI consists of 10 subscales: quality policy documents, hospital governance board activities, quality resources, quality management, evidence-based medicine protocols, preventive protocols, internal quality methods for general activities, personnel, clinical practice and patients. Each item is rated on a four-point Likert scale (range 1–4).

QMCI measures the managerial aspects of quality improvement in hospitals on three subscales: quality planning, monitoring patient and professional opinions and quality control and monitoring. CQII quantifies implementation of quality systems at hospital level including whether systems exist, to what extent implementation has been monitored and whether implementation is sustainable. CQII consisted of seven subscales: preventing and controlling healthcare-associated infections, medical safety, preventing patient falls, preventing pressure injuries, routine assessment and diagnostic testing of patients in elective surgery, safe surgery that includes an approved checklist and recognizing and responding to clinical deterioration in acute healthcare. For QMCI and CQII, items are rated on a five-point Likert scale (range 0–4).

The final DUQuA instruments are similar, but not identical, to the tools used in the DUQuE study. For example: some questions in the QMSI survey are worded differently to align with Australian terminology and processes, and a small number of additional questions have been included.

Development of the department scales

Following a similar process, the DUQuA team modified the DUQuE measures [5] (SER, EBOP, PSS and CR) for the Australian context. SER explores how clinical responsibilities were assigned for a particular condition; EBOP measures whether department processes, such as admission, acute care, rehabilitation and discharge, were organized to facilitate evidence-based care recommendations; PSS measures the use of clinical practice guidelines, and CR assesses the integration of audit and systematic monitoring with department quality management mechanisms. SER, CR and PSS used identical measures for the three conditions, while EBOP has the same structure for each condition-specific department (AMI, stroke and hip fracture) but the content varies to reflect the condition-specific evidence recommendation.

There are differences between the DUQuE and DUQuA department scales. As in the QMSI survey, some audit items are worded differently to align with Australian terminology and processes. Unlike DUQuE, the DUQuA project recognizes the Emergency Department (ED) as a common entry point to the hospital for the majority of AMI, hip fracture and stroke patients. ED condition-specific measures were developed for SER, EBOP and CR, to ensure relevance to the ED context; thus, for example, the ED assessment of SER comprised questions for each of AMI, hip fracture and stroke. For ED, the PSS items are all generic (i.e. not condition-specific).

Setting, participant recruitment and data collection

The DUQuA project commenced in February 2014 in 32 large Australian public hospitals, and data collection was completed by November 2017. In each state and territory, general public hospitals that met the following criteria were recruited: (i) with approximately 200 beds or more, (ii) with an ED and (iii) that regularly admit more than 30 each of stroke, AMI and hip fracture patients over a period of 3 months. A group of 78 hospitals were invited to participate, 70 that met all inclusion criteria and eight that were close to the specified thresholds or lacked at least one of the chosen departments; 62 hospitals initially agreed (89%). Due to factors associated with healthcare leadership changes, hospital relocations and obtaining ethics approval, a number of hospitals withdrew; however, 32 hospitals participated until completion of the study. The QMSI questionnaire was assessed by the senior person responsible for the coordination of quality improvement activities in each of the 32 participating hospitals.

Hospitals responding to an initial research invitation were provided with a formal letter detailing the study’s background, data collection procedure and timeline. A Local Principal Investigator (LPI) was nominated by each hospital as the study point of contact. The role of the LPI was to maintain collaborative relationships between the hospital staff and the research team, to contact the quality manager to complete the QMSI questionnaire and to coordinate a site visit for accredited surveyors to conduct an external quality assessment (EQA) visit, during which the two organization-level measures (QMCI and CQII) and the four department-level measures (SER, EBOP, PSS and CR) were assessed. Details about surveyors’ visits are reported elsewhere in this Supplement [4, Appendix A]. Questionnaires were completed anonymously, and EQA was conducted as a two-day site visit in the hospital, including the three condition-specific wards (AMI, hip fracture and stroke) and the ED. Data collection commenced in February 2016 and concluded in November 2017.

Statistical analysis

DUQuE applied the following rules in assessing reliability: Cronbach’s |$\alpha$| >0.7 [6] and item-total correlation coefficient >0.4 [6] were considered acceptable evidence of internal consistency and consistency with the subscale construct, respectively, and Pearson’s correlation coefficient <0.7 between subscale scores was accepted as demonstrating non-redundancy of a subscale [5, 6]. DUQuA adopted the same rules, except that Cronbach’s |$\alpha$| >0.8 was deemed preferable, based on the work of Nunnally [7]. These thresholds were used to indicate a need for consideration of model re-specification for the three organization-level measures (QMSI, QMCI and CQII). The four department-level measures are single factor scales, so model re-specification was not considered, but we nevertheless report the test-values for internal reliability to inform interpretation of the measures. Further details on statistical analysis methods used for DUQuA have been published elsewhere in this Supplement [8]. Confirmatory Factor Analysis was not feasible because of the small sample size.

Data were analysed in SAS/STAT 9.4 (Cary, North Carolina, USA). Characteristics of the hospitals were summarized. We used a combination of theoretical grounds, item-total correlation and Cronbach’s |$\alpha$|⁠, to refine the theoretical model.

After reaching the final model structure, each organization-level subscale was calculated as the mean of all non-missing items. If more than 50% of the items within the subscale were missing, the subscale was set to missing. We performed 100 multiple imputations, using Markov Chain Monte Carlo (MCMC) method [9], to impute missing organization subscales. After imputation, we calculated the organization scales by summing all subscales (for QMSI, eight was subtracted from the sum, in line with DUQuE procedures). To assess the degree of redundancy, Pearson’s correlation coefficient was calculated between pairs of subscales within each scale; the analysis was repeated 100 times for each imputation and the SAS/STAT MIANALYZE procedure [10] was used to obtain the pooled parameter estimates and standard errors, if any imputed values were involved in the calculations. For department-level and ED-level scales, each department-level scale was the mean of all applicable non-missing items. If more than 50% of the applicable items within the scale were missing, the scale was set to missing.

Results

Characteristics of participants

Although 32 hospitals participated in DUQuA, two hospitals shared a quality manager and QMS, and therefore, 31 QMS questionnaire surveys were included for analysis. All 32 hospital provided audit data. The characteristics of participating hospitals are summarized in Supplementary eTable A1, Appendix A.

Organization scales—structure, reliability and non-redundancy

Missing data

For QMSI, there were no missing data at subscale level. For QMCI, the percentage of missing subscales was 1.6% and ranged from 0 to 3.1% per subscale, affecting one hospital only. For CQII, the percentage of missing subscales was 0.4% and ranged from 0 to 3.1% per subscale; again, only one hospital was affected.

Descriptive statistics for QMSI, QMCI and CQII subscales

The descriptive statistics for each subscale of each of the three organization-level measures are summarized in Table 1. Subscales for all three measures were left-skewed with all but one scale having a mean value above 3.0. For QMSI, the subscale mean ranged from 3.2 to 3.7; for QMCI, both subscales had a mean of 3.5, and for CQII ‘Routine assessment and diagnostic testing of patients in elective surgery’ had a mean of 2.3 with the remaining subscale means ranging from 3.3 to 3.9. Item-level summary statistics for each measure can be found at Supplementary eTables A24, Appendix A.

Table 1

Descriptive statistics of organization subscales

ScaleSubscalenMeanSDMedianMinMaxIQRa
QMSIQuality policy313.20.903.31.341
Hospital governance—board activities313.40.533.6240.8
Quality resources313.40.373.32.440.4
Quality management313.50.483.7240.7
Preventive protocols313.60.403.72.540.6
Internal quality methods—general activities313.40.483.4240.7
Performance monitoring313.30.703.31.741.3
Internal quality methods—patients313.70.424340.5
QMCIMonitoring patient and professional opinions313.50.763.80.640.8
Quality control and monitoring323.50.633.80.840.7
CQIIPreventing and controlling healthcare-associated infections323.90.2043.240
Medication safety323.90.2143.240.2
Preventing patient falls323.90.1343.640.1
Preventing pressure injuries323.80.5141.240.3
Routine assessment and diagnostic testing of patients in elective surgery322.31.562.5043.3
Safe surgery that includes an approved checklist313.31.294041
Recognizing and responding to clinical deterioration in acute healthcare323.90.1843.240
ScaleSubscalenMeanSDMedianMinMaxIQRa
QMSIQuality policy313.20.903.31.341
Hospital governance—board activities313.40.533.6240.8
Quality resources313.40.373.32.440.4
Quality management313.50.483.7240.7
Preventive protocols313.60.403.72.540.6
Internal quality methods—general activities313.40.483.4240.7
Performance monitoring313.30.703.31.741.3
Internal quality methods—patients313.70.424340.5
QMCIMonitoring patient and professional opinions313.50.763.80.640.8
Quality control and monitoring323.50.633.80.840.7
CQIIPreventing and controlling healthcare-associated infections323.90.2043.240
Medication safety323.90.2143.240.2
Preventing patient falls323.90.1343.640.1
Preventing pressure injuries323.80.5141.240.3
Routine assessment and diagnostic testing of patients in elective surgery322.31.562.5043.3
Safe surgery that includes an approved checklist313.31.294041
Recognizing and responding to clinical deterioration in acute healthcare323.90.1843.240

aIQR, Interquartile Range.

Table 1

Descriptive statistics of organization subscales

ScaleSubscalenMeanSDMedianMinMaxIQRa
QMSIQuality policy313.20.903.31.341
Hospital governance—board activities313.40.533.6240.8
Quality resources313.40.373.32.440.4
Quality management313.50.483.7240.7
Preventive protocols313.60.403.72.540.6
Internal quality methods—general activities313.40.483.4240.7
Performance monitoring313.30.703.31.741.3
Internal quality methods—patients313.70.424340.5
QMCIMonitoring patient and professional opinions313.50.763.80.640.8
Quality control and monitoring323.50.633.80.840.7
CQIIPreventing and controlling healthcare-associated infections323.90.2043.240
Medication safety323.90.2143.240.2
Preventing patient falls323.90.1343.640.1
Preventing pressure injuries323.80.5141.240.3
Routine assessment and diagnostic testing of patients in elective surgery322.31.562.5043.3
Safe surgery that includes an approved checklist313.31.294041
Recognizing and responding to clinical deterioration in acute healthcare323.90.1843.240
ScaleSubscalenMeanSDMedianMinMaxIQRa
QMSIQuality policy313.20.903.31.341
Hospital governance—board activities313.40.533.6240.8
Quality resources313.40.373.32.440.4
Quality management313.50.483.7240.7
Preventive protocols313.60.403.72.540.6
Internal quality methods—general activities313.40.483.4240.7
Performance monitoring313.30.703.31.741.3
Internal quality methods—patients313.70.424340.5
QMCIMonitoring patient and professional opinions313.50.763.80.640.8
Quality control and monitoring323.50.633.80.840.7
CQIIPreventing and controlling healthcare-associated infections323.90.2043.240
Medication safety323.90.2143.240.2
Preventing patient falls323.90.1343.640.1
Preventing pressure injuries323.80.5141.240.3
Routine assessment and diagnostic testing of patients in elective surgery322.31.562.5043.3
Safe surgery that includes an approved checklist313.31.294041
Recognizing and responding to clinical deterioration in acute healthcare323.90.1843.240

aIQR, Interquartile Range.

QMSI—structure, reliability and non-redundancy

From the original 10-factor theoretical model for QMSI, we merged item Q7.1 to 7.6 from the ‘evidence-based medicine protocols’ construct with item Q7.7 to 7.13 from the ‘preventive protocols’ construct to form a 13-item ‘preventive protocols’ construct. Items Q8.1 to 8.12 originally forming three ‘internal quality methods’ constructs (general activities, personnel and clinical practice) were restructured to ‘internal quality methods—general’ and ‘performance monitoring’. The final eight-factor model is presented in Table 2, along with the Cronbach’s |$\alpha$| scores and item-total correlations. All subscales except ‘internal quality methods—patients’ achieved or nearly achieved internal consistency reliability (Cronbach’s |$\alpha$| >0.8). Twelve out of 51 items did not meet the desired threshold (>0.4) for item-total correlation, to demonstrate internal consistency within the subscale construct. Appendix A, Supplementary eTable A5, shows between subscale correlations for QMSI. Twenty-five pairs of correlations were within (0.16–0.69) the acceptable range of <0.7; the three scales with potential redundancy were ‘Quality resources’ (0.75), ‘Quality management’ (0.74) and ‘Internal quality methods—general activities’ (0.77), each of which correlated with ‘Hospital governance—board activities’.

QMCI—structure, reliability and non-redundancy

From the original three-factor theoretical model for QMCI, we moved two items into the ‘quality control and monitoring’ subscale: item XQ01 from the single item ‘quality planning’ subscale and XQ08 from ‘monitoring patient and professional opinions’. The final two-factor model can be found in Table 3. Both subscales showed adequate internal consistency through Cronbach’s |$\alpha$| without reaching our preferred threshold of 0.8. One item in each subscale had an item-total correlation <0.4, indicating less than ideal fit with the subscale construct. The between subscale correlation coefficient for QMCI (0.73) indicated non-redundancy.

Table 2

Cronbach’s alpha and item-total correlation for QMSI

Subscale and itemsCronbach’s |$\alpha$|Item-total correlation
Quality policy0.805
 Q3.1 A documented quality policy approved by the hospital governance board0.650
 Q3.2 Quality improvement (QI) plan at hospital level (translation of the quality objectives into concrete activities and measures designed to realize the quality policy)0.614
 Q3.3 Report on evaluation of QI activities (focusing on routine quality and safety indicators, e.g. clinical outcomes, finances, human resources and patient satisfaction)0.700
Hospital governance—board activities0.779
 Q4.1 The hospital governance board makes it clear what is expected from clinicians in regard to QI0.529
 Q4.2 The hospital governance board has established formal roles for quality leadership (visible in organizational chart)0.770
 Q4.3 The hospital governance board assesses on an annual or bi-annual basis whether clinicians comply with day-to-day patient safety procedures0.557
 Q4.4 The hospital governance board knows and uses performance data for QI0.516
 Q4.5 The hospital governance board monitors the execution of QI plans0.518
Quality resources0.755
 Q5.1 Clinicians attend at least one training session a year to further develop their professional expertise0.057a
 Q5.2 Clinicians receive information back on the results of their treatment of patients0.337a
 Q5.3 Clinicians are encouraged to report incidents and adverse events0.230a
 Q5.4 Clinicians’ registrations are reviewed by the hospital annually0.452
 Q5.5 The hospital provides training to clinicians0.485
 Q5.6 Clinicians are trained in teamwork0.507
 Q5.7 Middle management (e.g. NUM, Head of Department, etc.) is trained in QI methods0.676
 Q5.8 Clinicians are trained in QI methods0.763
 Q5.9 Clinicians are trained in patient safety procedures0.535
Quality management0.910
 Q6.1 Data used from clinical indicators0.782
 Q6.2 Data used from incident reporting system0.619
 Q6.3 Data used from patient interviews or surveys0.659
 Q6.4 Data used from assessment of guideline compliance0.705
 Q6.5 Data used from results of internal audits0.867
 Q6.6 Data used from audits of hand hygiene compliance0.673
 Q6.7 Data used from audits of patient identification0.858
Preventive protocols0.797
 Q7.1 An up-to-date hospital protocol for use of prophylactic antibiotics0.210a
 Q7.2 An up-to-date hospital protocol for medication reconciliation0.199a
 Q7.3 An up-to-date hospital protocol for the handover of patient information to another care unit0.475
 Q7.4 An up-to-date hospital protocol for use of a surgical checklist0.485
 Q7.5 An up-to-date hospital protocol for recognizing and responding to clinical deterioration in acute healthcare0.137a
 Q7.6 An up-to-date hospital protocol for routine assessment and diagnostic testing of elective surgery patients0.597
 Q7.7 Prevention of central line infection0.672
 Q7.8 Prevention of surgical site infection0.642
 Q7.9 Prevention of healthcare-associated infections0.353a
 Q7.10 Prevention of ventilator-associated pneumonia0.351a
 Q7.11 Prevention of medication errors0.560
 Q7.12 Prevention of patient falls0.696
 Q7.13 Prevention of patient pressure injuries0.611
Internal quality methods—general activities0.836
 Q8.1 Root-cause analysis of incidents is conducted according to legislation or policy, and within recommended timeframes0.233a
 Q8.2 Risk management, consisting of a systematic process of identifying, assessing and taking action to prevent or manage clinical risks in the care process, is undertaken in all units0.530
 Q8.3 Internal audit, consisting of periodical review of all components of the quality system, is undertaken in all units0.602
 Q8.7 Staff workplace satisfaction is measured and monitored at least annually0.341a
 Q8.8 Multidisciplinary CR in all units to assess and improve the results of care delivery0.866
 Q8.9 Patient record review in all units to determine incidents and priorities for quality improvement0.653
 Q8.10 Development of care pathways or process redesign0.524
 Q8.11 Benchmarking clinical practice against other departments within the hospital0.655
 Q8.12 Benchmarking clinical practice against other hospitals0.549
Performance monitoring0.791
 Q8.4 Executive ‘walk-arounds’ are frequently conducted to identify safety and quality issues0.669
 Q8.5 Performance of individual doctors is monitored at least annually0.638
 Q8.6 Performance of individual nurses is monitored at least annually0.660
Internal quality methods—patients0.449
 Q8.13 Patient satisfaction or experience is measured and monitored at least annually0.303a
 Q8.14 Periodical evaluation of patient complaints is used to drive improvements0.303a
Subscale and itemsCronbach’s |$\alpha$|Item-total correlation
Quality policy0.805
 Q3.1 A documented quality policy approved by the hospital governance board0.650
 Q3.2 Quality improvement (QI) plan at hospital level (translation of the quality objectives into concrete activities and measures designed to realize the quality policy)0.614
 Q3.3 Report on evaluation of QI activities (focusing on routine quality and safety indicators, e.g. clinical outcomes, finances, human resources and patient satisfaction)0.700
Hospital governance—board activities0.779
 Q4.1 The hospital governance board makes it clear what is expected from clinicians in regard to QI0.529
 Q4.2 The hospital governance board has established formal roles for quality leadership (visible in organizational chart)0.770
 Q4.3 The hospital governance board assesses on an annual or bi-annual basis whether clinicians comply with day-to-day patient safety procedures0.557
 Q4.4 The hospital governance board knows and uses performance data for QI0.516
 Q4.5 The hospital governance board monitors the execution of QI plans0.518
Quality resources0.755
 Q5.1 Clinicians attend at least one training session a year to further develop their professional expertise0.057a
 Q5.2 Clinicians receive information back on the results of their treatment of patients0.337a
 Q5.3 Clinicians are encouraged to report incidents and adverse events0.230a
 Q5.4 Clinicians’ registrations are reviewed by the hospital annually0.452
 Q5.5 The hospital provides training to clinicians0.485
 Q5.6 Clinicians are trained in teamwork0.507
 Q5.7 Middle management (e.g. NUM, Head of Department, etc.) is trained in QI methods0.676
 Q5.8 Clinicians are trained in QI methods0.763
 Q5.9 Clinicians are trained in patient safety procedures0.535
Quality management0.910
 Q6.1 Data used from clinical indicators0.782
 Q6.2 Data used from incident reporting system0.619
 Q6.3 Data used from patient interviews or surveys0.659
 Q6.4 Data used from assessment of guideline compliance0.705
 Q6.5 Data used from results of internal audits0.867
 Q6.6 Data used from audits of hand hygiene compliance0.673
 Q6.7 Data used from audits of patient identification0.858
Preventive protocols0.797
 Q7.1 An up-to-date hospital protocol for use of prophylactic antibiotics0.210a
 Q7.2 An up-to-date hospital protocol for medication reconciliation0.199a
 Q7.3 An up-to-date hospital protocol for the handover of patient information to another care unit0.475
 Q7.4 An up-to-date hospital protocol for use of a surgical checklist0.485
 Q7.5 An up-to-date hospital protocol for recognizing and responding to clinical deterioration in acute healthcare0.137a
 Q7.6 An up-to-date hospital protocol for routine assessment and diagnostic testing of elective surgery patients0.597
 Q7.7 Prevention of central line infection0.672
 Q7.8 Prevention of surgical site infection0.642
 Q7.9 Prevention of healthcare-associated infections0.353a
 Q7.10 Prevention of ventilator-associated pneumonia0.351a
 Q7.11 Prevention of medication errors0.560
 Q7.12 Prevention of patient falls0.696
 Q7.13 Prevention of patient pressure injuries0.611
Internal quality methods—general activities0.836
 Q8.1 Root-cause analysis of incidents is conducted according to legislation or policy, and within recommended timeframes0.233a
 Q8.2 Risk management, consisting of a systematic process of identifying, assessing and taking action to prevent or manage clinical risks in the care process, is undertaken in all units0.530
 Q8.3 Internal audit, consisting of periodical review of all components of the quality system, is undertaken in all units0.602
 Q8.7 Staff workplace satisfaction is measured and monitored at least annually0.341a
 Q8.8 Multidisciplinary CR in all units to assess and improve the results of care delivery0.866
 Q8.9 Patient record review in all units to determine incidents and priorities for quality improvement0.653
 Q8.10 Development of care pathways or process redesign0.524
 Q8.11 Benchmarking clinical practice against other departments within the hospital0.655
 Q8.12 Benchmarking clinical practice against other hospitals0.549
Performance monitoring0.791
 Q8.4 Executive ‘walk-arounds’ are frequently conducted to identify safety and quality issues0.669
 Q8.5 Performance of individual doctors is monitored at least annually0.638
 Q8.6 Performance of individual nurses is monitored at least annually0.660
Internal quality methods—patients0.449
 Q8.13 Patient satisfaction or experience is measured and monitored at least annually0.303a
 Q8.14 Periodical evaluation of patient complaints is used to drive improvements0.303a

aItem-total correlation coefficient is less than the acceptable value of 0.4.

Table 2

Cronbach’s alpha and item-total correlation for QMSI

Subscale and itemsCronbach’s |$\alpha$|Item-total correlation
Quality policy0.805
 Q3.1 A documented quality policy approved by the hospital governance board0.650
 Q3.2 Quality improvement (QI) plan at hospital level (translation of the quality objectives into concrete activities and measures designed to realize the quality policy)0.614
 Q3.3 Report on evaluation of QI activities (focusing on routine quality and safety indicators, e.g. clinical outcomes, finances, human resources and patient satisfaction)0.700
Hospital governance—board activities0.779
 Q4.1 The hospital governance board makes it clear what is expected from clinicians in regard to QI0.529
 Q4.2 The hospital governance board has established formal roles for quality leadership (visible in organizational chart)0.770
 Q4.3 The hospital governance board assesses on an annual or bi-annual basis whether clinicians comply with day-to-day patient safety procedures0.557
 Q4.4 The hospital governance board knows and uses performance data for QI0.516
 Q4.5 The hospital governance board monitors the execution of QI plans0.518
Quality resources0.755
 Q5.1 Clinicians attend at least one training session a year to further develop their professional expertise0.057a
 Q5.2 Clinicians receive information back on the results of their treatment of patients0.337a
 Q5.3 Clinicians are encouraged to report incidents and adverse events0.230a
 Q5.4 Clinicians’ registrations are reviewed by the hospital annually0.452
 Q5.5 The hospital provides training to clinicians0.485
 Q5.6 Clinicians are trained in teamwork0.507
 Q5.7 Middle management (e.g. NUM, Head of Department, etc.) is trained in QI methods0.676
 Q5.8 Clinicians are trained in QI methods0.763
 Q5.9 Clinicians are trained in patient safety procedures0.535
Quality management0.910
 Q6.1 Data used from clinical indicators0.782
 Q6.2 Data used from incident reporting system0.619
 Q6.3 Data used from patient interviews or surveys0.659
 Q6.4 Data used from assessment of guideline compliance0.705
 Q6.5 Data used from results of internal audits0.867
 Q6.6 Data used from audits of hand hygiene compliance0.673
 Q6.7 Data used from audits of patient identification0.858
Preventive protocols0.797
 Q7.1 An up-to-date hospital protocol for use of prophylactic antibiotics0.210a
 Q7.2 An up-to-date hospital protocol for medication reconciliation0.199a
 Q7.3 An up-to-date hospital protocol for the handover of patient information to another care unit0.475
 Q7.4 An up-to-date hospital protocol for use of a surgical checklist0.485
 Q7.5 An up-to-date hospital protocol for recognizing and responding to clinical deterioration in acute healthcare0.137a
 Q7.6 An up-to-date hospital protocol for routine assessment and diagnostic testing of elective surgery patients0.597
 Q7.7 Prevention of central line infection0.672
 Q7.8 Prevention of surgical site infection0.642
 Q7.9 Prevention of healthcare-associated infections0.353a
 Q7.10 Prevention of ventilator-associated pneumonia0.351a
 Q7.11 Prevention of medication errors0.560
 Q7.12 Prevention of patient falls0.696
 Q7.13 Prevention of patient pressure injuries0.611
Internal quality methods—general activities0.836
 Q8.1 Root-cause analysis of incidents is conducted according to legislation or policy, and within recommended timeframes0.233a
 Q8.2 Risk management, consisting of a systematic process of identifying, assessing and taking action to prevent or manage clinical risks in the care process, is undertaken in all units0.530
 Q8.3 Internal audit, consisting of periodical review of all components of the quality system, is undertaken in all units0.602
 Q8.7 Staff workplace satisfaction is measured and monitored at least annually0.341a
 Q8.8 Multidisciplinary CR in all units to assess and improve the results of care delivery0.866
 Q8.9 Patient record review in all units to determine incidents and priorities for quality improvement0.653
 Q8.10 Development of care pathways or process redesign0.524
 Q8.11 Benchmarking clinical practice against other departments within the hospital0.655
 Q8.12 Benchmarking clinical practice against other hospitals0.549
Performance monitoring0.791
 Q8.4 Executive ‘walk-arounds’ are frequently conducted to identify safety and quality issues0.669
 Q8.5 Performance of individual doctors is monitored at least annually0.638
 Q8.6 Performance of individual nurses is monitored at least annually0.660
Internal quality methods—patients0.449
 Q8.13 Patient satisfaction or experience is measured and monitored at least annually0.303a
 Q8.14 Periodical evaluation of patient complaints is used to drive improvements0.303a
Subscale and itemsCronbach’s |$\alpha$|Item-total correlation
Quality policy0.805
 Q3.1 A documented quality policy approved by the hospital governance board0.650
 Q3.2 Quality improvement (QI) plan at hospital level (translation of the quality objectives into concrete activities and measures designed to realize the quality policy)0.614
 Q3.3 Report on evaluation of QI activities (focusing on routine quality and safety indicators, e.g. clinical outcomes, finances, human resources and patient satisfaction)0.700
Hospital governance—board activities0.779
 Q4.1 The hospital governance board makes it clear what is expected from clinicians in regard to QI0.529
 Q4.2 The hospital governance board has established formal roles for quality leadership (visible in organizational chart)0.770
 Q4.3 The hospital governance board assesses on an annual or bi-annual basis whether clinicians comply with day-to-day patient safety procedures0.557
 Q4.4 The hospital governance board knows and uses performance data for QI0.516
 Q4.5 The hospital governance board monitors the execution of QI plans0.518
Quality resources0.755
 Q5.1 Clinicians attend at least one training session a year to further develop their professional expertise0.057a
 Q5.2 Clinicians receive information back on the results of their treatment of patients0.337a
 Q5.3 Clinicians are encouraged to report incidents and adverse events0.230a
 Q5.4 Clinicians’ registrations are reviewed by the hospital annually0.452
 Q5.5 The hospital provides training to clinicians0.485
 Q5.6 Clinicians are trained in teamwork0.507
 Q5.7 Middle management (e.g. NUM, Head of Department, etc.) is trained in QI methods0.676
 Q5.8 Clinicians are trained in QI methods0.763
 Q5.9 Clinicians are trained in patient safety procedures0.535
Quality management0.910
 Q6.1 Data used from clinical indicators0.782
 Q6.2 Data used from incident reporting system0.619
 Q6.3 Data used from patient interviews or surveys0.659
 Q6.4 Data used from assessment of guideline compliance0.705
 Q6.5 Data used from results of internal audits0.867
 Q6.6 Data used from audits of hand hygiene compliance0.673
 Q6.7 Data used from audits of patient identification0.858
Preventive protocols0.797
 Q7.1 An up-to-date hospital protocol for use of prophylactic antibiotics0.210a
 Q7.2 An up-to-date hospital protocol for medication reconciliation0.199a
 Q7.3 An up-to-date hospital protocol for the handover of patient information to another care unit0.475
 Q7.4 An up-to-date hospital protocol for use of a surgical checklist0.485
 Q7.5 An up-to-date hospital protocol for recognizing and responding to clinical deterioration in acute healthcare0.137a
 Q7.6 An up-to-date hospital protocol for routine assessment and diagnostic testing of elective surgery patients0.597
 Q7.7 Prevention of central line infection0.672
 Q7.8 Prevention of surgical site infection0.642
 Q7.9 Prevention of healthcare-associated infections0.353a
 Q7.10 Prevention of ventilator-associated pneumonia0.351a
 Q7.11 Prevention of medication errors0.560
 Q7.12 Prevention of patient falls0.696
 Q7.13 Prevention of patient pressure injuries0.611
Internal quality methods—general activities0.836
 Q8.1 Root-cause analysis of incidents is conducted according to legislation or policy, and within recommended timeframes0.233a
 Q8.2 Risk management, consisting of a systematic process of identifying, assessing and taking action to prevent or manage clinical risks in the care process, is undertaken in all units0.530
 Q8.3 Internal audit, consisting of periodical review of all components of the quality system, is undertaken in all units0.602
 Q8.7 Staff workplace satisfaction is measured and monitored at least annually0.341a
 Q8.8 Multidisciplinary CR in all units to assess and improve the results of care delivery0.866
 Q8.9 Patient record review in all units to determine incidents and priorities for quality improvement0.653
 Q8.10 Development of care pathways or process redesign0.524
 Q8.11 Benchmarking clinical practice against other departments within the hospital0.655
 Q8.12 Benchmarking clinical practice against other hospitals0.549
Performance monitoring0.791
 Q8.4 Executive ‘walk-arounds’ are frequently conducted to identify safety and quality issues0.669
 Q8.5 Performance of individual doctors is monitored at least annually0.638
 Q8.6 Performance of individual nurses is monitored at least annually0.660
Internal quality methods—patients0.449
 Q8.13 Patient satisfaction or experience is measured and monitored at least annually0.303a
 Q8.14 Periodical evaluation of patient complaints is used to drive improvements0.303a

aItem-total correlation coefficient is less than the acceptable value of 0.4.

Table 3

Cronbach’s alpha and item-total correlation for QMCI

Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Monitoring patient and professional opinions0.730
 XQ03 The results of patient satisfaction or experience surveys were formally reported to the hospital governance board0.571
 XQ04 The hospital governance board received results of surveys of staff satisfaction0.509
 XQ09 Patient satisfaction or experience are measured and evaluated0.638
 XQ10 Patient complaints and feedback are investigated and acted upon0.672
 XQ11 Staff opinion or perception on organizational quality and safety culture are measured and evaluated0.239a
Quality control and monitoring0.785
 XQ01 The hospital governance board approved a current program for quality improvement (QI)0.535
 XQ02 The hospital governance board received regular, formal reports on quality and safety (Q&S)0.334a
 XQ05 Clinical leaders received regular, formal reports on Q&S0.715
 XQ06 There is an active clinical guideline register0.706
 XQ07 Processes for implementation and evaluation of clinical guidelines against practice0.532
 XQ08 Clinical incidents (adverse events) are analysed and evaluated0.541
Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Monitoring patient and professional opinions0.730
 XQ03 The results of patient satisfaction or experience surveys were formally reported to the hospital governance board0.571
 XQ04 The hospital governance board received results of surveys of staff satisfaction0.509
 XQ09 Patient satisfaction or experience are measured and evaluated0.638
 XQ10 Patient complaints and feedback are investigated and acted upon0.672
 XQ11 Staff opinion or perception on organizational quality and safety culture are measured and evaluated0.239a
Quality control and monitoring0.785
 XQ01 The hospital governance board approved a current program for quality improvement (QI)0.535
 XQ02 The hospital governance board received regular, formal reports on quality and safety (Q&S)0.334a
 XQ05 Clinical leaders received regular, formal reports on Q&S0.715
 XQ06 There is an active clinical guideline register0.706
 XQ07 Processes for implementation and evaluation of clinical guidelines against practice0.532
 XQ08 Clinical incidents (adverse events) are analysed and evaluated0.541

aItem-total correlation coefficient is less than the acceptable value of 0.4.

Table 3

Cronbach’s alpha and item-total correlation for QMCI

Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Monitoring patient and professional opinions0.730
 XQ03 The results of patient satisfaction or experience surveys were formally reported to the hospital governance board0.571
 XQ04 The hospital governance board received results of surveys of staff satisfaction0.509
 XQ09 Patient satisfaction or experience are measured and evaluated0.638
 XQ10 Patient complaints and feedback are investigated and acted upon0.672
 XQ11 Staff opinion or perception on organizational quality and safety culture are measured and evaluated0.239a
Quality control and monitoring0.785
 XQ01 The hospital governance board approved a current program for quality improvement (QI)0.535
 XQ02 The hospital governance board received regular, formal reports on quality and safety (Q&S)0.334a
 XQ05 Clinical leaders received regular, formal reports on Q&S0.715
 XQ06 There is an active clinical guideline register0.706
 XQ07 Processes for implementation and evaluation of clinical guidelines against practice0.532
 XQ08 Clinical incidents (adverse events) are analysed and evaluated0.541
Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Monitoring patient and professional opinions0.730
 XQ03 The results of patient satisfaction or experience surveys were formally reported to the hospital governance board0.571
 XQ04 The hospital governance board received results of surveys of staff satisfaction0.509
 XQ09 Patient satisfaction or experience are measured and evaluated0.638
 XQ10 Patient complaints and feedback are investigated and acted upon0.672
 XQ11 Staff opinion or perception on organizational quality and safety culture are measured and evaluated0.239a
Quality control and monitoring0.785
 XQ01 The hospital governance board approved a current program for quality improvement (QI)0.535
 XQ02 The hospital governance board received regular, formal reports on quality and safety (Q&S)0.334a
 XQ05 Clinical leaders received regular, formal reports on Q&S0.715
 XQ06 There is an active clinical guideline register0.706
 XQ07 Processes for implementation and evaluation of clinical guidelines against practice0.532
 XQ08 Clinical incidents (adverse events) are analysed and evaluated0.541

aItem-total correlation coefficient is less than the acceptable value of 0.4.

CQII—structure, reliability and non-redundancy

We retained the original seven-factor theoretical model for CQII. The model can be found in Table 4 along with the Cronbach’s |$\alpha$| and item-total correlations. Only three factors (‘preventing pressure injuries’, ‘routine assessment and diagnostic testing of patients in elective surgery’, and ‘Safe surgery that includes an approved checklist’) achieved internal consistency through Cronbach’s |$\alpha$| and item-total correlations. The other four subscales and their constituent items failed to meet preferred thresholds. We explored removing items to improve Cronbach’s |$\alpha$|⁠, but were unable to significantly improve results. This was not surprising, as the majority of hospitals scored the maximum plausible value of four for these items (the mean of each of these four subscales was 3.9). We decided to retain these items for theoretical reasons (presented in the Discussion); however, the limited range in scale scores meant that numerous item-total correlations were not calculable. Supplementary eTable A6 provides the intra-scale correlation coefficients for CQII. The correlation coefficient between the ‘Medication safety’ and ‘Recognizing and responding to clinical deterioration in acute healthcare’ subscales was 0.88, the only subscale pairing indicating potential redundancy.

Table 4

Cronbach’s alpha and item-total correlation for CQII

Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Preventing and controlling healthcare-associated infections (HAI)0.257
 XC01.1 Existence of a committee responsible for preventing and controlling HAINCa
 XC02.1 Existence of hospital policy or guidelines for preventing and controlling HAINCa
 XC03.1 Monitoring of compliance with policy or guidelines for preventing and controlling HAI0.192b
 XC04.1 Sustainability of system for preventing and controlling HAI0.111b
 XC05.1 Improvement focus for preventing and controlling HAI0.315b
Medication safety0.160
 XC01.2 Existence of a committee responsible for medication safetyNCa
 XC02.2 Existence of hospital policy or guidelines for medication safetyNCa
 XC03.2 Monitoring of compliance with policy or guidelines for medication safety0.152b
 XC04.2 Sustainability of system for medication safety0.086b
 XC05.2 Improvement focus for medication safety0.226b
Preventing patient falls0.177
 XC01.3 Existence of a committee responsible for preventing patient fallsNCa
 XC02.3 Existence of hospital policy or guidelines for preventing patient falls0.212b
 XC03.3 Monitoring of compliance with policy or guidelines for preventing patient fallsNCa
 XC04.3 Sustainability of system for preventing patient falls0.220b
 XC05.3 Improvement focus for preventing patient falls0.010b
Preventing pressure injuries0.804
 XC01.4 Existence of a committee responsible for preventing pressure injuriesNCa
 XC02.4 Existence of hospital policy or guidelines for preventing pressure injuries0.847
 XC03.4 Monitoring of compliance with policy or guidelines for preventing pressure injuries0.841
 XC04.4 Sustainability of system for preventing pressure injuries0.739
 XC05.4 Improvement focus for preventing pressure injuries0.490
Routine assessment and diagnostic testing of patients in elective surgery (ESP)0.920
 XC01.5 Existence of a committee responsible for routine assessment and diagnostic testing of ESP0.678
 XC02.5 Existence of hospital policy or guidelines for routine assessment and diagnostic testing of ESP0.891
 XC03.5 Monitoring of compliance with policy or guidelines for routine assessment and diagnostic testing of ESP0.825
 XC04.5 Sustainability of system for routine assessment and diagnostic testing of ESP0.842
 XC05.5 Improvement focus for routine assessment and diagnostic testing of ESP0.745
Safe surgery that includes an approved checklist0.956
 XC01.6 Existence of a committee responsible for safe surgery0.884
 XC02.6 Existence of hospital policy or guidelines for safe surgery0.970
 XC03.6 Monitoring of compliance with policy or guidelines for safe surgery0.961
 XC04.6 Sustainability of system for safe surgery0.872
 XC05.6 Improvement focus for safe surgery0.784
Recognizing and responding to clinical deterioration in acute healthcare (CDAHC)−0.003
 XC01.7 Existence of a committee responsible for recognizing and responding to CDAHCNCa
 XC02.7 Existence of hospital policy or guidelines for recognizing and responding to CDAHCNCa
 XC03.7 Monitoring of compliance with policy or guidelines for recognizing and responding to CDAHC0.129b
 XC04.7 Sustainability of system for recognizing and responding to CDAHC−0.104b
 XC05.7 Improvement focus for recognizing and responding to CDAHC0.004b
Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Preventing and controlling healthcare-associated infections (HAI)0.257
 XC01.1 Existence of a committee responsible for preventing and controlling HAINCa
 XC02.1 Existence of hospital policy or guidelines for preventing and controlling HAINCa
 XC03.1 Monitoring of compliance with policy or guidelines for preventing and controlling HAI0.192b
 XC04.1 Sustainability of system for preventing and controlling HAI0.111b
 XC05.1 Improvement focus for preventing and controlling HAI0.315b
Medication safety0.160
 XC01.2 Existence of a committee responsible for medication safetyNCa
 XC02.2 Existence of hospital policy or guidelines for medication safetyNCa
 XC03.2 Monitoring of compliance with policy or guidelines for medication safety0.152b
 XC04.2 Sustainability of system for medication safety0.086b
 XC05.2 Improvement focus for medication safety0.226b
Preventing patient falls0.177
 XC01.3 Existence of a committee responsible for preventing patient fallsNCa
 XC02.3 Existence of hospital policy or guidelines for preventing patient falls0.212b
 XC03.3 Monitoring of compliance with policy or guidelines for preventing patient fallsNCa
 XC04.3 Sustainability of system for preventing patient falls0.220b
 XC05.3 Improvement focus for preventing patient falls0.010b
Preventing pressure injuries0.804
 XC01.4 Existence of a committee responsible for preventing pressure injuriesNCa
 XC02.4 Existence of hospital policy or guidelines for preventing pressure injuries0.847
 XC03.4 Monitoring of compliance with policy or guidelines for preventing pressure injuries0.841
 XC04.4 Sustainability of system for preventing pressure injuries0.739
 XC05.4 Improvement focus for preventing pressure injuries0.490
Routine assessment and diagnostic testing of patients in elective surgery (ESP)0.920
 XC01.5 Existence of a committee responsible for routine assessment and diagnostic testing of ESP0.678
 XC02.5 Existence of hospital policy or guidelines for routine assessment and diagnostic testing of ESP0.891
 XC03.5 Monitoring of compliance with policy or guidelines for routine assessment and diagnostic testing of ESP0.825
 XC04.5 Sustainability of system for routine assessment and diagnostic testing of ESP0.842
 XC05.5 Improvement focus for routine assessment and diagnostic testing of ESP0.745
Safe surgery that includes an approved checklist0.956
 XC01.6 Existence of a committee responsible for safe surgery0.884
 XC02.6 Existence of hospital policy or guidelines for safe surgery0.970
 XC03.6 Monitoring of compliance with policy or guidelines for safe surgery0.961
 XC04.6 Sustainability of system for safe surgery0.872
 XC05.6 Improvement focus for safe surgery0.784
Recognizing and responding to clinical deterioration in acute healthcare (CDAHC)−0.003
 XC01.7 Existence of a committee responsible for recognizing and responding to CDAHCNCa
 XC02.7 Existence of hospital policy or guidelines for recognizing and responding to CDAHCNCa
 XC03.7 Monitoring of compliance with policy or guidelines for recognizing and responding to CDAHC0.129b
 XC04.7 Sustainability of system for recognizing and responding to CDAHC−0.104b
 XC05.7 Improvement focus for recognizing and responding to CDAHC0.004b

aNC, not calculated; item-total correlations were not derived for items with zero variance.

bItem-total correlation coefficient is less than the acceptable value of 0.4.

Table 4

Cronbach’s alpha and item-total correlation for CQII

Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Preventing and controlling healthcare-associated infections (HAI)0.257
 XC01.1 Existence of a committee responsible for preventing and controlling HAINCa
 XC02.1 Existence of hospital policy or guidelines for preventing and controlling HAINCa
 XC03.1 Monitoring of compliance with policy or guidelines for preventing and controlling HAI0.192b
 XC04.1 Sustainability of system for preventing and controlling HAI0.111b
 XC05.1 Improvement focus for preventing and controlling HAI0.315b
Medication safety0.160
 XC01.2 Existence of a committee responsible for medication safetyNCa
 XC02.2 Existence of hospital policy or guidelines for medication safetyNCa
 XC03.2 Monitoring of compliance with policy or guidelines for medication safety0.152b
 XC04.2 Sustainability of system for medication safety0.086b
 XC05.2 Improvement focus for medication safety0.226b
Preventing patient falls0.177
 XC01.3 Existence of a committee responsible for preventing patient fallsNCa
 XC02.3 Existence of hospital policy or guidelines for preventing patient falls0.212b
 XC03.3 Monitoring of compliance with policy or guidelines for preventing patient fallsNCa
 XC04.3 Sustainability of system for preventing patient falls0.220b
 XC05.3 Improvement focus for preventing patient falls0.010b
Preventing pressure injuries0.804
 XC01.4 Existence of a committee responsible for preventing pressure injuriesNCa
 XC02.4 Existence of hospital policy or guidelines for preventing pressure injuries0.847
 XC03.4 Monitoring of compliance with policy or guidelines for preventing pressure injuries0.841
 XC04.4 Sustainability of system for preventing pressure injuries0.739
 XC05.4 Improvement focus for preventing pressure injuries0.490
Routine assessment and diagnostic testing of patients in elective surgery (ESP)0.920
 XC01.5 Existence of a committee responsible for routine assessment and diagnostic testing of ESP0.678
 XC02.5 Existence of hospital policy or guidelines for routine assessment and diagnostic testing of ESP0.891
 XC03.5 Monitoring of compliance with policy or guidelines for routine assessment and diagnostic testing of ESP0.825
 XC04.5 Sustainability of system for routine assessment and diagnostic testing of ESP0.842
 XC05.5 Improvement focus for routine assessment and diagnostic testing of ESP0.745
Safe surgery that includes an approved checklist0.956
 XC01.6 Existence of a committee responsible for safe surgery0.884
 XC02.6 Existence of hospital policy or guidelines for safe surgery0.970
 XC03.6 Monitoring of compliance with policy or guidelines for safe surgery0.961
 XC04.6 Sustainability of system for safe surgery0.872
 XC05.6 Improvement focus for safe surgery0.784
Recognizing and responding to clinical deterioration in acute healthcare (CDAHC)−0.003
 XC01.7 Existence of a committee responsible for recognizing and responding to CDAHCNCa
 XC02.7 Existence of hospital policy or guidelines for recognizing and responding to CDAHCNCa
 XC03.7 Monitoring of compliance with policy or guidelines for recognizing and responding to CDAHC0.129b
 XC04.7 Sustainability of system for recognizing and responding to CDAHC−0.104b
 XC05.7 Improvement focus for recognizing and responding to CDAHC0.004b
Scale and itemsCronbach’s |$\alpha$|Item-total correlation
Preventing and controlling healthcare-associated infections (HAI)0.257
 XC01.1 Existence of a committee responsible for preventing and controlling HAINCa
 XC02.1 Existence of hospital policy or guidelines for preventing and controlling HAINCa
 XC03.1 Monitoring of compliance with policy or guidelines for preventing and controlling HAI0.192b
 XC04.1 Sustainability of system for preventing and controlling HAI0.111b
 XC05.1 Improvement focus for preventing and controlling HAI0.315b
Medication safety0.160
 XC01.2 Existence of a committee responsible for medication safetyNCa
 XC02.2 Existence of hospital policy or guidelines for medication safetyNCa
 XC03.2 Monitoring of compliance with policy or guidelines for medication safety0.152b
 XC04.2 Sustainability of system for medication safety0.086b
 XC05.2 Improvement focus for medication safety0.226b
Preventing patient falls0.177
 XC01.3 Existence of a committee responsible for preventing patient fallsNCa
 XC02.3 Existence of hospital policy or guidelines for preventing patient falls0.212b
 XC03.3 Monitoring of compliance with policy or guidelines for preventing patient fallsNCa
 XC04.3 Sustainability of system for preventing patient falls0.220b
 XC05.3 Improvement focus for preventing patient falls0.010b
Preventing pressure injuries0.804
 XC01.4 Existence of a committee responsible for preventing pressure injuriesNCa
 XC02.4 Existence of hospital policy or guidelines for preventing pressure injuries0.847
 XC03.4 Monitoring of compliance with policy or guidelines for preventing pressure injuries0.841
 XC04.4 Sustainability of system for preventing pressure injuries0.739
 XC05.4 Improvement focus for preventing pressure injuries0.490
Routine assessment and diagnostic testing of patients in elective surgery (ESP)0.920
 XC01.5 Existence of a committee responsible for routine assessment and diagnostic testing of ESP0.678
 XC02.5 Existence of hospital policy or guidelines for routine assessment and diagnostic testing of ESP0.891
 XC03.5 Monitoring of compliance with policy or guidelines for routine assessment and diagnostic testing of ESP0.825
 XC04.5 Sustainability of system for routine assessment and diagnostic testing of ESP0.842
 XC05.5 Improvement focus for routine assessment and diagnostic testing of ESP0.745
Safe surgery that includes an approved checklist0.956
 XC01.6 Existence of a committee responsible for safe surgery0.884
 XC02.6 Existence of hospital policy or guidelines for safe surgery0.970
 XC03.6 Monitoring of compliance with policy or guidelines for safe surgery0.961
 XC04.6 Sustainability of system for safe surgery0.872
 XC05.6 Improvement focus for safe surgery0.784
Recognizing and responding to clinical deterioration in acute healthcare (CDAHC)−0.003
 XC01.7 Existence of a committee responsible for recognizing and responding to CDAHCNCa
 XC02.7 Existence of hospital policy or guidelines for recognizing and responding to CDAHCNCa
 XC03.7 Monitoring of compliance with policy or guidelines for recognizing and responding to CDAHC0.129b
 XC04.7 Sustainability of system for recognizing and responding to CDAHC−0.104b
 XC05.7 Improvement focus for recognizing and responding to CDAHC0.004b

aNC, not calculated; item-total correlations were not derived for items with zero variance.

bItem-total correlation coefficient is less than the acceptable value of 0.4.

Department scales—structure, reliability and non-redundancy

Missing data

There were 119 participating departments (27 AMI, 29 hip fracture, 32 stroke and 32 ED). For EBOP and CR, there were no missing data; for SER, missing items were restricted to hip fracture departments, with 1.1% of hip fracture items missing (ranging from 0 to 3.5% per item); and for PSS, the percentage of missing items was 0.4% (ranging from 0 to 3.8%) per item in AMI and 0.4% (ranging from 0 to 3.5%) per item in hip fracture. There were no departments with a scale missing.

Descriptive statistics for SER, EBOP, PSS and CR

Descriptive statistics for SER, EBOP, PSS and CR are summarized in Table 5 and for their items in Supplementary eTable A7. Items were left-skewed with all means calculated at two or above (in a 0–4 plausible range). Consistently across the three departments, PSS had the highest (3.4–3.6) and CR had the lowest (2.2–2.9) mean scores.

Table 5

Descriptive statistics for department-level scales: condition-specific departments

DepartmentScaleNMeanSDMedianMinMax
AMISER272.90.932.70.74
EBOP273.00.773.314
PSS273.60.343.62.94
CR272.31.502.704
Hip fractureSER292.51.162.70.34
EBOP292.61.062.60.24
PSS293.50.373.42.74
CR292.21.602.704
StrokeSER313.30.743.71.34
EBOP313.20.893.40.84
PSS313.40.333.32.64
CR312.91.253.304
DepartmentScaleNMeanSDMedianMinMax
AMISER272.90.932.70.74
EBOP273.00.773.314
PSS273.60.343.62.94
CR272.31.502.704
Hip fractureSER292.51.162.70.34
EBOP292.61.062.60.24
PSS293.50.373.42.74
CR292.21.602.704
StrokeSER313.30.743.71.34
EBOP313.20.893.40.84
PSS313.40.333.32.64
CR312.91.253.304
Table 5

Descriptive statistics for department-level scales: condition-specific departments

DepartmentScaleNMeanSDMedianMinMax
AMISER272.90.932.70.74
EBOP273.00.773.314
PSS273.60.343.62.94
CR272.31.502.704
Hip fractureSER292.51.162.70.34
EBOP292.61.062.60.24
PSS293.50.373.42.74
CR292.21.602.704
StrokeSER313.30.743.71.34
EBOP313.20.893.40.84
PSS313.40.333.32.64
CR312.91.253.304
DepartmentScaleNMeanSDMedianMinMax
AMISER272.90.932.70.74
EBOP273.00.773.314
PSS273.60.343.62.94
CR272.31.502.704
Hip fractureSER292.51.162.70.34
EBOP292.61.062.60.24
PSS293.50.373.42.74
CR292.21.602.704
StrokeSER313.30.743.71.34
EBOP313.20.893.40.84
PSS313.40.333.32.64
CR312.91.253.304
Table 6

Descriptive statistics of ED-level scales and items

ConditionScalenMeanSDMedianMinMax
GenericPSS323.40.433.62.34
AMISER312.51.092.514
EBOP313.60.513.82.34
CR312.01.622.304
Hip fractureSER311.91.33204
EBOP312.51.27304
CR311.31.32104
StrokeSER322.81.24304
EBOP323.40.723.71.34
CR322.01.64204
ConditionScalenMeanSDMedianMinMax
GenericPSS323.40.433.62.34
AMISER312.51.092.514
EBOP313.60.513.82.34
CR312.01.622.304
Hip fractureSER311.91.33204
EBOP312.51.27304
CR311.31.32104
StrokeSER322.81.24304
EBOP323.40.723.71.34
CR322.01.64204
Table 6

Descriptive statistics of ED-level scales and items

ConditionScalenMeanSDMedianMinMax
GenericPSS323.40.433.62.34
AMISER312.51.092.514
EBOP313.60.513.82.34
CR312.01.622.304
Hip fractureSER311.91.33204
EBOP312.51.27304
CR311.31.32104
StrokeSER322.81.24304
EBOP323.40.723.71.34
CR322.01.64204
ConditionScalenMeanSDMedianMinMax
GenericPSS323.40.433.62.34
AMISER312.51.092.514
EBOP313.60.513.82.34
CR312.01.622.304
Hip fractureSER311.91.33204
EBOP312.51.27304
CR311.31.32104
StrokeSER322.81.24304
EBOP323.40.723.71.34
CR322.01.64204
Table 7

Cronbach’s alpha and item-total correlation for department-level scales: condition-specific

DepartmentScale and itemsCronbach’s |$\alpha$|Item-total correlation
AMISER0.524
 AS01 There is a strategic committee within the hospital responsible for the overall clinical management of AMI0.458
 AS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of AMI patients0.287b
 AS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with AMI0.374b
EBOP0.516
 AE01 There is a specialist (consultant) doctor available at all times to determine whether fibrinolysis or percutaneous coronary intervention (PCI) is appropriate−0.063b
 AE02 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.188b
 AE03 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.529
 AE04 There are written criteria and procedures to ensure information on episode of care is provided to usual clinical provider0.717
PSS0.107
 AP01 Patients are identified by bracelet−0.038b
 AP02 Safety boxes for disposal of injection devices are available−0.115b
 AP03 Promotional hand hygiene reminders are on display in the workplace0.101b
 AP04 Readily accessible hand sanitizer is provided at the point of patient care−0.088b
 AP05 No concentrated potassium chloride (KCl) stored on the ward−0.088b
 AP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.131b
 AP07 Each emergency crash cart has a completed checklist of equipment and supplies0.184b
 AP08 There is a system to report clinical incidents (adverse events)0.212b
 AP09 Peer review included analysis of reported clinical incidents (adverse events)−0.047b
CR0.883
 AC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ward or department0.738
 AC02 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ward or department0.816
 AC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.771
Hip FractureSER0.517
 HS01 There is a strategic committee within the hospital responsible for the overall clinical management of hip fracture0.266b
 HS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of hip fracture patients0.499
 HS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with hip fracture0.252b
EBOP0.803
 HE01 There is a clear clinical path for management of hip fracture surgery0.510
 HE02 There are written criteria and procedures to ensure appropriate prophylactic medication is administered0.645
 HE03 There are written criteria and procedures for patient mobilization following surgery0.623
 HE04 There are written criteria and procedures to ensure fall prevention assessment is provided post-surgery and prior to discharge0.634
 HE05 There are written criteria and procedures to ensure treatment for secondary fracture prevention on discharge0.572
PSS0.167
 HP01 Patients are identified by bracelet0.129b
 HP02 Safety boxes for disposal of injection devices are availableNCa
 HP03 Promotional hand hygiene reminders are on display in the workplace0.099b
 HP04 Readily accessible hand sanitizer is provided at the point of patient care0.163b
 HP05 No concentrated potassium chloride (KCl) stored on the ward0.035b
 HP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.113b
 HP07 Each emergency crash cart has a completed checklist of equipment and supplies0.017b
 HP08 There is a system to report clinical incidents (adverse events)0.028b
 HP09 Peer review included analysis of reported clinical incidents (adverse events)0.012b
CR0.867
 HC01 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ward or department0.663
 HC02 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ward or department0.754
 HC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.832
StrokeSER0.318
 SS01 There is a strategic committee within the hospital responsible for the overall clinical management of stroke0.187b
 SS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of stroke patients0.041b
 SS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with stroke0.456
EBOP0.747
 SE01 There is a specialist (consultant) doctor available at all times to determine whether intravenous rt-PA is appropriate−0.160b
 SE02 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.657
 SE03 There are written criteria and procedures for allocation of ischaemic stroke patients to the stroke unit0.614
 SE04 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.589
 SE05 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.746
PSS−0.120
 SP01 Patients are identified by bracelet0.459
 SP02 Safety boxes for disposal of injection devices are available−0.181b
 SP03 Promotional hand hygiene reminders are on display in the workplace0.129b
 SP04 Readily accessible hand sanitizer is provided at the point of patient care−0.044b
 SP05 No concentrated potassium chloride (KCl) stored on the ward−0.315b
 SP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart−0.112b
 SP07 Each emergency crash cart has a completed checklist of equipment and supplies0.251b
 SP08 There is a system to report clinical incidents (adverse events)0.118b
 SP09 Peer review included analysis of reported clinical incidents (adverse events)0.082b
CR0.744
 SC01 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ward or department0.528
 SC02 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ward or department0.685
 SC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.511
DepartmentScale and itemsCronbach’s |$\alpha$|Item-total correlation
AMISER0.524
 AS01 There is a strategic committee within the hospital responsible for the overall clinical management of AMI0.458
 AS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of AMI patients0.287b
 AS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with AMI0.374b
EBOP0.516
 AE01 There is a specialist (consultant) doctor available at all times to determine whether fibrinolysis or percutaneous coronary intervention (PCI) is appropriate−0.063b
 AE02 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.188b
 AE03 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.529
 AE04 There are written criteria and procedures to ensure information on episode of care is provided to usual clinical provider0.717
PSS0.107
 AP01 Patients are identified by bracelet−0.038b
 AP02 Safety boxes for disposal of injection devices are available−0.115b
 AP03 Promotional hand hygiene reminders are on display in the workplace0.101b
 AP04 Readily accessible hand sanitizer is provided at the point of patient care−0.088b
 AP05 No concentrated potassium chloride (KCl) stored on the ward−0.088b
 AP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.131b
 AP07 Each emergency crash cart has a completed checklist of equipment and supplies0.184b
 AP08 There is a system to report clinical incidents (adverse events)0.212b
 AP09 Peer review included analysis of reported clinical incidents (adverse events)−0.047b
CR0.883
 AC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ward or department0.738
 AC02 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ward or department0.816
 AC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.771
Hip FractureSER0.517
 HS01 There is a strategic committee within the hospital responsible for the overall clinical management of hip fracture0.266b
 HS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of hip fracture patients0.499
 HS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with hip fracture0.252b
EBOP0.803
 HE01 There is a clear clinical path for management of hip fracture surgery0.510
 HE02 There are written criteria and procedures to ensure appropriate prophylactic medication is administered0.645
 HE03 There are written criteria and procedures for patient mobilization following surgery0.623
 HE04 There are written criteria and procedures to ensure fall prevention assessment is provided post-surgery and prior to discharge0.634
 HE05 There are written criteria and procedures to ensure treatment for secondary fracture prevention on discharge0.572
PSS0.167
 HP01 Patients are identified by bracelet0.129b
 HP02 Safety boxes for disposal of injection devices are availableNCa
 HP03 Promotional hand hygiene reminders are on display in the workplace0.099b
 HP04 Readily accessible hand sanitizer is provided at the point of patient care0.163b
 HP05 No concentrated potassium chloride (KCl) stored on the ward0.035b
 HP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.113b
 HP07 Each emergency crash cart has a completed checklist of equipment and supplies0.017b
 HP08 There is a system to report clinical incidents (adverse events)0.028b
 HP09 Peer review included analysis of reported clinical incidents (adverse events)0.012b
CR0.867
 HC01 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ward or department0.663
 HC02 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ward or department0.754
 HC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.832
StrokeSER0.318
 SS01 There is a strategic committee within the hospital responsible for the overall clinical management of stroke0.187b
 SS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of stroke patients0.041b
 SS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with stroke0.456
EBOP0.747
 SE01 There is a specialist (consultant) doctor available at all times to determine whether intravenous rt-PA is appropriate−0.160b
 SE02 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.657
 SE03 There are written criteria and procedures for allocation of ischaemic stroke patients to the stroke unit0.614
 SE04 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.589
 SE05 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.746
PSS−0.120
 SP01 Patients are identified by bracelet0.459
 SP02 Safety boxes for disposal of injection devices are available−0.181b
 SP03 Promotional hand hygiene reminders are on display in the workplace0.129b
 SP04 Readily accessible hand sanitizer is provided at the point of patient care−0.044b
 SP05 No concentrated potassium chloride (KCl) stored on the ward−0.315b
 SP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart−0.112b
 SP07 Each emergency crash cart has a completed checklist of equipment and supplies0.251b
 SP08 There is a system to report clinical incidents (adverse events)0.118b
 SP09 Peer review included analysis of reported clinical incidents (adverse events)0.082b
CR0.744
 SC01 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ward or department0.528
 SC02 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ward or department0.685
 SC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.511

aNC, not calculated; item-total correlations were not derived for items with zero variance.

bItem-total correlation coefficient is less than the acceptable value of 0.4.

Table 7

Cronbach’s alpha and item-total correlation for department-level scales: condition-specific

DepartmentScale and itemsCronbach’s |$\alpha$|Item-total correlation
AMISER0.524
 AS01 There is a strategic committee within the hospital responsible for the overall clinical management of AMI0.458
 AS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of AMI patients0.287b
 AS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with AMI0.374b
EBOP0.516
 AE01 There is a specialist (consultant) doctor available at all times to determine whether fibrinolysis or percutaneous coronary intervention (PCI) is appropriate−0.063b
 AE02 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.188b
 AE03 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.529
 AE04 There are written criteria and procedures to ensure information on episode of care is provided to usual clinical provider0.717
PSS0.107
 AP01 Patients are identified by bracelet−0.038b
 AP02 Safety boxes for disposal of injection devices are available−0.115b
 AP03 Promotional hand hygiene reminders are on display in the workplace0.101b
 AP04 Readily accessible hand sanitizer is provided at the point of patient care−0.088b
 AP05 No concentrated potassium chloride (KCl) stored on the ward−0.088b
 AP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.131b
 AP07 Each emergency crash cart has a completed checklist of equipment and supplies0.184b
 AP08 There is a system to report clinical incidents (adverse events)0.212b
 AP09 Peer review included analysis of reported clinical incidents (adverse events)−0.047b
CR0.883
 AC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ward or department0.738
 AC02 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ward or department0.816
 AC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.771
Hip FractureSER0.517
 HS01 There is a strategic committee within the hospital responsible for the overall clinical management of hip fracture0.266b
 HS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of hip fracture patients0.499
 HS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with hip fracture0.252b
EBOP0.803
 HE01 There is a clear clinical path for management of hip fracture surgery0.510
 HE02 There are written criteria and procedures to ensure appropriate prophylactic medication is administered0.645
 HE03 There are written criteria and procedures for patient mobilization following surgery0.623
 HE04 There are written criteria and procedures to ensure fall prevention assessment is provided post-surgery and prior to discharge0.634
 HE05 There are written criteria and procedures to ensure treatment for secondary fracture prevention on discharge0.572
PSS0.167
 HP01 Patients are identified by bracelet0.129b
 HP02 Safety boxes for disposal of injection devices are availableNCa
 HP03 Promotional hand hygiene reminders are on display in the workplace0.099b
 HP04 Readily accessible hand sanitizer is provided at the point of patient care0.163b
 HP05 No concentrated potassium chloride (KCl) stored on the ward0.035b
 HP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.113b
 HP07 Each emergency crash cart has a completed checklist of equipment and supplies0.017b
 HP08 There is a system to report clinical incidents (adverse events)0.028b
 HP09 Peer review included analysis of reported clinical incidents (adverse events)0.012b
CR0.867
 HC01 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ward or department0.663
 HC02 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ward or department0.754
 HC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.832
StrokeSER0.318
 SS01 There is a strategic committee within the hospital responsible for the overall clinical management of stroke0.187b
 SS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of stroke patients0.041b
 SS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with stroke0.456
EBOP0.747
 SE01 There is a specialist (consultant) doctor available at all times to determine whether intravenous rt-PA is appropriate−0.160b
 SE02 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.657
 SE03 There are written criteria and procedures for allocation of ischaemic stroke patients to the stroke unit0.614
 SE04 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.589
 SE05 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.746
PSS−0.120
 SP01 Patients are identified by bracelet0.459
 SP02 Safety boxes for disposal of injection devices are available−0.181b
 SP03 Promotional hand hygiene reminders are on display in the workplace0.129b
 SP04 Readily accessible hand sanitizer is provided at the point of patient care−0.044b
 SP05 No concentrated potassium chloride (KCl) stored on the ward−0.315b
 SP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart−0.112b
 SP07 Each emergency crash cart has a completed checklist of equipment and supplies0.251b
 SP08 There is a system to report clinical incidents (adverse events)0.118b
 SP09 Peer review included analysis of reported clinical incidents (adverse events)0.082b
CR0.744
 SC01 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ward or department0.528
 SC02 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ward or department0.685
 SC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.511
DepartmentScale and itemsCronbach’s |$\alpha$|Item-total correlation
AMISER0.524
 AS01 There is a strategic committee within the hospital responsible for the overall clinical management of AMI0.458
 AS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of AMI patients0.287b
 AS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with AMI0.374b
EBOP0.516
 AE01 There is a specialist (consultant) doctor available at all times to determine whether fibrinolysis or percutaneous coronary intervention (PCI) is appropriate−0.063b
 AE02 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.188b
 AE03 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.529
 AE04 There are written criteria and procedures to ensure information on episode of care is provided to usual clinical provider0.717
PSS0.107
 AP01 Patients are identified by bracelet−0.038b
 AP02 Safety boxes for disposal of injection devices are available−0.115b
 AP03 Promotional hand hygiene reminders are on display in the workplace0.101b
 AP04 Readily accessible hand sanitizer is provided at the point of patient care−0.088b
 AP05 No concentrated potassium chloride (KCl) stored on the ward−0.088b
 AP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.131b
 AP07 Each emergency crash cart has a completed checklist of equipment and supplies0.184b
 AP08 There is a system to report clinical incidents (adverse events)0.212b
 AP09 Peer review included analysis of reported clinical incidents (adverse events)−0.047b
CR0.883
 AC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ward or department0.738
 AC02 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ward or department0.816
 AC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.771
Hip FractureSER0.517
 HS01 There is a strategic committee within the hospital responsible for the overall clinical management of hip fracture0.266b
 HS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of hip fracture patients0.499
 HS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with hip fracture0.252b
EBOP0.803
 HE01 There is a clear clinical path for management of hip fracture surgery0.510
 HE02 There are written criteria and procedures to ensure appropriate prophylactic medication is administered0.645
 HE03 There are written criteria and procedures for patient mobilization following surgery0.623
 HE04 There are written criteria and procedures to ensure fall prevention assessment is provided post-surgery and prior to discharge0.634
 HE05 There are written criteria and procedures to ensure treatment for secondary fracture prevention on discharge0.572
PSS0.167
 HP01 Patients are identified by bracelet0.129b
 HP02 Safety boxes for disposal of injection devices are availableNCa
 HP03 Promotional hand hygiene reminders are on display in the workplace0.099b
 HP04 Readily accessible hand sanitizer is provided at the point of patient care0.163b
 HP05 No concentrated potassium chloride (KCl) stored on the ward0.035b
 HP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.113b
 HP07 Each emergency crash cart has a completed checklist of equipment and supplies0.017b
 HP08 There is a system to report clinical incidents (adverse events)0.028b
 HP09 Peer review included analysis of reported clinical incidents (adverse events)0.012b
CR0.867
 HC01 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ward or department0.663
 HC02 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ward or department0.754
 HC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.832
StrokeSER0.318
 SS01 There is a strategic committee within the hospital responsible for the overall clinical management of stroke0.187b
 SS02 There are clinical leaders with specialist training who are formally recognized as having principal responsibility for overall clinical care of stroke patients0.041b
 SS03 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by clinicians for the management of patients with stroke0.456
EBOP0.747
 SE01 There is a specialist (consultant) doctor available at all times to determine whether intravenous rt-PA is appropriate−0.160b
 SE02 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.657
 SE03 There are written criteria and procedures for allocation of ischaemic stroke patients to the stroke unit0.614
 SE04 There are written criteria and procedures to ensure appropriate medication is prescribed on discharge0.589
 SE05 There are written criteria and procedures to ensure arrangements for ongoing care on discharge0.746
PSS−0.120
 SP01 Patients are identified by bracelet0.459
 SP02 Safety boxes for disposal of injection devices are available−0.181b
 SP03 Promotional hand hygiene reminders are on display in the workplace0.129b
 SP04 Readily accessible hand sanitizer is provided at the point of patient care−0.044b
 SP05 No concentrated potassium chloride (KCl) stored on the ward−0.315b
 SP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart−0.112b
 SP07 Each emergency crash cart has a completed checklist of equipment and supplies0.251b
 SP08 There is a system to report clinical incidents (adverse events)0.118b
 SP09 Peer review included analysis of reported clinical incidents (adverse events)0.082b
CR0.744
 SC01 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ward or department0.528
 SC02 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ward or department0.685
 SC03 Clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.511

aNC, not calculated; item-total correlations were not derived for items with zero variance.

bItem-total correlation coefficient is less than the acceptable value of 0.4.

Descriptive statistics for ED-level scales

Descriptive statistics for ED-level SER, EBOP, PSS and CR are summarized in Table 6 and for their items in Supplementary eTable A8. Items were left-skewed with ceiling effect. The PSS, which was assessed using generic rather than condition-specific questions, had a mean score of 3.4. Across all three conditions, EBOP consistently had the highest mean scores (2.5–3.6) and CR had the lowest mean score (1.3–2.0); mean scores for hip fracture were lower than for either AMI or stroke.

SER, EBOP, PSS, CR—structure, reliability and non-redundancy

Department-level SER, EBOP, PSS and CR. The models are shown in Table 7 along with the Cronbach’s |$\alpha$| and item-total correlations. Internal consistency reliability through Cronbach’s |$\alpha$| was high for CR (0.74–0.88 across the three conditions), and this was supported by all item-total correlations exceeding the desired threshold. For EBOP, Cronbach’s |$\alpha$| was acceptable for hip fracture (0.80) and stroke (0.76), but only moderate for AMI (0.52); as suggested by these results, item-total correlations were poor for one item in stroke (−0.16) and two items in AMI (−0.06 and 0.19). For PSS, Cronbach’s |$\alpha$| suggested its constituents were effectively unrelated (−0.12 to 0.17), with virtually all item-total correlations poor. For SER, Cronbach’s |$\alpha$| showed a moderate association amongst its constituent items (0.32–0.52), again with relatively few item-total correlations above 0.40.

Table 8

Cronbach’s alpha and item-total correlation for ED-level scales

ConditionScale and itemsCronbach’s |$\alpha$|Item-total correlation
GenericPSS0.438
 EP01 Patients are identified by bracelet0.087b
 EP02 Safety boxes for disposal of injection devices are availableNCa
 EP03 Promotional hand hygiene reminders are on display in the workplace0.183b
 EP04 Readily accessible hand sanitizer is provided at the point of patient care0.454
 EP05 No concentrated potassium chloride (KCl) stored on the ward−0.118b
 EP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.257b
 EP07 Each emergency crash cart has a completed checklist of equipment and supplies0.343b
 EP08 There is a system to report clinical incidents (adverse events)0.484
 EP09 Peer review included analysis of reported clinical incidents (adverse events)0.405
AMISER0.016
 ES01 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of AMI0.009b
 ES04 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with AMI0.009b
EBOP0.455
 EE01 There are written criteria and procedures for fast track admission and treatment of patients presenting with acute chest pain0.399b
 EE02 There are written criteria and procedures to ensure that eligible ST-Elevation Myocardial Infarction patients receive fibrinolysis0.296b
 EE03 There is a clear procedure in the ED to enable immediate transport or transfer for Percutaneous Coronary Intervention (PCI) for eligible STEMI patients0.261b
 EE11 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether fibrinolysis or PCI is appropriate0.221b
CR0.888
 EC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ED0.848
 EC04 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ED0.700
 EC07 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.802
Hip fractureSER0.402
 ES03 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of hip fracture0.253b
 ES06 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with hip fracture0.253b
EBOP0.744
 EE09 There are written criteria and procedures for initial pain score and pain relief for patients with suspected hip fracture0.595
 EE10 There are written criteria and procedures for fast track admission and treatment of patients presenting with hip fracture0.595
CR0.804
 EC03 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ED0.596
 EC06 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ED0.714
 EC09 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.675
StrokeSER0.407
 ES02 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of stroke0.271b
 ES05 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with stroke0.271b
EBOP0.781
 EE04 There are written criteria and procedures for ensuring that patients with suspected stroke are screened and assessed for eligibility to receive intravenous rt-PA0.616
 EE05 There are written criteria and procedures to ensure that eligible ischaemic stroke patients receive intravenous rt-PA0.689
 EE06 There is a clear procedure in the ED to enable immediate transport or transfer for brain imaging (e.g. CT scan, Magnetic Resonance Angiogram)0.443
 EE07 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.462
 EE08 There are written criteria and procedures for fast track admission and treatment of ischaemic stroke patients0.774
 EE12 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether intravenous rt-PA is appropriate0.166b
CR0.927
 EC02 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ED0.925
 EC05 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ED0.856
 EC08 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.775
ConditionScale and itemsCronbach’s |$\alpha$|Item-total correlation
GenericPSS0.438
 EP01 Patients are identified by bracelet0.087b
 EP02 Safety boxes for disposal of injection devices are availableNCa
 EP03 Promotional hand hygiene reminders are on display in the workplace0.183b
 EP04 Readily accessible hand sanitizer is provided at the point of patient care0.454
 EP05 No concentrated potassium chloride (KCl) stored on the ward−0.118b
 EP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.257b
 EP07 Each emergency crash cart has a completed checklist of equipment and supplies0.343b
 EP08 There is a system to report clinical incidents (adverse events)0.484
 EP09 Peer review included analysis of reported clinical incidents (adverse events)0.405
AMISER0.016
 ES01 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of AMI0.009b
 ES04 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with AMI0.009b
EBOP0.455
 EE01 There are written criteria and procedures for fast track admission and treatment of patients presenting with acute chest pain0.399b
 EE02 There are written criteria and procedures to ensure that eligible ST-Elevation Myocardial Infarction patients receive fibrinolysis0.296b
 EE03 There is a clear procedure in the ED to enable immediate transport or transfer for Percutaneous Coronary Intervention (PCI) for eligible STEMI patients0.261b
 EE11 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether fibrinolysis or PCI is appropriate0.221b
CR0.888
 EC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ED0.848
 EC04 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ED0.700
 EC07 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.802
Hip fractureSER0.402
 ES03 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of hip fracture0.253b
 ES06 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with hip fracture0.253b
EBOP0.744
 EE09 There are written criteria and procedures for initial pain score and pain relief for patients with suspected hip fracture0.595
 EE10 There are written criteria and procedures for fast track admission and treatment of patients presenting with hip fracture0.595
CR0.804
 EC03 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ED0.596
 EC06 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ED0.714
 EC09 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.675
StrokeSER0.407
 ES02 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of stroke0.271b
 ES05 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with stroke0.271b
EBOP0.781
 EE04 There are written criteria and procedures for ensuring that patients with suspected stroke are screened and assessed for eligibility to receive intravenous rt-PA0.616
 EE05 There are written criteria and procedures to ensure that eligible ischaemic stroke patients receive intravenous rt-PA0.689
 EE06 There is a clear procedure in the ED to enable immediate transport or transfer for brain imaging (e.g. CT scan, Magnetic Resonance Angiogram)0.443
 EE07 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.462
 EE08 There are written criteria and procedures for fast track admission and treatment of ischaemic stroke patients0.774
 EE12 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether intravenous rt-PA is appropriate0.166b
CR0.927
 EC02 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ED0.925
 EC05 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ED0.856
 EC08 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.775

aNC, not calculated; item-total correlations were not derived for items with zero variance.

bItem-total correlation coefficient is less than the acceptable value of 0.4.

Table 8

Cronbach’s alpha and item-total correlation for ED-level scales

ConditionScale and itemsCronbach’s |$\alpha$|Item-total correlation
GenericPSS0.438
 EP01 Patients are identified by bracelet0.087b
 EP02 Safety boxes for disposal of injection devices are availableNCa
 EP03 Promotional hand hygiene reminders are on display in the workplace0.183b
 EP04 Readily accessible hand sanitizer is provided at the point of patient care0.454
 EP05 No concentrated potassium chloride (KCl) stored on the ward−0.118b
 EP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.257b
 EP07 Each emergency crash cart has a completed checklist of equipment and supplies0.343b
 EP08 There is a system to report clinical incidents (adverse events)0.484
 EP09 Peer review included analysis of reported clinical incidents (adverse events)0.405
AMISER0.016
 ES01 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of AMI0.009b
 ES04 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with AMI0.009b
EBOP0.455
 EE01 There are written criteria and procedures for fast track admission and treatment of patients presenting with acute chest pain0.399b
 EE02 There are written criteria and procedures to ensure that eligible ST-Elevation Myocardial Infarction patients receive fibrinolysis0.296b
 EE03 There is a clear procedure in the ED to enable immediate transport or transfer for Percutaneous Coronary Intervention (PCI) for eligible STEMI patients0.261b
 EE11 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether fibrinolysis or PCI is appropriate0.221b
CR0.888
 EC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ED0.848
 EC04 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ED0.700
 EC07 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.802
Hip fractureSER0.402
 ES03 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of hip fracture0.253b
 ES06 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with hip fracture0.253b
EBOP0.744
 EE09 There are written criteria and procedures for initial pain score and pain relief for patients with suspected hip fracture0.595
 EE10 There are written criteria and procedures for fast track admission and treatment of patients presenting with hip fracture0.595
CR0.804
 EC03 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ED0.596
 EC06 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ED0.714
 EC09 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.675
StrokeSER0.407
 ES02 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of stroke0.271b
 ES05 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with stroke0.271b
EBOP0.781
 EE04 There are written criteria and procedures for ensuring that patients with suspected stroke are screened and assessed for eligibility to receive intravenous rt-PA0.616
 EE05 There are written criteria and procedures to ensure that eligible ischaemic stroke patients receive intravenous rt-PA0.689
 EE06 There is a clear procedure in the ED to enable immediate transport or transfer for brain imaging (e.g. CT scan, Magnetic Resonance Angiogram)0.443
 EE07 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.462
 EE08 There are written criteria and procedures for fast track admission and treatment of ischaemic stroke patients0.774
 EE12 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether intravenous rt-PA is appropriate0.166b
CR0.927
 EC02 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ED0.925
 EC05 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ED0.856
 EC08 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.775
ConditionScale and itemsCronbach’s |$\alpha$|Item-total correlation
GenericPSS0.438
 EP01 Patients are identified by bracelet0.087b
 EP02 Safety boxes for disposal of injection devices are availableNCa
 EP03 Promotional hand hygiene reminders are on display in the workplace0.183b
 EP04 Readily accessible hand sanitizer is provided at the point of patient care0.454
 EP05 No concentrated potassium chloride (KCl) stored on the ward−0.118b
 EP06 Diagrammatic instructions for resuscitation are displayed in resuscitation areas or attached to crash cart0.257b
 EP07 Each emergency crash cart has a completed checklist of equipment and supplies0.343b
 EP08 There is a system to report clinical incidents (adverse events)0.484
 EP09 Peer review included analysis of reported clinical incidents (adverse events)0.405
AMISER0.016
 ES01 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of AMI0.009b
 ES04 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with AMI0.009b
EBOP0.455
 EE01 There are written criteria and procedures for fast track admission and treatment of patients presenting with acute chest pain0.399b
 EE02 There are written criteria and procedures to ensure that eligible ST-Elevation Myocardial Infarction patients receive fibrinolysis0.296b
 EE03 There is a clear procedure in the ED to enable immediate transport or transfer for Percutaneous Coronary Intervention (PCI) for eligible STEMI patients0.261b
 EE11 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether fibrinolysis or PCI is appropriate0.221b
CR0.888
 EC01 Peer review within the last 12 months included analysis of clinical indicators for the management of AMI in the ED0.848
 EC04 There was a multidisciplinary review within the last 12 months of practice against the AMI guidelines in the ED0.700
 EC07 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the AMI guidelines0.802
Hip fractureSER0.402
 ES03 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of hip fracture0.253b
 ES06 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with hip fracture0.253b
EBOP0.744
 EE09 There are written criteria and procedures for initial pain score and pain relief for patients with suspected hip fracture0.595
 EE10 There are written criteria and procedures for fast track admission and treatment of patients presenting with hip fracture0.595
CR0.804
 EC03 Peer review within the last 12 months included analysis of clinical indicators for the management of hip fracture in the ED0.596
 EC06 There was a multidisciplinary review within the last 12 months of practice against the hip fracture guidelines in the ED0.714
 EC09 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the hip fracture guidelines0.675
StrokeSER0.407
 ES02 A clinician from the ED is a member of the strategic committee within the hospital responsible for the overall clinical management of stroke0.271b
 ES05 Evidence-based clinical guidelines have been disseminated by the hospital and formally adopted by ED clinicians for the management of patients with stroke0.271b
EBOP0.781
 EE04 There are written criteria and procedures for ensuring that patients with suspected stroke are screened and assessed for eligibility to receive intravenous rt-PA0.616
 EE05 There are written criteria and procedures to ensure that eligible ischaemic stroke patients receive intravenous rt-PA0.689
 EE06 There is a clear procedure in the ED to enable immediate transport or transfer for brain imaging (e.g. CT scan, Magnetic Resonance Angiogram)0.443
 EE07 There are written criteria and procedures to ensure that ischaemic stroke patients are screened and assessed for dysphagia0.462
 EE08 There are written criteria and procedures for fast track admission and treatment of ischaemic stroke patients0.774
 EE12 There is immediate access in the ED at all times to a specialist (consultant) doctor to determine whether intravenous rt-PA is appropriate0.166b
CR0.927
 EC02 Peer review within the last 12 months included analysis of clinical indicators for the management of stroke in the ED0.925
 EC05 There was a multidisciplinary review within the last 12 months of practice against the stroke guidelines in the ED0.856
 EC08 ED clinicians receive direct feedback on results within the last 12 months, following audit or review of their practice against the stroke guidelines0.775

aNC, not calculated; item-total correlations were not derived for items with zero variance.

bItem-total correlation coefficient is less than the acceptable value of 0.4.

While exploratory factor analysis was used to reduce and determine which items would be aggregated to build a scale for SER and CR, the items comprising EBOP and PSS were determined based on theoretical importance and background knowledge. It was not possible to build one generic scale for the EBOP, because of the different items across pathways. The other scales developed in this analysis used the same items to compute scores for each pathway. Despite the same items being used across pathways for the quality measure PSS, no generic scale for the four pathways was revealed after factor analysis.

The final models for ED measures are shown in Table 8 along with the Cronbach’s |$\alpha$| scores and item-total correlations. PSS in the ED, assessed through a single set of generic questions, had only a moderate Cronbach’s |$\alpha$| (0.44), but was clearly stronger than in the condition-specific departments. For all three conditions, CR in the ED was again the most reliable scale in terms of both Cronbach’s |$\alpha$| (0.80–0.93) and item-total correlation (all items). EBOP in the ED showed a similar pattern to the condition-specific results, with an acceptable Cronbach’s |$\alpha$| for hip fracture (0.80) and stroke (0.78), but only a moderate score for AMI (0.46). The two SER items relating to AMI in the ED were essentially unrelated (Cronbach’s |$\alpha$| of 0.02) and moderate (0.40–0.41) for the other conditions.

Discussion

Interpretation of results

This study aimed to refine and validate, in the context of the Australian healthcare system, three scales for measuring quality improvement at organization level (QMSI, QMCI and CQII), and four scales for measuring quality improvement at hospital care pathway-level for AMI, hip fracture and stroke conditions (SER, EBOP, PSS and CR).

The final QMSI scale consists of eight subscales, measuring: quality policy, hospital governance board activities, quality resources, quality management, preventive protocols, internal quality methods, general activities and performance monitoring. The final QMCI consists of two subscales, measuring: monitoring patient and professional opinions and quality control and monitoring. The final CQII consists of seven subscales, measuring: preventing and controlling healthcare-associated infections, medical safety, preventing patient falls, preventing pressure injuries, routine assessment and diagnostic testing of patients in elective surgery, safe surgery that includes an approved checklist and recognizing and responding to clinical deterioration in acute healthcare. In addition to the differences with the DUQuE scales noted in the introduction, some items are grouped within different subscales following subscale analysis.

The final CQII scale could not be validated due to the scores clustering at the high end of the scale. This is likely a consequence of mandatory accreditation requirements for Australian hospitals, as there is a strong alignment between the topics addressed by the CQII subscales and accreditation measures. Nevertheless, we believe it is important to retain the CQII, as the performance assessed has been shown via research to be important for patient safety [9, 10], and the scale needs to allow for the prospect that not all future hospitals will be as high performers as those in DUQuA.

Within each scale, we retained some subscales that had Cronbach’s |$\alpha$| lower than the desired 0.8, where those subscales measured aspects of quality that were considered to be a critical component of hospital quality management, or where the lower correlation could be explained by understanding how hospital care was structured. For example, in the QMSI scale, all subscales but the ‘internal quality methods—patients’ achieved (or nearly achieved) internal consistency reliability using the cut-off value. This subscale consisted of ratings of whether patient satisfaction or experience is measured and monitored at least annually, and whether periodical evaluation of patient complaints is used to drive improvements. The DUQuE QMSI scale found a similar result for this subscale [11], and was retained for both studies, recognizing the importance of the patient voice in hospital care. There are good reasons to keep items and scales within the measurement instruments because they are theoretically important, even if they did not fulfil the high statistical standards set before the analysis. While this may seem to undermine the importance of these criteria and the methods used, it is of practical relevance for the final scales to be able to measure an adequate range of elements that make up quality management and quality improvement activities in hospitals.

In many scales and subscales we also had some items below our acceptable correlation cut-off of 0.4. Similarly, we retained items that had lower correlation with other items within the same factor where those items were considered to be an important part of hospital activity. For example, within the QMSI subscale of ‘preventative protocols’, removing the item rating prevention of ventilator-associated pneumonia would have improved Cronbach’s |$\alpha$| from 0.797 to >0.8, but we resisted this because patients in Australian hospital are normally only ventilated in the Intensive Care Unit, during surgery, and—occasionally—in the ED, potentially explaining why it did not correlate well with other activities that are performed more broadly across the hospital. In another example within the same subscale, we retained the item ‘an up-to-date hospital protocol for recognizing and responding to clinical deterioration in acute healthcare’ due to its importance in patient care, despite an item-total correlation of only 0.137. This is an element of care that has received strong endorsement from State health departments over the last decade [12, 13], and the data were strongly left skewed due to high performance in this aspect of care in most participating hospitals.

For department-level care pathway scales, the final scales were not modified as a result of our analysis. We developed a generic scale for PSS, but it was not possible to build generic scales for SER, EBOP and CR, as the processes are different depending on the specific condition and the department; for this reason, we also developed condition-specific versions of each of these scales for use in the ED.

Limitations

While most of the scales are based on independently audited data, the QMSI scale is based on perceptions of the quality manager. In mitigation, we worded the questions in the quality manager survey to include only questions on facts, to minimize bias associated with self-report. Additionally, while we have shown how the organization and department scales have been modified from their European antecedents, the DUQuA scales have only been validated for the Australian context, and their generalizability to other settings remains to be demonstrated. Differences between the DUQuA and DUQuE scales must be taken into account when comparing findings from Australian and European hospitals.

Implications for research, policy and practice

It can be difficult for hospitals to access validated measurement tools for assessing quality management that are powerful yet easy to use. The DUQuA scales are now publicly available to fill this niche. The scales could also be used to collect longitudinal data, for example before and after an intervention, used to collect information to assist in designing a quality improvement intervention, or as part of a comparison of hospitals or hospital departments.

At present, the CQII scale is likely to just confirm what is known from accreditation results. More work is required to develop or refine a tool that is more able to discriminate between hospitals with high implementation indices due to participation in external accreditation processes. Collection of data from international hospitals that are not subject to mandatory accreditation may assist this process.

Conclusions

The DUQuA organization and department scales can be used by Australian hospital managers to assess and measure improvement in quality management at organization and department levels within their hospitals and are readily modifiable for other health systems depending on their needs.

Acknowledgements

Members of the original DUQuE team (Oliver Gröne and Rosa Suñol) provided extensive input and advice in the initial stages of the project, supporting design modifications from DUQuE and the planned DUQuA study approach. Dr Annette Pantle provided expert advice on revision and development of DUQuA measures, and Professor Sandy Middleton, Associate Professor Dominique Cadillac, Kelvin Hill, Dr Carmel Crock and Professor Jacqueline Close provided input into the development of our stroke, AMI and hip fracture clinical process indicator lists. Nicole Mealing and Victoria Pye provided statistical advice in the initial phases of the project. We greatly appreciate their efforts.

Funding

This research was funded by the National Health and Medical Research Council (NHMRC) Program Grant APP1054146 (CI Braithwaite).

Authors’ contributions

The research team consists of experienced researchers, clinicians, biostatisticians and project managers with expertise in health services research, survey design and validation, large-scale research and project management, sophisticated statistical analysis, quality improvement and assessment, accreditation, clinical indicators, policy and patient experience. JB conceived the idea, led the research grant to fund the project and chairs the steering committee. RCW and NT co-led the detailed study design, managed the project across time and contributed to the development of the manuscript. HPT and GA provided statistical expertise for the study design and developed the analysis plan for the manuscript. TW contributed to the logistics of project management, the refinement of measures and the development of the manuscript.

Ethics approval

Ethical approvals were secured from State and Territory human research ethics committees in New South Wales (#14/206), Victoria (#15/36), the Australian Capital Territory (#15/131), South Australia (#15/260), the National Territory (#15-2509), Tasmania (#H0015383) and Queensland (#15/361). Site-specific authorizations, including permission for external researchers to collect data in hospitals, were granted. We complied with confidentiality requirements of national legislation or standards of practice of each jurisdiction.

Data sharing statement

Data will be made publicly available to the extent that individual participants or participating hospitals cannot be identified, in accordance with requirements of the approving Human Research Ethics Committees.

References

1.

Taylor
N
,
Clay-Williams
R
,
Hogden
E
et al.
Deepening our understanding of quality in Australia (DUQuA): a study protocol for a nationwide, multilevel analysis of relationships between hospital quality management systems and patient factors
.
BMJ Open
2015
;
5
:e010349.

2.

Secanell
M
,
Groene
O
,
Arah
OA
et al.
Deepening our understanding of quality improvement in Europe (DUQuE): overview of a study of hospital quality management in seven countries
.
Int J Qual Health Care
2014
;
26
:
5
15
.

3.

Wagner
C
,
Groene
O
,
Thompson
C
et al.
Development and validation of an index to assess hospital quality management systems
.
Int J Qual Health Care
2014
;
26
:
16
26
.

4.

Braithwaite
J
,
Clay-Williams
R
,
Taylor
N
et al.
Deepening our Understanding of Quality in Australia (DUQuA): An overview of a nation-wide, multi-level analysis of relationships between quality management systems and patient factors in 32 hospitals
.
Int J Qual Health Care
2020
;
32
:
8
21
.

5.

Wagner
C
,
Thompson
CA
,
Arah
OA
et al.
A checklist for patient safety rounds at the care pathway level
.
Int J Qual Health Care
2014
;
26
:
36
46
.

6.

Wagner
C
,
Groene
O
,
Dersarkissian
M
et al.
The use of on-site visits to assess compliance and implementation of quality management at hospital level
.
Int J Qual Health Care
2014
;
26
:
27
35
.

7.

Nunnally
JC
. An overview of psychological measurement. In:
Wolman
BB
(ed.).
Clinical Diagnosis of Mental Disorders
.
Boston, MA
,
Springer
,
1978
,
97
146
.

8.

Arnolda
G
,
Winata
T
,
Ting
HP
et al.
Implementation and data-related challenges in the Deepening our Understanding of Quality in Australia (DUQuA) study: implications for large-scale cross-sectional research
.
Int J Qual Health Care
2020
;
32
:
75
83
.

9.

Gilks
WR
,
Richardson
S
,
Spiegelhalter
D
.
Markov Chain Monte Carlo in Practice
.
Boca Raton, FL
,
Chapman and Hall & CRC Press
,
1995
.

10.

Rubin
DB
.
Multiple Imputation for Nonresponse in Surveys
.
New York, NY
,
John Wiley & Sons
,
2004
.

11.

Wagner
C
,
Groene
O
,
Thompson
CA
et al.
DUQuE quality management measures: Associations between quality management at hospital and pathway levels
.
Int J Qual Health Care
2014
;
26
:
66
73
.

12.

Hillman
KM
,
Rubenfeld
GD
,
Braithwaite
J
.
Time to shut down the acute care conveyor belt?
Med J Aust
2015
;
203
:
429
30
.

13.

Hughes
C
,
Pain
C
,
Braithwaite
J
et al.
‘Between the flags’: implementing a rapid response system at scale
.
BMJ Qual Saf
2014
;
23
:
714
7
.

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://dbpia.nl.go.kr/journals/pages/open_access/funder_policies/chorus/standard_publication_model)