Justification, rationale and methodological approaches to realist reviews

Abstract

Realist reviews offer a method to understand why an intervention is successful or not. Many factors influence how complex healthcare interventions are delivered and this makes understanding what works difficult. Effectiveness depends on delivery context, and success in one setting does not guarantee the same result in alternate settings. How an intervention works (the underlying mechanisms) in a particular setting for one population group, may not work in the same way for a different group. A realist review provides an iterative theory-driven approach to help understand how and for whom and under what conditions an intervention works.

Graphical Abstract

Open in new tabDownload slide

Introduction

Healthcare decisions at patient, organizational, and policy level are made using the most robust and highest level of available evidence. Systematic reviews and meta-analyses of randomized controlled trials (RCTs) are at the peak of the traditional hierarchy of evidence, which appraises the quality of evidence alongside risk of bias.¹ Direct application of the traditional hierarchy pyramid may oversimplify levels of evidence, and may assume that all systematic reviews and RCTs are of equal quality. Despite this, evidence obtained from systematic reviews and/or meta-analyses of RCTs is commonly referred to as the ‘gold standard’ level of evidence, and in many settings the only means of demonstrating the superiority of an intervention and/or treatment compared with usual care.

Meta-analyses and RCTs limit the outcome of an intervention to effective, not effective, or inconclusive. For example, a systematic review investigating the effects of using decision aids to facilitate shared decision making for oral anticoagulation decisions in patients with atrial fibrillation resulted in inconclusive evidence into the effectiveness of decision aids on the prevention of stroke and major bleeding complications in this population.² Authors made suggestions that the design of decision aids may have influenced the outcomes of some studies, or another variable may have been the timings in which the decision aids were used, i.e. before or during consultations. Within systematic reviews, there is no scope to identify what elements of the intervention being studied may lead to success or failure (e.g. healthcare setting, resource allocation, sociodemographic of target populations, patient involvement, and treatment burden). This means that an intervention that was deemed effective within a clinical trial context may not be successful when implemented in a real-world clinical setting.³ This is particularly problematic when considering complex interventions where interventions consist of multiple variables implemented at different levels within the healthcare system.⁴ Therefore, there is a need for a more pragmatic approach to review the literature to provide an explanatory analysis of the intervention being studied. The updated MRC framework for complex interventions has moved away from research that is driven by the primary outcome of ‘effectiveness’. Instead, the emphasis is on asking whether interventions are implementable, cost effective, transferable, and scalable across contexts.⁴

Realist reviews offer a method to understand why something is, or is not, successful in a naturalistic setting, and whether it would work when adopted in different settings.⁵ Realist reviews, therefore, offer the scope to develop practical solutions to complex healthcare questions.

What is a realist review?

A realist review is an iterative theory-driven approach used to understand how and for whom and under what conditions an intervention works. The method examines sets of ‘context-mechanism-outcome’ (CMO) configurations.⁵ In one context, a particular mechanism is evoked and results in a successful outcome. In another context, the same mechanism may result in a different outcome or may not work. Realist review includes building an initial programme theory of why and how an intervention is believed to work and testing and refining this by examining the literature from varying sources. These can include primary qualitative and quantitative research, grey literature, i.e. write up of programmes on websites or project initiation documents,⁶ and local, national, and global policy documents. Multiple searches may be required as the programme theory is refined. Realist review appraises the quality of evidence based on two elements. First, relevance (do the descriptions of programme processes and contexts contribute to explaining how, for whom, and why the intervention works)? Second, rigour (are the conclusions aligned to the research design)? Data included in the final analysis are then used to refine the initial programme theory, created in the development stage of review, and can contribute to one or multiple elements of the theory to explore and refine descriptive concepts into a final framework.

Realist reviews should not be confused with mixed-method systematic reviews, which also combine the findings of qualitative and quantitative studies. A mixed-method review analyses and synthesizes the literature that meets specific inclusion and exclusion criteria to answer a specific question, and includes a traditional quality appraisal.⁷ In contrast, a realist review develops a theory about how, for whom, and under what conditions interventions work based on CMO configurations and includes a realist quality appraisal.⁵

Step-by-step approach: how to perform a realist review

A realist review follows Pawsons five defined steps:⁵

• ȃ(1) Define the scope and clarify the purpose of the review.

• ȃ(2) Develop the initial programme theory.

• ȃ(3) Evidence search and appraisal.

• ȃ(4) Extract and synthesize findings.

• ȃ(5) Draw conclusions and make recommendations.

But what does this mean for researchers and how do we apply this to fulfil the remit of a robust realist review?

Steps 1 and 2: Define the scope and develop the initial programme theory

An initial programme theory that addresses the purpose of the review is developed via a preliminary scoping review and discussion within an expert group.⁵ The formation of the initial programme theory can be time-consuming, messy, iterative, and uncertain but this is fundamental for a rigourous realist review. This theory should hypothesize why an intervention (e.g. a clinical treatment, service, or health policy) is believed to work. It should question if we deliver a programme under certain conditions, how, and for whom it works. Initial programme theory should ideally be grounded in a middle-range theory.⁶ The term ‘middle-range’ describes a theory that involves some abstraction but is close enough to the observed data to be incorporated in propositions that permit empirical testing.⁸ This includes formal theories such as the Chronic Care Model⁹ that describes, explains, or predicts the effects of a complex intervention. Initial searches may yield a diversity in research designs, health topics, definitions, and concepts that make the identification of a middle-range theory impossible at the outset of the review.¹⁰ If this is the case, the appropriateness of middle-range theories should be reviewed during identification, selection, appraisal, and synthesis to aid conceptualization.¹⁰

Contexts are the physical, social, psychological, cultural, and institutional circumstances in which an intervention is delivered.¹¹ They are dynamic, embedded, and uncontrollable. Mechanisms are the less tangible and hidden components that bring about change.⁵ They exist before the intervention but remain silent until implemented within given contexts.¹¹ Understanding potential mechanisms and how they work in different contexts can help to illuminate how an intervention may be adapted to fit within different conditions.^5,11,12 Outcomes are changes that result from the delivery of an intervention in certain contexts using certain mechanisms. Outcomes may be short or long term, and at the individual or system level.

The following example considers the inner workings of why nurse-led clinics may be helpful to manage chronic diseases. It is too simplistic to say that the nurse is the mechanism that results in improved healthcare outcomes for patients. The nurse forms the building blocks of the intervention, which in realist review terminology is referred to as the programme architecture. To identify potential mechanisms, it is helpful to breakdown them down into available resources and the response to these resources.¹³ This can be done by formulating if/then statements with the resources and responses.

Worked examples

Example 1: Nurse-led clinics

A potential resource available within a nurse-led clinic is the chance for patients to have additional 1:1 time with a healthcare professional. A prospective response to this could be facilitating opportunity for the medication review. An if/then statement relating to this potential combination is:

If nurse-led clinics offer 1:1 time with patients (resource) then patients will have additional opportunities to speak with healthcare professionals to review their medication (response).

We can also add a proposed outcome on the end of these statements to begin to understand what change this mechanism brings about:

If nurse-led clinics offer 1:1 time with patients (resource) then patients will have additional opportunities to speak with healthcare professionals to review their medication (response) leading to improved patient experience (outcome) (Figure 1).

There may be several if/then statements related to 1:1 time with a healthcare professional. Thinking in this way can be confusing. Preconceptions are challenged as we start to dig deeper into the causal nature of why an intervention is, or is not, successful. This is a normal part of the process and discussion within the expert panel should be used to clarify thinking. It is easy to see how quickly a realist review expands. Each resource/response that forms a proposed mechanism is affected by multiple contexts that influence outcomes, like a complex engine with many interlocking cogs (Figure 2). The evolving of initial and refined theory helps a realist reviewer tease out the nuances of previously unidentified causation.

Example 2: Telehealth

A potential tool for telehealth is the use of a patient mobile app. A resource provided by an app is the incorporated educational content included within the apps functionality. A prospective patient response to this is learning about their health condition. An if/then statement relating to this potential combination is:

If patients have access to a mobile health app for their chronic condition (resource) then patients will have the opportunity to learn about their condition and understand how to self-manage at home (response).

Again, we can begin to expand the statements to begin to understand what change this mechanism brings about:

If patients have access to a mobile health app for their chronic condition (resource) then patients will have the opportunity to learn about their condition and understand how to self-manage at home (response) leading to patients taking an active role in their condition management (short-term outcome) resulting in reduced symptom burden (mid-term outcome) and risk of hospitalization (long-term outcome).

Step 3: Evidence search and appraisal

Once the initial programme theory has been agreed by the expert panel, the literature searching should be designed to identify sources of evidence that can develop, support, or refute the initial programme theory. Traditional database searching is used, and a diverse range of sources is included.^5,14–16 RCTs, observational studies, qualitative studies, case studies, protocols, and even systematic reviews may be involved. During screening, papers are selected based on whether they describe or discuss one or more context, mechanism, or outcome from the initial programme theory. Once potential papers have been identified, the quality appraisal is used to assess each source for its rigour and relevance. Rigour is defined as ‘whether a particular inference drawn by the original researcher has sufficient weight to make a methodologically credible contribution’.⁵ Relevance is defined as ‘Not about whether the study covered a particular topic, but whether it addressed the theory under test’.⁵ A study may be screened and initially categorized as containing pertinent data, however, as the theory evolves it may become apparent that it does not in fact address any element of the theory and therefore must be excluded from the final data. Additionally, a new element of the theory may be identified necessitating a new database search be carried out identifying a new pool of references to work from.

Step 4: Data extraction and evidence synthesis

It may be useful to the reviewer to extract study demographics using conventional data extraction tables. However, each realist review will also require its own unique data extraction form to tease out any elements that address the relevant context, mechanisms, and outcomes contained within the theory. Ideas can be assimilated through note taking, mind mapping, and discussion in addition to traditional data extraction methods. It is a complex process to record how each piece of information may or may not contribute to the contexts, mechanisms, or outcomes. This step can also be titled ‘Theory Testing’.^5,11 This involves revisiting and expanding on the initial theory based on the evidence synthesis and beginning to construct a refined programme theory built from an interlocking web of contexts, mechanisms, and outcomes (Figure 3).

For Example 1, nurse-led clinics: If nurses with specialist knowledge in specific disease management with additional prescriber qualifications (Context 1) offered 1:1 time with patients (Resource 1) who have complicated polypharmacy (Context 2) then patients would have additional opportunities to speak with healthcare professionals to review their medication (Response 1) within a less intimidating environment (Context 3) leading to reduced patient anxiety (short-term outcome). Patients would be more encouraged to ask questions (Response 2) leading to improved patient knowledge (mid-term outcome) and greater patient experience (mid-term outcome) and improved health outcomes (long-term outcome).

For example 2, Telehealth

If patients have access to a mobile health app for their chronic condition that incorporates behaviour change techniques (resource) then patients will have the opportunity to learn about their condition and track their symptoms (response) leading to patients taking an active role in their condition management (short-term outcome), improving nurse–patient communication (short-term outcome) resulting in long-term health behaviour changes (mid-term outcome) and reducing the risk of hospitalization (long-term outcome).

Step 5: Draw conclusions and make recommendations

Reviewers should summarize the main findings and suggest actionable recommendations to help inform policy and practice. These may be specific to the intervention examined or more generalized if applicable. Where possible, recommendations should be underpinned with the middle-range theory used to help guide the synthesis.⁶ If there is a lack of evidence, or there are missing evidence links in the reasoning behind the refined theory, then recommendations should be made cautiously.

Real-world examples

There are very few realist reviews in the field of cardiovascular healthcare. Given the complex nature of many cardiac services, there is great potential to develop understanding and improve service delivery by using this methodology. One good example of an easy to follow and relevant review is: a realist review of shared medical appointments: how, for whom, and under what circumstances do they work?¹⁷

A realist review for digitally facilitated self-care for clinical sexual and reproductive health information focused on the question ‘What information supports whom to effectively access and use self-care technologies in what circumstances and why?’¹⁸ The authors were able to build eight context-mechanism-outcome statements that refined their initial programme theory into an improved programme theory. Contexts around the type of sexual health disease, age, and geographical area were interwoven with mechanisms and outcomes based on access, information searching, trustworthiness, and presentation of information. From this, six specific recommendations to improve online clinical information for user self-care were generated:

• ȃ(1) Sexual health information needs to be endorsed by trusted services.

• ȃ(2) Providers should recognize that interactive digital information appears to be as effective as clinician-delivered information.

• ȃ(3) Services should support young people to find information online through signposting.

• ȃ(4) Services should train clinicians on how to assess the value of external sites and recommend suitable sites based on quality.

• ȃ(5) Sites should provide the capacity for users to speak anonymously with a clinician without the need for registration.

• ȃ(6) Self-care information should be provided in a variety of media.

It is not a requirement of realist review to suggest recommendations. Instead, Pawson et al. (2005) states realist review should be used to describe relationships between the interventions being studied and potential contexts. That said, it is common to find authors specify recommendations to aid policy makers and key stakeholders to inform future service development.

Reporting

Realist review should be undertaken in accordance with the Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES I and RAMESES II).^6,16 These standards have been designed to create consistency in the quality across reviews. The explanatory and theory-driven focus of a realist review means typical journal word counts and formatting requirements may constrain realist explanations. This means that authors should consider how to present findings visually, submit supplementary files and appendices, and/or publish a realist review protocol, which reports Step 1 and Step 2 of Pawson’s realist review methodology.⁵

Challenges

Realist reviews are complex but rewarding to conduct, although the methodology is time-consuming, and identifying what constitutes a context, mechanism, and outcome is a difficult task. Realist reviews can take anywhere from 12 to 18 months¹⁶ and require substantial resources to identify, develop, and test programme theories. This compares with a mean of 15.5 months to undertake systematic review.¹⁹ Although rapid realist review can be undertaken by researchers who have extensive knowledge of the available literature, it is not advised as it may limit theory refinement.

Those who are not used to conducting realist reviews should be aware of the differences between this methodology and a traditional systematic review, so that pertinent literature containing casual process information is not discarded. Primary studies may not transparently report the intervention processes, instead choosing to focus on the outcomes. Reviewers should make all efforts to contact corresponding authors to clarify content, however, identifying sufficient evidence to refine a theory and isolate causal pathways is not guaranteed.

Conclusion

The popularity of realist reviews is increasing because they are useful to help understand how and why an intervention may or may not work, for whom, and under what circumstances. This makes it possible to identify possible adaptations required to successfully implement an intervention within different settings. Although based on Pawson’s five-steps, approaches may vary between reviews. Authors should be guided by the intervention/issue being explored whilst staying true to the philosophy that underpins realist review.

Funding

There was no funding involved for this work.

References

Author notes