Effective mission-oriented research: A new framework for systemic research impact assessment

Abstract

Mission-oriented research combines a wide array of natural and social science disciplines to offer solutions for complex and multi-dimensional challenges such as climate change, loss of biodiversity, and scarcity of natural resources. The utilization of the outputs of mission-oriented research aims for changes in behavior, policy and practice resulting in real world impacts. Systematically assessing such research impacts and impact-generating processes is novel and offers great potential to plan for impactful research. This article develops a framework for systemic research impact assessment (RIA) on the basis of a literature review taking natural resource management (NRM) research as an example. The review compiles and analyzes 70 relevant RIA approaches. The resulting framework combines four components for improving societal impacts (1) an integrated component enabling reflection of impacts on all sustainability dimensions, (2) a missions component orienting toward societal goals to ensure societal relevance, (3) an inclusive component enabling wide participation to ensure legitimacy of research and its impact, and (4) a strategic component to choose appropriate assessment scales and time dimensions to ensure effectiveness. We provide suitable examples for the framework and we conclude with a call for an increased use of systemic and formative RIA that incorporate participatory strategies for research priority setting as well as socially deliberated target systems (e.g. SDGs), to plan for impactful mission-oriented research.

1. Introduction

Efforts to address contemporary challenges to society, such as climate change, biodiversity loss and natural resource scarcity, are increasingly formulated in terms of universal goals and missions. These range from cross-sectoral and multi-dimensional goals such as the 17 United Nations Sustainable Development Goals (SDGs), to sectoral targets such as the Paris Agreement to reduce greenhouse gas emissions, to very context-specific missions such as the new five EU Missions in Horizon Europe (Climate Adaptation, Cancer, Water, Climate-Neutral Cities and Healthy Soils) (European Commission (EC) 2022). Mission-oriented research integrates natural and social science disciplines in order to contribute solutions to complex and multi-dimensional challenges by informing changes in behavior, policy and practice. Thus, research aims to contribute to these common missions, goals and targets, by creating impact beyond the mere extension of the scientific knowledge base. This so-called ‘research impact’ can be divided into more specific impact dimensions, covering social, environmental and economic and policy/governance aspects—following the traditional sustainability dimensions. Depending on the approach, these dimensions can be further extended (e.g. ‘sanitary’ dimension in ASIRPA (Joly et al. 2015)) or assessed separately (e.g. only policy impact in Reed, Bryce and Machen (2018)). So called research impact assessment (RIA) approaches can have formative ambitions to monitor, analyze and learn from research activities, and responsibly shape and promote impactful research, or summative intentions to evidence and account for contributions of research activities to societal impacts (Reed et al. 2021). Summative RIA and formative RIA are not to be thought of separately, but emphasize two streams of motivation for conducting RIA, and are often practiced together.

Systematically anticipating and assessing societal impacts, the contributions to common societal goals and the underlying impact-generating processes of research is a comparably novel practice. However, this approach offers great potential to plan for impactful research. In this vein, ‘mission-oriented innovation policies’ (MOIPs) are ‘a new type of systemic intervention that a growing number of countries has implemented in order to tackle mounting societal challenges’ and to correct ‘the lack of holistic strategic orientation and policy co-ordination, and fragmented policy mixes’ (Larrue 2021: 3). In addition to policy coordination and strategic orientation, Larrue (2021) identifies and analyzes ‘policy implementation’ as a central feature of MOIPs. Hence, to ‘ensure (…) consistency and effectiveness of the (…) intervention’ one needs to enable ‘evaluability’ to assess research results with potential to improve policy, ‘reflexivity’ to use results from evaluation and monitoring to make decisions, and MOIPs reforms to achieve the targeted mission (Larrue 2021: 17). Assessments incorporating societal missions can thus be useful as an internal learning tool to enhance the contribution of research to intended impacts (formative RIA) and as an external tool to account for evidence of generated societal impacts (summative RIA).

For research to meaningfully support societal change and meet societal needs, knowledge created by science ultimately needs to be utilized in civil society, policy and practice. Cash et al. (2002) summarize three criteria crucial for scientific information to be effectively translated into action: credibility, salience/relevance and legitimacy. Credibility of scientific information refers to its scientific standard often tested through the quality-control mechanism of peer-review. It persists to be a ‘traditional criteria of scientific rigor’ (Belcher et al. 2016: 1). The second criteria, salience, refers to the relevance that the scientific information has for the user (Cash et al. 2002). The third criteria, legitimacy, refers to the unbiased and fair representation of all relevant stakeholder interests. Formative RIA, focusing on assessing but also achieving actual or potential contributions of research outputs to problem solving and societal change, requires a balanced consideration of all three aspects (Cash et al. 2002; Belcher et al. 2016).

The general RIA approach aims for a balanced consideration of different aspects ‘to demonstrate and measure the impact of (…) research beyond academia’ (Fryirs, Brierley and Dixon 2019: 1).

Though in the 1970s the impact of research beyond academia was already considered in assessments of research utilization and knowledge mobilization, this orientation toward impact dissipated in the 1980s, when performance-based research funding became mainstream (ibid.), leading to the predominant assessment of research based on scientific output and credibility. Bornmann (2013) summarizes three historical phases of the development of RIA approaches since the 1980s that increasingly assess societal impact, defined earlier by Donovan (2008). The first phase expands the assessment of scientific credibility to include the economic impact of research. The aim is to deduct accountability of research funding by calculating monetary return on investment (summative RIA). The second phase expands beyond economic impact and starts assessing the social impact of research (Bornmann 2013). The third phase is characterized by a case-study approach, in which a range of quantitative and qualitative indicators are linked, allowing for an integrated analysis of all types of societal impacts (economic, social, environmental) of research studies (formative RIA) (ibid.).

Adjustments to RIA approaches are ultimately linked to changes in research policy. The shift toward societal impacts in RIA is repeatedly linked to the Lund Declaration in 2009, laying a foundation stone for mission-oriented research and innovation policy in Europe (Matt et al. 2017; Chams, Guesmi and Gil 2020). Since then, national RIA frameworks have been introduced in Europe, USA, Australia, Canada and Hong Kong as an attempt to focus national research on ‘facilitating national prosperity through the transfer, translation and commercialization of knowledge, combined with the integration of research findings into government policy-making’ (Fryirs, Brierley and Dixon 2019: 2). Most prominent is the UK’s Research Excellence Framework (REF) (2014 and 2021), which assesses impact of publicly-funded research in terms of relative significance and reach, linking government funding to outcomes (UK Research and Innovation (UK RI) 2022) and thus having a strong summative RIA alignment. A further manifestation of mission orientation in research policy are the previously mentioned Horizon Europe Missions, striving to steer research activities, as well as policy measures and legislative initiatives (European Commission (EC) 2022).

In contrast to traditional research assessment approaches based on scientometric impact factors quantifying research outputs and credibility (e.g. H-Index), contemporary RIA broadens the focus to transformative effects, addressing the sequence of generated products (outputs), uses (practical and political integration/application) and societal impacts (changes in society) (Bornmann 2013) as well as considering impact-promoting processes (Daedlow et al. 2016). This results in the widespread use of logical frameworks in contemporary RIA approaches, commonly referring to inputs, activities, outputs, outcomes and impacts along an impact pathway.

Assessing and anticipating impact is easier for applied scientific disciplines, especially when the ‘impact agenda aligns well with the norms and practices’ of the discipline and researchers (Reed et al. 2021: 2). Therefore, it is not surprising that the use of RIA approaches thrives in disciplines aiming for tangible impact and those with long traditions of utilizing goals and missions, such as the WHO Resolution to Health Promotion in the health sciences. A recent review of RIA models has revealed that 70% of approaches continue to be developed in health sciences (Razmgir et al. 2021). An analysis of a repository of impact assessment studies in the international development sector revealed that studies published in health science journals had a share of over 80% until 2000, which has since slowly decreased in favor of studies in education, social protection and some studies in agriculture (Cameron, Mishra and Brown 2016).

This article will emphasize on these newly emerging sectors outside the health sciences that increasingly employ the RIA model, by focusing on developed or applied RIA approaches for research concerned with the use, conservation and management of natural resources, referred to here as natural resource management (NRM) research. While there is a growing body of literature on RIA approaches in NRM contexts, the approaches differ in comprehensiveness, strategic objectives and mission orientation. A clear analytical framework that highlights differences between approaches and supports the analysis of which approach may suit best for which purpose is missing.

The objective of this article is to assess existing approaches with a focus on suitability to strategically plan impactful research through a reflexive, internal process (formative RIA). This paper applies a systematic literature review and grounded theory analysis (Section 2) to develop a systemic RIA framework supporting impactful mission-oriented research and gives some suitable examples (Section 4). It offers an overview and a deeper understanding of 70 assessment approaches for analyzing societal impacts of NRM research by integrating multiple dimensions of sustainability and/or impact-generating processes, hereafter referred to as ‘integrated RIA approaches’ (Section 3). We focus on the source of the assessment, the aim of the assessment, and analyze the impact dimensions and definitions, as well as the use of societal goals, and prioritization and participation tools. The conclusion shares some considerations of the possible merits of applying RIA approaches aligned with our framework (Section 5).

2. Materials and methods

We applied a systematic literature review to capture a comprehensive list of RIA approaches in NRM research. We used grounded theory analysis (Glaser and Strauss 1967) to dissect, analyze and interpret the identified approaches and finally to deduce a systemic RIA framework for impactful mission-oriented research. While the review process was based on defined selection criteria, it is not separated from the personal interpretation of the authors when selecting suitable approaches and in their qualitative analysis.

2.1 Literature search

The systematic literature review performed (1) a keyword database search in the Web of Science Core Collection to comprehensively capture documents about integrated RIA approaches in the field of NRM research; and a (2) structured review of relevant conference content, training programs and networks, followed by a scoping of relevant organizational websites to capture information provided only in grey literature.

2.1.1 Peer-reviewed literature

We selected the Web of Science Core Collection because it includes an array of scientific indices allowing us to search for the application of impact assessment of NRM research across a wide range of disciplines. Web of Science was chosen in favor of Scopus due to access availability through a full institutional subscription. Also, Scopus and Web of Science have a strong overlap in articles included in their database (Martín-Martín et al. 2018; Zhu and Liu 2020). The selected keywords were searched for in the ‘Topic’ function of the Web of Science search interface as a search string. The ‘Topic’ function scans the article’s title, keywords and abstracts for the search terms.

The search terms were identified collectively in the research team through the collection of relevant keywords, as well as informed through previous search strings in relevant literature (Deeming et al. 2017; Weißhuhn, Helming and Ferretti 2018; Kamenetzky and Hinrichs-Krapels 2020). The search strategy was continuously tested and modified to finally include all previously identified benchmark articles (see Supplementary Table S1). To filter for the NRM disciplines we made use of the discipline filters ‘Web of Science Categories’ and omitted all health science categories, making the literature count more manageable (see Figure 1). As mentioned in the introduction, health sciences constitute the most prominent source for RIA approaches, however, the focus of this study are approaches in context of NRM research.

The defined search string (see Figure 1), filtering for research impact assessment with integrated approaches (referring to multiple sustainability dimensions and/or impact-generating processes), resulted in 1660 articles on 16 August 2021. Excluding health sciences reduced the count to 664 articles (40%). Titles, keywords and abstracts were screened for articles discussing (1) methods of research/societal impact assessment in the (2) field of NRM research. The resulting 157 articles were assessed in full-text. We excluded articles that did not deal with RIA, NRM, integrated approach (more than one sustainability dimension covered) or only reviewed but not developed or applied RIA methodologies. Of the remaining 78 articles, approximately half were either solely descriptive or empirical studies applying existing approaches, leaving 37 articles describing conceptual RIA approaches which were used for the in-depth analysis.

2.1.2 Grey literature

Since RIA happens at the interface between academia and science management, we assume that many approaches are likely only available in grey literature, particularly on the websites of assessing or assessed organizations. Therefore, we screened for organizations and researchers, hereafter referred to as actors, active in RIA, as well as presented RIA approaches in conference programs, training programs and networks related to research impact identified in preliminary desk research (see full list in Figure 1).

We retrieved 1586 potential RIA actors who were combined with selected actors identified in the peer-reviewed literature (see Figure 1) into an actor database. After removing duplicates, 666 organizational websites were screened for RIA approaches from September 2021 to December 2021. Of these, only 154 organizations revealed to have documents concerning RIA. The document screening left 40 organizations with a total of 38 RIA approaches (60 documents) meeting the search criteria. Integrated approaches that had a clear NRM reference and were applied to research activities were included in the analysis (see Figure 1, Box ‘Records excluded’).

The grey literature search revealed an overlap in actors providing RIA approaches and research for development (R4D) impact assessments. Since the area of international development has a long tradition of developing and applying impact assessment approaches (Cameron, Mishra and Brown 2016), it offers a large abundance of approaches that inform RIA in related disciplines. R4D impact assessment approaches that were applied in/developed for an apparent NRM research context were included in our analysis.

2.2 Material

Combining the approaches identified in the grey and peer-review literature, the literature search resulted in a total of 70 conceptual RIA approaches (94 documents; see Supplementary Table S3 for full list) for the in-depth analysis.

The earliest integrated RIA approaches for NRM research were published in 1995 (Bantilan and Johansen 1995), 1998 (Riley and Alexander 1998) and 2003 (Baur et al. 2003). Only after 2007 we observe a slow but steady increase in publications (Figure 2). Since 2017, two years after the introduction of the SDGs, there has been a steep increase in integrated RIA approaches, also in comparison to the total Web of Science publications from 1995 to 2021.

The largest proportion of RIA approaches was developed and applied by international organizations and research institutions, half of them originating from either the Consultative Group on International Agricultural Research (CGIAR) or instutions affiliated with CGIAR (Figure 3). Countries with a robust national RIA tradition, such as the UK, Canada, Australia, the USA and the EU and EU member states, contributed to ∼75% of the integrated RIA approaches in NRM research (Figure 3). The category ‘others’ encompasses addional other EU countries, Brazil, New Zealand, South Africa, and Uganda.

The largest proportion of the identified approaches was developed to assess research in agricultural contexts (31%), followed by general approaches without specific thematic contexts (e.g. CSIRO Impact Framework) (21%), and approaches in development (14%) and sustainability (8%) studies (Figure 4). Other sectors covered in the RIA approaches are marine and freshwater ecosystems, climate change, water management, forestry, and biodiversity and conservation. While agriculture appears to be the most strongly represented sector, a significant proportion of these agricultural approaches, along with some in other sectors, fall under the umbrella of research for development (R4D).

2.3 Qualitative analysis

The relevant documents underwent coding and analysis in the qualitative data analysis tool MAXQDA 2022. We employed open coding and grounded theory analysis (Glaser and Strauss, 1967) to deduce reoccurring characteristics in the identified integrated RIA approaches. The initial round of analysis resulted in coded sections and notes on (1) analytical approach, (2) assessment source, (3) assessment aim, (4) impact areas/dimensions, (5) methods and tools applied, (6) role of sustainability in assessment, and (7) assessment scale (see Table 1). The first level coding was applied to all 138 eligible documents, comprising 78 from Web of Science and 38 from grey literature (see Figure 1, eligibility stage). Corresponding sub-codes were either developed during the initial coding or introduced in the second round, focusing exclusively on conceptual approaches (n = 70), excluding documents that only empirically applied existing approaches. The results of the qualitative analysis were ultimately utilized to derive a systemic RIA framework for societally impactful research.

3. Results and discussion

The analysis of the 70 integrated RIA approaches resulted in a comprehensive review of the definitions of research outcomes and impacts, as well as the classification of scales, sources, impact dimensions, aims, and timing dimensions. Further, we identified the results on roles of societal goals and priority setting in these RIA approaches. These findings are discussed in relation to their relevance for impactful research, with reference to two of the three Cash criteria for effective research: relevance and legitimacy (Cash et al. 2002). The third Cash criteria, ‘credibility’, is well established and captured in traditional research assessments that measure so called scientific impact (e.g. publication and citation counts; impact factors). As such this criteria is not the focus of this analysis. Good scientific practice and resulting scientific impact are assumed as preconditions, expanded upon through societal impact assessment.

3.1 Approach attributes

3.1.1 Source of assessment

There are four types of sources for societal impact assessment of research (Bornmann 2013):

• Source Type 1: national evaluation systems,

• Source Type 2: research-funding organizations,

• Source Type 3: research projects assessing their impact (case study), and

• Source Type 4: research projects reflecting on approaches to measure impact.

The analysis revealed an additional source of RIA, Source Type 5: approaches developed by research organizations. This type exhibits similarities to both Type 3 and Type 4, aiming to develop and provide a context-dependent framework to assess and/or advance the impact of a research organization.

The analyzed RIA approaches were predominately developed by research projects conceptualizing research impact (Type 4) (40%) and organizations (Type 5) developing their own approach and assessing their contributions to societal impacts (39%) (Figure 5). Case studies (Type 3) and research funders (Type 2) each account for 1/10 of the remaining analyzed RIA approaches. The reviewed literature does not include national evaluation systems (Type 1) due to their generic and cross-disciplinary character. However, the reviewed literature does include some cases of organizations offering impact assessment approaches motivated or inspired by such national evaluation systems.

3.1.2 Aim of assessment

The International School on Research Impact Assessment (ISRIA) has developed comprehensive guidelines for effective RIA and emphasizes the need to reflect on the purpose of the assessment (Morgan and Grant 2013; Adam et al. 2018). They identify four core motivations for RIA: the accountability of impact, the analysis of impact and impact-generating processes, the efficient allocation of funding for research, and the advocacy for financial support of proposed research.

The main motivational aims for our analyzed RIA approaches are analysis (77%) and accountability (54%) of research impact, often considered together (37%) (see Figure 5). While purely analytical motivations are more common in RIAs conceptualized in projects (Type 4), organizational RIAs (Type 5) more often follow a mixed approach aiming for accountability as well as analysis. Research funders have the strongest focus on accountability (38%). Only four RIA approaches indicated advocacy and two research funders had allocation motivations (included in ‘other’ in Figure 5). Across all approach sources, the analyzed RIAs, in most cases, have some formative motivation. Particularly, organizational and research project approaches (Type 4 & 5) follow a purely summative assessment of impact accountability only in 10% of the cases.

The aim of the assessment depends on the source of the assessment and their objectives, and further informs which research components the assessment focuses on. While research projects (Type 4) trying to conceptualize and understand research impact are predominantly concerned with analytical and formative approaches, research organizations (Type 5) also want to address their summative accountability objectives. Some, in particular larger, organizations, therefore use an array of different complementary RIA approaches depending on different contexts and purposes (see CGIAR and FAO).

There are two different approaches in linking research activities to actual or intended impacts. Technical and applied innovations, such as medical or developmental interventions, discuss the direct attribution of research activity to usually rather narrow and specific intended impacts. In this case, research is assumed as a sufficient cause for the impact. When assessing wider societal impacts or systems-oriented scientific activities, RIA rather assesses the contributions of such research activities. It assumes the assessed activities to be a necessary and important contributing but not sufficient factor to these impacts (Reed et al. 2021). Also, the analyzed 70 RIA approaches, being integrated and predominantly discussing wider societal and sustainability impacts of research, focus on contribution rather than attribution of the research activity to the impact.

3.1.3 Impact dimensions

Since RIA approaches make use of different definitions, we summarize some of the terminology, levels, and dimensions used for research impact and outcome in the context of NRM research. Most commonly, research outcome is defined as ‘intended or desired short-to-medium term effects/change expected to be realized from successful delivery of research output’ (CSIRO 2020: 42), hence the uptake of scientific knowledge in policy, practice and public. Impact describes the less tangible and indirect effects of research activities on ‘change or benefit to the economy, environment or society beyond those contributions to academic knowledge’ (ibid.: 42). In some parts of the analyzed RIA approaches, the definitions of impact deviate from this classical definition by introducing additional levels or scales, applying deeper foci on certain aspects of research impact and, in some cases, having blurry boundaries between impact and outcome. In their scaling pathway approach, the IDRC (2017) refrains from the phrase ‘impact’ altogether and scales the societal changes enabled through research activities into ’immediate outcomes’ for primary users, ’intermediate outcomes’ beyond intended users, and ‘development outcomes’.

Some approaches apply different orders of impacts to distinguish between direct and intended impact, and further indirect, unintended, up-scaled or generalized impacts and spill-over effects (Douthwaite et al. 2008; Joly et al. 2015; IDRC 2017; Barret et al. 2018; Fryirs, Brierley and Dixon 2019; Eriksson et al. 2020). The ImpresS approach by CIRAD, for example, defines first level impacts as impacts on ‘actors interacting directly or indirectly with the research community and/or major actors of the innovation’, and second level impacts as the ‘result from spillover effects (indirect impacts) or changes of scale: horizontal (scaling out) and vertical (scaling up)’ (Barret et al. 2018: 12). The ‘direction of impact’ (Edwards and Meagher 2020), hence whether the impact is positive or negative, is only explicitly considered in 1/6 of the analyzed approaches (e.g. Walker et al. 2008; European Commission (EC) 2018; Eriksson et al. 2020; Faure et al. 2020).

Certain approaches focus on only one impact level. The case study approach of Edwards and Meagher (2020), for example, focuses on the immediate and direct impact of research activities. It is exclusively concerned with who or what changed within the duration and proximity of a research project. They further assess in detail which stakeholder group changed and how these changes occurred.

Regarding the impact dimensions, we found three types of RIA approaches: those focusing primarily on assessing societal impact (‘What impact is achieved?’) (52%), those focusing on impact processes (‘How is impact achieved?’) (28%), and those looking at both together (20%) (see Supplementary Table S3).

When analyzing what impact has been achieved or is anticipated, the RIA approaches most commonly discuss impacts of one or more of the four sustainability dimensions. The impacts in the economic, environmental and social dimensions are treated with a similar frequency in the approaches (44 to 54%), while policy or governance impacts are only explicitly assessed in a few (14%) (Figure 5). Only about half cover at least three sustainability dimensions in a highly integrated manner. Most commonly, these approaches integrated the economic, environmental and social impact dimensions (27%). In 10% of the cases all four sustainability dimensions were considered, while 14% explicitly assess ‘sustainability impact’. Nineteen of the analyzed RIA approaches included other impact areas than covered by the predefined sustainability or impact process dimensions. These covered scientific impacts (8), research quality (4), technology or innovation impact (4), intermediate, short-term or direct impacts on intended stakeholders (4), and viability of the assessed research unit (2) or applied a general research strategy (2), applied individual impact scores, or explicitly considered the societal relevance of the research activities.

Approximately half of the approaches consider the underlying processes that lead to impact in their assessment. These can be summarized as processes enabling the (1) dissemination, (2) utilization and (3) application and implementation of scientific knowledge. Reoccurring categories are instrumental impacts on policy and practice, conceptual impacts on attitudes and awareness, capacity building impacts on skills and decision making, and connectivity impacts on network structures (Tilley, Ball and Cassidy 2018; Wyborn et al. 2018; Stockholm Environment Institute (SEI), 2019; Edwards and Meagher 2020; Meadow and Owen 2021).

Eleven RIA approaches showed a high integration of sustainability dimensions and considered impact processes in their assessment. GIZ, for example, uses the SDGs as an evaluation dimension and combines the objectives of ‘results orientation’ as well as ‘utilization focus’ in their evaluation portfolio (GIZ Evaluation Unit 2018). The ImpresS approach by CIRAD includes impact domains ‘access and use of information’ and ‘capacity to innovate’ and ‘institutions and policy’ to not only report on social, environmental and economic impacts but also on impact enabling processes (Faure et al. 2018; Faure et al. 2020). Finally, the integration of impact-orientation and impact processes can improve the way we ‘understand the process and interactions that led to an impact, to evidence the causal mechanisms at play, and to identify the contribution of research in this process’ (Faure et al. 2020:  2).

3.1.4 Time dimension

Research impact can be assessed after the conclusion of a research activity (ex post), anticipated before (ex ante), and monitored during. Across all source types, anticipatory ex ante approaches had the smallest proportion. Only 14% followed a purely ex ante approach, while 41% followed an ex post approach and 40% followed a mixed approach, however with dominating focus on ex post evaluations only vaguely considering ex ante assessment for strategic impact planning. The remaining 4% solely apply monitoring of running research activities.

Around half of the approaches with multiple assessment timings, having a predominantly summative focus to account for their research’s contribution to certain societal impacts, are starting to consider the analytical potential of ex ante assessment for strategic impact planning to different degrees. For example, the International Livestock Institute envisioned to increasingly conceptualize ‘research programmes and projects (…) by “starting with the end in mind” (…), by identifying the development challenge(s) we want to meet—the impact we want to make, where and on whom’ (ILRI 2020: 727).

Only three approaches served exclusively a monitoring function during the research activity. While impact monitoring during a research activity played a role in around half of the mixed assessment timing approaches, it was only considered in respectively 20% of the ex ante or ex post approaches.

The limited representation of ex ante assessment has been acknowledged and criticized in previous studies. The focus on summative accounting for impacts of past activities overlooks the opportunity ‘to be more instructive (…) [and] directly guide the design of research toward maximizing beneficial impacts’ (Weißhuhn, Helming and Ferretti 2018: 40). Chams, Guesmi and Gil (2020: 11) emphasize that a ‘complementary approach based on ex post and ex ante evaluation merits further attention to improve RIA’s ability (…) to understand and report on the mechanisms that generate impact’. CIRAD has recognized these benefits and developed its ImpresS ex ante approach, based on its earlier ex post approach (Blundo-Canto et al. 2018).

3.1.5 Role of societal goals

For research outcomes to be relevant and lead to societal impacts, they need to be linked to political agendas (Pahl-Wostl et al. 2013; Vinke-de Kruijf, Pahl-Wostl and Knieper 2020) commonly formulated in universal societal goals. Societal goals and targets (in nearly all cases sustainability and the SDGs) are considered in 72% of the analyzed RIA approaches, albeit in different forms (Figure 5). One third of the approaches considered societal goals/sustainability solely implicitly in the assessment as an underlying principle of their organization or research activity. Another third went further and assigned societal goals a guiding role in the impact assessment by either making such impacts on societal goals an objective of the RIA, or even aligning the RIA with the SDGs.

Organizations with a long R4D tradition are at the forefront of comprehensive research impact strategies that now also link their research activities to the SDGs. For instance, CGIAR link the 2006 report ‘When will we ever learn? Improving lives through impact evaluation’ by the Center for Global Development to the mission of R4D (Stevenson, Macours and Gollin 2018). The report highlighted ‘long-term issues in (…) [the] understanding of aid effectiveness and led to the creation of the International Initiative for Impact Evaluation (3ie)’ toward ‘improving aid effectiveness’ (ibid.: 11). This joint mission orientation has led to further R4D impact networks, such as the RDI Network. Hence, an inherent mission orientation seems to have facilitated both reflection on impact of research and alignment of these with the SDGs as a societal target system. Sustainable development and the SDGs offer suitable goals and mission orientations (Chapman et al. 2020), and the engagement with SDGs in academia can further advocate for SDGs in the public awareness (SDSN Australia/Pacific 2017; FAO OED 2020, 2021).

The societal goals or missions to which a RIA approach commits can be more context specific and go beyond general sustainability or the SDGs. For instance, the research activities of IWMI are prioritized by a context specific overarching vision (‘A water secure world’), an organizational mission (‘To provide water solutions for sustainable, climate-resilient development’) and a transformative demand on their research (‘Science for a transformative agenda’) (IWMI, 2019: 4). They use these guiding principles to define three response areas (food, climate, growth) linked to the SDGs, addressing global water challenges and assign detailed research priorities that are context dependent yet integrated.

One core feature of the SDGs is the consideration of synergies and trade-offs between different targets. Twelve approaches explicitly discuss trade-offs between societal targets and/or impact dimensions. For example, the IDRC’s scaling pathway approach recognizes that scaling impact to meet real-world challenges ‘produces a collection of impacts, and we must consider the trade-offs between them to determine the magnitude, sustainability, variety, and equity of impact at optimal scale’ (OTT Consulting 2021: 4). Also FAO assessed such trade-offs and synergies between SDG targets, examining their activities in close detail (FAO OED 2020, 2021). They acknowledge a well-established ‘focus on interactions and trade-offs between sectors’ for work on food security, however, identify a lack of such systematic approach to trade-offs within the area of climate change. Henzler et al. (2020: 19) suggest the use of their SDG-based sustainability assessment ‘to identify benefits and trade-offs, and to derive further recommendations for optimization that can contribute to sustainable and future-proof innovations’. Although considered in only half of the approaches aligned with societal goals, the assessment of trade-offs and synergies is inevitable for a comprehensive understanding of societal impact and requires additional capacities and resources.

3.1.6 Role of prioritization and stakeholder engagement

3.1.6.1 Prioritizing research activities

The SDG Stakeholder Guide of UN Sustainable Development Solutions Network (SDSN) emphasizes the integrative approach of the SDGs claiming that ‘the challenges addressed by the SDGs (…) must be pursued in combination (…) and cannot be ordered by priority’ (UNSDSN 2015: 9). While this holds true for a general application of the SDGs, prioritization depending on spatial, organizational, and thematic context is inevitable for research activities. Fifty-seven percent (57%) of the analyzed approaches discuss methods to prioritize research topics. While approximately half of these approaches derive their prioritization solely from internal reviews, e.g. by aligning research activities with international agreements (SDGs, Paris Agreement) as proxies for ‘civic priorities’ (SDSN Australia/Pacific 2017; IWMI, 2019; Blundo-Canto et al. 2020; Chapman et al. 2020), the other half established participatory methods to develop targets or research priorities with partners and stakeholders. Various approaches are employed to deal with the prioritization of research with stakeholders. These include: general collaborative research approaches calling for the co-production and co-implementation of research activities to ensure an iterative common determination of priorities for action (Belcher et al. 2016; SDSN Australia/Pacific 2017; IWMI, 2019; Mach et al. 2020; Graham et al. 2021) and the use of existing partnerships to define and verify identified priorities (ASARECA 2014; The World Bank 2019; Morell 2020); detailed participatory methods, such as problem-tree-analysis, to identify main problems with stakeholders (Douthwaite et al. 2007; Blundo-Canto et al. 2020); multi-stakeholder outcome mapping and the RACI matrix to define collaborative strategies for research activities (Blundo-Canto et al. 2020); the organization of a hackathon (Morell 2020).

3.1.6.2 Prioritizing assessment objects

Forty-seven percent (47%) of the approaches discuss prioritizing case studies and assessment components, such as impact areas or indicators. Again, only half of these approaches involve stakeholders. Some approaches prioritize the assessment of high-impact cases in order to analyze and learn from good practice examples (Joly et al. 2015; Faure et al. 2018; FAO OED 2020, 2021), thereby informing future research. RIA and the reported impacts should be designed to meet the needs of the intended users of research (IDRC 2017). Hence, it is necessary to identify the users und their needs (CSIRO 2020). This can be achieved through the participatory development of impact pathways and monitoring and evaluation plans (Douthwaite et al. 2007; Steger et al. 2021) or through the systematic consideration of different stakeholders’ interests (Saari and Kallio 2011), with developmental evaluation (Saari and Kallio 2011; University of Guelph 2019) or stakeholder analysis (Reed, Bryce and Machen 2018; Tilley, Ball and Cassidy 2018; Edwards and Meagher 2020; Reed et al. 2021). This allows for the consideration of different perspectives, the identification of priority impact areas, and the participatory weighing of research successes. Twenty percent (20%) of the analyzed RIA approaches use systematic tools to prioritize and select impact indicators, commonly referred to as materiality analysis. In the SDG Impact Assessment Tool (Eriksson et al. 2020), researchers prioritize SDGs and define/negotiate the predominant directions of impact they have on them. Roughly linking activities and outcomes to the SDGs can help identify priorities, opportunities, and gaps (European Commission (EC) 2022). The approaches of the UN-affiliated SDSN and FAO both make use of the SDG targets and indicators (SDSN Australia/Pacific 2017; FAO OED 2020, 2021). Quick-indicators for SDG monitoring can be useful for RIA, however need to be expanded by additional indicators aligning with context-specific priorities, which ideally build on existing reporting monitoring (SDSN Australia/Pacific 2017). In this case, data availability is a strong driver for selected indicators. Other approaches use indicator catalogues and assessment matrices to select relevant impact dimensions, and self-assess and weigh impact indicators (Riley and Alexander 1998; Hansen et al. 2013; ASARECA 2014; Reed, Bryce and Machen 2018; University of Calgary 2021). Further RIA approaches go beyond document analyses and state that stakeholders should be consulted for identifying indicators, as they are most likely able to explain how and what change has occurred (Boshoff and Esterhuyse 2016). These approaches involve stakeholders via workshops or interviews in the collection and prioritization of indicators (Madeira et al. 2011; Joly et al. 2015; Hoffmann, Klein and Pohl 2019), and only in the case of ImpresS, also in the quantification of these (Barret et al. 2018; Faure et al. 2018, 2020).

3.1.6.3 Stakeholder Engagement

Forty percent of the analyzed approaches integrate stakeholder (views) to inform prioritization processes. While transdisciplinary projects already include actors outside of academia, Hoffmann, Klein and Pohl (2019) emphasize the necessity to iteratively integrate various target groups beyond the project team. The analyzed approaches showed varying degrees of stakeholder engagement. Some implicitly give attention to stakeholder views or needs through stakeholder analyses, e.g. stakeholder network maps (Douthwaite et al. 2007; Chams, Guesmi and Gil 2020), developmental evaluation (Saari and Kallio 2011; University of Guelph 2019), interest-benefit-matrixes (Reed et al. 2021), alignment-interest-influences-matrixes (AIIM) (Tilley, Ball and Cassidy 2018), frameworks to assess levels of interest in and influences over the intervention (Edwards and Meagher 2020), and 3i-frameworks (Reed, Bryce and Machen 2018). Many of these tools aim to find ways to present or transfer knowledge to stakeholders, rather than giving them a truly participatory role in the development, implementation, or assessment of research. Only a few approaches explicitly include stakeholder opinions through workshops or surveys (Douthwaite et al. 2007; Faure et al. 2018; Chams, Guesmi and Gil 2020; FAO OED 2020), although the active inclusion can help build common understanding and commitment with stakeholders (Douthwaite et al. 2007). The ImpresS approach suggests to combine participatory and non-participatory tools to ‘unpack long and complex multi-stakeholder innovation processes and to systemically assess impacts achieved, from the perspectives of diverse stakeholders’, while still being frugal with time, human and financial resources (Faure, et al. 2020: 9). Implicit as well as explicit stakeholder integration can increase the legitimacy of the planned research activity, impact assessment, and its results.

3.1.7 System boundaries

When drawing system boundaries for research assessment, Reed et al. (2021: 8) advise that impact ‘may be evaluated over different time horizons, at different social scales (from individuals to society), spatial scales (from local to international) and across multiple domains (including social, economic, environmental, health and wellbeing, and cultural)’. Nevertheless, we identified an additional scale relevant for RIA in NRM research, namely a scale of the assessed research activity. Just like the time horizon, spatial or social scale, the research activity scale also ranges from small to large. Unlike the scales identified by Reed, focusing on areas research has an impact ‘on’, our additional scale sets a boundary to the research activity the RIA approach assesses the impact ‘of’.

The scales used for the units of assessment in the analyzed integrated RIA approaches reach from single interventions to international research communities (see Supplementary Table S3). The scale of research activities can be divided into (1) single interventions, such as single studies, projects, or methods; (2) organizational research communities, such as entire research institutes or internal cross-sectional research topic or working groups; (3) national research communities, connected through research programs, networks or common policy; and (4) international research community, connected through international research networks, disciplines, or considered as research in general. The largest proportion (39%) of approaches applies the assessment to single research interventions, predominantly research projects (see Supplementary Figure S1 for details). The second largest proportion (20%) applies the assessment on multiple levels, in all but one case starting at a single research intervention and scaling the assessment findings up to the scale of the organizational, national, or international research community. The third largest proportion (15%) applies the assessment to the organizational research community, mainly upscaling from research clusters to the whole research institute.

Two-thirds of the approaches developed by research institutes to assess their own research apply a scale beyond single research interventions. These scales are often up-scaled from single interventions to an organizational, national or international level, depending on the reach of the institution. With different levels, the aim of the assessment can change. At the International Development Research Centre (IDCR), for example, assessments on an organizational level have a mainly formative purpose, while the aim of accountability of impact increases with smaller scales (IDRC 2017). Also, the ImpresS approach by CIRAD follows a scaling process from ‘project’ to ‘project cluster’ to ‘research themes’ to the ‘research institute’, and points out that assessments ‘at “non-project” scales help elucidate partnership strategies from different angles’ (Blundo-Canto et al. 2020: 63).

4. Framework for systemic research impact assessment

A new type of RIA has emerged recently. An increased focus on societal responsibilities of human action and in particular the introduction of the SDGs has triggered the development and application of RIA approaches assessing the contribution of research to societal change and transformation. Over the past decade, there has been a growing trend of incorporating societal target systems into RIA, necessitating the integration of stakeholder viewpoints and priorities to also address trade-offs between societal targets. When applied in an ex ante approach, such integrated RIA can be instrumental for strategic research planning. To address this evolving landscape, we developed a comprehensive RIA framework (Figure 6) aimed at supporting research in assessing and strategically planning contributions to societal transformation. It combines a (1) missions component aligning the assessment with societal goals, (2) inclusive component applying systematic and participatory priority setting, (3) strategic component selecting assessment types and time dimensions with (4) an integrated component to impact dimensions. Finally, we present suitable examples identified in the analysis of 70 integrated RIA approaches.

4.1 Missions component—enable relevancy of research

Linking research activities to universal societal goals, targets, and missions can enhance the relevancy of research in addressing current complex problems. Societal goals may implicitly guide research planning when recognized as principles within the research organization or the research activity itself, or explicitly when RIA approaches assign a guiding role to these goals in their assessment. These goals can either be of general/universal character, such as the SDGs, or they can be more context specific, e.g. the water related vision, mission, and transformative demand of IWMI (2019).

4.2 Inclusive component—enable legitimacy of research

Engaging relevant stakeholders in all stages of research, namely the problem formulation, development, and conduction of a research activity, as well as in the (iterative) assessment of goals and successes, can enable the legitimacy of research. The extent of the engagement can vary from passive, e.g. analyzing stakeholder interests, to active roles, e.g. co-creation. Stakeholder interests can thus prioritize the research agenda as well as the goals the research’s success will be measured by. Beyond the intrinsic motivations of research organizations and researchers to apply inclusive approaches in research, also funders can require co-production or ‘engage users at multiple stages of the funding life cycle to help prioritize research questions, review proposals, and assess outcome’ (Mach et al. 2020: 34).

4.3 Strategic component—enable effectiveness of research

Depending on the purpose of the RIA, one needs to strategically choose the scales, aims, and timing of the assessment. Motivations for RIA vary across assessment sources, informing further characteristics of the approach. While assessing a single research intervention may suffice for planning a new project, strategic planning of an organization’s research area or nationally/internationally funded research programs requires larger scales. While accounting for impact evidence of single interventions is feasible, this task becomes progressively challenging as the assessment scale grows larger. The analysis of research impact is achievable and carried out across various assessment scales. The main aim of RIA, applied for the planning of impactful research, should be the formative analysis and anticipation of impact, strengthening the role of ex ante assessments and reflexive learning. Using previous research for learning or employing analytical RIA processes to set up a succeeding accounting framework can be a useful complement. Results of ex post assessments can help understand conditions and processes benefiting impact. The ex ante assessment can also be followed by (periodical) reflection and monitoring of anticipated and achieved impacts, and concluded with an ex post assessment.

4.4 Integrated component—enable systemic reflection

Considering impacts from all four sustainability dimensions, along with the preconditions and processes enabling societal impact, is crucial for gaining a systemic understanding of the different contributions research makes to transforming complex socio-ecological systems. Through systemic reflection, RIA approaches can anticipate direct, indirect, positive, and negative impacts, pay attention to synergies and trade-offs, and thereby plan research that consciously contributes to system transformation.

4.5 Interconnected components of the framework

The components of the framework are not exclusive but are interconnected. For instance, aligning RIA with societal goals constitutes an integrated consideration of impact dimensions and can inform priority setting of research topics. However, even in very systemic and integrated RIAs, one needs to prioritize the considered impacts and indicators to be frugal of resources. The framework makes apparent that it is not ‘one size fits all’ solution, and that the source and motivation of the assessment will result in different RIA characteristics. Therefore, some, especially larger, organizations use an array of different complementary RIA approaches depending on different contexts and purposes (see CGIAR and FAO). When designing a RIA, one should initially consider all components of the framework but assign them varying weights based on the purpose and scale of the assessment. For a large-scale meta-analysis of research activities at an organizational level, for example, direct stakeholder engagement may be less feasible or necessary than for an ex ante assessment of a proposed individual research project.

4.6 Suitable examples

The framework was applied to characterize the analyzed integrated RIA approaches concerning their suitability for systemic (formative) assessments (see Supplementary Table S3). The scoring applied for this purpose does not imply a quality assessment of the tools themselves but rather a suitability evaluation of the systemic nature of the approaches in relation to a combination of the four the components outlined above:

• The ImpresS approach by CIRAD applies both ex ante (Blundo-Canto et al. 2018; Blundo-Canto et al. 2020) and ex post (Barret et al. 2018; Faure et al. 2018, 2020) assessments to analyze and account for its research impact. It links its impacts to the SDGs, considers all impact dimensions, as well as impact-generating processes, and even made ‘building a culture of impact’ a core aim of their approach (Blundo-Canto et al. 2019). Research as well as assessment topics are prioritized by political agendas, using analytical tools like problem tree, as well as participatory tools such as stakeholder surveys and workshops. The scale of assessment can be adjusted to fit different aims.

• Although applying only an ex post assessment, the impact assessment of agricultural research on sustainable development by EIARD (Baur et al. 2003) is one of the first integrated RIAs that assigns sustainable development a guiding role in the assessment of impact. The varying aims of accountability and analysis are achieved through up-scaled units of assessment, ranging from assessments of single projects to the aggregation to EU programs. Stakeholders were included in verifying the impact.

• The Impact Assessment within the Horizon 2020 program (Arhus University 2016; European Commission (EC) 2018) only explicitly addresses social and economic impact, however, requires the outline of (potential) impact of research on EU priorities, which include meeting the SDGs. Ex ante self-assessment can be used to advocate for the funding of ones suggested research project and the presented ex ante assessment results inform the prioritization and allocation of research funding. Monitoring and ex post analyses, in turn, steer future research innovation policy.

• Also, the Scaling Pathway by IDRC (IDRC 2017; OTT Consulting 2021) uses different assessment scales depending on assessment aims. The aims for different scales are participatorily identified and the assessment designed to meet the needs of the users of research. Although not giving special attention to environmental impacts, this approach explicitly investigates trade-offs between intended and unintended impacts.

The framework provides an overview of the essential components of systemic RIA and their manifestations and should be consulted when designing a new RIA. Applying systemic RIA and paying attention to all components of our framework can change the planning, execution and outcomes of NRM research. A participatory and anticipatory (ex ante) reflection on societal goals and their integrated impacts can become a routine process, informing research proposals that address inter- and transdisciplinary diversity to solve complex issues. When applying for funding sources that increasingly demand the consideration of societal impact (e.g. EU), presenting results of such RIA can constitute a competitive advantage. The early integration of systemic RIA before or during a research activity allows for a systematic documentation of necessary partners, stakeholder, enablers and barriers. The early agreement on anticipated impacts and connected indicators allows to monitor successes and shortcoming during the research activity, and inform necessary adaptations. The reflection and/or inclusion of interested, impacted and/or influential stakeholders and the early planning of tailored transfer products will allow for a more successful application and implementation of the created knowledge in society, policy and practice.

5. Conclusions

One of the first peer-reviewed articles analyzed in this review, from 1998, already recognized that ‘research impact assessment involves a long-term management approach’ that relies on incorporating ex ante impact assessment already in the project design (Riley and Alexander 1998: 99). It further pointed out the ability of medical and health research to use suitable indicators linked to societal targets (e.g. good quality-of-life and sustainability) and suggested to learn from them. Already then, the authors advocated for integrated approaches and criticized one-dimensional assessments.

Our review has revealed that, since then, systemic RIA approaches in NRM research remain scarce and are predominately clustered around R4D organizations, suggesting that common missions fuel the strategic consideration of impact in research practices. We observe a growing trend of incorporating impact assessment throughout all cycles of research and call for further efforts to expand the use of ex ante methods for planning impactful research. Finally, we have developed a systemic framework to use RIA for varying purposes, combining a missions component, an inclusive component, a strategic component, and an integrated component.

We do not claim that the list of analyzed approaches is complete, but we made a concerted effort to systematically collect a representative compilation of integrated RIA approaches in the NRM context in peer-reviewed as well as grey literature. While our compilation may not cover every approach in existence, we believe that the number and diversity of the included approaches are sufficient for our in-depth analysis to draw meaningful assumptions about the characteristics of such RIA approaches. Similar analyses in other sectors, such as R4D, may provide further insights into the tools developed and applied in RIA.

In conclusion, using ex ante approaches, along with common societal targets for mission-orientation, and participation and prioritization techniques, can serve as effective tools to enhance the impact of planned research activities. This principle applies across all discussed sources of RIA, including applications by researchers in research projects, by research-funding organizations or by research organizations alike. Funders gain the ability to allocate investment more advantageously, research organizations can learn and strategically plan their research focus and pathways, and researchers can identify and propose relevant research questions. It is crucial to emphasize that the proposed framework in no way diminishes the essential importance of explorative, blue sky research. However, when it comes to the contribution of research to solving societal challenges, the suggested approach may increase efficiency.

Finally, our framework is an invitation to all individuals engaged in conducting or facilitating mission-oriented research to actively participate in anticipating societal impact, to reflect priorities of different stakeholder groups, and to align research actions with common societal targets. The framework will allow for the careful design of RIA approaches tailored for differing purposes and already serves as the foundation for an ex ante RIA approach developed for and implemented in a research project in which the authors are involved in.

Supplementary data

Supplementary data are available at Research Evaluation Journal online.

Acknowledgements

The authors would like to thank Sina Trölenberg and Katharina Schuster for their support in retrieving documents and editing visualizations in the article. Furthermore, the authors would like to thank Alexandra Bussler, Ian Overton and Sebastian Ferse for their comments on a draft version of the manuscript.

Author contributions

L.P.: Conceptualization; Methodology; Formal analysis; Writing—Original Draft, Review & Editing; Visualization. K.H.: Conceptualization; Writing—Original Draft, Review & Editing.

Funding

This work was supported by the project LeNa Shape (Forschen in gesellschaftlicher: Verantwortung—Gestaltung, Wirkungsanalyse, Qualitätssicherung) (grant number: FKZ 01UV2110G), funded by the German Federal Ministry of Education and Research (BMBF).

Conflict of interest statement. None declared.

References