Capabilities for transdisciplinary research Open Access

Abstract

Problems framed as societal challenges have provided fresh impetus for transdisciplinary research. In response, funders have started programmes aimed at increasing transdisciplinary research capacity. However, current programme evaluations do not adequately measure the skills and characteristics of individuals and collectives doing this research. Addressing this gap, we propose a systematic framework for evaluating transdisciplinary research based on the Capability Approach, a set of concepts designed to assess practices, institutions, and people based on public values. The framework is operationalized through a mixed-method procedure which evaluates capabilities as they are valued and experienced by researchers themselves. The procedure is tested on a portfolio of ‘pump-priming’ research projects in the UK. We find these projects are sites of capability development in three ways: through convening cognitive capabilities required for academic practice; cultivating informal tacit capabilities; and maintaining often unacknowledged backstage capabilities over durations that extend beyond the lifetime of individual projects. Directing greater attention to these different modes of capability development in transdisciplinary research programmes may be useful formatively in identifying areas for ongoing project support, and also in steering research system capacity towards societal needs.

1. Introduction

Research framed to address global, grand, and societal challenges has brought fresh impetus to calls by funding agencies for increased transdisciplinary research capacity (Bammer et al. 2013; Lyall and Fletcher 2013; Belmont Forum 2020; European Commission 2020; Belcher and Hughes 2021). Complex and cross-domain problems such as climate change, public health programmes, and efforts to meet the UN Sustainable Development Goals, provide rationales for research design that go beyond mono- or even interdisciplinary solutions (Werlen 2015). However, transdisciplinary (TD) research cannot be produced on demand. Appropriate TD research capacity is required.

We understand TD research as knowledge production activities spanning across disciplinary boundaries and meaningfully involving non-academic partners in research design, operation, analysis, publication, and the practice of attendant methods. TD research is problem-oriented, transcends separate disciplinary sectors, transgresses disciplinary and institutional boundaries and is context specific (Nowotny 2000; Klein 2006; Huutoniemi 2010). It facilitates diverse and mutually accountable interactions between participants and different styles of knowledge. In doing so, TD research potentially fosters transformative scientific and technological progress through social innovations and represented marginalized interests (Lang et al. 2012; Stirling 2015; Smith and Stirling 2018). TD research practices and evaluations are therefore ‘generative processes of harvesting, capitalizing, and leveraging multiple kinds of expertise’ (Klein 2008). While there are numerous discussions on the relations between interdisciplinarity and transdisciplinarity (Klein 2008; Pohl et al. 2011; Lyall Meagher Bruce 2015), they remain outside the scope of this article. Instead, we focus on the aforementioned features of transdisciplinarity as a mode of research contingent on cultivating relevant capabilities.

For expertise to be useful it must be instantiated as a kind of capability. Research capacity then is the aggregate capabilities and resources required for organizations, groups, and individuals to mobilize expertise and perform research. Building, maintaining, and evaluating TD research capacity is important in several regards. First so that a state of readiness might be reached and maintained (Klein 2008; Falk-Krzesinski et al. 2011). Readiness to do research, address wicked research problems quickly, or make complex research feasible—feasibility and risk can be an inhibiting feature of research systems (Luukkonen 2012). Second, and responding to the need for readiness, building, or enhancing capacity has become a major challenge for funders (Lyall and Meagher 2012). This has been a significant aim of major research initiatives over the past decade, including major UK projects such as RELU, UK SPINE, and UKRI CCF. Finally, capabilities and enhanced capacity are themselves a product of research, a first-order or intermediate outcome of transdisciplinary activity (Wiek et al. 2014) making TD capacity building an important impact goal of some projects (Hansson and Polk 2018).

Assessing capacity and underlying capabilities is important because they both contribute to a broader evaluation of the complex and often non-quantifiable criteria of interactions between science and society in TD research (Ernø-Kjølhede and Hansson 2011; van der Ploeg 2011). Such interactions include, but are not limited to, collaboration, integration of knowledge, learning processes, and the performance of cognitive and social functions. Moreover, evaluating TD research is important for funders normatively in measuring processes and outcomes; for demonstrating return on investment and legitimizing funding programmes (Mosse 2004). It is also analytically interesting for evaluators and researchers for what it tells us about what kind of TD research is being done and how society and research systems are reconfigured through collaborative, integrative, and ontological work (Frederiksen, Hansson and Wenneberg 2003; Molas-Gallart et al. 2016).

Methods of evaluating research programmes tend to focus on quantitative metrics of research outputs, qualitative network analysis (Oancea, Florez Petour Atkinson 2017), examining ‘productive interactions’ in research between academics and non-academics (Molas-Gallart and Tang 2011; Spaapen and van Drooge 2011), and qualitative evaluations of policy interventions (Smutylo 2001). These efforts however tend not to explicitly focus on capabilities. Nor do these efforts systematically evaluate how policy interventions build capabilities for TD research. The aim of this article is therefore to develop a framework with which to conceptualize and systematically evaluate capabilities for TD research attentive to these gaps. Our entry point is from the perspective of UK research funding agencies, and their institutional obligation to expand capacity on the academic side of transdisciplinary collaborations. As such, our evaluation centres on the capabilities of academic researchers in the first instance.

We discuss the particulars of these capabilities and how they might arise in Section 2, where we also provide an account of the evaluative framework’s conceptual underpinnings using the Capabilities Approach (Sen 1999; Nussbaum 2001). In order to locate, explore and assess the presence of research capabilities in the field, we propose a novel framework of evaluation. Our research question is: what are the different capabilities valued by researchers performing TD research, and how can these be mapped in specific research projects?

We describe the procedural tasks of evaluation and propose a novel methodological approach in Section 3. In Section 4, we demonstrate the framework with a case study of a TD research programme, the ESRC Nexus Network+ (NN+)¹. We discuss the analysis and implications for evaluation practice, theory, and policy in Section 5. Section 6 offers a short conclusion.

2. Theory: evaluating capabilities for transdisciplinary research

2.1. Locating transdisciplinary research capabilities in practice and theory

What can we learn from how evaluators have conceptualized capabilities to date? Here we understand capabilities as an individual or group’s ability to do valuable acts or reach valuable states of being in pursuit of TD research. For instance, skills, aptitudes (Lyall and Meagher 2012), competences (Hoffmann et al. 2017; Hansson and Polk 2018), and human capital (Bozeman et al. 2001) are all relevant to human capabilities and feature in procedures evaluating the impacts and outputs of research. Our primary interest in outputs is in how they contribute to research capacity and readiness levels and the capabilities that are variously constitutive of these outputs or required in their production. In this way, scientific and technological human capital (Bozeman et al. 2001; Bozeman and Corley 2004) can be understood as outputs and inputs of TD research. These include skills and career prospects of expert researchers measured using bibliometric techniques and content analysis of CVs. Similar techniques have been used to assess capabilities produced in graduate courses and on research teams (de Oliveira et al. 2019).

Capabilities also feature in evaluations of research processes. Processes and constituent research practices include the cognitive disciplinary work of research as well as the integrative (Boix-Mansilla 2006) and ontological (Barry, Born and Weszkalnys 2008) work of TD research. They also include social and collaborative practices across project groups and with stakeholders in society. These processes are underpinned in part by what Klein (2008) calls ‘competencies’. That is, how well methods for decision-making, knowledge distribution, and networking are managed across projects and sub-projects. These methods in turn require management, coaching, and leadership capabilities.

Cutting across evaluations of outputs and processes are issues of quality (Feller 2006; Huutoniemi et al. 2010; de Jong et al. 2011; Belcher et al. 2016). Collaboration for example, can be evaluated as a process, and also be used as an indicator of research quality (de Jong et al. 2011). Collaborative practices in particular draw our attention to capabilities that are inherently collective, that is situated across groups. TD quality can be enhanced through the enacting of certain capabilities: mastering multiple disciplines, emphasizing integration, critiquing disciplinarity (Huutoniemi 2010). Effectiveness is one useful criterion of quality with which evaluators can assess social learning (the change in knowledge, attitudes, and skills at the individual, group, and institutional level) and the building of what they call societal capacity (Belcher et al. 2016).

A common feature of TD research is that assessments of quality are relative to the goals, expectations, norms, and values of research stakeholders and thus vary from one TD research context to another (Huutoniemi 2010). De Jong (2011) for example evaluates performance of a research groups against their own research aims. More broadly, the role of context in TD research has formed an important theme within evaluation literature (Feller 2006; Stokols et al. 2008; Hansson and Polk 2018). Evaluators that assess context draw attention to the role of norms, institutions, rules, and conditions of enquiry in shaping, enhancing, and diminishing individual and group capabilities (Mormina 2019). Context is also the location for interventions that might enhance capabilities and capacity for integration and collaboration (Klein 2008).

This brief discussion indicates some of the features a framework to evaluate capabilities needs to incorporate. It needs to understand capabilities as constitutive of research practice, as well as outputs of research processes. It needs to account for social as well as cognitive capabilities and those performed by individuals as well as groups. It needs to be sensitive to the effect of contextual features on research practice and the complexity of situations in which they arise. It needs to take into account the aims of specific research projects and constituent researchers. And it needs to acknowledge the subjectivity and politics inherent in the values and decisions that determine which knowledge and methods are made to matter through processes of reflection, deliberation, and negotiation (Wiek et al. 2014) or otherwise. To accomplish this, we conceptualize capabilities using the Capability Approach and operationalize the framework using a realist evaluation technique (Section 3).

2.2. The capability approach—evaluating what researchers are effectively able to do

What are capabilities exactly, and what capabilities are required to carry out these TD research practices? Moreover, what can an assessment of capabilities tell us that current evaluative procedures cannot? Concepts about human capabilities by development scholars such as Amartya Sen (1999) and Martha Nussbaum (2001) are useful here. The Capability Approach (CA) is a framework with which to assess institutions and practices based on one or several public values, conceptualized in capability terms (Robeyns 2016). Emerging from assessments of public values related to quality of life (Sen) and social justice (Nussbaum), the CA has been used for a range of theoretical and conceptual work (Robeyns 2016). These include small-scale project evaluation (Alkire 2005) and assessing human capabilities in collaborative workshops (O’Donovan and Smith 2020). Much research has been published on education and capabilities (Saito 2003; Walker 2003) but rather less on the practices of research. Addressing this gap, we develop a conceptualization of capabilities for TD research that are valued by researchers and other interested agents.

The core characteristic of the approach is its focus on what people are effectively able to do and be; that is, on their capabilities (Robeyns 2005 p. 94). For example, the research techniques available for researchers to choose to practice—such as the capability to do ethnographic fieldwork, or to be an environmental economist. In other words, something discrete and measurable, so a research project’s capability set can be thought of like a budget set—the capabilities valued and available to that project. The CA is normative in that it supports analysis and decisions based on value judgements about how research groups ought to behave in order to create research with and for society. A major strength of the CA is that it allows us account for the normative assumptions of the research subjects, in our case TD researchers.

To be clear, in our evaluation we focus on the cultivation and availability of capabilities to act, rather than the achievements of the group being evaluated—what capability theorists call achieved functionings. Robeyns (2005) provides a comprehensive overview of concepts and procedures used in theorizing and operationalizing capability frameworks, of which the most relevant to the aims of this article are sketched out below. ²

2.3. Our conceptual framework

We now introduce a conceptual framework that builds on the evaluative features of the capability approach in order to address the issues of TD research discussed in Section 2.1. TD research projects form the evaluative space—what we will refer to as a capability space—for our framework. But what capabilities matter within these projects?

Our analysis follows Sen (1999) and Robeyns (2005) in seeing the capabilities valued by and available to researchers as a matter of empirical identification. Moreover, taking the situatedness of TD research seriously directs us to paying attention to how researcher might value specific capabilities differently depending on the context of the research. After all, TD research gains in accountability precisely because of the situated nature of the socially robust knowledge it might generate.

Nevertheless, we do not start from an entirely blank slate. The literature discussed in Section 2.1 gives us some clues as to what kind of capabilities are likely to be valued by researchers in TD research projects generally. We use this literature as a starting point with which to map the capability spaces of the NN+ projects specifically.

To guide our exploration of these spaces we propose a four-part, conceptually driven heuristic. These parts account for (i) individual capabilities held by researchers; (ii) collective capabilities held within the capability set of the research projects; (iii) cognitive capabilities required to perform academic practices and again collectively held in the context of transdisciplinary research; (iv) contextual features which shape the capability space of research projects. The idea is to use this heuristic to direct our attention to where and how capabilities are most likely to be situated, whilst—again following Robeyns (2005)—inductively locating them through bibliometric techniques, grounded methods, and careful analysis.

2.3.1 Individual capabilities

Many capabilities required for research are more ‘social or political than simply cognitive’ (Bozeman and Rogers 2001, p. 418). These include ‘the strategies and policies of knowing that are not codified in textbooks but nevertheless inform expert practice’ (Knorr Cetina 1999, p. 2). Examples include skills, tacit knowledge, and experiential knowledge acquired by individual researchers (Heckman and Corbin 2016) and performed in the social setting of research projects. Tacit, social, and cognitive factors interact, requiring management of information and decision-making (Klein 2006). Such tacit individual capabilities are influenced by personal background and include often individualistic ambitions like the capability to have an identity as a researchers (Lau and Pasquini 2008).

2.3.2 Collective capabilities

Bozeman and Rogers considered the human and technical capital required for research to include the ‘sum total of scientific, technical, and social knowledge and skills and capabilities available to a particular individual’ (Bozeman and Rogers 2001). In TD practice though, individual researchers rarely, if ever, cultivate all of these capabilities required to carry out research. Research project groups facilitate the division of labour, share expertise, and contribute mutual or collective capabilities and are an everyday feature of scientific research.

CA scholars have shown how the provision and achievement of capabilities is held, used and influenced by group, social, and environmental factors (Stewart 2005; Ibrahim 2006). Moreover, a set of capabilities available to an individual or group can be expanded through collective action as well as individual effort (Stewart 2005; Roy 2012). Two points are important here. First, we can expect that a research group’s capability set contains the aggregate of individual capabilities. Second, the capability set also contains capabilities that are mutually held and not dividable to individual researchers. In other words, there are collective capabilities of a group that are not features of individuals on their own.

For example, the production of scientific knowledge, legitimacy and individual expertise is influenced socially (Cuevas Garcia 2016) and culturally (Corley et al. 2019). Collective capabilities that might be valued by researchers include capabilities for collaboration and coordination (Hohl, Knerr and Thompson 2019), capabilities to build and continuously maintain networks (Spaapen and van Drooge 2011; Hansson and Polk 2018) and to promote interpersonal and intrapersonal learning that can foster enhanced understanding and capabilities to respond to complex questions, issues, or problems (Lyall and Meagher 2012).

2.3.3 Cognitive capabilities

Indeed, the expansion of collective cognitive capabilities is an evaluative impact of research in itself (Molas-Gallart, Tang and Rafols 2014). These are the capabilities required to perform scientific and technical work include formal educational endowments acquired through graduate and post-graduate training, usually encompassed in concepts of ‘human capital’ (e.g. Becker 1964). These cognitive capabilities variously include the skills and formally taught techniques required, for instance, to take soil samples, conduct interviews, or perform econometric modelling and to perform advanced knowledge tasks such as differentiating, reconciling and synthesizing (Lyall and Meagher 2012).

The provision of these capabilities usually follows disciplinary lines in undergraduate education and often through post-graduate training. These demarcations are also evident in academic journals, which remain, for the most part, closely organized along disciplinary lines (Rafols et al. 2012). Critically for this study, the capability to publish in a disciplinary journal is an important indicator of the capabilities available collectively to researchers in the context of the specific research projects we investigate.

2.3.4 Contextual influences on capabilities

The social and material contexts in which capabilities are put into practice in the production of knowledge are multiple, patchy and heterogeneous (Pickering 1992, p. 8). These include offices in academic departments, disciplinary norms, methods, campus rules, research ethics frameworks, field locations and the myriad of instruments, tools, knowledges, and relations at and between these sites (Feller 2006; Hansson and Polk 2018). In evaluation, flexibility and sensitivity to this diversity and context is critical (Langfeldt 2006).

The normative evaluation of capabilities in context was part of Sen’s motivation in early developments of the CA (Couldry 2019). This milieu of social institutions, norms, environmental factors, guiding policy visions, regulations, and people’s behaviours are what CA analysts call conversion factors that mediate capability inputs, such as knowledge, finance, resources, goods, and services that in turn facilitate the cultivation of capabilities within a capability set (Robeyns 2005). The degree, distribution, and quality of capabilities within a research project is shaped by these factors. Exactly how is, again, a matter for empirical research. Our framework is attentive on the one hand to how researchers might value capabilities differently depending on the context of specific projects. And on the other, to how contextual phenomena influence the availability of capabilities valued by researchers within a capability space. To be clear, the contextual influences that form the final part of our heuristic are not strictly capabilities, but rather the broader social contexts and socio-material configurations of research projects that influence capabilities (Robeyns 2005; Oosterlaken 2011; Zheng and Stahl 2011). Heuristic components are summarised in Table  1.

3. Methods

Based on our conceptualization of transdisciplinary research projects, we need a methodology that accomplishes a series of analytic tasks.

• Stage I: identify inductively a list of capabilities associated with TD research practices.

• Stage II: appraise how the capabilities claimed for TD research sites are actually experienced by researchers.

• Stage III: compare the capabilities in Stage I with those mapped in Stage II in order to identify expected and absent capabilities.

To achieve this, we follow a typical evaluation pathway (Gilmore et al. 2019): establishing expectations, determining reality, and locating differences—in this case between expected capabilities and those which are experienced by researchers. This is a theory-driven approach that lets us locate discrete sets of capabilities, while also acknowledging the limitations, situatedness, and contextual features of those capabilities.

3.1 Realist evaluation and the capabilities approach

The three tasks outlined above are typical of a Realist Evaluation (RE) in which the core expectations of an Initial Programme Theory (IPT) are set a priori (Stage I) (Nurjono et al. 2018). Data appropriate to examining that IPT is then collected (Stage II). Then IPT expectations are recursively compared with reality (Stage III). Importantly for our purposes, while alternative forms of theory-driven evaluation (for instance programme-theory evaluation or theory-of-change evaluation) ask questions about ‘what works?’, RE is focused on ‘what works for whom and in what contexts?’ (Pawson and Tilley 1997). As such, RE enables us to not only discover what capabilities enabled researchers to conduct TD research, but how the situatedness of those capabilities shaped their usefulness. We detail the methodology for each of these stages in turn.

3.2.1 Stage I—compiling a list of expected capabilities

Adapting the CA for analytical purposes requires careful explanation and justification of the relevant capabilities identified and the methods used. Robeyns (2003) sets out identification criteria: they should be explicit, discussed and defended. The level of abstraction should be appropriate to the study context and project objectives. Ideal sets of capabilities must become a pragmatic list that can be studied and should include all important elements non-reducible to the other elements, even if there exists some overlap.

Guided by these criteria we identified a list of capabilities expected in the test programme, arrived at by searching for signals of our conceptual framework in a focused review of the literature on conducting and evaluating TD research. We began the task by searching the literature for signals and expressions of our conceptual framework—cognitive, tacit, collective beings, and doings. We refined this list using project documentation (project proposals, annual reports, etc.) from within the NN+ partnership programme. Finally, we integrated observations from two workshops organized by the NN+, ‘Transdisciplinary Methods for Developing Nexus Capabilities’. Details of these sources of evidence are reported in Section 4.1 (Table  2) and identified capabilities are reported in Section 4.3.

3.2.2 Stage II—mapping capabilities at research sites

Following Step 2 in the realist evaluation procedure and further guided by the heuristic, we next gathered data on actual capabilities valued by researchers within each case. In order to map the collective cognitive capabilities available at each research project, bibliometric profiles for the respective projects were compiled. These consisted of aggregate records of peer-reviewed journal publications of constituent researchers, collected on 20th August 2018 (no time boundary). Mapping the bibliometric profiles onto Web of Science categories (Leydesdorff, Carley and Rafols 2013) gave us indicators of different cognitive capabilities and disciplines that constituted each of the NN+ projects. ³ Furthermore, and despite some limitations, for each of these project profiles, a co-authorship map gave us inference into potential relations between project researchers.

Knowledge outputs such as bibliometric indicators, however, only partially reveal characteristics of a knowledge community (Strathern 2004) and are insufficient to capture the nature of transdisciplinary programmes (Koier and Horlings 2015); for instance reduced efficacy for early career researchers and non-academic participants (who have fewer publications). These limitations were mitigated through close analysis of the project proposals and the self-reported biographies of project researchers. Also, findings derived from bibliometric analysis were explicitly triangulated and augmented, first at workshop 2 (see Table  2) and over the course of researcher interviews.

In parallel, a questionnaire was deployed to all researchers in the five NN+ projects and this article reports 26 respondents. The questionnaire asked about the nature and frequency of interpersonal interactions within and across projects in order to map further individual, collective and group capabilities. It also sought to generate data on how capabilities were valued and experienced by researchers. This added qualitative data from individuals who were not available for interview, triangulating project narratives. Moreover, this provided an insight into the collective and contextual capabilities in the projects (Ayre and O’Donovan 2018). Data from the questionnaire and bibliometric analyses formed the basis for 15 semi-structured interviews with project researchers from across the five NN+ projects. PIs, Co-Is, and research associates as well as and practitioner partners were interviewed. These interviews provided further feedback loops within the evaluation framework. They also allowed identification of research practices and capability experiences that were important to researchers actually doing the research.

3.2.3 Stage III—comparing the list of expected capabilities with evidence from research projects

The final step of realist evaluation is in practice multiple iterations of the same step—to iteratively reflect upon our initial list of capabilities as our data collection and analysis evolves (Gilmore et al. 2019). Stage III occurred in parallel with Stages I and II. This interplay between expectations and reported reality is where realist evaluation highlights those aspects which are structural, unexpected, or absent entirely. As more data was gathered, project stories became thicker, more detailed, and increased in specificity. This allowed us to infer more accurately the capabilities valued by researchers within each project and therefore refine our Initial Programme Theory (the capabilities list).

4. Evaluating transdisciplinary capabilities in the nexus network+

4.1 The nexus network+ partnership programme

In 2014, the ESRC Nexus Network+ was awarded £1.5 m in funding by the UK’s Economic and Social Research Council in order to build a network of researchers capable of examining interactions within and between domains of water, energy, and food, the WEF nexus (Cairns, Wilsdon and O’Donovan 2017; UKRI 2020). The Network + was the second iteration of a new ‘pump-priming’ model introduced by UK public funding agencies which included the provision of refunding budget for onward distribution over the course of several programmes of activities. Characteristics of the programme and projects are introduced in Table  2, and further bibliometric data are available in the Supplementary material. In the following sections, we report on capabilities valued by researchers across the entire portfolio. We begin by demonstrating the framework in action by reporting in-detail on Project 1.

4.2 Mapping the capabilities within NN+ partnership programme Project 1

The aim of Project 1 was to facilitate the transfer of research findings from environmental social sciences into the UK policy context (Table  2). Bibliometric analysis indicates disciplinary characteristics of the set of collective cognitive capabilities such as capabilities in geography and other social sciences with environmental sciences and civil engineering (Figure  1).

Figure  1 reveals the presence of social science capabilities on the left-hand side in line with the ESRC’s goal for the NN+ to ‘engage the social science community with these complex ‘Nexus challenges’ and link them to research users from business, government, and civil society’ (UKRI 2020). This already indicates the possibility of an incursion of social science methods and capabilities into the WEF nexus idiom, traditionally composed of engineering and environmental science expertise, for instance capabilities for optimizing, modelling, and quantifying.

The large clusters on the right-hand side indicate researchers with published outputs in the natural and environmental sciences and engineering indicating a plurality of cognitive capabilities available collectively at the project. Nevertheless, interpretive social scientists with a shared interest in the domestic use of water, energy and food were in the majority. For these researchers, collaboration with practitioners was more significant than the integration of knowledge across academic disciplines:

‘Insofar as our project was transdisciplinary, it was transdisciplinary in terms of involving practitioners as well as academics. Personally, the term I would use would probably be ‘collaborative research’ because it was collaborating with practitioners’ (Project 1 researcher).

Another explanation for why integration is underplayed is that researchers did not collect novel environmental science and engineering data—their analytic focus was elsewhere.

Qualitative data also revealed the presence of other collective capabilities in the project. For instance, capabilities to translate, integrate, and operationalize diverse methods. These are collective because they involve researchers from different disciplines collaborating. These also included capabilities to allow researchers collectively reflect on previous collaborations which were required to effectively translate social science perspectives to practitioner partners in water, energy, and food industries.

Augmenting the project’s co-authorship network with practitioner relations found in the data, we found that the project’s capability set was influenced by three distinct networks (shown in Figure  2). First, the network illustrates what one interviewee—referring to the region in which their colleagues worked—called a ‘disciplinary North-West hub’ of predominantly social theorists, whose expertise is located in various domains of the water-energy-food nexus. Second, a social science—engineering network is evidenced, revealing working relations established over a decade. One co-investigator (the relatively isolated ‘water cluster Co-I’ at the top of the graph) is distant from the network’s core because their publications are mostly collaborations with engineers. Third, integrated across these two networks is a set of academic-practitioner relationships (labelled clusters, Figure  2).

In the context of the project, this co-authorship network indicates a set of emergent collective capabilities. Collectively, relations underpinned trustworthy collaborations and effective communication of complexity surrounding environmental science and policy. Furthermore, already-established relations allowed researchers build their project team, including non-academic partners, at short notice and effectively manage the group through the NN+ funding process:

‘Longstanding links between our two universities is the bottom line. We know each other personally; we tend to work in the same sort of field.

‘[…] It’s a completely pre-existing academic and stakeholder network, which is handy to have when you’ve got to turn around a proposal in a tight timeframe. So, you know, if you’ve got a proto-team already in place you can mobilize it quickly in response to things’ (Project 1 researcher).

These networks indicate not only existing cross-disciplinary capabilities (established connections), but also the potential for new collaborations, based on long-established networks, but made more likely by being funded together. Indeed, this was an aim of the Partnership Grant, to convene and realize transdisciplinary potential. One North-west hub researcher told us that the project was exciting exactly because it was an opportunity to work with co-is that they had long-known, but not published with previously.

Turning to research practice, researchers reported capabilities of pluralism, in the sense of valuing different kinds of knowledge:

‘It’s very easy to have a critical or a meta orientation towards science if you were versed in rudimentary sociology. I don’t think that does you much good. (…) economists or scientists may have models of human action that we find objectionable as social scientists. Rather than going, ‘Okay, well they’re stupid, let’s dismiss it,’ they’re not stupid. Have the courtesy to understand what it is that other people do.’ (Project 1 researcher)

Furthermore, researchers praised the egalitarian ethos in the team (9), demonstrated through collaborative approaches to writing:

‘I don’t think there was a particular hierarchy between us. I would come up with the same copy, and depending on colleagues’ availability, would do the rounds between them. It would come back to me with the edges smoothed and some suggestions for further work to do…. it was a nice mix.’ (Project 1 research associate)

Summarizing, researchers cultivated group capabilities to be humble, plural, egalitarian and reflexive. In particular, our analysis revealed the value of professional networks within academia and with practitioners. While interviewees did not emphasize cross-disciplinary cognitive capabilities, capabilities that allowed them to transgress institutional boundaries were valued. Also valued were capabilities that allowed them to manage the project team, communicate complexity across nexus domains and sustain their own livelihoods through each round of competitive funding.

4.3. Mapping and comparing capabilities across the NN+ partnership programme

4.3.1 Cognitive capabilities

We now report on the capabilities in projects across the portfolio using the heuristic (Section 2, Table  1). The analysis indicates cognitive capabilities are valued in at least three ways. First, the cognitive capability to do research beyond the boundary of researchers’ home disciplines. This capability is about having available the cognitive ability to use or, critically, to learn new ways of doing and knowing research. However, while individual researchers value the possibilities of publishing in new disciplines and journals, this possibility is only available to them because they are part of a research team. The capability to publish across disciplinary lines is collectively held.

Second, within nexus research settings we found radically different kinds of organizations, procedures, stakeholders, power relations, purposes, and wider political–cultural contexts. These included international assessments, government enquiries, regulatory committees, participatory processes, NGO studies, or social movement activities and accessing data in firms. Researchers valued a cognitive capability to apply known tools and frameworks in situations such as these, in which they were sometimes working for the first time.

Third, we found researchers in Projects 1 and 4 valued a cognitive capability for a sustained appreciation of the particular—assessing social and material practices in specific contexts. This capability urges caution over the implications of generalization. Methodologically this is a capability when investigating interconnections between WEF nexus and represents a novel contribution to traditional quantitative and aggregative nexus methods.

4.3.2 Individual tacit capabilities: engaging with power and structure

Individual capabilities valued by researchers included capabilities of humility, valued in colleagues who ‘[don’t] think somewhere in the back of their minds that they are superior’ (researcher P4), the implication here being that superiority arose when uncertainty was located in the work of others, but not their own; capabilities of pluralism in colleagues who are ‘not particularly holding to any Western way of understanding the data’ (researcher P3); and capabilities to be egalitarian, in colleagues who can ‘(…) learn to listen what the stakeholders need and understand how you can respond to their needs, and not just the project’s’ (researcher P1). In other words, a researcher who strives to identify, engage with or alter existing research priorities power structures.

Leadership was mentioned by several researchers in terms of being able to build and manage a research team and also in terms of instilling an attitude of openness, balance. In the context of these research projects we find it more useful to think about leadership in terms of separate capabilities: managing research, being egalitarian in relation to project participants and being able to sustain values of pluralism in the project.

Researchers value being able to break down the hierarchies traditionally constituted by academic disciplines, social status, seniority and uneven geographies—even if they are not always able to fully realize this. A researcher in Project 4 said:

‘a lot of the funding is directed through the [global north] universities, so although we were able to give our partners money through subcontracts, we were still controlling it. So, in a sense there is some kind of implicit hierarchy that goes with it. (…) Are there ways in which we could support research partners in the global south more directly?’.

What researchers are acknowledging here is a capability to cede power, not merely respecting global south interjections, but actively opening and creating spaces for them. They note that capabilities that might better empower overseas colleagues were absent, or structural impediments were too strong to overcome.

4.3.3 Collective capabilities: valuing networks

Project 1 shows us how capabilities are influenced by the configuration of people, knowledge and resources within research sites. Throughout the portfolio, capabilities like egalitarianism, team management, and trust in collaborations were experienced through practices such as ‘encouraging to ask silly questions’ (researcher P2); ‘not being precious during collaborative writing’ (researcher P1); and ‘managing tensions with respect and positivity’ (practitioner P5). It seems that not only are these capabilities individually held, they are also cultivated through shared concerns, interests, and research cultures.

Significantly, networks provide positive feedback for researchers, strengthening capabilities to conduct TD research:

‘I guess these things have a life of their own. Once you start engaging people and you find people you like and can work with, then you want to do more’ (researcher P2).

Networks can be established in numerous ways, by primary investigators opening access to established academic and policy networks (P1 and P3), through networking grants (P1 and P2), or conferences (P4 and P5). Project networks demonstrate the critical importance of nurturing connections over time. Participants reflected that without pre-established personal networks it is challenging to sustain collaborations once the project finishes. This was the experience of the investigators in P4 and P5, where the connections between the researchers and practitioners were established only shortly before the project start and through professional events rather than personal networks:

‘It's very hard to keep networks running after a project has ended. With all the other drawings on our time, other things tend to take priority, especially then if they're to do with the lifeblood of running an institute i.e., income streams’ (researcher P5).

4.3.4 Contextual influences: a need for transdisciplinary research policy

Participants pointed out the need to enhance capability building at the institutional level through training and learning for TD practices, more flexible procedures for bidding for research funding, provision of career progression mitigating risk for those engaged in TD research, and investment in cross-institutional networks were seen as necessary enablers for TD research. They facilitate the cultivation of cognitive capabilities as well as those that enhance livelihood, pluralism, and capabilities to challenge-power, a capability that requires mitigation of risk. When asked about existing capabilities for TD research at university and research council level, all participants emphasized the lack of appropriate structures and policies.

Researchers perceived TD research as a risk to the academics and stakeholders, who, without the institutional support to build long-term partnerships and secure careers, are left to the project-based and precarious working conditions. Similarly, under the current research evaluation framework, transdisciplinarity was seen as a risky endeavour for the universities, which often retreat to mono-disciplinary research and commercial partnerships as they are easier to prove impactful and lucrative according to a researcher from P4:

‘The [Research Excellence Framework] is still siloed. When you put together projects and publications, PI’s and Co-I's and are becoming increasingly pickier as to what publications they are putting together, what sorts of projects. Publications can tick boxes. Sometimes they can be put in a tick box for engineering, sometimes for geography. I think REF conflates the whole TD agenda because it's still very much siloed.’.

Frustrations about the lack of support for capabilities to sustain a livelihood from TD research was put even more succinctly by this researcher from Project 1:

‘if these grand challenges are so bloody important, then how do we invest in people’s careers that are also transdisciplinary without increasing precarity and casualization?’.

In summary, mapping capabilities through each of the heuristic levels we found a plurality of capabilities that are valued and experienced (summarized in Table  3) and some that appear in Table  1 that are absent entirely. In the following section we discuss some points particularly salient for the provision of policy in these areas.

5. Discussion: implications for evaluators and funders

From the capabilities reported in Table  3, we draw several implications for funders and evaluators of TD research. At a programme level, the Nexus Network+ appears to have achieved its goal. The network convened a host of cognitive capabilities, anchored in social science disciplines, in a problem domain that is dominated by natural and systems sciences. At first glance, this represents a significant achievement for the NN+, and their funders, the ESRC, whose goal was to prime social science capacity and capabilities in WEF Nexus domains. By convening a portfolio of projects that were constitutive of cognitive capabilities, the NN+ extended ‘nexus thinking’ putting social science firmly on the knowledge map (see Figure  1).

Using a capability approach however, our evaluation has revealed a more ambiguous picture. The value of the framework is that it emphasizes the skills and characteristics of individual researchers and collectives that are not included in typical programme evaluations. For example, individual and collective capabilities that underpin processes of learning are valued by researchers (as expected in Table  1) but often reported absent. As noted in Project 4 (Table  2), the funding did not allow for formal training. Learning itself can be considered a capability (as evidenced in the heuristic, Table  1). However, we found that in projects learning was more usefully understood as a process of capability building that is directed by policy and programme rules and is enhanced through available resources such as knowledge, finance, and time. These resources were not sufficiently provisioned under the NN+ Partnership Programme.

This has implications for this kind of pump-priming model as a means of organizing and controlling research and ultimately readying capacity—at least as it is instantiated here which we argue resembles a form of research projectification (Fowler, Lindahl and Sköld 2015; Torka 2018). In this case, we find that model of research control that emphasizes discreet projects over short durations is not sufficient to meet broader ambitions for TD research as valued by researchers themselves.

Furthermore, calls for societal legitimacy in knowledge production and more latterly societal impact have been a feature of TD research programmes. These are motivating capability drivers for researchers in our study, expressed through the collective capabilities they value such as being able to critically interact with power and being able to contribute to societal capacity for democratic life. For instance, through the maintenance of accountability relationships between science and society (Strathern 2004). By way of example, the principal investigator in Project 4 was attentive to power imbalances between the UK team and the overseas partners, imbalances they could not successfully address. The implications are twofold, first, given the noted absence, it appears collective capabilities requires greater structural support at the institutional level.

Second, with regard to evaluation literature, these findings fit with ideas about the transformative potential of research such as agonistic–antagonistic logics (Barry, Born and Weszkalnys 2008), transformative values (Huutoniemi and Rafols 2017) and transformative understandings (Marres and de Rijcke 2020). This suggests an ambition for TD research that ‘springs from a self-conscious dialogue with, criticism of or opposition to the intellectual, ethical or political limits of established disciplines or the status of academic research in general’ (Barry, Born and Weszkalnys 2008, p. 29)⁴.

It is a limitation of our framework that we are not permitted to evaluate the actual transformative impact of the research. Rather, we observe a set of collective capabilities that might contribute to such (political) goals of TD researchers. Of course, this in itself poses a challenge to evaluators given such normative and often contested ambitions. However, the usefulness of our capability framework is in fact to reveal the values that underpin such positions, making visible the normativity of these value-decisions in pursuit of transformation. Indeed, mono- or multi-disciplinary approaches are no less normative simply because they align with more deeply situated epistemic incumbencies (Stirling 2019).

Epistemic, institutional, or broader societal transformation is not always achieved in TD research. Nevertheless, we propose that there is use in drawing attention to capabilities valued by researchers in pursuit of transformative goals and following agonistic logics. We cannot be confident these capabilities on their own will lead to transformative outcomes (for instance, being able to challenge power in knowledge; being able to contribute to societal capacity for democratic life), and so we characterize them as transgressive capabilities.

We understand transgressive capabilities as the collective capability of experts to come together to transgress the boundaries of disciplinary knowledge and challenge institutional or structural constraints. To be clear, these are goals in themselves, but the collective capabilities that are valued for their contribution achieving transformative goals. While we find compelling evidence that transgressive capabilities are valued in NN+ projects, we find less evidence for their actual presence, as reported in Table  3. This suggests the need for a long-term TD research interventions at the institutional level.

Returning to the case of the PI in project 4, they found it difficult to support capability development of overseas partners from the UK side. By comparison, researchers in Project 3 had spent over a decade building up these kinds of capabilities through the establishment of new networks of local people. Although highly contextual, this comparison illustrates an issue of capability building across the portfolio. Where transdisciplinary capabilities have been well-cultivated, in P1 and P3 and to a lesser extent P2, these capabilities have been supported by long-established networks of actors, knowledge, relations and other capabilities. Capability development has taken place through other means, that is, outside of the project boundaries established through the NN+ Partnership Programme. In these cases, we find evidence of capability building over a multi-project timeframe initiated prior to the NN+ Programme. In the case of Project 3 for example, this backstage capability space, was established by researchers over the course of a decade by building up networks and maintaining relations and the provision of new resources such as funding and researchers.

The current funding system in the UK treats research capacity as held in disembodied projects that may be turned on and off as demand dictates (Vaesen and Katzav 2017)—bringing us back to the issue of projectification. This is simply not the case. Our framework facilitates a more nuanced understanding of what is going on and the extent to which capabilities are being convened, cultivated, or maintained under a buzzword like nexus, or indeed sustainability.

This has implications and applications in particular for how research is organized to address emerging hot topics. It also draws attention to the importance of capabilities for network building, noted in Table  1 (Spaapen and van Drooge 2011; Hansson and Polk 2018). But this idea goes beyond highlighting the value of network building. It draws attention to how network relations make available a store of social and material resources—conversion factors in the language of the Capability Approach—that contribute to capabilities.

Finally, a strength of the capability framework is in how it emphasizes contingency and situatedness of capabilities. This brings with it at least two limitations. First is the generalizability of capabilities from one capability space to another. Following Robeyns (2005), we are resistant to the idea of a master list of capabilities for TD research. However, we suggest the evaluation literature would benefit from further analysis of what factors cultivate or close down commonly valued capabilities. For that, more comparative research is required. And in the case of collective capabilities, forms of observational analysis such ethnography or situated analysis might be usefully employed in this task.

Further research might also address a second limitation, which is a lack of epistemic and explanatory power over why certain capabilities arise in some situations and not others. Empirically derived, these capabilities may in some cases be conceptually ambiguous. Work is needed to define and explain such capabilities with social theory and methods appropriate to the evaluation setting. In line with this, analyses such as bibliometric studies that rely on indicators of research outputs would benefit from being conducted at greater time intervals after the research to better reflect publishing norms in academic practice. Or, put another way, the capability mapping framework proposed in this article might usefully complement established evaluation procedures (see Section 1 and 2) that already focus on issues of impact, quality or transformation.

6. Conclusions—towards a framework for evaluating capabilities

This article has proposed a novel framework for evaluating how researchers value capabilities for TD research. It reveals capabilities of researchers typically not included in programme evaluation. Our main empirical finding is that under a ‘nexus research’ framing, the NN+ Partnership Programme convened a diversity of cognitive capabilities. However, these capabilities on their own were insufficient to support TD research. We noted two other features of capability development. The value placed on transgressive capabilities by researchers and the maintenance of a backstage capability space required for TD research.

One important observation from this research is to show that developing expertise for real word problem solving (be they framed as grand challenges, societal missions, or agendas more modestly named) is not about assembling the best experts from a rage of narrow academic domains. Rather, it is about fostering a set of capabilities that will contribute to knowledge production that gains legitimacy and accountability from the plurality of contributing actors, the wider societal contexts in which it is produced, and interconnected relations between these actor and contexts. We have contributed a set of methods that can evaluate these practices and procedures. However, our focus on researcher capabilities neglects a systematic study on the capabilities of other participants of TD research project. Beyond the scope of this article, such studies are urgently required.

Nevertheless, a capability approach demonstrates both the importance and absence of systematic capability evaluation in research evaluation frameworks to date. By expanding the scope of our inquiry beyond outputs, quality, and processes, we show how an evaluation of capabilities can contribute to evaluations of epistemic and social contexts of knowledge production. In this, we draw on examples from other recent evaluation contributions (Belcher et al. 2016; Bone et al. 2020). A point of similarity in these approaches is opening the black box of research projects to explore knowledge production downstream of traditional indicators. Future work might explore complementarities between these approaches.

We expect this framework to be useful for funding agencies and public research organizations, especially in midterm or ongoing evaluations. Our framework looks at capabilities as expressed in practice, and thus concerns itself with how projects enact transdisciplinary capabilities. As such, a capabilities approach to evaluation is best used formatively in an effort to identify areas for support to achieve project success. Also, we believe these findings to have particular salience in pump-priming instruments such as sandpits and UKRI Network+ model where the urgency of developing capabilities to research societal issues are stressed, yet uncertainties regarding the most appropriate action remain. Examples of issues in US, European, and UK research policy systems include climate change, artificial intelligence, and economic productivity.

The evaluation approach might also be extended to other forms of participatory knowledge production such as co-creation and co-design. The cultivation and maintenance of transgressive capabilities in particular is a prudent way in which funders might plan for future urgencies, the details of which are often difficult to anticipate. Critically, such capabilities will augment research system capacity only if appropriately resourced and configured. We believe this evaluation framework can contribute in formatively identifying, cultivating, and maintaining such capabilities.

Acknowledgements

The authors thank the participants at the workshops mentioned in the article. We sincerely thank researchers from the Nexus Network Partnership Grant programme who agreed to be interviewed. We are grateful to Becky Ayre, Marion Clarke, Gordon Mackerron and James Wilsdon, and to Frédérique Bone and Andy Stirling. Thanks also to the journal's editors and two anonymous reviewers for their substantive engagement with this work.

Supplementary data

Supplementary data are available at Research Evaluation Journal online.

Funding

This work was supported by the Economic and Social Research Council, ESRC [ES/L01632X/1].

Conflict of interest statement. None declared.

Endnotes

References