Impacts of policy-driven public procurement: a methodological review

Abstract

In this paper, we review recent research on the impact of public procurement with a focus on methods and data. The growing interest in mobilizing procurement for strategic purposes, such as innovation, economic growth, social value, and sustainable development, has brought to light significant knowledge gaps on the impact of public procurement on products, solutions, actors, and markets. Using a comprehensive approach to analyse scholarly understandings of procurement, we find several notions of policy-driven public procurement and identify challenges in distinguishing between strategic and ‘regular’ public procurement. We then provide a critical discussion on data, examining the currently available data sources and highlighting the need for greater data integration and linkage at the firm level to enable the causal identification of innovation and other impacts from participation in procurement. To address these gaps, we propose a set of actions for research and practice.

1. Introduction

Public procurement—the purchase of goods, services, and works by public authorities—is increasingly being discussed and adopted as a demand-side instrument to achieve strategic objectives related to sustainable development and socio-economic well-being. The size of public procurement was estimated as 12 per cent of Gross Domestic Product (GDP) for Organisation for Economic Co-operation and Development (OECD) countries in 2016 and $6.4 trillion in 2021 at the level of cities around the world (Haselmayer 2021), lending itself to positioning public procurement as a powerful tool that governments can use to influence the economy and society. In many countries, this has been translated into targets of a minimum share of public procurement devoted to driving innovation. For instance, Finland set a government target that 5 per cent of all public procurement should be innovative (Kuuttiniemi and Lehtomäki 2017). The 2023 UK Science and Technology Framework suggests setting a minimum proportion of government procurement to directly support innovation in critical technologies (DSIT 2023).

There are several theoretical rationales for linking public procurement and innovation (PPI), such as (1) innovation risks and market failure necessitating government investment, (2) the systemic nature of innovation and technology development, or (3) the role of public procurement in convening and catalysing innovation in public and private organizations by focusing on grand challenges or ‘missions’ (Edler and Georghiou 2007; Marens 2008; Martin 2016; Sánchez-Carreira, Peñate-Valentín, and Varela-Vázquez 2019; Caravella and Crespi 2020). In addition, public procurement is being re-envisioned as a tool to build domestic industrial capabilities. The current geopolitical environment has highlighted the risks of strategic dependencies in critical sectors and technologies such as semiconductors and critical minerals. The Coronavirus pandemic (COVID-19) exposed the fragility of global value chains (Gereffi 2020). In response, policymakers are increasingly pursuing industrial policies (e.g. the US Inflation Reduction Act or EU Green Deal) aimed at enhancing supply chain resilience and reducing reliance on foreign suppliers, especially for essential goods (Mazzucato and Kattel 2020).

In the last 10 years, procurement frameworks and legislation in different countries have adopted environmental and social criteria in response to societal challenges and the climate crisis. For example, green public procurement (GPP) criteria have existed at the European level since 2008, and these were further strengthened in the 2014 EU procurement directives (Directives 2014/24/EU and 2014/25/EU). In the UK, the Public Services (Social Value) Act came into force in 2013 and required public service commissioners to consider social, economic, and environmental benefits from government contracts. The existence of such guidance, frameworks, and legislation suggests that practitioners and policymakers also consider public procurement as a tool or mechanism to promote economic development, innovation, growth and sustainability, and other socially desired outcomes.

However, despite the apparent consensus regarding the proactive role that public procurement can play in developing technological, social, and environmental innovations, the empirical evidence linking PPI is limited. According to the OECD (2017), while a majority of OECD Member States have developed public procurement strategies to foster innovation, only half of these nations are currently assessing the efficacy of their initiatives. Empirical research on innovative and strategic public procurement has yet to be published systematically, making it challenging to consolidate findings and impacts. Furthermore, the evidence base tends to be largely limited to the Global North, such as the UK, USA, and European countries (Kundu, James, and Rigby 2020; Rejeb et al. 2024), partly because evidence availability is facilitated to researchers by relatively well-organized institutions’ practices, data, and records on public procurement.

This knowledge gap creates several problems First, it is difficult to establish the precise route through which procurement affects innovation, making it challenging to design procurement policy. The concerns expressed by Geroski (1990) continue to elude the research community—‘what is controversial is not so much the proposition that procurement policy can affect innovation, as understanding why it does so when it does, and how that effect can be maximized by an appropriately designed policy’ (p. 185). Secondly, the scattered nature of existing empirical evidence and limited evaluation of public procurement as a strategic policy leads to confusion about the benefits and risks for stakeholders and society. Such limitations in understanding are especially unhelpful when the public sector is perceived to be risk-averse (Cinar, Trott, and Simms 2019) and public procurement is portrayed negatively in the public imagination (Flynn and Harris 2022).

In this paper, we seek to understand the limitations of the existing evidence base, consolidate the empirical research, and review the data sources and methodologies associated with the study of public procurement. In doing so, we evaluate approaches used to assess public procurement impacts. We adopt a comprehensive approach to procurement, traversing the notions of procurement purpose and procurement process and the range of qualitative and quantitative research methods. Thus, our review presents the diverse and growing literature and offers a timely intervention for discussions on using public procurement as a strategic policy tool. This review aims to inform a broader debate on evaluation and assessment frameworks for public procurement impacts.

We contribute to the current literature and debates on procurement by providing a detailed survey of empirical research on PPI, which differs from other significant reviews on the topic, which tend to be conceptual (Boon and Edler 2018; Chicot and Matt 2018; Obwegeser and Müller 2018). We analyse quantitative and qualitative studies, data, and methodologies, considering pros and cons to provide a starting point for newcomers or a springboard for more experienced researchers to expand research on how to improve policy-oriented and outcome-focused public procurement. Our review of the evidence reveals a magnitude of definitions that are helpful in bringing stakeholders together but can be detrimental to developing a shared understanding. A typical example is the conflation in research and policy circles between regular and policy-led procurement. Finally, we provide extensive discussion on data and methodological challenges, such as the development of linked datasets and the identification of potential causal linkages between procurement and innovation outcomes.

In the next section, we provide a background on public procurement and its emergence (or re-emergence) as a demand-side innovation policy instrument. Section 3 analyses the existing knowledge on public procurement impacts and maps out the thematic areas of interest and the geography of research before diving more deeply into methods and approaches that prominently feature in empirical research. Section 4 synthesizes existing knowledge and highlights the issues that currently limit our understanding of the impacts of public procurement. The final section outlines a future research agenda and concludes.

2. The need for a methodological review

For over a decade, there has been a growing recognition of the role played by public procurement and government demand in supporting innovation and facilitating technological transitions. There have been developments in innovation policy discourse as well, which has broadened the framing of innovation from a narrow focus on science and research and development (R&D) for growth, to national systems of innovation in the 1980s, and now to the view of innovation for transformative change (Schot and Steinmueller 2018). Consequently, there are different theoretical approaches linking PPI, ranging from market failure and market creation, to infant-industry arguments, often in national systems of innovation framing, and to more evolutionary economics and structuralist approaches, as we discuss below.

However, PPI is very different from the neoliberal understanding of public procurement as a least-cost-based, transactional activity where procurement performance evaluation is based on objective criteria like price or cost efficiency (Flynn and Davis 2014; Uyarra and Kundu 2022). How then do we measure and evaluate the impact of public procurement on society? While there has been some evaluation of public procurement’s ability to promote innovation outcomes (Aschhoff and Sofka 2009; Guerzoni and Raiteri 2015) constitute some examples, the findings have not been systematically consolidated. Innovation impacts of public procurement can also be influenced when it is applied in combination with other science policy instruments, such as technology-push measures or other supply-side innovation policy tools (Caravella and Crespi 2020). However, evaluation and empirical evidence are critical to policy activity, especially because PPI and its practice, among other things, require strong political commitment which may hinge on the quality of evidence (European Commission and PwC 2021).

While public procurement is being discussed as a strategic policy tool to help realize objectives like innovation and sustainability, there has yet to be a study of strategic public procurement from a methodological perspective to understand how data are collected and analysed to arrive at conclusions about the impact of public procurement. A methodological survey allows the research community to examine the available data and tools. We survey the diversity of research methods and approaches that have been used to gather empirical evidence on public procurement and its impacts. A critical exploration of existing empirical research is intended to inform future research on this topic.

Unlike previous reviews of the PPI literature (Obwegeser and Müller 2018; Lenderink, Halman, and Voordijk 2019; Kundu, James, and Rigby 2020), our focus is methodological rather than conceptual. We are not seeking to propose a new typology to make sense of public procurement or innovation policy (Uyarra and Flanagan 2010; Boon and Edler 2018) but, instead, focus on the types of data used to identify PPI activities and their impacts. We hope that this methodology-oriented review on PPI will invite researchers, those who are both familiar and unfamiliar with the topic, to contribute further to knowledge development. Our methodological review and the invitation to a wider community emerge at a particularly pivotal moment in time to build on new developments in data availability, the advances in computational capacity, and the recent global experience of the COVID-19 pandemic, which has highlighted the importance of procurement.

2.1 Theoretical approaches

In the late 20th century, a number of researchers explored the role of government demand in promoting technological development (Pavitt and Walker 1976; Rothwell 1981), stemming from an interest in industrial development and with a particular interest in the effect that public procurement can play in increasing public and private R&D spending (Lichtenberg 1988).¹ Further critical explorations, prompted by Rothwell and Zegveld (1981), helped establish different theoretical justifications for the use of PPI, ranging from market-based neoclassical reasoning like market failure with respect to innovation, to evolutionary economics, structuralist, and infant-industry arguments, which suggest a more active and integral role for public procurement and public policy in general (Mazzucato 2016).

For example, Dalpé, DeBresson, and Xiaoping (1992) demonstrated that the government is often the ‘first user’ of new products and technologies, which has been further developed into the idea that the government can act as a ‘lead user’ to support the diffusion of innovation (Edler and Georghiou 2007; Alic 2008). Innovation, by its nature, is fraught with risk and uncertainty, which can lead to underinvestment or suboptimal allocation of resources. PPI research builds on this rationale to argue that government has a critical role to play in the innovation process by prompting and/or assuring demand and thus correcting the market failures associated with innovation and technological development (Marens 2008; Mazzucato 2016; Chicot and Matt 2018).

At the same time, innovation policy discourse itself has shifted towards a more structuralist or systems-based approach (Fagerberg 2017; Schot and Steinmueller 2018), highlighting that the development and diffusion of innovation rely on the feedback or needs of users. The systems of innovation approach suggests the use of procurement as a demand-side innovation policy instrument (Edquist and Hommen 1999; Edler and Georghiou 2007). Innovation management and policy scholars have made several contributions in recent years to the field by studying procurement and innovation policy in specific spatial and sectoral contexts (Caravella and Crespi 2020; Divella and Sterlacchini 2020), particularly regarding the impact of public procurement on innovation and industrial change (Crespi and Guarascio 2019).

PPI research also draws on evolutionary economics, taking institutions and complex interdependencies into account to study public purchasing or commissioning. Evolutionary perspectives have been used to study the role of procurement in market formation (Bleda and Chicot 2020), innovation processes, and regional innovation systems (Dale-Clough 2015). Gee and Uyarra (2013) highlight that innovation procurement requires a great degree of coordination among users, producers, and suppliers. Moreover, public authorities may be under tight budget constraints, may be risk-averse when confronted with procedural complexities, and may lack incentives or the capacity to undertake innovative public procurement (Georghiou et al. 2014; Uyarra et al. 2014). Kattel and Lember (2010) critically discuss the possibility of applying PPI in different countries where the interpretive flexibility and discretionary element of PPI may poorly interact with corruption or regulatory capture by private interests. This makes it particularly interesting to probe the geography of PPI research.

Over the years, PPI research has drawn upon different theoretical lenses, linking public procurement with industrial development and supply chain resilience in a globalized world (Gereffi 2020; Mazzucato and Kattel 2020; Smith 2023). There has also been an emphasis on promoting social value and social innovation through public procurement and identifying mechanisms and conditions that can help improve the participation of social enterprises in public contracts (Jaehrling et al. 2018; Hughes, Morrison, and Ruwanpura 2019; Barraket 2020). Public procurement has also been positioned as a policy tool to meet ‘grand challenges’ or complex societal problems (Edquist and Zabala-Iturriagagoitia 2012; McCue, Prier, and Swanson 2015; Grandia and Meehan 2017). Such policy-led public procurement at a time of climate emergency has led to a strong research interest in the role of public procurement in protecting the environment, some of which focus on the role of demand in promoting ‘clean’ or environment-friendly innovations (Veugelers 2012; Orsatti et al. 2020). Cheng et al. (2018) provide a systematic literature review of GPP, which shares some commonalities with PPI (the importance accorded to public procurement) but also diverges from PPI (in that GPP is neutral to innovation concepts).

2.2 Procurement processes

It is important to expand upon GPP in our predominantly PPI-centric discussion on policy-driven procurement so far, not only because both GPP and PPI research are at the frontier of public procurement research (Rejeb et al. 2024) or because there is a strong body of literature on green and sustainable public procurement,² but mainly because the evolution of research on GPP closely mirrors the interests and concerns of PPI research, especially on the different procurement processes.

The European Commission defines GPP as ‘a process whereby public authorities seek to procure goods, services and works with a reduced environmental impact throughout their life cycle when compared to goods, services and works with the same primary function that would otherwise be procured’ (European Commission 2016). GPP draws attention to the long-term effects of procurement by focusing on the life cycle of products and services, and like PPI, it signals a significant departure from the view of procurement as a transactional activity driven solely by cost efficiency. Environment-friendly products and services can be encouraged through technical specifications on public tenders, like chemical content, material choice, origin of materials, eco-labels, and emissions to air or water.

Empirical research has revealed the practical challenges of implementing GPP. These range from lack of legislation to concerns about legality in a complex legal and policy environment, lack of incentives for green purchasing, lack of financial support, and lack of information on environmental impacts (Thomson and Jackson 2007; Zhu, Geng, and Sarkis 2013; Ahsan and Rahman 2017). Testa et al. (2016) identify ‘awareness’ about GPP techniques and procedures as the most important facilitator (and conversely, the lack of awareness as the greatest barrier) for designing environment-friendly tenders. Additionally, similar to the evolutionary work in PPI, other studies point towards the complex relationships among stakeholders, the tensions arising from a diversity of interests in prioritizing and influencing decision-making, and how such pressures shape outcomes (Johnson and Robert 2022). Critically, GPP researchers also point out the challenges in identifying or classifying public procurement activity as ‘green’ or sustainable (Nissinen, Parikka-Alhola, and Rita 2009; Grandia and Kruyen 2020) in the absence of a clearly defined GPP procurement process. This is similar to the case of PPI research where there is no clearly defined procurement process either.

There are multiple terms to describe different types of procurement procedures and mechanisms of PPI in academic research and practice. ‘Public procurement of innovation’ (Rolfstam 2012) comprises issues on modernizing and improving public services through innovation, while ‘public procurement for innovation’ focuses on procurement as an innovation policy tool (Obwegeser and Müller 2018). ‘Innovation-friendly procurement’ is defined as practices and competencies that ensure that innovative solutions are not excluded or disadvantaged (Uyarra and Flanagan 2010). Another term, ‘pre-commercial procurement’, refers to the purchase of R&D with the aim to stimulate innovation, although this may or may not lead to procurement ultimately (Rigby 2013). Caravella and Crespi (2020) use the term innovation-inducing public procurement for procurement contracts that require innovation activities.

The proliferation of terms in academic literature makes any review exercise challenging, but our criticism is not directed at these different conceptualizations of PPI. Theories and concepts help us make sense of the world, and the richness and diversity of theoretical constructs may be a testament to the energy and enthusiasm of the research community to engage with this topic. Different theoretical approaches and procurement processes suggest that different methodological approaches can be used to understand the prevalence and impact of policy-driven public procurement. Our review contributes to the research corpus in a novel way by adopting a methodological perspective. A critical examination of data sources and methodologies captures and emphasizes the diversity of research and helps to suggest directions for future research on the topic.

3. Survey of empirical research

In order to conduct a comprehensive survey of empirical research, we searched for relevant academic articles on the Web of Science and Scopus, which are both popular sources of indexed academic work. The Boolean search expression was ‘public procurement’ AND (‘innovation’ or ‘technology’), published between 2007 and 2023 to limit our study to the recent wave of research on this topic. We also filtered the results by imposing the criteria of ‘Journal Article’, since publications in peer-reviewed journals are considered the gold standard for academic discourse. Furthermore, we limited our review to articles published in the English language, attempted to remove any article that mentioned ‘literature review’ or ‘systematic review’ in the title (since we wanted to make sure articles were comparable for analysis), and restricted our search to relevant subject areas. For the Web of Science, the following subject categories were included: Management, Public Administration, Environmental Studies, Business, Economics, Regional Urban Planning, and Political Science. In Scopus, the following subject categories were included: Social Science, Business Management and Accounting, Environmental Science, and Economics, Econometrics and Finance. The number of hits and the filtering process are described in Table 1.

This search yielded 561 unique records. Since our interest was in empirical research, we removed articles that did not contain an empirical element, for example, conceptual papers, bibliometric studies, or theoretical modelling without empirical evidence. We also excluded papers that were of a technical nature (e.g. legal notes and technology evaluation in health and engineering). This was done manually by reading the abstracts (and the full paper in some cases) by all co-authors. The manual selection criteria for inclusion were as follows: (1) the paper focuses on the impact or effects of procurement, as broadly defined in Section 2.2, and (2) the paper contains original empirical research. The final corpus consisted of 227 journal articles, of which six could not be accessed. Hence, our analysis is limited to 221 journal articles. The logic for restricting our review to academic journal articles is the relative uniformity of format in which evaluation and impact are discussed and published, i.e. there is a statement of research methods used, data sources, and discussion of findings.³

3.1 Geography of research

While it is tempting to map the intellectual structure of existing research by analysing authorship, keywords, publication outlets, and themes, we have not done that here in order to preserve space for discussing data sources and methodological approaches.⁴ However, we have made a note of the countries in which research has been conducted to get a sense of the diversity of institutional and political contexts in which public procurement impacts have been studied. In our corpus, the public procurement activities of sixty-six countries have been covered. Sixteen papers look at EU procurement more generally, so the 66th country category in our data is ‘EU’.⁵ We have classified the papers according to country, but the perspectives include both national, regional, and local-level procurement and its impacts. Some research papers draw data from more than one country and conduct cross-country comparisons. Figure 1 displays the number of papers on each country in the corpus on a geographical map. The top five countries, or geographical contexts for PPI research, are the UK (thirty-one), Sweden (twenty-seven), the USA (twenty-three), Italy (twenty-three), and Spain (twenty). There are several more countries on the list compared to a systematic review of PPI conducted previously in 2018 (Kundu, James, and Rigby 2020), mainly from Eastern Europe and the African continent, and a marked increase in the number of studies from China.

3.2 Methods and data sources

Since our review focuses on methods, we classified the journal articles preliminarily into three broad categories—qualitative, quantitative, and mixed—based on the research method that had been used. This information could be gleaned from the abstract itself in most cases as it is standard practice to describe the data and methods used in research. We followed the authors’ own description to classify papers and found the distribution to be 113 qualitative, 103 quantitative, and five mixed-methods papers. Plotting the distribution over time leads us to observe a rising trend in quantitative research papers (Fig. 2).

Quantitative research studies draw information from surveys, tender and contract data (like Tenders Electronic Daily), and other administrative data like public spending or government subsidies. Qualitative research studies draw information from interviews, case studies, analysis of reports and documents, and illustrative examples. From 2017, we can observe a growing trend in the use of quantitative data, particularly firm-level data or performance indicators and the linking of tender and contract data to firm performance indicators or patent generation to study the impact of public procurement. Statistical techniques like regression, propensity score matching, difference-in-difference, and other experiment-based methods become particularly prevalent. We found very few mixed-methods studies in our review. Such studies may draw on a combination of surveys and in-depth interviews [see, for example, Castelnovo and Dal Molin (2020) for a study set in Italy and Mengistu, Beyene, and Wudineh (2023) for a study set in Ethiopia]. We also observe the prevalence of certain research methods in certain geographical contexts, which may be linked to data availability or pervasive methodological schools (Fig. 3). For example, the share of papers using quantitative research methods is higher in North America (mainly driven by the USA) and in Asia (led by China). However, the different theoretical approaches in PPI benefit from the use of both qualitative and quantitative research methods, as we show in our discussion of advantages and limitations in Table 2. Rather than positioning any one type of research method as a ‘gold standard’ or ‘best practice’ in PPI research, we think that there should be a more balanced approach in choosing research methods so that a more comprehensive picture of PPI impacts can be gained in different geographical contexts.

Overall, we find three main sources of data being used for empirical research on public procurement—case studies, surveys, and administrative data like tender, contracts, and public spending statistics. We describe them below.

3.2.1 Empirical insights derived from case study research.

Several academic publications have detailed public procurement of innovation cases across different sectors and countries. These qualitative explorations derive information from document analysis (e.g. policy documents, internal reports, evaluations, and newspapers), interviews, site visits, and observations and take a sectoral perspective like health (Phillips et al. 2007; Kalvet and Lember 2010; Yeow and Edler 2012), construction (Caerteling et al. 2009; Da Silva Cader et al. 2018; Sparrevik et al. 2018), and public transport (Aldenius 2018; Alhola et al. 2019). Although case studies often consider procurement at national or central government levels, a few studies have also looked at procurement by regional and local public authorities (Telles 2012; Gee and Uyarra 2013; Mwesiumo et al. 2019; Winden and Carvalho 2019).

A common characteristic of these works is their evaluative focus on the process and events preceding and during the purchase of solutions aimed at fostering innovation and the conditions that led (or did not lead) to the successful use of public money. Most of the studies do not explicitly differentiate between ‘regular’ and ‘innovative’ procurement. Instead, they offer a rich analysis of the challenges and good practices associated with tapping the transformative potential of public procurement. For instance, they indicate that cost-saving is still the main criterion driving decision makers. Still, long-term benefits, such as increased recycling levels, energy savings, and accumulation of knowledge, can be some of the other outcomes resulting from public purchases (Gee and Uyarra 2013; Mwesiumo et al. 2019; Winden and Carvalho 2019).

Due to the in-depth and highly detailed examination of the reviewed cases, the impact of PPI ranges from economic outcomes (e.g. cost savings and employment creation) to organizational and societal impacts, such as the co-creation of knowledge, the establishment of contact networks in strategic sectors, or the reduction of environmental impact, among others. These impacts are sometimes embodied in the practices, awareness, and thinking of the stakeholders, whose participation in PPI as buyers or suppliers creates a space to reconsider their capacities for innovation. These qualitative investigations capture new forms and opportunities for impact, shedding light on the transformational power of PPI beyond the traditional economic impact. At the same time, the context-specific nature of case study research often makes it difficult to draw general implications or routes to impact.

3.2.2 Firm-level surveys on innovation and public sector involvement.

In several quantitative studies of PPI, the source of data is a survey conducted in a representative sample of firms, although some studies also conduct surveys with other stakeholders in the procurement landscape (Amann and Essig 2015; Mwesiumo et al. 2020; Soares et al. 2021). Rocha (2019) assesses the impact of public procurement on innovation by using data from the Brazilian Innovation Survey (PINTEC). In Europe, the most common innovation survey is the Community Innovation Survey (CIS). The CIS is a voluntary exercise conducted by national governments to gather innovation statistics and coordinated by the European Commission. Additionally, the European Commission’s Eurobarometer service provides innovation surveys through its Innobarometer series and some additional Flash Eurobarometer surveys. CIS surveys can be found from as early as 1992, covering specific aspects of the public sector’s contribution to innovation, and have been used by researchers to study the impacts of public contracts. For example, Aschhoff and Sofka (2009) use the CIS to illustrate whether private sector firms considered public customers an important source of innovation. In the CIS 2012 wave, questions 10.1 and 10.2 asked firms more explicitly about whether they had any public procurement contracts with domestic or foreign public sector organizations and whether innovation was required by the contract, which has been used as a proxy for innovation procurement [see Divella and Sterlacchini (2020) for an example].

The outcome or impact of public procurement is measured as the change in firm turnover, collected under the CIS section on ‘basic economic information about your enterprise’. It is also possible to relate the responses to public sector contracts with other variables like product and process innovations, organizational and marketing innovations, and environmental benefits. While the CIS is a comprehensive survey on innovation determinants, its representativeness at the subnational and subsectoral levels is far from perfect. Another drawback is that the anonymous nature of the data and changes to the questionnaires over time means it is impossible to join multiple surveys and derive insights for the medium or long term. Some authors, like Caravella and Crespi (2020), have been able to integrate different waves of the CIS with other commercial datasets such as the AIDA-Bureau van Dijk balance-sheet data for Italy.

Some researchers employing survey instruments have designed their own questions for firms, like Georghiou et al. (2014) for the UK, Tammi, Saastamoinen, and Reijonen (2020) for Finland, and Changalima et al. (2023) for Tanzania. Such surveys have greater flexibility in understanding the procurement context. Tammi, Saastamoinen, and Reijonen (2020) included geographical disaggregation, allowing differentiation between public sector customers based on their location (own municipal region versus other municipal areas). In the survey conducted by Georghiou et al. (2014), the respondent firms identified several procurement-related practices, such as outcome-based specification, as being more conducive to innovation. Changalima et al. (2023) focus on small and medium enterprises (SMEs) and survey SME managers to understand how procedural capabilities affect participation in public procurement contracts and the effects of public procurement participation on firm sales performance. Bespoke surveys help to interrogate procurement practices further as compared to CIS or large-scale firm surveys where the role of procurement is self-identified, and therefore its estimation may vary. Some of the researcher-designed surveys also ask firms to reflect on the importance of the public sector customer in their innovation activities, introducing Likert scale questions (Reijonen, Saastamoinen, and Tammi 2018).

3.2.3 Administrative data on public spending, tenders, and contracts.

A key source of information on procurement activity is administrative data that are routinely produced like tender announcements and contract award notices. Data from procurement notices can be used for benchmarking the performance of public sector enterprises (Agrahari and Srivastava 2019), determining impacts on competition and cost efficiency (Gadepalli, Gumireddy, and Bansal 2022), and even cross-country comparisons of procurement systems (Bento et al. 2022). Researchers have also used tender data to derive the impact of tendering requirements like sustainability and innovation on firms (Nemec, Kubak, and Dzupka 2021; Rutkowski et al. 2022).

Another form of administrative data that are used to study procurement is public spending data. This helps measure the magnitude of public procurement, treating the volume of spending as a continuous variable rather than a binary variable (whether public procurement took place or not), which is often the case when drawing data from surveys and tenders. However, the major caveat of such data is its quality, mainly when dealing with spending data at the subnational or sectoral level or over time. The data series tend to present gaps or may be collected in a different format, which may be problematic for comparative analyses.

Existing research that captures public procurement through public spending data attempts to measure strategic procurement in different ways—based on either the category of public spending (Orsatti et al. (2020) distinguish ‘green public procurement’ from overall public procurement expenditure data by looking at items being procured) or their technology or industrial sector (Slavtchev and Wiederhold 2016; Crespi and Guarascio 2019). Innovative public procurement is measured by adjusting procurement expenditure figures with innovation propensity at a sector or country level (Detelj, Jagrič, and Markovič-Hribernik 2016; Crespi and Guarascio 2019). There are also cases where no distinction is made (Pang, Dou, and Huan 2020).

The impact of public procurement spending can be linked to patent creation (Crespi and Guarascio 2019; Orsatti et al. 2020) and private levels of R&D spending (Slavtchev and Wiederhold 2016). These impact measures are particularly interesting for two reasons. First, they show that public procurement can have an impact on both innovation ‘outputs’ and innovation ‘inputs’ (patents and R&D spending can generate further innovation). Secondly, the findings imply that strategic public spending creates a virtuous cycle of investment rather than ‘crowding out’ private spending.

3.3 Summary

Empirical research on the impact of public procurement indicates a strong interest in understanding the effects of public spending and whether it delivers the promised outcomes in terms of innovation, sustainability, or social value. Despite the prominence of EU countries as the research context, research on public procurement of innovaton encompasses all continents with a particular focus on how participation in public procurement or targeted procurement strategies can support local firms [see, e.g., Hoekman and Sanfilippo (2018) for sub-Saharan Africa and Windapo, Olugboyega, and Adediran (2019) for South Africa]. A variety of research methods have been used, and the corpus features both the collection of primary data (interviews, surveys, and site observations) and the exploitation of secondary data (tender notices, public spending data, and firm-level data). While we identify these main types of data sources and methods, they are by no means the only ways of studying public procurement. Research on the impact of policies on electric vehicle adoption makes use of vehicle purchase and usage data (Egnér and Trosvik 2018; Liu et al. 2020), several empirical papers use experimental techniques to compare the effect of special procurement policy adoption (Siminica et al. 2020), and Bernal, San-Jose, and Retolaza (2019) adopt the Delphi technique to conduct surveys with experts to understand the impact of social and sustainable procurement.

From the methodological survey, a primary problem in evaluating the impact of public procurement seems to be identifying when PPI, GPP, or similar policy-driven procurement is taking place at all. Researchers tend to look at specific policies or projects that explicitly embrace a policy-driven idea of public procurement, but it is generally difficult to distinguish ‘regular’ public procurement from policy-led procurement. This is partly due to the multiple terms used in the field (Uyarra et al. 2023), and these different formulations make it challenging to agree on what is PPI and what is not. Therefore, we remain unsure about the specific route or process through which public procurement has an impact.

The next question to be posed from the methodological survey is ‘impact on what?’. Figure 4 lists out the impact variables of interest in general terms.⁶ There is a great degree of interest in understanding the impact of public procurement on technology adoption, with a particular focus on sustainability transitions and achieving environment-friendly outcomes, although it can also be argued that many empirical studies also operationalize impact at the level of firms, i.e. firm R&D, firm economic performance, and SMEs (Fig. 4). These studies usually draw their data from firm surveys like the CIS or by matching tender and contract data with firm-level data (Fernández-Sastre and Montalvo-Quizhpi 2019; Ravenda et al. 2022). Naturally, much of PPI research focuses on the impact of public procurement on the development, uptake, and diffusion of innovation, but there are also several studies (generally qualitative) on the role of public procurement in sustainable transitions and climate-friendly socio-technical transformation (Bugge, Coenen, and Branstad 2018; Shin, Yeo, and Lee 2020; Lingegård et al. 2021; Rainville 2021) and on organizational learning and public sector innovation (Selviaridis 2020; Demircioglu and Vivona 2021). It is interesting to note a schism in the methods and data sources to study impact—case study research is more commonly associated with innovation adoption, sustainability, and organizational learning, surveys are mostly used to identify procurement impact on firms, and administrative data are harnessed to note the impact on patenting activity, economic development, and cost efficiency.

Table 2 summarizes the main empirical approaches described earlier. It highlights the operationalization of public procurement, the identification or distinction between ‘regular’ and innovative procurement, the understanding of impact, and the advantages and limitations.

4. Discussion

Demand-side innovation measures, such as public procurement, have attracted the attention of scholars and policymakers in the last few years in several countries, with a particular interest in understanding the impact of procurement on innovation and economic development. Our review of the recent empirical research on innovative and strategic public procurement reveals some common methodological approaches that have been used to study the impact of public procurement, such as interview-based or qualitative case studies, firm-level surveys, and analysis of administrative data. With each approach, one can observe similarities and differences in the understanding of ‘public procurement’ and ‘impact’. Public procurement may be studied in the form of projects (case study), firm participation in public contracts (surveys), contracting activity, or public spending (administrative data). A wide variety of outcome variables can be found in the PPI literature, particularly innovation measured as R&D spending, patents, adoption of new products and processes, and environment-friendly outcomes such as adoption of environment-friendly technologies and green transitions.

Below, we discuss three key issues for research on public procurement impacts. The first is about defining PPI where we suggest that new methods like text analysis and machine learning may lead us to a more inductive identification of innovation procurement from non-innovative or ‘regular’ procurement. Secondly, there are several challenges in trying to establish causal links between public procurement and firm outcomes, which is an important economic impact of interest (Fig. 4). Thirdly, we raise the question about the quality and accessibility of procurement data for research and generating insights.

4.1 ‘Regular’ Procurement and PPI

From a methodological and policy-making perspective, it is particularly interesting to note how researchers differentiate ‘regular’ procurement from strategic or policy-driven public procurement (and whether they can do so at all). A key challenge in understanding the impact of public procurement is that there is no clear consensus in the literature regarding how PPI is defined and measured. While the literature has focused on deliberate and direct attempts to drive innovation through public procurement, in practice, it is very difficult to differentiate between regular procurement and procurement that is more strategic or innovation-friendly. This makes comparing the performance of firms involved in PPI vis-à-vis regular procurement a challenging task. This is compounded by the fact that, unlike other policy instruments such as R&D subsidies or tax breaks, PPI may not be a single instrument but a combination of subtasks, practices, and procedures (Uyarra 2016).

In a limited number of cases, an explicit PPI scheme is in place (such as the Korean ‘excellent product’ programme, see Shin and Lee 2021), the implementation of which can be evaluated in relation to normal procurement or other interventions. Beyond specific schemes, researchers often rely on proxy measures for innovative procurement, for instance, by asking through surveys whether firms carried out innovation activities in response to a public contract. In these surveys, however, there is heterogeneity in how relevant questions are asked in different years. Innovation surveys like the CIS in Europe ask firms whether they were involved in public contracts and whether innovation was required by the contract, which is then used as a proxy for innovation procurement. According to Appelt and Galindo-Rueda (2016), the question provides a ‘simple mechanism for firms to identify and self-report’ on the link between procurement and innovation. These surveys enabled the identification of innovative firms that engaged in PPI and how that related to different types of innovation (product or process and radical or incremental) and R&D spending (Ghisetti 2017; Radicic 2019).

Other studies use variations of these questions. Crespi and Castillo (2022) draw from the ENIIMESIC survey in Peru, which included two questions regarding public procurement, namely whether the firm had been selected in any public procurement contract, and if so, whether it had carried out innovation activities in the context of that public procurement contract (regardless of whether it was required or not and regardless of whether those activities led to innovation or not). The UK UNDERPINN study (Georghiou et al. 2014) had a slightly different question, namely whether firms’ reported innovations were the result of bidding for public sector contracts. This was intended to capture the signalling effect of public procurement, i.e. public procurement incentivizing bidding firms to innovate, whether they win the contract or not.

Recently, there has been an interest in using text analysis techniques to identify PPI or GPP from tender and contract data. Such studies (Grandia and Kruyen 2020; Yu, Morotomi, and Yu 2020; European Commission and PwC 2021) draw upon advances in computational text analysis and artifical intelligence to train algorithms to identify PPI. This is done either by constructing a vocabulary explicitly or by manually classifying a smaller tranche of the data and allowing the machine to ‘learn’ from this set and apply its learning to classify the remaining dataset. This may lead us to a more inductive distinction between ‘regular’ procurement and PPI, where rather than attempting to define these concepts and match contracts to one of the two categories, the dataset is classified based on the implicit understanding and assumptions that took place when classifying a smaller portion of the contracts or tender data. Such an approach could be made transparent and validated for wider usage.

4.2 Attribution issues

Another challenge lies in establishing a causal relationship between public procurement and observed outcomes obscuring the evaluation of public procurement policies. One of the major problems in estimating the causal relationships between the effects of public funding, including public procurement, and firm-level performance outcomes is the possibility of endogeneity bias. The fact that firms successfully acquire public-funding research is not an exogenous and randomized treatment. Firms that participate actively in public procurement projects are likely affected by endogenous factors that affect the decision to apply and successfully participate in these funding programmes.

To overcome endogeneity problems, most econometric analyses on the effects of public funding and public procurement on innovation and firm-level innovation output or performance measure use the Propensity Score Matching (PSM) estimation methodology (Guerzoni and Raiteri 2015; Rocha 2019; Caravella and Crespi 2020; Liu et al. 2020; Shin and Lee 2021). PSM allows for comparing performance measures of participating firms (suppliers) versus a control group of nonparticipating firms (nonsuppliers) before and after they take part in the funded programme (treatment). The technique estimates a firm’s propensity to participate in a public-funding programme based on relevant observable determinants that are considered factors, which influence the likelihood of becoming a supplier. Among the most commonly used internal determinants of firms, we can find the firm age, size, ownership structure, credit availability, innovation, productivity, or export performance pre-treatment. Apart from these, firms’ internal characteristics, location, market, and industry information are also used as additional control factors to construct a counterfactual group. It then performs a comparison of performance between suppliers and this counterfactual group to estimate the impact of public procurement on firm performance.

Another strand of evaluation studies utilizes discrete choice models that explain the probability of becoming suppliers to the public sector and its impact on firm performance (Guerzoni and Raiteri 2015; Fernández-Sastre and Montalvo-Quizhpi 2019; Czarnitzki, Hünermund, and Moshgbar 2020; Stojčić, Srhoj, and Coad 2020), censored models (Aschhoff and Sofka 2009), and quadratic estimations that try to capture the potential nonlinear effects of public procurement on performance outcomes (Tammi, Saastamoinen, and Reijonen 2020). The presence of in-house R&D activities and the educational level of employees (Divella and Sterlacchini 2020), the local skills composition (Orsatti et al. 2020), or the role of inter-organizational supplier networks (Saastamoinen, Reijonen, and Tammi 2018) have been found to affect the probability of winning public contracts.

Limitations to such approaches are linked to the fact that groups (suppliers and nonsuppliers) should be perfectly randomized. This means that the likelihood of becoming a supplier must not be correlated with any of the included firm’s internal characteristics due to potential self-selectiveness and biased selection of firms participating in these policy programmes. Caravella and Crespi (2020) also introduce the ‘hidden treatment effect’ caused by not considering the compound effects occurring when more than one policy tool is affecting the same firm (‘policy mix effect’).

4.3 Accessibility of procurement data

Another challenge relates to data sources and data quality. While there is interest and great potential in harnessing public procurement to achieve policy objectives, there is a need to match this ambition with detailed, granular data that allow researchers and analysts to identify and evaluate the mechanisms through which procurement leads to positive outcomes for businesses and communities. There is variation in access and quality of data collected on electronic government procurement systems in different countries and a severe gap in the integration of different levels of procurement data to make meaningful analysis possible (Cocciolo, Samaddar, and Fazekas 2023).

In Europe, where CIS serves as a popular source of data on private sector innovation, researchers have noted several limitations. CIS data are anonymized, and it is not possible to conduct follow-ups or complement it with other data. As such, the detail provided on the role of public procurement is limited and often concentrated on innovation grants rather than standard procurement per se. Additionally, because the questions keep changing and the surveys cannot be joined, it has been impossible to construct larger panels and study the medium- and long-term effects of public procurement so far (noted by Aschhoff and Sofka 2009; Radicic 2019; Stojčić, Srhoj, and Coad 2020). Furthermore, large-scale surveys like CIS are often not comprehensive and coverage of the subnational and subsectoral levels is far from perfect.

Several studies of the impact of public procurement undertake their own bespoke surveys (Fernández-Sastre and Montalvo-Quizhpi 2019; Dai, Li, and Chen 2020; Tammi, Saastamoinen, and Reijonen 2020). Due to cost, these surveys tend to be smaller in size and last for as long as a research project is designed for. They benefit from being designed for a specific purpose and contain quantitative and often qualitative information that can appropriately address a set of research objectives and questions but need to provide a consistent approach to procurement evaluation in time, space, sector, or combinations of these attributes. We anticipate that a methodological review such as ours can generate discussions and lead to consensus about survey tools for the research community.

Finally, in recent years, the advent of data science techniques has led to the emergence of proprietary data that contain information on public procurement contracts awarded, often matched to spending and firm-level data. The advantage of these datasets is that they exclusively focus on tracking procurement spending from a range of organizations and thus provide the most comprehensive and unique picture of public procurement origin and destination. However, they are not freely accessible, and the drawbacks include a lack of clarity (due to the proprietary nature) of the methodology and coverage of the datasets, as well as the lack of information on subcontracting.

5. Conclusion

Our review captures the diversity of research on public procurement impacts. The empirical evidence is generally positive regarding the impact of public procurement on several economic and social outcomes. Researchers assume that PPI is beneficial in its own right but fail to illustrate the types of direct and indirect impacts that PPI policy can produce. Qualitative studies, however, have made a significant effort in this regard. Their exploratory nature helps reveal different impacts (tangible and intangible) and the contexts, challenges, and practices surrounding the implementation of innovative and strategic public procurement policy.

It is challenging to evaluate the impacts of PPI on innovation and other social outcomes. Partially, this stems from the ambiguity of concepts and multiple definitions. It is difficult to identify what is (or what is not) policy-driven public procurement. Moreover, strategic public procurement has only been recently implemented in many countries, and the quality of available data also makes it difficult to evaluate and monitor the implementation of policy measures. In addition to the difficulty involved in defining the instrument to be evaluated, another challenge when it comes to understanding PPI impacts is the problem of establishing a causal link between the dependent outcome (economic, social, and environmental) and the independent variable (usually some form of participation in public procurement).

Evaluations also tend not to consider whether procurement is a more cost-effective way to achieve innovation objectives than other innovation policies (Warwick and Nolan 2014). Procurement may be part of a ‘policy mix’ , requiring evaluations that consider synergies between instruments over time (Flanagan, Uyarra, and Laranja 2011; OECD 2011). Some researchers have investigated this by comparing public procurement with other innovation policy instruments like R&D grants and subsidies, technical standards and regulations, information, and networking support services, among others. The impact of public procurement continues to be positive, with some researchers claiming that public procurement continues to be effective as a stand-alone policy as well as within a policy mix, unlike government subsidies for R&D (Guerzoni and Raiteri 2012, 2015; Radicic 2019).

Our analysis illustrates the variety of data and methodologies used to capture the impact of public procurement. Methodologies to capture the impact of public procurement on economic outcomes range from descriptive and deep-dive case study analyses to large-scale surveys. Existing empirical research reflects the diversity of theoretical underpinnings of PPI as it is possible to find studies on the market-shaping effects of public procurement, the role of public procurement in mission-oriented grand societal challenges like climate change, and the transformative nature of public procurement leading to organizational learning and changes. Recent attempts to explore the effect of public procurement on economic performance have benefited from the increased digitization of administrative data. However, the quality and comparability of the data still needs improvement. Besides data availability problems, multiple definitions, biased selection processes (picking winners), heterogeneous policy mix effects, and endogeneity problems linked to self-selectiveness make evaluating public procurement impacts challenging.

5.1 The future research agenda

It is encouraging to see PPI research expanding in methods, scope, and geography. For example, the impact of PPI is being considered in a larger number of nations, demonstrating a universal interest in maximizing the value of public spending. The impact of procurement is studied over a variety of factors like economic development (SMEs and firm performance), environment-friendly outcomes and sustainability transitions, and improving public service delivery and organizational learning. Impacts are not limited to the uptake and diffusion of innovation alone. We also note a growing diversity in research methods used to study the impacts of PPI, moving towards the presence of both qualitative and quantitative approaches. Our paper aims to inspire further research in public procurement and its links to innovation. There are a number of directions for future research, both from within the existing scholarly community and for those new to the field.

First, the body of knowledge will benefit from creatively using primary and secondary data through qualitative and quantitative analysis. Quantitative endeavours could improve the linkage of procurement data with longitudinal firm-level information to provide more accurate and causal estimates of the potential impact of different procurement programmes on firm-level innovation. Qualitative studies can add nuance to a better understanding of the mechanisms behind the involvement of firms with public procurement and the impact of procurement on both the buyer and the supplier.

Secondly, there is an opportunity for broadening the geography of PPI research beyond the USA, UK, Western Europe, and China. It is likely that public procurement is even more important for innovation and other outcomes in developing economies with more fragmented markets and less transparent market mechanisms. More research in these countries is likely to shed light on improving the impact of public procurement both from within these countries but also in the form of overseas development assistance.

Finally, discussions on the design and impact of public procurement must increasingly take into account strategic interdependencies and supply chain resilience. In the domain of innovation, this is particularly relevant with respect to the idea of technological sovereignty (Edler et al. 2023), where it will be important to position public procurement carefully such that it aligns with broader policy objectives, which often include a nation’s industrial, environmental, and social value goals. Robust evidence-based research on prevailing practice and coproduction of knowledge on the enablers and barriers to change will help the scholarship make informed recommendations for the future.

Footnotes

Conflict of interest statement.

The authors have no potential conflict of interest to report.

Funding

The authors would like to acknowledge support from (1) The Productivity Institute (ESRC grant number ES/V002740/1), (2) Policy@Manchester, and (3) Consortium for Research in Innovative and Strategic Public Procurement (CRISPP), which is a partnership among the University of Manchester, the University of Birmingham, and Connected Places Catapult. Tasos Kitsos would like to acknowledge funding from the British Academy Innovation Fellowship (IF23RBP\230040).

References