‘I want to be able to do what I know the tools will allow us to do’: Practicing evaluative bibliometrics through digital infrastructure

Abstract

The proliferation of quantitative research assessment has been accompanied by an increasing growth and diversification of digital infrastructure for evaluative bibliometrics. Since the beginning of the 2000s, insights into academic performance provided by a variety of new databases and devices significantly exceed the capacities of the former Science Citation Index and embedded metrics. Going beyond the research on the construction, uses, and consequences of bibliometric indicators, we therefore posit that a perspective on bibliometric infrastructure is crucial for understanding how evaluative bibliometrics is put into practice. Drawing on interviews with academic librarians on the increasing provision and implementation of bibliometric infrastructure in the years 2013 and 2014, we analyse how the entanglement of technology and its users shapes how evaluative bibliometrics is understood and practiced.

Introduction

The landscape of bibliometric infrastructure, designed to measure, evaluate, and judge about academic performance, has constantly increased within the last two decades. Besides the well-established Web of Science (WoS), resold from Thomson Reuters to Clarivate (formerly Clarivate Analytics) in 2016, several new citation databases have emerged on the scene. First competitors to the market monopoly exercised by WoS such as Scopus from Elsevier or Google Scholar (GS) powered by Google emerged in 2004, followed by Microsoft Academic Search (MAS) by Microsoft in 2006. The latest addition is the Dimensions database owned by Digital Science. The growing number of data sources (Waltman and Larivière 2020) is furthermore accompanied by a growing diversification of software tools to collect, process, and analyse publication and citation data. These tools include commercial bibliometric analytics suites, visualization programs, and current research information systems (CRIS). They are technical devices that do not only allow for informing about and reporting on research activities but also allow for constructing metrics and indicators of academic performance. The former area of ‘research information’ has thus been turned into a new ‘market place’ (de Rijcke and Rushforth 2015) of providers and tools for ‘research analytics’ (Glänzel, Chi and Debackere 2020).

The technological development of bibliometric infrastructure is strongly intertwined with the development of the field of evaluative bibliometrics, both spurred by a growing attribution of importance to quantitative academic performance measurement (Costas 2017). The idea of performative accountability, demanding the demonstrable maximization of outputs relative to inputs, permeates these quantitative research evaluation practices (Oancea 2008; Roberts 2013). Starting in the late 1970s and gaining traction and widespread adoption as of the 1990s, metrics and indicators have become crucial devices to make academic performance visible and tangible. In particular, bibliometric indicators have proliferated as a widely used yardstick of academic performance monitoring and evaluation at the macro, meso, and micro level ranging from metrics-based performance funding mechanisms to individual-level bibliometrics (Whitley and Gläser 2007; Hicks 2012; Hammarfelt et al. 2016; Nicholas et al. 2020). Benchmarking tools such as international university rankings have increased considerably in their global reach and local significance for higher education organizations (Hazelkorn 2011; Espeland and Sauder 2016; Brankovic, Ringel and Werron 2018).

Previous research has put a focus on indicator construction and use emphasizing that indicators are value-laden, context-sensitive devices of knowledge production and evaluation carrying particular assumptions of research practices and conceptions of academic quality within them. As subjects of constant negotiation and debate, their constitution in practice and their consequences have widely been studied (Barré 2010, 2019; Hammarfelt and Rushforth 2017). However, the growth and diversification of bibliometric infrastructure raises an important and so far neglected question: How does digital infrastructure itself influence and shape evaluative bibliometric practices? Instead of putting the construction and use of bibliometric indicators centre-stage, we therefore address the digital infrastructure that enables and shapes the practices of measuring academic performance.

With this perspective on bibliometric infrastructure, we draw on research in infrastructure studies (Star and Ruhleder 1996; Bowker and Star 1999; Kornberger et al. 2019). We argue that research on academic performance measurement profits from such a perspective because infrastructures must not be understood as a neutral technical support for working processes. Instead, they already contain assumptions about their application inscribed in their interfaces and the evaluative options, categories, and classification systems these interfaces provide.

For studying digital infrastructure in the context of evaluative bibliometrics it is, however, not only necessary to sketch out the technological development of bibliometric infrastructure, highlighting the array of functionalities and technical performance. It is furthermore important to shed light on its usage in distinct application settings (Pollock, Williams and Procter 2003; McCoy and Rosenbaum 2019) and, therefore, to conceptualize bibliometric infrastructure as entangled with its users in producing evaluations (Orlikowski and Scott 2008). In this paper, we therefore focus on the entanglement (Orlikowski and Scott 2008) of technology and its users, that is the community of practice (Lave and Wenger 1991; Star and Ruhleder 1996; Wenger 1999) of librarians in their particular settings, and ask how they ascribe meaning to bibliometric infrastructure and how vice versa bibliometric infrastructure influences their evaluative practices.

We study this question by proceeding in four steps. First, we briefly review the literature on the field of evaluative bibliometrics highlighting the predominant focus on indicators as devices of evaluation and the existing knowledge about bibliometric practitioners and user groups but also the lack of research on digital infrastructure, which puts the indicators into use. We then sketch out the development of bibliometric infrastructure since the year 2000 when the monopoly of WoS became increasingly challenged. In a second step, we present our theoretical approach to digital infrastructure drawing on research from infrastructure studies and organization research. Third, based on a qualitative secondary analysis of interviews with British and German librarians—as a branch of the information profession that has not only been integral to the nascent field of bibliometrics but has also developed multi-faceted support activities in evaluative bibliometrics—and posts from the Jisc bibliometrics mailing list both from the years 2013 to 2014, we discuss how academic librarians put bibliometric infrastructure into practice. Finally, we draw conclusions from our analysis regarding the influence of technology and its uses on the social practice of evaluative bibliometrics.

Developments in evaluative bibliometrics

The uses and users of evaluative bibliometrics

The development of the Science Citation Index (SCI) by Eugene Garfield is unanimously regarded as the birth hour not only of the research field of quantitative science of science but also of ‘evaluative bibliometrics’ (Narin 1976; De Bellis 2014) which provides methods, tools, and techniques for the systematic quantitative assessment of academic performance in terms of its impact and output (Furner 2014). Despite having been conceived initially as a tool for information retrieval, the citation database quickly evolved into the core evaluative resource for assessing not only the quality of scientific journals but even more so for the performance evaluation of individual researchers, research groups, and research organizations (Wouters 1999).

In the formative years of scientometrics, the development of novel approaches in citation analysis by pioneering research groups and companies in the USA and Europe in the 1970s and 1980s already spanned research performance measurement at the level of national science systems, research organizations and groups down to the individual researcher, fuelled by increasing demands from US and European science policy (Gingras 2016; van Raan 2019). As of the 1990s, quantitative research assessment has been institutionalized indicating that evaluative bibliometrics has become an accepted, yet not uncontested practice (Bornmann and Leydesdorff 2014). The proliferation of metrics and indicators has, however, always been accompanied by critical discussions already since the inception of the field of evaluative bibliometrics (Weingart 2005; van Raan 2019). While this ‘quantification’ (Porter 1995; Espeland and Stevens 2008; Desrosières 2015) has raised general concerns about the reactivity and performativity of quantitative indicators for qualitative characteristics, a vast bulk of the bibliometrics literature is marked predominantly by methodological and technical discussions of validity and robustness of indicators, the nature of citation distributions as well as scope, coverage, and data quality of citation databases amongst others (Glänzel 2008). Recently, several proposals have been made to transform the use of indicators in research evaluation and policy also from within the bibliometrics community by for example the need to contextualize scientometric indicators or to explore academic value beyond the divide of scientific and societal impact in evaluative inquiry (Lepori, Barré and Filliatreau 2008; Barré 2010, 2019; Waltman and Van Eck 2016; Hammarfelt and Rushforth 2017; Ràfols 2019; de Rijcke et al. 2019).

While these approaches put the critical reflection of bibliometric indicators and their application centre-stage, some research has also been done on bibliometric practitioners and user groups. The distinction between expert or professional and amateur users thereby marks early (Gläser and Laudel 2007) as well as more recent approaches (Leydesdorff, Wouters, and Bornmann 2016; Petersohn 2019), with the latter introducing typologies distinguishing data producers, bibliometricians and research managers (Leydesdorff, Wouters, and Bornmann 2016) or (aspiring) professional expert organizations and groups such as academic librarians (Petersohn 2016). In addition, some in-depth studies have been conducted on the use of metrics in research assessment by academic librarians (Petersohn 2016), review panels (Hammarfelt and Rushforth 2017), expert organizations such as the CWTS (Petersohn and Heinze 2018), and research managers (Söderlind and Geschwind 2019).

However, little attention has been paid so far to the digital infrastructure in evaluative settings besides technical and descriptive approaches (Moed 1996; Costas 2017; Cabezas-Clavijo and Torres-Salinas 2021). Yet, complementing the study of bibliometric indicators and actors and relating it to the digital infrastructure, which is underpinning it, is vital. This becomes particularly apparent when taking into account the enormous growth and diversification of the landscape of digital infrastructure since the 2000s (Åström 2016; Petersohn and Heinze 2018; Krüger 2020).

The landscape of bibliometric infrastructure since the 2000s

Within the last two decades, new databases, software tools, and sources and types of data have emerged: More than 40 years after its inception, the monopoly of WoS, resold from Thomson Reuters to Clarivate in 2016, was challenged by two very different competitors in 2004. Scopus, launched by Elsevier, is a traditional subscription-based bibliographic database using a curated list of source journals, whereas GS, powered by Google, is a freely accessible web search engine that contains both citations and bibliographic metadata with no restrictions regarding included publication types (Delgado López-Cózar, Orduña-Malea and Martín-Martín 2019; Bauer 2021). MAS, produced by Microsoft, was a short-lived, also freely accessible web engine as a direct competitor to GS from 2006 until 2012, which has been succeeded by Microsoft Academic in 2016 (Harzing 2016).¹ The latest data source for evaluative bibliometrics is more than a citation database: Dimensions, developed by Digital Science in 2018, covers not only scientific publications but also grants, clinical trials, and patents among others (Thelwall 2018).

In addition to these databases, information companies, web service providers, and publishers have also started to provide software tools for data analysis, some of them as derivative products from their databases, such as the analytic software suites Publish or Perish (2006) by Anne-Wil Harzing, SciVal (2009) by Elsevier, InCites (2009) by Clarivate, or most recently Dimensions Analytics (2019) by Digital Science (Cabezas-Clavijo and Torres-Salinas 2021). With the exception of Publish or Perish, they are subscription-based analytic tools using the underlying citation data to benchmark and evaluate institutional productivity and output, to generate metrics, and to display them in advanced visualizations. Both, the databases and derived software products incorporate central metrics and indicators such as the Journal Citation Reports (JCR), Journal Impact Factor, h-index, and many more. Such software tools are, furthermore, not only developed by commercial providers. Especially in the realm of quantitative science studies, software solutions and packages for mapping and analysing bibliometric networks flourish. Among the most well-known tools are VosViewer and CitNetExplorer, developed by the Centre for Science and Technology Studies at Leiden University in the Netherlands, BibExcel by the University of Umea in Sweden, and the Sci² Tool by Indiana University, USA (Moral-Muñoz et al. 2020). The third group of products is represented by CRIS. They are integrated database and information systems for the main purpose of research reporting with optional evaluative functionalities (Sivertsen 2019). Two prominent CRIS, Pure and Converis, were acquired by Elsevier 2012 and Thomson Reuters (now Clarivate) in 2013 from the Danish and German companies Atira and Avedas. With Symplectic Elements, Digital Science has also set up its own CRIS.

While the providers of WoS or Scopus have put some effort into the diversification of indexed publications enlarging their scope towards other publication types such as conference proceedings or books and other languages than English, there is a growing amount of additional data types and sources related to the digitization of academic publishing and the rise of social media outside the closed universe of citation databases. In particular, these two developments have fuelled the aggregation and use of new data such as downloads, likes, tweets, views, blog posts, and many more for the construction of usage indicators and altmetrics (Moed 2017). These new data are also exploited through software tools and altmetric data aggregators constructed by start-ups such as Altmetric and PlumAnalytics (both founded in 2011) and bought by Digital Science and Elsevier later on (Haustein 2016).

It is beyond the scope of this paper to provide an encompassing overview of providers and tools; however, this short overview has already demonstrated the increasing growth and diversification of bibliometric infrastructure within the last two decades. Yet, the implications of the development of this multi-faceted landscape for evaluative bibliometric practices have not yet been explored.

Theoretical approach: practicing infrastructure

In the research literature on digital infrastructure as large operational and information systems, Geoffrey Bowker, Susan Leigh Star, and Karen Ruhleder have discussed infrastructure as far beyond a mere support and optimization of workflows. Star and Ruhleder have instead delineated important features of digital infrastructure, in particular, that it is embedded into social arrangements and other technologies, that it embodies certain standards, which makes it easily adaptable to other technological devices, and that it is learned as part of the membership to a specific community of practice² and thereby shapes and is shaped by their conventions (Star and Ruhleder 1996: 113). Bowker and Star (1999, 2000) have furthermore shown that digital infrastructure is shaped through normative assumptions about its functionality and efficiency and simultaneously shapes the ways this functionality can be practiced.

Recent research on ‘thinking infrastructures’ argues in a similar vein. With the attribute ‘thinking’, Martin Kornberger, Geoffrey Bowker, Julia Elyachar, Andrea Mennicken, Peter Miller, and Joanne Randa Nucho highlight that digital infrastructures do not only format our practices but similarly ‘structure attention, shape decision-making and guide cognition’ through ‘concepts, classifications, categorizations, commensurations and evaluations’ (Kornberger et al. 2019: 1). Digital infrastructures thus constitute ‘orders of worth’ inscribing valuations in the practices and outcomes they are used for (Kornberger et al. 2019: 4).

While this research mainly discusses the inscription of normative assumptions into infrastructure and its performative effects on practices, people, and their perception of reality, further research in science and technology studies has suggested that technology itself has to be considered an ‘actant’ (see also Callon 1984; Latour 1988, 2005) because it poses restrictions on deliberate attempts to inscribe specific normative assumptions and functionalities into it. The construction of technology is understood as ‘a dialectic process of accommodation and resistance’ (Orlikowski and Scott 2008: 459) making it necessary to shed light on the sociomaterial entanglement of technology, its providers, and its users in which entities ‘acquire form, attributes, and capabilities through their interpenetration’ (Orlikowski and Scott 2008: 455). Infrastructures are thus relationally constituted in practice.

In the realm of research governance and quality assessment, there are already few empirical studies that analyse how users apply specific technologies and how users’ practices are simultaneously shaped through their entanglement with digital infrastructure. In his study on the application of Pure as a CRIS at Danish universities, Miguel Antonio Lim (2021) has shown how researchers, who are obliged to feed this infrastructure with information about their publication record, are turned into ‘bibliometric selves’ particularly because ‘it is not an automatic data capture’ (Lim 2021: 11) but instead ‘individual academics need to actively engage with it’ (Lim 2021). Likewise, in a study by Leonardo Piromalli (2019), the CRIS used by most Italian universities, called IRIS, outlines courses of actions for its users by means of its technical features and, at the same time, it ‘perform(s) ideas about what researchers and research should look like’ (Piromalli 2019: 314).

In addition to these studies, McCoy and Rosenbaum (2019) concentrate on what Tarlton Gillespie has called ‘domestication’ (Gillespie 2014: 185–6). In their case study on the application of a decision support system data dashboard at a university, which was implemented to improve data-based decision-making, they show how research administrators—with the words of Gillespie – ‘ma[d]e [these technologies] their own, embedding them in their routines, imbuing them with additional meanings that the technology provider could not have anticipated’ (Gillespie 2014: 186). Analysing the ‘shadow practices’ of users in dealing with this dashboard, McCoy and Rosenbaum highlight that digital infrastructure and its meaning are constantly enacted in the practices of their users.

Infrastructure studies therefore provide two essential analytical notions for our empirical case of bibliometric infrastructure: first, digital infrastructure must not be understood as a neutral technical support for working processes and practices but instead needs to be conceptualized as entangled with providers and users within a particular context through which it derives its meaning. Infrastructure is therefore not to be considered in isolation from the communities of practice and the conventions in which they are embedded. Second, the sociomaterial entanglement of communities of practice and infrastructure guides and shapes cognition and attention and formats its practices.

Data and methods

Academic librarianship features as a central profession providing support services in evaluative bibliometrics both to research management and academics (Petersohn 2016), making it an important user group to investigate in the context of our study. As the information profession has been integral to the nascent field of bibliometrics, libraries are often considered to be the ‘natural place’ for licencing citation databases (Gross and Gross 1927). Our study therefore builds on a qualitative secondary analysis (Beck 2019) of 24 interviews that were conducted with librarians at universities and research organizations in 2013 and 2014 as well as of posts to the Jisc bibliometrics mailing list³ from the same time. Fifteen interviews were held in Germany and nine interviews in the UK.

This material was initially gathered to investigate academic librarians’ claims of professional expertise in evaluative bibliometrics by looking into the types of support provided and the knowledge and skills employed to offer these services (Petersohn 2016). The digital infrastructure used to provide bibliometric services was part of the data collection; however, it has not been analysed in-depth in the context of the previous study. Yet, at the time of initial data collection in the years 2013 and 2014, digital bibliometric infrastructure started to gain traction in research organizations and subsequently also became subject to increasing commercial exploitation. Doing a secondary data analysis of this empirical material, therefore, allows for obtaining a primary first-hand perspective of librarians as users of the developing bibliometric infrastructure instead of a retrospective perception of past developments gained by interviewing librarians in the present.

Using the software tool MaxQDA 2020 (VERBI 2021), we extended and augmented the original category system into a new inductive code system, significantly departing from the original context of data analysis, with a focus on the integration of bibliometric infrastructure into the evaluative practices of the academic librarians. We operationalized the understanding and practices of evaluative bibliometrics by librarianship as communities of practice by coding the fields of application at the individual and organizational level in which evaluative bibliometric practices were performed as well as the librarians’ motivation, knowledge, and understanding of research quality and evaluation processes underpinning these practices. The core category ‘impact of digital infrastructure’ opened up our data with regard to the sociomaterial entanglement of technology and its users. To carefully navigate around the risk of technological determinism, we analyzed not only how librarians were affected by the possibilities and limits in the application of technology, but furthermore how they actively dealt with these affordances and constraints imposed by technology.

The original study investigated evaluative bibliometric practices in academic librarianship against the backdrop of the two differing German and British science and evaluation systems and national settings for librarianship as a profession (Petersohn 2016). This national scope of the initial data set is also relevant for our secondary analysis, since the ‘strong’ and ‘weak’ evaluation systems of Great Britain and Germany (Whitley 2007) provide both institutionalized structures and policies that act as mediating factors for the use and relevance of bibliometric infrastructure. We do not, however, perform a comparative analysis but seek to highlight that across these two different contexts bibliometric infrastructure has started spreading and thus influencing the practice of evaluative bibliometrics.

Analysis: the role of bibliometric infrastructure in practicing evaluative bibliometrics

Librarians’ perspectives on the uses and users of bibliometric infrastructure

Today, quantitative performance measurement can be regarded as commonly shared practice across research organizations (de Rijcke et al. 2016). Yet, by the years 2013 and 2014, among librarians at German and British research institutions, evaluative bibliometrics and its underlying digital infrastructure were not experienced as part of their routine body of work. When asked how much of their daily working practices comprised bibliometrics, many interviewees described offering evaluative bibliometrics rather as a ‘reactive’ (D B4, UK B3) service for occasional demands. However, we also found that some librarians were already beginning to offer analyses and reports proactively approaching researchers and research management. One librarian even reported to be

‘interest[ed] to go and influence these people. Talk to them and see what their perspective is (…) because in the end I want the university to win. I can help them to win and I'll be delighted to do it.’ (UK B2)

Yet, while evaluative bibliometrics and respective services were still found to be ‘a very unstructured (…) process’ (UK B2), a majority of librarians felt a growing demand for it ‘as people become more aware of the service and as the use of citations for evaluation becomes more widespread’ (UK B3). One librarian even reported that the incoming demands were already exceeding his/her knowledge base as people were ‘wanting things which I haven't been able to provide’, yet suspecting that ‘that kind of thing might happen more frequently’ (UK B3). The librarians thus perceived that ‘the next growth area [was] going to be research analytics or citation analysis and bibliometrics’ (UK B8). At some research organizations, the interviewees even reported that new positions were offered and new pilot projects and units were set up for providing evaluative bibliometrics. Nonetheless, the overall majority of the librarians regarded this field as ‘still very, very new in terms of what we’re going to do with bibliometrics and how we’re going to apply it at the university’ (UK B8) and as ‘just kind of testing the waters’ (UK B7).

Motivations and usage contexts for evaluative bibliometrics in academic libraries

In Germany and in the UK, librarians recognized that ‘the interest does definitely come from the institution at a strategic sort of level’ (UK B7), indicating that the demand side for evaluative bibliometrics was primarily represented by research management and administration ‘for being able to demonstrate towards external actors who is with us, who has published what, [and] where something has been published’ (D B2). Yet, besides external demands, librarians also noticed that key performance indicators had become ‘much more popular within the last years’ (D B15) for internal strategic use as well. Questions that especially mattered in both countries focused on measuring research performance for knowing ‘how we are just doing generally in terms of keeping our mission’ (UK B3) or doing benchmarking exercises to identify potential partners for research collaborations as well as metrics being used ‘to evaluate the performance of our researchers’ (D B14).

For the British case, librarians mentioned, in particular, the Research Assessment Exercise (RAE), in 2014 renamed in the Research Excellence Framework (REF), as a trigger for the growing awareness of evaluative bibliometrics. Yet, the degree of reliance on metrics for the RAE was already debated vividly in 2003 but rejected after a pilot phase prior to the RAE 2008. Nonetheless, one interviewee described that universities were still using evaluative bibliometrics in that context because

‘universities are always looking to see what their strengths are and what kind of competitive advantage they can have from the publications they have. So we’ll be looking at working or making sure we do well on those.’ (UK B8)

Also, in the modified REF 2014, citation counts were introduced for informed peer review in the deliberations of some sciences review panels, when applicable (Hinze et al. 2019; Szomszor et al. 2020).

In the UK, ‘not only doing well with the REF but also doing well in university rankings’ (UK B11) was another key driver for a growing demand of evaluative bibliometrics. Research organizations were reported to ‘becom[e] much more competitive and want to understand the nature of their competitors and benchmark themselves against them’ (UK B7). According to the librarians, senior management regarded rankings and university league tables both as a strategic form of intelligence about the university’s overall performance and a means to improve research performance by informing ‘activities which hopefully would make the university rank higher (…) and improve our citation ratings in those league tables’ (UK B3).

In Germany, some of the interviewees referred to the ‘Exzellenzinitiative’ (excellence initiative), a federal competition among universities for extra-funding (Möller, Schmidt and Hornbostel 2016) as one reason for a growing demand of research monitoring. With the Exzellenzinitiative, it became necessary that university leaders no longer only thought ‘I want to know what is going on’ but instead ‘I have to know it’ (D B3), with some insights being based on bibliometric indicators. Librarians also reported about a growing interest in performance metrics that were either used for recurrent evaluations of extramural research organizations and their ‘differences compared to universities’ (D B9) or ‘to evaluate professors’ (D B2) in the context of performance-based funding or appointments. One librarian also noted that such performance metrics had become ‘much more popular within the last years and also more differentiated into journal and article-based metrics’ (D B15).

However, the librarians did not only perceive individual researchers to be mere objects of measurement and evaluation by the research management. Instead, they highlighted them as another group of increasingly active ‘consumers’ of bibliometric indicators with an increasing awareness about the measurement of their individual performance in requests such as ‘how do I calculate my Hirsch index’ (D B9) and how to ‘know […] where to publish and how to give examples of their research impact’ (UK B8). They noticed a growing consciousness among researchers about assessing and displaying their individual performance for ‘maximiz[ing] the visibility of their research’ (UK B8). Some librarians also mentioned how they were

‘supporting these young people in scientific publishing so that they keep in mind that later they will be evaluated based on these publications or that they could even use this knowledge smartly for boosting their career.’ (D B14)

They were thus alerting them to the importance of strategically employing bibliometrics. However, they often found that researchers ‘didn’t really know how to use [citations and impact factors] to their advantage’ (UK B4). In this context, more than half of the interviewees already mentioned Altmetrics as a new development complementary to existing indicators and ‘a very interesting field of experimentation’ (D B6). Particularly in the context of the self-representation of individual researchers, usage metrics and academic social networking sites and profiles such as ResearchGate or GS Profiles were partly embraced as a new way for measuring or even augmenting research impact and visibility ‘by providing people with these measures about their own research and the skills of a tool to find those measures out’ (UK B11). Still, others recognized that ‘altmetrics is one of the things that I don’t feel fully up to speed with’ (UK B3).

In sum, the interviews and posts from the years 2013 and 2014 suggest, first, that librarians perceived a growing demand for evaluative bibliometrics at the organizational level as well as at the level of the individual researcher. Second, some librarians furthermore reported that they had already started to advise researchers and their research management on the importance of bibliometrics for their self-representation and how new possibilities in data production and assessment had motivated them to advocate bibliometrics more proactively. Third, others mentioned that to some extent these demands were already exceeding their resources and sometimes even their competencies in this field.

Yet, no matter if librarians felt external pressure to deal with bibliometrics or were themselves motivated to proactively engage with it, in all cases, librarians had started to search, investigate, and implement new databases and devices for supporting their work. Therefore, to analyse how evaluative bibliometrics was understood and practiced at that time, we need to shed light on their entanglement with new data, databases, and devices, which they increasingly put into use.

Putting bibliometric infrastructure into practice

Our findings on the usage contexts and motivations of librarian’s evaluative bibliometric practice show that these developments cannot be separated from the growing landscape of bibliometric infrastructure. The interviewees recognized a diversification of data, databases, and data assessment devices and reported on how they were dealing with these developments. Not all librarians embraced them unequivocally. In a few cases, they still questioned the general need for evaluative bibliometrics or their own role in it as an academic library; in other cases, they criticized the uninformed use of devices and indicators by researchers and research management. Yet, in every case in which a librarian had started to engage with these developments, in whatever direction, we found that the awareness about the new developments in bibliometric infrastructure had already started to shape librarians’ practice of evaluative bibliometrics.

Looking for more data sources

Although all libraries maintained in-depth knowledge about and licences for the large citation databases WoS or Scopus, the starting point for establishing evaluative bibliometrics often rested on institutional publication repositories and databases. They provided not only data about the publications from their own research organization. Furthermore, as one librarian put it, ‘usually the person who is managing the repository is doing a little bit of bibliometric work’ (UK B4), too. In general, such publication data were reported to become constantly more important, analysed for various purposes from the calculation of the h-index to comparative citation analyses, and linked to other types of research information such as third-party funding. Consequently, most librarians recognized the need for better data sources to gather data about academic performance at their research organizations.

Already in 2013 and 2014, we observe that librarians strove to extend, renew and interconnect the data sources, especially in those cases with recurring requests from research management and academics. A librarian reported that the historical tradition within this community of practice to record publications was succeeded by an integrated CRIS (D B9), whereas another one described the setting up of an MS Access database combined with Excel spreadsheets as a common scenario (D B12). These efforts to build up a more complex data infrastructure reflect a growing demand for automated data gathering and processing among librarians.

Automating data assessment

Yet, the librarians did not only report a need to establish and broaden databases but also to perform analytic tasks and visualizations. A few librarians had therefore already programmed their own data extraction and processing routines and scripts for extracting information from data provided by commercial databases such as WoS. Others were using free devices such as Anne-Wil Harzing’s Publish or Perish. However, the acquisition and (planned) use of commercial analytic software suites such as InCites by Thomson Reuters or SciVal by Elsevier featured prominently across the interviews and posts, with an emphasis on British libraries.⁴

Several interviewees reported that they had recently purchased at least one of these devices or were in the process of deciding upon which product from which provider to subscribe to. As one librarian put it:

‘I recently saw a demo of SciVal, Elsevier’s answer to InCites. It’s an expensive product, but it looks quite promising in its ability to analyze institutional research strengths, discover funding opportunities, and measure output and impact against multiple benchmarks.’ (Jisc 32)

However, some librarians mentioned that testing a new device often followed the trails of already having purchased others, as this posting to the Jisc mailing list illustrates:

‘We have been looking at InCites too, and the main driver for this is we have just purchased [the] Current Research Information System Pure.’ (Jisc 18)

Others reported that these devices had started to become engrained into evaluative bibliometric practices to such an extent that they had even started to shape the description of new job positions at libraries for doing academic performance measurement simultaneously to a subscription to SciVal (UK B8).

The main tenet in the use of analytics software suites and tools was the easy and intuitive handling, which often amounted to a ‘plug and play’-like practice and attitudes that one librarian expressed in the following way: ‘I go to InCites and it tells me’ (UK B1). Another librarian also disclosed that ‘I tend to use those sorts of things that are calculated for me. I don’t do lots of calculations other than that’ (UK B1). With regard to common problems such as scarce resources and understaffing, workflows such as ‘Just search for an institution, select all articles and then press the View citations overview button’ (Jisc 45) made offering bibliometric research support services achievable, given the availability of the respective devices and licences.

The devices also enabled the partial automatization and implementation of complex processes like field normalization and therefore made it possible to broaden the service range. Some librarians expressed their wish to ‘let people know that we now have a product that can tell us lots of different bits of useful information’ (UK B8). Another librarian noted that

‘in order to be able to achieve anything normalized I need to use (…) SciVal (…) which we started subscribing to in 2013. (…) And then (…) there is (…) a second generation version of SciVal which Elsevier has just produced in which (…) you can actually see a percentile figure for a particular article.’ (UK B3)

Whereas the easy handling can therefore partly be considered as a facilitation and also as a way to enlarge analytical capacities for librarians, some also underlined that they were not the only beneficiaries from this newly established bibliometric infrastructure. It had also become much easier for researchers and research managers themselves to do bibliometric analyses because ‘they don’t want to go through all this slug of gathering all the data’ (UK B2). They rather wanted to understand and quickly grasp the results that were now provided through graphs and visualizations instead of mere tables and numbers as, for example ‘with the new version of InCites there will be things on maps (…) and more graphical interfaces’ (UK B1). While some librarians positively acknowledged the easy handling of the new devices for academics and research managers, others expressed their concern about the dangers of an un-reflected ‘quick and dirty’ use of these tools, which was, according to a librarian, practiced by some senior researchers in a ‘quick and clandestine manner at the computer’ during meetings (D B1).

Implications for librarian’s evaluative bibliometric knowledge and practice

Many librarians carefully gauged especially the use of analytical devices against possible caveats, pitfalls, and disadvantages, displaying a critical-reflexive stance towards it. Despite the fact that ‘the tools that you could use to calculate things (…) have so many features and facets (…) [that] [y]ou could do so many different things’ they posed the question: ‘But just because you could do it does that mean you should do it and why would you do [it]?’ (UK B 11) Some librarians even feared that the commercial providers of databases and tools were ‘de-skilling’ (UK B2) bibliometrics by offering seemingly fully fledged bibliometric analyses at the cost of simply a few clicks to anyone (D B5) and ‘interpret[ing] them as meaning one thing but without understanding how the data is being constructed behind the scenes’ (UK B7). Thus, although many analytical tools were perceived as offering ready-made solutions, some librarians maintained that using them also meant to learn how they operated to be able to understand the information they provided.

A significant impediment was experienced by the limited access to bibliometric databases and tools due to high licencing fees and restricted terms of use of many commercial providers. Some reflected critically that

‘[t]he more you use these things the more you realize not they are limited but they are controlling the amount of information that they want to give to you. So as an ordinary user you wouldn't notice it but as you begin to ask yourself more searching questions you realize, whilst the data is there in both Scopus and WoS, actually extracting it is not quite so easy.’ (UK B2)

Such experiences led librarians to manage their expectations

‘in the sense that I want to be able to do what I know that the tools will allow us to do and if I happen to know that there is something we can’t do because the tools don’t allow that capability then I tend not to look for being able to do that.’ (UK B3)

This suggests that many academic librarians were acutely aware that technical functionalities of bibliometric infrastructure thus literally provided visibility to certain notions of academic performance while making others become invisible.⁵ Analytical software tools limit evaluative bibliometric practices to prescribed functionalities, thus, only shedding light on a distinct scope and understanding of academic performance, thereby leaving other areas not covered by data, benchmarks, and visualizations in the shadow.

One librarian also reported that limited access made free databases such as GS and Altmetric more attractive, although the interviewee admitted at the same time that these databases did not contain data valid enough for proper evaluation (D B14). Such databases, also including MAS, were recognized as viable, yet not widely applicable alternatives to the major proprietary citation databases. In the case of altmetrics, their providers had just begun to become investigated by the librarians. Yet, analytics tools such as SciVal and InCites were not considered to be at experimental stages anymore but rather as devices whose functionalities and benefits had been scrutinized and trialled by many libraries for potential licencing.

In sum, the interviews and posts not only suggest that bibliometric infrastructure has become firmly engrained and intertwined within the practices of librarians but also how they facilitated and constrained academic performance measurement in libraries and how librarians made sense of them. This infrastructure was described, first, as opening up new possibilities for doing evaluations and facilitating the production and handling of information, for instance, through visualizations. Librarians who reported to struggle with new demands for evaluative bibliometrics embraced the easy ‘plug and play’-like applicability thereby being led by the functionalities of the devices. Second, librarians who already had a higher affinity towards bibliometrics highlighted the growing possibilities for data analysis combined with an increasing interest in expanding their bibliometric services. Still, many interviews and posts point, third, to a broad awareness of issues of data quality and the need to critically reflect upon validity and reliability. However, we also see, fourth, that decisions about which database or device to use often did not depend on functional and methodological requirements per se but on path dependencies and lock-in effects in terms of existing subscription bundles acquired by Elsevier and Thomson Reuters or even simply on their purportedly easy application and handling.

Conclusion

Since the beginning of the 2000s, insights into academic performance have become offered through a variety of new databases and devices, increasing significantly the scope and abilities of what had been offered before by the SCI and the JCR. In this article, we have therefore argued that to analyse how evaluative bibliometrics is understood and practiced it is indispensable to shed light on the entanglement of bibliometric databases, software tools, and analytical devices with their users.

Based on a secondary analysis of interviews and posts from the years 2013 and 2014, we have concentrated our analysis on the early years when these devices and databases became increasingly implemented, but their application was still at a rather experimental stage. Our analysis has shown how during these years librarians were starting to become familiar with these new technologies and to make sense of them. In the interviews and posts, they reflected about the growing demands for evaluative bibliometrics. Moreover, in some interviews, we found a genuine interest in actively promoting bibliometrics to researchers and research management. Overall, librarians had started to search, investigate, and implement bibliometric infrastructure for being able to comply with these demands as well as to extend their services, thereby further contributing to the promotion of evaluative bibliometrics to researchers and research management.

Nevertheless, we posit that these devices and databases cannot be simply conceived of as mere technical tools for supporting librarians’ work. Instead, they firmly inscribed themselves into the evaluative practices in three essential ways. At the level of practice, we find that the offered functionalities of the devices had started to frame how librarians were doing evaluative bibliometrics through technical affordances and constraints as well as through the providers’ commercial strategies, irrespective of their actual attitude towards evaluative bibliometrics. New evaluation possibilities, easy application and handling, and the visualization of results allowed for and facilitated evaluative bibliometric practices for librarians who did not feel fully prepared to comply with serving the growing demands, but also for librarians who were eager to demonstrate the usefulness of bibliometrics to researchers and research management. Others were also experiencing limitations through restricted access and path dependencies created by commercial product strategies. In some cases, these affordances and constraints were critically reflected, suggesting that librarians were also questioning the ‘shiny surfaces’ of the analytical tools contrasting them in light of their professional conventions and knowledge about data quality to the underlying and often problematized databases and indicators. Nonetheless, nearly all interviewed librarians made use of them for being able to practice evaluative bibliometrics to a larger extent than before.

At the level of expertise, the increasing application of bibliometric infrastructure had not only started to format how evaluative bibliometrics was performed but also shaped the knowledge that was needed for practicing it. Some librarians had started to expand their knowledge based on the inscribed functionalities of the infrastructure and the operations that had now become possible. Yet, others feared for a loss of professional expertise through the affordances of ‘plug and play’-like devices that were experienced by some librarians rather as constraints than as enabling evaluative bibliometrics. At the same time, instead of ‘de-skilling’, others claimed the necessity for obtaining new or even more specific skills tailored to operating bibliometric infrastructure and, moreover, to be able to critically reflect and understand the results they provided.

At the level of the organization, the increasing entanglement of librarianship with the bibliometric infrastructure also shifted organizational attention to the library and evaluative bibliometrics. On the one hand, closely connected to the central tenet of their community of practice to provide information literacy, academic librarians employed the digital infrastructure as devices of information to foster bibliometric literacy among researchers and research managers. On the other hand, it fostered positioning the library closer to research management and leadership by providing the grounds for decision-making. Also, with regard to researchers, librarians had started to use these technologies not only for supporting the research process but for enabling them to manage their self-representation, thereby also contributing to the self-representation of the organization. External demands and active engagement in the promotion of evaluative bibliometrics thus contributed to a gradual shift of the librarians’ community of practice towards becoming managers of organizational and individual self-representation.

While, in our analysis, we have drawn on data from the years 2013 and 2014 and thus from the initial phase where the implementation of increasingly growing and diversified bibliometric infrastructure started to gain traction, looking at current developments highlights that the growth and diversification of bibliometric infrastructure is an even more prevailing issue today. As the ‘Responsible Metrics State of the Art Survey 2020’ conducted by the JISC bibliometrics community highlights, a majority of librarians and research managers is currently making use of bibliometric tools and devices in their evaluative practice (Robinson-Garcia 2021). The recently updated competency model for bibliometric work by the British information professional community (Lancho Barrantes, Vanhaverbeke and Dobre 2021) also illustrates that bibliometric infrastructure has become a central pillar of evaluative bibliometrics by different user groups and practitioners from many countries. In this new edition, the model highlights specifically the role of ‘technical skills’ for operating different software packages and tools as one of the four main areas of crucial bibliometric competencies. At three levels, technical skills are set to comprise different competencies ranging from the ‘download, storage, analysis and graphical presentation’ of bibliometric data to ‘know[ing] the basic software packages of different artificial intelligence techniques’.

Although this model does not name particular devices, it nonetheless becomes clear that dealing with digital infrastructure has become institutionalized as an integral part of librarians’ evaluative bibliometrics practice.⁶ In addition, the claim to be able to critically reflect and understand the results analytical devices provided, also perpetuated in the competency model, has not lost any of its recency. It has been renewed by the library community by formulating responsible use guides for tools such as SciVal and InCites (Price and Gray 2020).

Yet, at the same time, we currently witness that the providers of bibliometric infrastructure have been advancing and branching out not only their databases and devices but also their commercial strategies. While such strategies were already visible in the years 2013 and 2014, path-dependencies and lock-in effects have been constantly reinforced. The major providers of bibliometric infrastructure such as Clarivate or Elsevier have built closed ecosystems of databases, software tools and data analytics devices that do not allow for exchange with devices from other companies, but are instead increasingly sold as ‘service bundles’ of advanced data analytics (Aspesi and Brand 2020).

However, contract research institutes, like the Centre for Science and Technology Studies in Leiden, the Netherlands or Science Metrix in Canada as well as technology company Digital Science’s most recent citation database Dimensions, also decisively contribute to the growth and development of the bibliometric infrastructure (Petersohn and Heinze 2018; Herzog, Hook and Konkiel 2020). In addition, many individual researchers and research groups produce software that can be used both for evaluative and scientific purposes, especially in the realm of bibliometric mapping and visualization. Therefore, a venue for future research consists in investigating both different user groups of evaluative bibliometrics such as research managers but also providers and developers and their specific engagement with the digital infrastructure.

Taken together the growing bibliometric infrastructure and how it is put into practice, we suggest that further research on the entanglement of the providers with their technical products and their users could be a promising way to continue research on the effects of digital infrastructure on evaluative bibliometrics.

Acknowledgements

We kindly thank the organizers and all participants of the Special Issue Author Workshop for their helpful comments on our manuscript. The support provided by the editorial team of the Special Issue throughout the whole process up to the review phase during the exceptional times of writing an article in the middle of a pandemic was highly appreciated. We are very grateful for the constructive and thorough reviews we received from our two anonymous reviewers.

Funding

This work was supported by the German Federal Ministry of Education and Research. We gratefully acknowledge funding received from the DZHW for publishing this article under an open access license.

Conflict of interest statement. None declared.

Endnotes

References