Google Scholar – Platforming the scholarly economy

Introduction

Bibliometrics have been a major part of academic publishing since the mid-20th century when Eugene Garfield developed citation indexing as a way to automate the organisation of scientific information (Garfield, 1955). Citation indexing paved the way for evaluative bibliometrics, whose centrality has long been controversial in the academic world (see e.g. European Commission, 2019; Nygaard & Bellanova, 2017; Crane & Glozer, 2022). Despite long-held concerns, research evaluators and managers, such as funders and university hiring committees, continue to use metrics to quickly approximate research and researcher quality, and the commercial academic publishing industry continues to develop evaluative metrics and analytics as next generation products. Rather than add to the vast literature critiquing academic metrics and the metrification of academic work, this article interrogates the consequences of academic metrics that are provided by platform businesses like Alphabet through services like Google Scholar.

Google Scholar is not simply a search interface. It has leveraged its system of citation counting into a tool that coordinates the scholarly economy. The service’s metrics drive the distribution, consumption and evaluation of both research and researchers. Like other digital platforms that use scoring to rank content and actors across markets, Google Scholar uses citation counting to control the visibility of research and researchers, and intermediate data flows between researchers, research managers and repositories. As its gatekeeping role (see e.g. Lynskey, 2017) in the academic domain expands, it infuses the scholarly context with platform dynamics in ways that trouble a set of contextual norms around academic autonomy.

In this article we describe how Google Scholar participates in refashioning the academic world in the image of the platform. Much has been said about the ways academic publishing replicates the logics of platform capitalism (see e.g. Mirowski, 2018), which includes commercial academic publishers’ turn to analytics as next generation products. However, perhaps because its relation to valorisation and profit is non-obvious, less attention has been paid to the specific dynamics that drive Google Scholar. These include a “free” service that enrolls scholars in systems of surveillance, multi-sided markets and behavioural advertising; Google Scholar’s user experience design that frustrates researchers’ ability to self-define search ‘relevance’; and the specific ways that Google Scholar exploits the academic open access movement as a vehicle for value extraction (vis-à-vis how commercial publishers have taken advantage of open access (see e.g. Mirowski, 2018; Meagher, 2021). We argue that these dynamics generate a particularly non-accountable and non-transparent evaluative environment that, through the sociological theory of Bourdieu (2004), presents a political and ethical challenge to the academy. In particular, as Google Scholar continues to be an intermediate in academic life, interjecting its opaque and trade-secret-protected systems of scholarly evaluation, it has begun to challenge the ability of the academy to understand how it evaluates itself.

After outlining the analytic method, this article proceeds by contextualising Google Scholar citation counts within the broader history of citation indexing and its coordinating function in the scholarly economy. The article then describes Google Scholar’s services, including its search, citation counting and scholar profiling tool. Finally, the article outlines Google Scholar’s platform dynamics, including its provision of free services, its user experience design and its exploitation of open access ideals and rhetorics.

Scholarly platforms and academic norms

The analysis of scholarly platforms and their relationship to metrics, evaluation and rankings has accelerated in recent years (Chen & Chan, 2021; Crane & Glozer; 2022). Our analysis supplements this literature on the performance, affordances and political economy of metrics and scholarly platforms, with specific attention given to the provision of evaluative bibliometrics by Google Scholar.

Our analysis uses the theoretical framework of ‘Handoff’ (Mulligan & Nissenbaum, 2020; Goldenfein et al., 2020) in order to parse the stakes of Google Scholar’s role in academic practice. The Handoff model affords a lens for a political and ethical analysis of sociotechnical systems that exposes the consequences when the performance of a given function shifts from one type of actor to another, where the inclination is to understand the latter actor as performing the same function as the former. Here, the function analysed is the provision of indicators that work as tools for coordinating the scholarly economy as they move between the world of commercial academic publishers and platform services like Google Scholar. The analysis requires a look at the different components of the system (be they humans, organisations, interfaces, users, etc.) in each arrangement, paying attention to how components act-on or engage each other (i.e. by force - including the direct constraints imposed by a technical system, or affordance, such as the ways user reactions are elicited by exploiting human actors’ tendencies to respond to cues in systematic ways). The lens “decomposes” the way these sets of actors and components engage each other, with the goal of showing the ethical and political significance of the shift from one configuration to another (Mulligan & Nissenbaum, 2020).

To that end, the analysis here compares the organisational, technical and ethical arrangements of bibliometric evaluation, especially with respect to their scholarly market coordination function, in the forms of Journal Impact Factor and Google Scholar. We note that this is particularly difficult in the context of a technical system like Google Scholar that, by design, resists efforts at systematic analysis. That absence of transparency, however, has become central to the normative stakes of Google Scholar’s popularity in the academic field.

Evaluative bibliometrics in the scholarly economy

Analysing Google Scholar’s reconfiguration of academic practices requires understanding, to whatever degree possible, the system that Google Scholar interrupts, as well as its political and ethical orientations. Historical work on scholarly publishing typically identifies its normative functions as community formation, research dissemination and the establishment of intellectual priority (Csiszar, 2018; Nissenbaum, 2002).1 Evaluative bibliometrics, however, are the product of their own set of historical forces. Geopolitics and competition in post WWII industrial science and technology meant a dramatic increase in funding for scientific research. As the quantity of research output increased, the need for better and faster indexing emerged, along with a demand for abstracting, reviewing and library services. Not long after the war, the UK Royal Society convened the Empire Scientific Conference, wherein it was reported that abstracting services only covered about half of published research papers (Shaw, 1948). The rest remained more or less undiscoverable. Support for scientific information services had not kept up with support for science research. Eventually the problem of insufficient digesting was understood to be an impediment that was slowing down scientific discovery (Bowles, 2000).

Concurrent developments in computation offered potential solutions to this crisis of information organisation, but required a method to code scientific information in a manner amenable to automated processing. Eugene Garfield’s answer to this problem was citation indexing. Rather than digesting research articles according to their semantic content as librarians had traditionally done, citation indexing enabled the organisation of articles according to the works they referenced. Characterising research according to its networks of references afforded a good proxy for content digesting, enabling researchers to trace the intellectual lineage of concepts while using a statistical language that could be parsed by computers and tabulating machines. Garfield commercialised this process in 1964, launching the Science Citation Index through his company, the Institute for Scientific Information (ISI) (which became part of Thomson Reuters in 1992 and Clarivate Analytics in 2018). The SCI initially included an index of 1.4 million citations from 613 journals. In 1972, Garfield published an article in Science titled ‘Citation Analysis as a Tool in Journal Evaluation: Journals can be ranked by frequency and impact of citations for science policy studies.’ It outlined his method for ranking journals listed in the SCI according to average citations per article over a period of time. The ranking unit developed was Journal Impact Factor (JIF).

Not only did JIF inform science policy by providing a shorthand for scientific authority and influence (Gingras, 2016, p. 7), in offering a proxy for research quality it became a mechanism for coordinating the scholarly economy. This is because JIF became a unit of equivalence capable of representing different forms of value pursued by different classes of actors in the academic field. For publishers, JIF became a proxy for economic capital through its determination of subscription prices. For scholars, reputational prestige was demonstrated through the JIF of a chosen publication forum, which for research managers, became a proxy for researcher quality (Fyfe et al., 2017; Cameron, 2005). As governments, foundations and other grant-giving institutions began to demand higher “accountability” from universities through the 1980s, JIF also became a way for universities to demonstrate value-for-money in research, often described as research “excellence” (Shore & Wright, 2000), as well as more broadly implementing science policy through competitive market dynamics under the imperatives of New Public Management (Strathern, 2000; Gläser & Laudel, 2007). In the context of such demands, universities became more interested in acquiring scholars that could demonstrate a capacity for measurable “impact”, for which JIF became the most convenient shorthand (Alexander, 2000; Salter & Tapper, 1994). JIF thus perpetuated academia’s reification into a status economy (Hamann & Beljean, 2017).

There are numerous components to the JIF evaluative ecosystem. These include the commercial businesses that derive indicators as commercial products from their database of journals and citations. Academic journals compete for entry into the SCI database, and for higher JIF scores by working to become attractive fora for highly cited scholars (and sometimes by manipulating citation counts). In order to pursue high prestige publications, individual researchers are sometimes (of course, not always) incentivised towards high JIF publications in their disciplines. Frequently, universities use journal quartile or quintile rankings (i.e. highest 25% or 20% of JIF ranked journals per discipline) as part of their research performance calculations for scholars. And JIF is commonly included in research quality evaluation for various research managers, stakeholders and funders.

The consequences of this arrangement have been clearly described in innumerable critiques of JIF and its impact on academia. These critiques are sometimes technical — i.e. the failure to adequately reflect citation performance (Russell & Rousseau, 2001); JIF’s failure to normalise citation counts across journal subject categories; its failure to correct for self-citations. They are sometimes epistemological — i.e. the impossibility of “measuring” academic quality; its incentivisation of “review” articles; its negative impact on knowledge production and agenda-setting, which includes channeling research towards universal rather than local domains (Bianco et al., 2016); its problematic cooptation of the notion of “impact” (Hecht et al., 1998). And the critiques are sometimes political — i.e. JIF’s inclusion of only a fraction of the world’s journals; its focus on certain publication formats over others; its privileging of English language content and associated geographies; its failure to take into account disciplinary citation conventions; its deployment as a marketing tool (Hecht et al., 1998); and its “off-label” use for the evaluation of individual researchers, programs and institutions.

The latter critique is related to JIF’s expanded reception in academic practice and its coordinating function. JIF was always intended to be a market coordination tool — but only between university libraries and journal publishers. But the affordances of JIF as an indicator of value enabled its coordinating function to be extended across academic interactions, such as funding decisions, the hiring and promotion of researchers, and even decisions about what materials were worth reading and citing, despite the fact that this type of evaluation was unintended by JIF’s inventors or owners. Notwithstanding repeated warnings that JIF should never be used to evaluate research performance for an article or of an individual (van Noorden, 2010, pp. 864-865; Glänzel & Moed, 2002), it is indisputably used this way (see, among others, Quan et al., 2017; Rushforth & de Rijcke, 2015; McKiernan et al., 2019). For this reason, even Garfield himself came to see JIF as a ‘mixed blessing’ (Garfield, 1999).

JIF’s coordination role was not inevitable, but it is very much a product of how academics and other stakeholders use it as an indicator. Most parties very clearly understand the limitations of this tool (Evidence Ltd, corp creators, 2007). Much like Google Scholar, JIF is not particularly sophisticated — it has been described as ‘a poor man’s citation analysis’ (Pendlebury & Adams, 2012). But its simplicity ironically augments its coordinating utility. Its uptake is a product of both technological affordance, i.e. the efficiency of indicators over subjective measures of value, and JIF’s explicit design as a market management tool, as well as the realities of academic practice, i.e. the reduction in labour necessary to engage in academic transactions. It is easy to use, access and comprehend by the various actors in the scholarly economy, enabling decisions without spending time to subjectively ascertain the value of researchers and their research output.

While JIF remains a key indicator in the contemporary evaluative ecology, we suggest these coordinating functions have now been supplemented by additional indicators — including Google Scholar. The Handoff model does not insist on the total substitution of one set of components or configuration with another. Rather, the goal is to identify “what may have changed in the reconfiguration of function across component actors” (Mulligan & Nissenbaum, 2020, p. 5), sometimes sequential and sometimes simultaneous, and unpack those differences to expose the political and ethical stakes of the shift. As we show below, the provision of indicators by Google Scholar augments the market logic that was facilitated and amplified by JIF (Shore & Wright, 2000; Shore, 2010; Burrows, 2012) with a range of platform dynamics (see e.g. Viljoen et al., 2021) that have their own political and ethical significance. For instance, whereas JIF worked to diffuse scholarly evaluation into a decentralised market for scholarly authority, Google Scholar’s interface simulates a market while introducing novel interfaces that opaquely manage the visibility of market participants. In that way, Google Scholar uses its gatekeeper power to distribute academic authority according to its own unchallengeable algorithmic designs.

It is important to note that JIF has been supplemented by numerous metrics systems, of which Google Scholar citation count is only one. Further, many of these bibliographic tools similarly enact platform-like features in explicitly profitable ways, such as channeling bibliometrics into institutional ranking systems (Chen & Chan, 2021). Those systems have been, and continue to be, subject to rigorous analysis. This will continue, as platformisation of universities, publishing and evaluative ecologies is the inevitable expression of informational capitalism’s dominant logic (Cohen, 2019). What we show below is how some of the features of platformisation, and especially the Google Scholar enactment of platformisation, have political and ethical consequences. And while Google Scholar has been persistently analysed for its comparative performance with other bibliometric services (Halevi et al., 2017), its political and ethical consequences are not frequently an analytical focus (for an exception, see van Dijck 2010).

The normative stakes

The primary ethical concerns exposed by our description of Google Scholar relate to concepts of academic autonomy. At one level, the evaluation of researchers and research centres has become central to the management of relationships between universities and their stakeholders — be they states or private funders. Financial independence often relies on performance outcomes. Indeed, if bibliometrics are central to evaluation, they are central to questions of institutional autonomy. But academic evaluation is also fundamentally connected to academic autonomy in the sense of self-governance (Whitley, 1984). For instance, beyond financial independence, academic evaluation implicates autonomy at the level of field formation — that is, the creation of a bounded academic identity and independence as a field of practice. Historically, academic autonomy, in this sense, refers to the ways in which professional scientists and scientific associations separated themselves from journalists and the broader public through conventions of authorship and evaluation, like anonymous refereeing. Csiszar (2018), for instance, describes how scientific identity and authority, what we might think of as precursors to recognition and capital, were minted originally through periodical authorship with its evaluative conventions. In this sense, autonomy reflected a boundary between the academic field and other social systems.

The social function of the university has transformed over recent decades, however. Universities’ shifting relationships with states and other stakeholders means that notions of academic autonomy have been recalibrated for the “knowledge economy” and informational capitalism (Olssen & Peters, 2005; Shore & Taitz, 2012; Cohen, 2019). Universities are now “economic engines”, expected to forge partnerships with industry and commercialise knowledge (Shore & Taitz, 2012) as a way to demonstrate efficiency and value (for money) to their various private and public stakeholders. This challenges the notion of a separation between society and the academic field. Autonomy has accordingly transformed, becoming a question of managing interactions with the market and other stakeholders, rather than avoiding industrial influences on knowledge production.

In this context, and somewhat ironically, autonomy has itself become a metricized accounting of national education policies in service to regional and global hegemonies of university performance (see e.g. Bennetot Pruvot & Estermann, 2017). Questions of independence from conflicted interests and the market, self-regulation or external influence on agendas, are subsumed into metricised levels of organisational autonomy, financial autonomy and staffing autonomy, through which autonomy is scored and compared across institutions and nations for the sake of university rankings.

But within this weird patchwork of contradictory meanings, autonomy continues to do some meaningful normative work — especially in the context of evaluation. Indeed, it has always been central to academic autonomy that the work of academics be evaluated by other academics. According to Bourdieu (2004), academic knowledge production can only be saved from its inherently contingent and political construction through transparency in the processes of evaluation, and through the pursuit of disinterested and peer-distributed forms of symbolic capital or credit. From here, Bourdieu makes the claim that:

The fact that [knowledge] producers tend to have as their clients only their most rigorous and vigorous competitors, the most competent and the most critical, those therefore most inclined and most able to give their critique full force, is for me the Archimedean point on which one can stand to give a scientific account of scientific reason, to rescue scientific reason from relativistic deduction and explain how science can constantly progress towards more rationality without having to appeal to some kind of founding miracle… The closure upon itself of the autonomous field constitutes the historical principle of the genesis of reason and the exercise of its normativity (p. 54).

That ideal has been meaningfully leveraged to limit whose evaluation of academic work and researchers ought to be considered meaningful and acceptable. For instance, Smith et al. (2011) describe, in the context of Australian and UK Research “Excellence” Frameworks, how “impact” requirements in research excellence assessments challenge academic autonomy because of the role of external, non-academic, industry assessors with the consequence of effectively replacing peer-review with ‘expert-review’ (p. 1370). Similar concerns have been raised with respect to the identity of individuals who determine the appointments of academic staff (Henkel, 2007, p. 92). We argue this imperative applies equally to the deployment of evaluative bibliometrics and their coordination function. We see academic autonomy, in this evaluative dimension, as one of the key values at stake in the handoff between systems of evaluative bibliometrics and their use as market indicators between commercial publishers and platform companies.

The use of metrics for evaluation, provided by any entity, raises complex issues. As noted above, there is no essential or inherent relationship between impact (defined by metrics) and quality. Metrics often smooth over geographic and disciplinary considerations, disconnecting those indicators from the real contexts of academic work (Bianco et al., 2016). To that end, the Leiden Manifesto for Research Metrics, for example, outlines a series of principles for the best practice use of metrics in evaluation (Hicks et al., 2015). Managers are enjoined to use metrics to support rather than usurp qualitative assessment; to localise evaluation against the mission of a research centre; to compare researchers against those in the same field; to ensure databases and data processing used for assessment are open and transparent; and to enable researchers to verify and rectify records. Central to these best practices are transparency and contestability in the construction of metrics.

The question then becomes whether Google Scholar metrics, which are used for scholarly evaluation and market coordination, challenge those ideals of transparency in evaluative mechanisms, tools and processes, and whether this is exemplified in the Google Scholar ecosystem when compared to the JIF system.

There is an unquestionable transparency deficit with JIF. On one hand, this pertains to criteria for inclusion in the relevant databases from which JIF is derived. Much like Google Scholar’s opaque definition of scholarliness, described below, building the SCIE and SSCI (the Science Citation Index Expanded and Social Sciences Citation Index from Clarivate Analytics) depends on opaque estimations of worthiness as determined by commercial operators. Similarly, despite the apparent simplicity of the calculation, JIFs have been challenged as non-reproducible (Anseel et al., 2004; Rossner et al., 2007). But subsequent research has refuted that argument. With access to the Web of Science Core Collection, authors have clearly demonstrated the reproducibility of JIF (Larivière & Sugimoto, 2019), and shown it to be an understandable and challengeable indicator. Critically, the databases from which this class of indicators derive are (commercially) available and interrogable — one can at least know if a forum is or is not included in the database. Ultimately, this means that JIF does not resist the academy’s capacity to understand how it is constructed, or how it operates as an evaluative metric. That is why scholars like Gingras (2016) suggest that the Web of Science and Scopus offer advantages over Google Scholar in that: ‘one can know the list of journals included at any time’, whereas Google Scholar content is not well defined and varies constantly. For Gingras, ‘this is problematic from an ethical point of view, as evaluations should be transparent when they affect people’s careers’ (Gingras, 2016, p. 64).

In addition, not only does the Google Scholar system incentivise, through technical affordance, the continued non-reflexive use of indicators by academics, it also, through technical and organisational constraints, challenges academia’s capacity to understand how this type of bibliometric evaluation occurs at all. As described below, and contrary to the Leiden Manifesto best practices, through the way Google Scholar reconfigures evaluative hierarchies and networks, and how scholars and research managers interact through its technical system, it encourages purely quantitative comparison. Such comparison is applied across an academic domain, with no contextual or disciplinary divisions, through non-discipline-normalised metrics that the service will not rectify. Through its system of evaluative bibliometrics and citation counting, it has managed to mint a new form of symbolic capital that is distributed according to its own opaque and non-challengeable algorithmic determinations. The scale of its index, the definition of scholarliness and the mechanism for extracting citations are not transparent. Those opaque and non-accountable systems of research and researcher evaluation explicitly challenge the academy’s understanding of how it evaluates itself.

In the sections that follows, we describe the Google Scholar ecosystem, and then how the political and normative consequences described above emanate from the way Google Scholar imports into scholarly evaluation platform logic like two-sided markets, intermediation and value extraction, non-transparent relevance ranking and problematic priorities in interaction design.

From JIF to Google Scholar citations

Google Scholar launched in 2004, the same year that Elsevier launched rival bibliometrics platform Scopus. Beginning in 2006, citation counts were included in search results. The scholar profiling service, “Google Scholar Citations”, was launched in 2011. Citation counting drives both search and scholar profiles in different ways, and we argue that it has become a new unit of value for coordinating the scholarly economy, albeit in a manner different from JIF. While JIF quite explicitly shifted research priorities and visibility around inclusion in the SCI (De Bellis, 2014), Google Scholar is much more granular and totalizing in its shaping of research and researcher visibility according to citation count.

Search

Google Scholar search scans its index of scholarly documents and provides results according to relevance ranking. Neither the boundaries of the index,2 nor the meaning of “relevance” are transparent. Typical academic search engines index explicit repositories of journals, often organised by discipline. The choice of forum dictates which search engines will index a publication, and which university library subscriptions will make it available. The visibility of research thus depends on hierarchies of journal quality and perceived value-for-money. Alphabet is not an academic publisher however, and Google Scholar’s index is repository and discipline agnostic — it offers results from multiple openly available and pay-walled repositories. Larger academic publishers opened their repositories to Google Scholar around 2009, both in response to the “open science” movement, and in recognition of Google Scholar’s growing centrality as a federated search tool. Direct access to publications was eventually offered through link resolvers (Acharya, 2012), meaning users could access PDF files from pay-walled journals their university library subscribed to directly from Google Scholar search without ever logging in to a publisher interface.

There are vague descriptions on the Google Scholar website of how “scholarliness” is determined by this system. Google Scholar employees have also offered some limited commentary.3 The tendency is towards inclusivity. However, the algorithm making such determinations remains an opaque trade secret. Studies have shown nonetheless, that the academic web indexed by Google Scholar is markedly larger than any other academic search engine (Martín-Martín et al., 2018; Gusenbauer & Haddaway, 2019). This means, query-parsing being equal, it returns a potentially greater range of “relevant” results and potentially a higher number of citations. However, the absence of an API, and the limitation of 1,000 results per query, means there is no way to accurately measure the size of the index.

Google Scholar also indexes a great deal of what is sometimes termed “grey literature”. This includes documents from journals, blogs, newspapers or free/unmoderated repositories like SSRN, ArXiv or Academia.edu. That reconfiguration of what materials are visible and accessible through scholarly indexes supplements legacy gatekeepers such as journal editors and peer-reviewers with novel actors capable of producing academic esteem and symbolic capital through their interaction with (i.e. participating in the publication of materials that are indexed by) the Google Scholar system. This trend may be generally correlated with evolutions in academic publishing in networked environments and the pluralisation of academic outputs. Google Scholar both leverages and perpetuates this phenomena.

Making available a much larger corpus of research necessitates new ways of relating that research to a search query. When Google Scholar launched, the majority of other academic search engines used keyword-in-context, or Boolean operations on specific words to determine search engine result rankings. They gave researchers the ability to self-sort results according to discipline, jurisdiction, date, keywords and other metadata, word frequency, syntax relationships and many other criteria. In other words, users defined their own parameters of relevance. Google Scholar however, offered researchers fewer options to customise results, limiting search only by date range, specific author or journal, but critically, not discipline, document type or syntax relationship.4 This document ranking system, Google Scholar’s “primary innovation” (van Noorden, 2014), remains opaque and trade-secret protected. However, empirical work suggests that relevance in the Google Scholar depends primarily on the number of times an article is referenced by other documents in the scholarly index (Beel & Gipp, 2009a; Beel & Gipp, 2009b; Rovira et al., 2019). That means the most important factor determining search ranking is estimated to be citation count.5

In this context, academic information retrieval is reorganised around a logic of aggregation and indexing as opposed to self-defined notions of relevance and publication prestige. The influence of citation counting moves academic evaluation away from the disciplinary “expertise” associated with editors and reviewers, and into new networked hierarchies of authority determined by distributed citation practices, though not necessarily constrained to the academy or publications evaluated by academics. Because citation counts influence search rankings, academic search itself then becomes the product of that distributed evaluative process. However, much like the algorithms that determine whether a work is scholarly or how relevant it is to a user, the algorithmic “parsing” systems used to count citations out of the scholarly index are opaque.

Profiles

The researcher profile interface “Google Scholar Citations” launched in 2011, the same year as Microsoft Academic’s metrics service (which closed in 2021).6 Scholar profiles are automatically generated in Google Scholar, but can be edited by scholars. To edit a profile, the author is required to have a personal or university-provided Google account. Academic users of Google Scholar are therefore subject to ordinary Google user terms of service (including around data collection, processing and advertising), rather than, for instance, the more protective and negotiable terms in ‘G Suite for Education’.7

Google Scholar Citations displays scholar profiles that include basic institutional and biographic information, lists publications with links and citation counts and shows a calculation of scholars’ h-index and i-10 index values. The i-10 index lists the number of documents with more than 10 citations. The h-index, developed in 2005 (Hirsch, 2005), is an indicator of “impact”, applied at the micro-level of individual researchers. It is based on the number of papers published by a scholar and the number of citations referring to those publications over a particular time frame. Unsurprisingly, the h-index presented on Google Scholar Citations uses Google Scholar’s system of citation counting for the calculation.

H-index has since become a dominant mechanism for evaluating scholars, albeit in some disciplines and institutions more than others.8 Aside from its dubious relationship to actual researcher quality (see e.g. Lehmann et al., 2005; Kolton & Hafner, 2021), much like JIF, it smoothes over disciplinary citation and collective authorship conventions, thus facilitating non-normalised researcher evaluations and comparisons (Bornmann & Daniel, 2008; Hirsch, 2005; Iglesias & Pecharromán, 2006; Kelly & Jennions, 2006; Malesios & Psarakis, 2017).

Some studies have sought to discern the distribution of h-indexes across certain disciplines, in particularly geographical contexts, using specific citation databases, at particular points in time (see e.g. Malesios & Psarakis, 2017). These indicate, for instance, that using the SCI, computer science has a lower h-index average than mathematics, which is lower than economics - with clinical medicine and chemistry at the top. However, different h-indexes across disciplines are also the product of different publication coverage for each discipline in the databases used to calculate the indicator. The more disciplinary fora included in the database, the higher the citation counts across the corpus that inform the h-index. In the sciences, Harzing (2010) has shown that Google Scholar, for instance, provides radically higher citation counts for computer science when compared to other databases, whereas many other sciences show roughly equivalent citation counts and h-indexes.

Various factors result in vastly different h-index averages across disciplines. At the same time, different disciplines have different interpretations and treatments of the indicator, and citation counting more broadly in their evaluative conventions. Similarly, different funding bodies and universities also have different treatments of citation counting systems, where anecdotal discussions with hiring committees suggest better-resourced institutions have the capacity to spend more time and engage more qualitative expertise and evaluation for the sake of hiring and promotion decisions. Nonetheless, metrics still inform decisions from the highest to the lowest level. Metrics are used in the evaluation of institutions (Wilsdon et al., 2015; Mingers et al., 2017) to dismiss or make scholars redundant (Else, 2021), as well as for hiring decisions. While we were not able to perform a comprehensive empirical study of the use of Google Scholar citations in academic review or promotion evaluations, as has been done previously for JIF (McKiernan et al., 2019), the escalating role of Google Scholar is both anecdotally and intuitively clear to researchers. Indeed, numerous academic job application interfaces now require Google Scholar research data.

And at the more quotidian level, citation count driven scholar profiles and search results inform ordinary decisions by scholars about who might be invited to speak, or whose work is worth reading, as well as the materials that students retrieve and analyse. While dismissing academic staff on the basis of Google Scholar metrics alone might be unthinkable, Google Scholar makes researcher evaluation extremely easy when compared to other bibliometric databases, thus centralising itself in the daily practice of academic work. That centrality is achieved, however, by infusing the scholarly world with platform dynamics that challenge scholarly norms like autonomy and transparency in evaluating researchers and their work.

Platform dynamics in the scholarly economy

“Free”

There are numerous commercial services that provide evaluative bibliometrics. One point of distinction between Google Scholar and other scholarly platforms is price. Large commercial academic publishers operate according to subscription models that increasingly bundle analytic and bibliometric products alongside content provision. That said, these business models are still highly opaque, often taking advantage of regulatory or funder-driven Open Access obligations, and coercive strategies for data and bibliometric valorisation through asymmetrical power relationships with universities. Open access fees are increasingly part of these bundle deals. Newer platforms like Academia.edu sometimes represent themselves as free, but they use predatory freemium models and charge for access to features, or like ResearchGate, openly disclose that they trade in user data.9 Google Scholar, however, does not charge users, and is less explicit about its treatment of user data, making its motivation in the scholarly space more difficult to decipher.

As far as we were able to discern, Google Scholar does not directly make revenue. Google scholar is free for users and does not display advertising. In an interview with Nature, co-creator Anurag Acharya described Google Scholar’s raison d’etre as follows: to ‘give back to the research community’ (van Noorden, 2014). Interviews reported in the New York Times in 2004, however, indicated an initial intention to include advertising (Markoff, 2004; Terdiman, 2004). To that end, we note analyses of “gift” structures in platform economics by Fourcade & Kluttz (2020), and of “free” online services by Hoofnagle & Whittington (2014). Describing how different business structures in the digital economy rely on non-monetary exchanges, those authors describe how zero-price exchanges enable platforms to intermediate and control data flows, which can then be taxed in different ways.

Google Scholar’s gift claim is thus better understood as a site of strategic contradiction. As described by Fourcade and Kluttz (2020, p. 6), it is more likely to be an ‘instrument to generate the primitive accumulation of data’, a mechanism to enroll users into a digital system under the guise of a benign process, to institutionalise platform approaches to daily academic practice and as a way to ‘mask[] structural asymmetry’ (p. 3). By intermediating transactions between researchers and content, Google Scholar’s data capture gives it the broadest view of academic research as a practice. From this vantage point, Alphabet becomes the best placed actor to provide “business intelligence” about trends in research and publishing, about topics and researchers obtaining traction, and about movement and flows of research between institutions.

Further, while there is no advertising presented on the Google Scholar service, when we asked Google Scholar whether Google Scholar user interaction and behavioural data was commercialised through data markets, behavioural advertising, or in the development of other commercial products pitched to academics, universities or publishers, they refused to answer. Data transparency requests subsequently lodged under Article 15 of the European General Data Protection Regulation clarified that Google Scholar data is governed by ordinary Google terms of service, and that data is processed according to the ordinary Google privacy policy. Alphabet is thus able to extract commercial value from Google Scholar through its “service/data-profile/advertising complex” (Lovink & Tkacz, 2015, p. 15), even though no ad inventory is explicitly sold on Google Scholar websites.10

At the same time, denying its commercial nature enables Google Scholar to avoid obligations of accountability to users. For instance, users are referred back to publishers to amend errors in article descriptions. Google currently suggests that updates in their system will “usually take 6-9 months” (Google Scholar Website, see also Internet Archive recordings from June 2010 and August 2021).11 Similarly, there are only limited ways of getting in contact with Google Scholar staff, making user inquiries extremely difficult. And there are no ways to challenge absent citations (even if citing papers are present on the Google Scholar index) or contest search rankings. This maintenance of distance may contribute to Google's resistance to critique. See, for instance, Google Scholar's continued 'trans-exclusionary and sexist design' as 'the most influential organisation in scientific publishing to not enter a partnership affirming the ability of trans authors to change their names' (Speer & Signers, 2021; see Nerzig (2021) for a list of parties to the partnership, including publishers discussed in the present research: Clarivate, Elsevier, Wiley and Scopus).

User experience

Google Scholar’s interaction design is a large part of its success. Because it is free, users are not required to go through library logins or publisher paywalls. The capacity to search “<researcher’s name> google scholar” in any search engine and immediately access publication history, citation counts and h-index makes researcher evaluation easy. Google is also able to leverage its dominance in web-searching by suggesting “relevant” Google Scholar results in ordinary Google searches. Other free academic services, like ResearchGate or Microsoft Academic, do not provide anything close to that level of usability for quick searches or researcher evaluation. With Google Scholar, accessing scholarly work and scholar profiles becomes part of the native internet browsing experience. However, some of these interaction design strategies contradict typical academic practices.

Bibliometricians, librarians and conscientious researchers initially critiqued the inability to manipulate Google Scholar search results according to users’ own criteria (Giglierano, 2008; Falagas et al., 2008; Lewandowski, 2010). However, the removal of that interactivity allows Google Scholar to more “transparently” intermediate scholarly reality (Day, 2014, p. 27). “Transparency” in interface and interaction design is, ironically, a design principle that substitutes complexity for intuitive representation and use. As Michael Black notes ‘we trust that interfaces present us with a faithful representation of the systems they provide access to or largely assume that those mechanisms they mask are hidden from us for good reasons’ (2020). Transparency is thus a way to align and structure user behaviour with an intended set of acceptable practices.

In this way, the capacity to perform targeted research is usurped by design practices with other priorities. For instance, the ability to search by subject area was once a feature of Google Scholar, but it was removed in 2012 in order to present a ‘new modern look’ (Harzing, 2016). Google Scholar also returns search results faster than its competitors — which research suggests is more effective for retaining users (Arapakis et al., 2014; Crescenzi et al., 2016; Brutlag, 2009). Users should not think that fewer options to manipulate results makes Google Scholar a less sophisticated or powerful tool. Rather, the Google Scholar interface dissimulates both the power of the tool, and the political consequences of the design choices made. Another example is the lack of a display for publication Digital Object Identifiers (DOI), and listing multiple versions of articles, each of which inflate citation counts for particular disciplines. Accepting these user experience strategies means accepting an ex-ante evaluation of scholarship premised on the invisible knowledge hierarchies built by Google, that are entirely opaque, unknowable and unchallengeable by the user.

While authorship and citation conventions in different fields elevate or decrease average citations, the decision to not normalise those counts by discipline in an a-contextual presentation also builds new disciplinary hierarchies. Studies suggest that while Google Scholar’s citations counts are generally larger than Web of Science and Scopus, Engineering disciplines have roughly double the number of citations on Google Scholar, whereas physical and life sciences are only boosted by approximately 50% (Harzing & Alakangas, 2016). These amplifications, alongside the much higher number of computer science researcher profiles on Google Scholar (Ortega & Aguillo, 2012), perpetuate a shift in intellectual topography through which computer science and computer engineering appear to be the disciplines around which all knowledge is organised.

Open access

A third Google Scholar dynamic that challenges certain academic politics and ethics is how it has leveraged “open access” publishing to make itself a new gatekeeper in the scholarly world. Heralded as a remedy to the predatory practices of commercial academic publishing, open access has become a pillar of science policy around the world (see e.g. European Commission, 2019). Supported by the Open Society Institute, the 2002 Budapest Open Access Initiative statement of principles declared:

An old tradition and a new technology have converged to make possible an unprecedented public good. The old tradition is the willingness of scientists and scholars to publish the fruits of their research in scholarly journals without payment, for the sake of inquiry and knowledge. The new technology is the internet. The public good they make possible is the world-wide electronic distribution of the peer-reviewed journal literature and completely free and unrestricted access to it by all scientists, scholars, teachers, students, and other curious minds (Budapest Open Access Initiative, 2002).

Library budget constraints and political intervention, especially in Europe, have made open access mainstream, and developments like Plan S indicate that it could become the default publishing mode (Aspesi & Brand, 2020). However, open access has not proved the utopian solution that early internet optimists had hoped for, and it has failed to decrease reliance on legacy commercial publishers (see e.g. Chen & Chan, 2021). Some suggest that open access has effectively morphed into its neoliberal antithesis (Mirowski, 2018), especially in its “hybrid” Green and Gold manifestations where researchers pay for open access, therefore improving publishers’ profit margins (Aspesi & Luong, 2014). More important to our purposes, open access has also made space for other commercial actors like Google Scholar to build value capturing platforms atop the infrastructure of commons-based open access content (see e.g. Bodó, 2019). As Mirowski (2018) reminds us: ‘Platform capitalism meets open science; romance ensues’. Indicative of its interest in those developments, the Google Scholar interface now polices open access mandates by offering a leaderboard of open access mandate compliance, and displaying open access information about articles on researcher profiles, even inducing scholars to upload articles to their Google Drive (see e.g. van Noorden, 2021).

To that end, the academic community has largely failed to translate open access publishing into its own value-added analytics or tools for coordinating the market for academic prestige. Instead of open access contributing to greater academic autonomy through standardised, transparent and accountable evaluative analytics, commercial platforms have managed to intermediate, make proprietary, and extract value from what the open access movement has made available. In making the content of open science available on its platform, Google Scholar intermediates the scholarly economy through which its own unit of scholarly capital is minted and flows. Open access’s failure to decrease reliance on commercial publishers has thus ironically created new dependencies on commercial platform providers with opaque business models.

Legacy publishers have followed similar strategies, for instance, acquiring pre-print platforms and open access repositories, developing new academic analytics products and services, and locking institutions into bundled access, publishing, analytics and multi-service portals. Wiley, for instance, acquired the open access repository and infrastructure, ‘Knowledge Unlatched’, in December 2021 (Fund, 2021). Prior to that, Elsevier purchased the free repository Social Science Research Network (SSRN), as well as Mendeley, Bepress/DigitalCommons, Plum Analytics and other institutional repositories.12 Although owning open access repositories may seem inconsistent with Elsevier’s business model, Elsevier describes these purchases as a move into research and technology data management (Elsevier, 2017). Publishers extract value from these acquisitions by channeling data flows acquired from open repositories into products and subscriptions sold to universities at high prices in the form of “research services”, “academic analytics” and “business intelligence” (see e.g. Kelty, 2016).13 Linking SSRN content and usage data to Scopus citation data, for example, offers a way to extract value from that commons-based infrastructure without charging for content provision.

Elsevier has also been able to make its tools indispensable to universities by integrating them with university ranking data infrastructures. In this way, Elsevier has become a university “ranking” business that, for instance, provides the exclusive metrics for the Times Higher Education rankings (see e.g. Chen & Chan, 2021). HeinOnline has taken similar steps, making substantial investments in citation counting to leverage its analytics directly into the US News University law school ranking system.14 Being the data and analytics channel for a primary law school ranking system enables HeinOnline to capture tremendous value from its catalogue, while simultaneously cementing its centrality as a repository for legal scholarship.

Chen & Chan (2021, p. 425) describe these moves as commercial publishing firms ‘leveraging rankings as part of their strategies to further extract rent and assets from the university beyond their traditional role as publishers and citation providers’. Google Scholar follows a different path, however. By offering a free platform that works as a federated search engine and simple evaluative tool, it is able to intermediate the totality of the academic corpus, capturing all the data generated as content flows between multiple institutional, commercial, pre-print and “grey” repositories and users. But how this control over scholarly capital is translated into economic capital is even less clear. Unlike Elsevier, whose products university executives, for better or worse, actively decide to purchase, Google Scholar has become an information gatekeeper, and indeed the owner and controller of a dominant and proprietary marketplace, without scholars or university administrators actively making that choice (see e.g. Tanczer et al., 2019).

While Google might claim it does not intend for Google Scholar to compete with Web of Science, Scopus or any other commercial bibliometric service, and is instead about providing an “open” platform for academic research (Butler, 2011), such comments need to be interpreted in light of the political, economic and normative configurations that this type of “openness” generates (see e.g. Powers & Jablonski, 2015). Google thrives on an “open” web, and by encouraging the web as the domain of all represented human thought, with authority determined by topology, it is restructuring the very meaning of authority and expertise in ways that are unexaminable to the academic world, let alone to other stakeholders. While JIF participated in building this market for authority, and has its own transparency deficits, Google Scholar amplifies these trends by distributing authority according to the opaque algorithmic rules of a proprietary market that it owns and controls. Google’s continuing centralisation of academic tools makes the academy more dependent (see e.g. Vaidhyanathan, 2011), more vulnerable and less autonomous.

Google Scholar and academic autonomy

The Google Scholar system reconfigures bibliometric evaluation and scholarly market coordination around a unit of value — Google Scholar citation counts — that it generates through its proprietary algorithmic system. Compared to previous systems performing those functions, like JIF, Google Scholar introduces new components and actors that are able to generate scholarly authority that operates beyond the boundaries of the academic field’s primary evaluative mechanism — peer review — in unexaminable and unknowable ways. The various components of the system also act on each other in novel ways. The affordances of Google Scholar’s interface alter the modes by which researchers interact with scholarly content, nudging researchers away from their own determination of relevance, in favour of accepting intellectual topographies and hierarchies of authority that are opaquely constructed and controlled by Google Scholar. The platform enrolls users through “free” services, leverages open access policies and rhetorics to intermediate data flows between researchers and repositories, and extracts behavioural data about scholars, all while offering nearly no accountability to users. Google Scholar’s usability, ironically limiting the degree of interaction and adjustment of its system, has centralised and institutionalised the system into a daily part of academic life, informing critical decisions about the quality of individual researchers, as well as what scholarly content is visible and authoritative.

Platforms bring new organisational and economic logic to the domains they intermediate. While large dominant platforms are the subject of a great deal of critique, they clearly bring useful tools. Improved information retrieval and processing across the academic field is a generally welcome development. But the analysis above suggests that the ways in which Google Scholar intermediates the scholarly field often contradicts what we understand to be academic best practices.

Google Scholar is not alone in its introduction of platform dynamics and minimal transparency in the scholarly context. Like Google Scholar, Web of Science also acts as a “platform” in the sense of rule-setting and market-making. Platforms like SSRN exercise profoundly non-accountable control over what is included in their index, with unclear standards for what is considered “scholarly” or of scholarly merit (Crawford, 2019), using non-transparent mechanisms for policing metrics. It is probably correct, as the journal PLOS submitted to the UK inquiry into metrics in research evaluation in 2014, that ‘there are no adequate sources of bibliometric data that are publicly accessible, usable, auditable and transparent’ (Wilsdon et al., 2015, p. 17).

Similarly, while Google Scholar enables the production of scholarly capital outside mechanisms of peer review, it is clear that peer-review is hardly ideal. There is a diversity of peer-review practices, some better and some worse, with positive developments occurring in the context of Open Review practices. But peer-review does not necessarily mean quality assurance. It also concentrates gatekeeping power in panels of editors and discipline experts, who dispense degrees, publications, rankings and funding, often for the sake of conservative disciplinary reproduction (see e.g. Sayer, 2014).

There are many studies on the limitations, biases and otherwise derogations from an ideal peer-review process (see e.g. Johnson & Hermanowicz, 2017), and unquestionably a great deal of important literature is created outside of such formal mechanisms (Lawrence et al., 2014). But despite its many problems, peer-review and other forms of collective editorship are at least performed by academics. From this perspective, even if peer-review is an inferior tool for evaluating quality and an instrument for exercising power, it at least separates scholarly evaluation from industrial interests. In other words, this is not a question of whether metrics or peer-review produce “better” evaluations. It is a question of whether it is possible to know (or influence) how a system of evaluation works, and to what degree it is accountable.

The economies of academic evaluation that emerged in the middle of the 20th century, while not necessarily transparent or independent, did not explicitly undermine the academy’s understanding of how it evaluated itself. In a changing political context, universities embraced the new marketised logic of evaluation. Within that suite of new practices and market dynamics, the evaluation of academic work and the hierarchies around it were still performed, more or less, by academics with their structures of gatekeeping and social reproduction intact. Where commercial publishers sought more editorial control, academics often resisted.15 Google Scholar’s tools, however, are less transparent and testable than the commercial alternatives, more ingeniously designed to achieve centralisation and less reliant on academic editorship and peer-review — the ‘least worst’ mechanism of academic evaluation (Wilsdon et al., 2015). Whereas JIF did not necessarily displace academic expertise at the centre of its evaluative economy, Google Scholar’s blurry operations perpetuate the belief in the epistemic superiority of the market. This neoliberal refrain: that the market knows best (echoed forcefully by the state and funding agencies), coupled with that market being entirely owned and controlled by Google, means the academy is less able to know itself.

Conclusion

Google Scholar does not have a monopoly on opacity. There is a dearth of transparency and accountability across the now numerous tools for evaluation and ranking. Google Scholar is not the only bibliometrics platform demonstrating these features or causing these issues. In many ways Google Scholar replicates issues already identified with JIF, especially its secrecy and lack of reproducibility (Rossner et al., 2007; Rossner et al., 2008). But as Onara O’Neill describes, old intermediaries are being replaced with new intermediaries ‘whose contributions are harder to grasp, and who are not and cannot be disciplined by the measures used to discipline the old intermediaries’ (O’Neill, 2020). Google Scholar insists on avoiding the accountability mechanisms that have at least some grasp on the explicitly commercial relationship between universities and academic publishers (see e.g. Davis, 2011; Davis, 2012; Davis, 2018).

Google’s ambitions in education are vast. Unquestioned reliance on such opaque measures in hiring, promotion and tenure committees, let alone decisions about what to read and who can speak, displace academic autonomy. If the academy cannot develop "an accountability relationship" with Google Scholar, it must find other ways to meet its own obligations (Rached, 2016).