{rfName}
Re

Indexed in

License and use

Altmetrics

Analysis of institutional authors

Schellaert, WAuthorMartínez-Plumed, FAuthorHernández-Orallo, JAuthor

Share

October 28, 2024
Publications
>
Editorial Material
No

Rethink reporting of evaluation results in AI Aggregate metrics and lack of access to results limit understanding

Publicated to:Science. 380 (6641): adf6369-138 - 2023-04-14 380(6641), DOI: 10.1126/science.adf6369

Authors: Burnell, Ryan; Schellaert, Wout; Burden, John; Ullman, Tomer D; Martinez-Plumed, Fernando; Tenenbaum, Joshua B; Rutar, Danaja; Cheke, Lucy G; Sohl-Dickstein, Jascha; Mitchell, Melanie; Kiela, Douwe; Shanahan, Murray; Cohn, Anthony G; Leibo, Joel Z; Hernandez-Orallo, Jose

Affiliations

- Author
DeepMind - Author
Google, Brain Team - Author
Harvard Univ, Dept Psychol - Author
Imperial Coll London, Dept Comp - Author
MIT, Dept Brain & Cognit Sci - Author
Shandong Univ - Author
Stanford Univ - Author
Univ Cambridge, Ctr Study Existential Risk - Author
Univ Cambridge, Dept Psychol - Author
Univ Cambridge, Leverhulme Ctr Future Intelligence - Author
Univ Politecn Valencia, Valencian Res Inst Artificial Intelligence - Author
See more

Abstract

Aggregate metrics and lack of access to results limit understanding. Copyright © 2023 The Authors, some rights reserved.

Keywords

ArticleArtificial intelligenceAssessment methodReproducibility

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal Science due to its progression and the good impact it has achieved in recent years, according to the agency WoS (JCR), it has become a reference in its field. In the year of publication of the work, 2023, it was in position 3/134, thus managing to position itself as a Q1 (Primer Cuartil), in the category Multidisciplinary Sciences. Notably, the journal is positioned above the 90th percentile.

From a relative perspective, and based on the normalized impact indicator calculated from World Citations provided by WoS (ESI, Clarivate), it yields a value for the citation normalization relative to the expected citation rate of: 7.33. This indicates that, compared to works in the same discipline and in the same year of publication, it ranks as a work cited above average. (source consulted: ESI Nov 14, 2024)

This information is reinforced by other indicators of the same type, which, although dynamic over time and dependent on the set of average global citations at the time of their calculation, consistently position the work at some point among the top 50% most cited in its field:

  • Weighted Average of Normalized Impact by the Scopus agency: 17 (source consulted: FECYT Feb 2024)
  • Field Citation Ratio (FCR) from Dimensions: 46.81 (source consulted: Dimensions Jul 2025)

Specifically, and according to different indexing agencies, this work has accumulated citations as of 2025-07-07, the following number of citations:

  • WoS: 40
  • Scopus: 34
  • Europe PMC: 8

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2025-07-07:

  • The use, from an academic perspective evidenced by the Altmetric agency indicator referring to aggregations made by the personal bibliographic manager Mendeley, gives us a total of: 57.
  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 66 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

  • The Total Score from Altmetric: 176.3.
  • The number of mentions on the social network Facebook: 1 (Altmetric).
  • The number of mentions on the social network X (formerly Twitter): 165 (Altmetric).
  • The number of mentions in news outlets: 10 (Altmetric).

Leadership analysis of institutional authors

This work has been carried out with international collaboration, specifically with researchers from: China; United Kingdom; United States of America.

There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: Last Author (Hernández Orallo, José).