{rfName}
Yo

Indexed in

License and use

Citations

21

Altmetrics

Analysis of institutional authors

Schellaert, WCorresponding AuthorMartínez-Plumed, FAuthorHernández-Orallo, JAuthor

Share

Publications
>
Article

Your Prompt is My Command: On Assessing the Human-Centred Generality of Multimodal Models

Publicated to:Journal Of Artificial Intelligence Research. 77 377-394 - 2023-01-01 77(), DOI: 10.1613/jair.1.14157

Authors: Schellaert, Wout; Martinez-Plumed, Fernando; Vold, Karina; Burden, John; Casares, Pablo A M; Loe, Bao Sheng; Reichart, Roi; hEigeartaigh, Sean O; Korhonen, Anna; Hernandez-Orallo, Jose

Affiliations

- Author
Technion Israel Inst Technol - Author
Univ Cambridge, Language Technol Lab LTL - Author
Univ Cambridge, Leverhulme Ctr Future Intelligence - Author
Univ Cambridge, Psychometr Ctr - Author
Univ Complutense Madrid - Author
Univ Politecn Valencia, VRAIN - Author
Univ Toronto, Inst Hist & Philosophy Sci & Technol, Toronto - Author
See more

Abstract

Even with obvious deficiencies, large prompt-commanded multimodal models are proving to be flexible cognitive tools representing an unprecedented generality. But the directness, diversity, and degree of user interaction create a distinctive human-centred generality (HCG), rather than a fully autonomous one. HCG implies that-for a specific user- a system is only as general as it is effective for the user's relevant tasks and their prevalent ways of prompting. A human-centred evaluation of general-purpose AI systems therefore needs to reflect the personal nature of interaction, tasks and cognition. We argue that the best way to understand these systems is as highly-coupled cognitive extenders, and to analyse the bidirectional cognitive adaptations between them and humans. In this paper, we give a formulation of HCG, as well as a high-level overview of the elements and trade-offs involved in the prompting process. We end the paper by outlining some essential research questions and suggestions for improving evaluation practices, which we envision as characteristic for the evaluation of general artificial intelligence in the future.

Keywords

Ai systemsCognitive adaptationCognitive toolEconomic and social effectsHuman-centered evaluationInformationInternetMemoryMultimodal modelsResearch questionsTrade offUser interactionUser interfaces

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal Journal Of Artificial Intelligence Research due to its progression and the good impact it has achieved in recent years, according to the agency Scopus (SJR), it has become a reference in its field. In the year of publication of the work, 2023, it was in position , thus managing to position itself as a Q1 (Primer Cuartil), in the category Artificial Intelligence.

From a relative perspective, and based on the normalized impact indicator calculated from World Citations provided by WoS (ESI, Clarivate), it yields a value for the citation normalization relative to the expected citation rate of: 1.28. This indicates that, compared to works in the same discipline and in the same year of publication, it ranks as a work cited above average. (source consulted: ESI Nov 14, 2024)

This information is reinforced by other indicators of the same type, which, although dynamic over time and dependent on the set of average global citations at the time of their calculation, consistently position the work at some point among the top 50% most cited in its field:

  • Weighted Average of Normalized Impact by the Scopus agency: 1.19 (source consulted: FECYT Feb 2024)
  • Field Citation Ratio (FCR) from Dimensions: 10.94 (source consulted: Dimensions Jun 2025)

Specifically, and according to different indexing agencies, this work has accumulated citations as of 2025-06-26, the following number of citations:

  • WoS: 9
  • Scopus: 12
  • OpenCitations: 7

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2025-06-26:

  • The use, from an academic perspective evidenced by the Altmetric agency indicator referring to aggregations made by the personal bibliographic manager Mendeley, gives us a total of: 16.
  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 16 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

  • The Total Score from Altmetric: 9.15.
  • The number of mentions on the social network X (formerly Twitter): 13 (Altmetric).

Leadership analysis of institutional authors

This work has been carried out with international collaboration, specifically with researchers from: Canada; Israel; United Kingdom.

There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: First Author (Schellaert, Wout Willy M.) and Last Author (Hernández Orallo, José).

the author responsible for correspondence tasks has been Schellaert, Wout Willy M..