Responsible Research Assessment and Research Information Management Systems

Responsible Research Assessment and Research Information Management Systems: History

Please note this is an old version of this entry, which may differ significantly from the current revision.

Contributor:

In the context of open science, universities, research-performing and funding organizations and authorities worldwide are moving towards more responsible research assessment (RRA). In 2022, the European Coalition for Advancing Research Assessment (CoARA) published an agreement with ten commitments, including the recognition of the “diversity of contributions to, and careers in, research”, the “focus on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators”, and the “abandon (of) inappropriate uses in research assessment of journal- and publication-based metrics”. Research assessment (RA) is essential for research of the highest quality. The transformation of assessment indicators and procedures directly affects the underlying research information management infrastructures (also called current research information systems) which collect and store metadata on research activities and outputs. This entry investigates the impact of RRA on these systems, on their development and implementation, their data model and governance, including digital ethics.

research assessment
responsible research assessment
research information management systems
open science

Research assessment (RA) encompasses a multifaceted process aimed at evaluating the quality, impact, and effectiveness of research endeavors. As highlighted by Robert K. Merton ^[1], “The activities of scientists are subject to rigorous policing”, indicating the inherent scrutiny applied to scientific endeavors. This scrutiny is manifest in the monitoring and evaluation of research performance, which permeates through individual, institutional, and governmental levels within the scientific community. RA is conducted by a variety of entities, each with their own motivations and objectives, such as academic institutions (promotion and tenure committees, departments), funding agencies (grant review panels), research councils (advisory committees, program managers), or government bodies (national assessment programs). Their main reasons are resource allocation, accountability, quality assurance, strategic planning on the institutional level, and recognition and reward of individual researchers.

At its core, RA serves to establish and uphold standards of research quality. It plays a pivotal role in shaping the allocation of resources by contributing to the efficient and accountable funding of projects, programs, and research teams. It affects the scholarly career of researchers in various ways, insofar that it forms the basis for decisions related to promotions and tenure in academic institutions and influences the likelihood of securing grants and funding (often based on number of publications, citation counts, h-index, and journal impact factors); a strong record in RA can enhance a researcher’s reputation in their field, lead to invitations to speak at conferences, collaborate on projects, and join editorial boards, further enhancing a researcher’s career prospects and competitivity for academic positions. Moreover, RA functions as a governance tool, fostering improvements in the quality of scholarship by promoting rigorous inquiry and adherence to established methodologies ^[2]^[3].

The landscape of RA is dynamic, continually evolving in response to the shifting demands of the scientific enterprise ^[4]. Emerging technologies, such as artificial intelligence (AI) and machine learning, offer opportunities to enhance assessment methodologies by facilitating the analysis of vast datasets with greater precision and efficiency ^[5]. Additionally, there is a continual refinement of metrics and indicators to encompass the diverse dimensions of research output and impact. Multidimensional assessment approaches have gained traction, recognizing the multifaceted nature of research performance and accounting for various aspects beyond traditional bibliometric measures ^[6].

Collaboration among stakeholders, including researchers, institutions, and funding agencies, is instrumental in advancing RA practices. By fostering synergies and sharing best practices, collaboration promotes methodological advancements and ensures that assessment frameworks align with the evolving needs and aspirations of the scientific community ^[7].

Recent discourse in the field underscores the importance of responsible research assessment (RRA), which advocates for transparency, diversity, quality, and open metrics ^[8]. This paradigm shift acknowledges the diverse contributions to research and underscores the imperative of qualitative evaluation while advocating for the judicious use of quantitative indicators. By embracing RRA principles, the scientific community can mitigate the unintended consequences of metric-driven assessment practices and foster a culture of responsible and equitable evaluation ^[9].

This entry is adapted from the peer-reviewed paper 10.3390/encyclopedia4020059

References

Merton, R.K. The Sociology of Science: Theoretical and Empirical Investigations; Chicago University Press: Chicago, IL, USA, 1973.
Gläser, J.; Laudel, G. Evaluation Without Evaluators. In The Changing Governance of the Sciences; Whitley, R., Gläser, J., Eds.; Springer: Dordrecht, The Netherlands, 2007; pp. 127–151.
Dennett, D.C. Quining qualia. In Consciousness in Contemporary Science; Marcel, A., Bisiach, C., Eds.; Oxford University Press: Oxford, UK, 1992; pp. 42–77.
Benbya, H.; Nan, N.; Tanriverdi, H.; Yoo, Y. Complexity and information systems research in the emerging digital world. MIS Q. 2020, 44, 1–17.
Xu, Y.; Liu, X.; Cao, X.; Huang, C.; Liu, E.; Qian, S.; Liu, X.; Wu, Y.; Dong, F.; Qiu, C.W.; et al. Artificial intelligence: A powerful paradigm for scientific research. Innovation 2021, 2, 100179.
Handoyo, S. Mapping the landscape of internal auditing effectiveness study: A bibliometric approach. Cogent Bus. Manag. 2024, 11, 2289200.
Kale, S.; Hirani, S.; Vardhan, S.; Mishra, A.; Ghode, D.B.; Prasad, R.; Wanjari, M. Addressing cancer disparities through community engagement: Lessons and best practices. Cureus 2023, 15, e43445.
The Agreement on Reforming Research Assessment. Coalition for Advancing Research Assessment (CoARA). Available online: https://coara.eu/ (accessed on 25 April 2024).
Sanders, J.; Moore, J.; Mountford-Zimdars, A. Operationalising Teaching Excellence in Higher Education: From ‘Sheep-dipping’ to ‘Virtuous Practice’. In Challenging the Teaching Excellence Framework; French., A., Thomas, K.C., Eds.; Emerald: Bingley, UK, 2020; pp. 47–94.

© Text is available under the terms and conditions of the Creative Commons Attribution (CC BY) license; additional terms may apply. By using this site, you agree to the Terms and Conditions and Privacy Policy.