Most researchers prefer their results to have a social impact. In addition, they show interest in knowing that impact. To this must be added the growing competition for financing, which makes it necessary to quantify said impact in order to opt for certain sources of financing. These reasons have motivated the efforts of the scientific community to adequately measure the social benefits of scientific research. For society, which finances scientific research, recognizing and properly rewarding the social approach to research is a fundamental objective.
Scientific research affects so many aspects of society that measuring the totality of its impacts is an arduous task. Furthermore, the most significant impacts are those that are most difficult to measure. Therefore, it is difficult to find good social impact metrics. However, at a time when the impact factors of scientific journals are being questionedthe search for new approaches that can measure the social impact is a primary task.
Evaluation with a social focus
Over the past decade, efforts by governments and funding agencies to better understand the return on investment in research have helped lay the groundwork for a more comprehensive assessment of impact. This is the case, for example, of the Research Excellence Framework (REF) conducted in 2014 and 2021, which measures the quality of the contributions of UK universities by assigning an impact score, which it defines as any effect or benefit on the “economy, society, culture, politics or public services, health, environment or quality of life beyond the academy” [1]. In this respect, it surpasses its predecessor, which regularly assessed UK research from 1986 to 2008, although without separately analyzing social impact. For REF 2021, the expert panels considered almost 6,800 applications from university departments, documents that try to explain the scope of their research in society and that can be consulted openly. [2].
In Spain we had Six-year Knowledge Transfer and Innovation [3], whose pilot test was carried out in 2018 and whose objective was to evaluate the performance of researchers in this matter. Assessing the social impact was challenging at the time, although the return on investment was high. This milestone laid the foundation for a more comprehensive evaluation of the research and the effects it has on politics, the economy, and other areas.
Impact on public policies
It is possible to systematically trace the connections between research and public policy. A citation in a public policy document may be even more significant than a citation in a widely read journal, as it is evidence that the research actually affected social behavior.
Recently, some researchers have found evidence that articles published in social sciences and humanities journals are more likely to be cited in public policy documents. [4]. Using data from Overton, a database of policy documents collected from governments, expert panels, intergovernmental organizations and NGOs, the nearly 37,000 articles in these fields that were published in 2010 generated nearly 260,000 policy citations in the following decade, a rate of almost seven citations per article. However, about 18,000 articles were published in engineering and physical sciences journals in 2010, generating approximately 63,000 policy citations in the same time period, a rate of about 3.5 citations per article.
The number of citations available suggests the possibility of creating impact metrics on public policies similar to the h-index, a number that measures academic impact through the number of documents published and citations received. However, while inexpensive and simple to implement, this indicator would likely oversimplify the impact of research and would not adequately discriminate against scientists who have actually contributed to changing the actions of governments, medical societies, or other organizations. Thus, the study that compared UK universities’ policy citations with their impact scores in the REF 2014 [4]found that institutions with the highest number of policy citations tended to receive higher scores from raters on the REF, although the correlation was not especially strong.
An indicator derived from a policy database may fall short and some form of peer review may need to be included to allow for a proper assessment of political influence. However, collecting and categorizing policy citations can complement and support expert judgment. For this, the database Altmetric is another alternative that collects citations in public policy documents.
Innovation and economic impact
Research often drives the development of new technologies and products, but tracking the impact of an individual researcher’s innovation can be a complicated process.
Innovations are essential for science and society, and it is increasingly important to identify the most innovative projects and researchers. However, the simplest metrics to measure innovation may not be enough. These metrics tend to measure the intermediate results of innovation reasonably well, through patents or citations, although with certain limitations. Just as some researchers often cite themselves in their publications to increase their own impact, inventors often cite their own work in patent applications. To this is added, in certain countries, the fact that inventors cite patents from that country exclusively, partly to avoid possible international copyright disputes.
University research can also have economic impacts that go far beyond patents or spin-offs, such as the creation of a qualified labor market. In fact, the differentiating aspect that universities generate from research and teaching are the graduates to the labor market.
Sustainable development and environmental impact
There are salient research impacts that are not economic and are more difficult to measure. Thus, for example, researchers in the humanities and social sciences carry out critical work that makes them rethink the things that we value as a society and denounce inequalities.
Scientific research has the potential to transform the planet by addressing global environmental crises, such as climate change, and improving the well-being of the poorest communities. The Sustainable Development Goals (SDGs) of the UN, a set of 17 targets that address the biggest global challenges, including poverty, hunger and clean energy, are increasingly used to measure that impact.
Universities and researchers often make visible their commitment to the SDGs, although a 2022 report suggests that research is largely falling short of these goals. said report [5] found that 95-98% of patents and inventions and 60-80% of academic publications are misaligned with the SDGs. Only 30-40% of research conducted in high- and middle-income countries has any connection to the SDGs, much less than 60-80% in low-income countries.
In this sense, the database DimensionsAI provides a classification system for these 17 objectives, most of which are interconnected. This automated classification of SDG-aligned posts is implemented using an expert-supervised natural language model [6].
Social media and cultural impact
Alternative metrics, or altmetrics, attempt to track the reach of research across social media, news sites, and other media. They were created to measure impact beyond academic citations, but they are still far from that goal. For example, counting mentions of an article on social media doesn’t tell you much about its true meaning. Anyone can share an article on social media, even those who don’t value the results, are doing so to criticize the research, or are spreading false information about its findings. If you take a look at the tweets about press news, it is clear that in most cases there is not a clear understanding or a deep reading of the news.
It is therefore worth asking how this evidence could be used in the evaluation of researchers. Some authors have found only a weak correlation between altmetric scores and expert-perceived research quality, a clear sign that the scores were a poor indicator of research quality. [7]. Alternative metrics must be refined and validated if they are to be a reliable basis for evaluation and funding decisions..
Despite these challenges, spanning the breadth of all types of research impact is not a lost cause. The scientific community dedicated to research evaluation is making significant efforts to identify the different dimensions of the impact of research, its meaning and the most appropriate metrics to quantify it.
*****
This article is sent to us Pablo Dorta-Gonzalez, Professor of Quantitative Methods at the University of Las Palmas de Gran Canaria. He currently works at the TiDES University Institute of Tourism and Sustainable Economic Development, where he maintains two lines of research open, one on quantitative methods in economics and business, and another on bibliometric indicators for scientific evaluation.
Scientific references and more information:
[1] Research Excellence Framework, REF 2021. https://www.ref.ac.uk/
[2] Impact case study database, REF 2021. https://results2021.ref.ac.uk/impact
[3] Six-year period of Knowledge Transfer and Innovation, 2018, ANECA. https://www.aneca.es/transferencia
[4] Szomszor M., Adie E. (2022) Overton: A bibliometric database of policy document citations. Quantitative Science Studies 3(3): 624–650. https://doi.org/10.1162/qss_a_00204
[5] Ciarli T. (2022) Changing Directions: Steering science, technology and innovation towards the Sustainable Development Goals, STRINGS, SPRU, University of Sussex. https://dx.doi.org/10.20919/FSOF1258
[6] Wastl J., Porte S., Draux H., Fane B., Hook D. (2020) Contextualizing sustainable development research, Digital Science Report. https://doi.org/10.6084/m9.figshare.12200081.v2
[7] Kassab O., Bornmann L., Haunschild R. (2020) Can altmetrics reflect societal impact considerations?: Exploring the potential of altmetrics in the context of a sustainability science research center. Quantitative Science Studies 1(2), 792–809. https://doi.org/10.1162/qss_a_00032
If you have an interesting article and you want us to publish it in Naukas as a guest contributor, you can contact us.