The EURECIA project has a number of key themes that cut across all of the work packages we are undertaking.
We have first asked ourselves what is wrong with the current ways in which the outcomes and impacts of research funding schemes are understood? What is wrong with how they are assessed? Second, after we have reviewed substantial amounts of relevant academic literature in important areas, we have not found material on how to better measure, attribute or assess impact. This could suggest that perhaps it cannot be done. Perhaps it is not worth doing? This could also mean EURECIA is doing something original. This is related to a third issue, which is a view of science (and social science) as disciplined dissent when it is at or pushing the frontiers of knowledge and when there is an aspect of risk to the research.
Overall we have drawn out three key themes related to impact. For each we have a number of key questions to address.
Measuring impact
Challenges
It is inherently difficult to measure the impact of a funding scheme on a complex system like science. First you need to understand what science is. Existing schools of thought have focused disproportionately on one aspect. There has been less attention to the whole system. The sociology of science has focused on science as a social organisation (e.g. scientific communities) and organisational rules. The sociology of knowledge has emphasised the social construction of scientific knowledge and knowledge dynamics in general. Science, technology and policy studies have focused on policy and its evolution but there is often no clear link between policy evolution and policy pressures and what actually happens in science. There is also a long tradition of measuring ‘change’ per se. Things can be measured at different points of time. Attributing this change to anything is a very interesting issue. Usually the change is not only produced by one policy measure but also is emergent from a whole range of factors. So measuring and attributing change is not trivial.
Our contribution
Clearly a lot of work has been done on these various aspects in isolation but far less research has been done on how to link these aspects together in a useful way. In EURECIA we have started by considering hypothetically the differences the ERC’s funding schemes could make for the entire research system (research, researchers and academic careers, research organisations, research landscapes, and epistemic networks), so as to consider both intended and unintended outcomes and impacts. We are interested in the causal mechanisms and boundary conditions for these differences to occur. We are also looking at differential impacts across different research fields, funding regime types, and for different kinds of research organisation around Europe. We are also not limiting ourselves to only capturing differences seen as objective changes in reality. We are also considering the subjective or symbolic impact of the ERC’s funding schemes on the research system.
Attributing impact
Challenges
Given the sums of public money spent on European research since the inception of the Framework Programmes it is perhaps not surprising that there has been growing attention paid to the impacts of research in Europe. There is a lot of activity about measuring impact but often research and evaluations have made questionable assumptions at the design stage that seriously affect their ability to properly attribute impacts to particular funding schemes or instruments. A significant amount can happen to a complex system like science during the term of a funding scheme’s operation. Changes seen in the European research system could easily be caused by other developments perhaps not connected to the funding scheme being evaluated.
Our contribution
Our assumption is that an impact has to be understood as a clearly attributable difference. Our key question here is therefore how can differences in the European research system over a period of time be attributed or traced back to the ERC’s funding schemes? What is the level of confidence at which this can be, if it can be done at all? How do we deal with issue of additionality as well as attribution? One of our main contributions is this area is our use of a control group so that we have very similar cohorts of researchers in almost every respect except the receipt of ERC grant funding. We will survey and compare ERC funding scheme grantees and our control group. In tandem with other parts of our integrated methodology which address developments elsewhere and throughout the European research system, we do hope to be able to attribute any differences we see to the ERC funding (or not).
Assessing impact
Challenges
When the impact of research funding schemes is assessed the focus of the evaluation process is very often on research outputs (e.g. research papers and reports arising from a research project are counted multiplied by the impact index of their place of publication or dissemination). However in the EURECIA project we believe that the actual impact is the difference made by these outputs on the research system. Impact is not only about the numbers of outputs that have been produced but also whether thinking has been changed. A book being written by a researcher is an output, whereas its impact is whether it redefines thinking about the world at that time.
Our contribution
The EURECIA project moves beyond this idea of assessing the impact of policy measures primarily on whether they have produced a certain number or type of outputs. For us the key issue is about the impact these outputs generate and then attributing that impact back to the policy measure. If the change cannot be attributed to the policy measure then this is not impact. With assessment there is also the distinction between ‘attributable difference’ and the actual process of judgement. How is the assessment of what is a positive or negative change made? Here the EURECIA project is interested in what the norms and criteria for assessment are and how they are formed. We will be exploring these issues during our fieldwork.
Share: