Effects of research and innovation policy measures - Impact measurement rarely takes place

Many of the evaluation studies on research and innovation policy measures carried out on behalf of the Federal Government do not allow any conclusions to be drawn as to whether the policy measures examined were effective and led to the desired results. This is shown by the Commission of Experts for Research and Innovation (EFI) in its new annual report, which it presented to Federal Chancellor Olaf Scholz at the end of February.

Many evaluation studies do not meet the methodological requirements

To what extent do the Federal Government's research and innovation policy measures - including project funding and innovation consulting - contribute to the creation of new findings, inventions and business models? And do these measures help to tap into new value creation potential and better manage transformation processes? These questions are becoming ever more pressing in times of empty public coffers and increasing pressure to transform. The evaluation studies commissioned by the German government should actually provide information on how effective the measures examined are. "So far, however, this has hardly been the case," says Professor Guido Bünstorf from the University of Kassel and member of the expert commission, "because most evaluation studies do not meet the methodological requirements for a meaningful impact measurement."

For its annual report, the Expert Commission examined 81 publicly accessible evaluation studies from the areas of responsibility of the BMBF and BMWK, which were written between 2009 and 2023. In 59 of these 81 studies, observed developments were causally interpreted as effects of the respective measures. However, only in seven cases did the methods used allow such statements about cause-and-effect relationships to be made at all. As a result, the evaluation studies examined contribute very little knowledge about whether the objectives of the respective measures were achieved. "With a better knowledge base, policy measures could be specifically adapted and their effectiveness improved," says the chairman of the expert commission, Professor Uwe Cantner from the University of Jena. "A lack of knowledge prevents policy learning."

Creating the conditions for a learning policy

In its report, the Expert Commission calls on the Federal Government to put evaluation studies out to tender in future in such a way that the terms of reference include a causal analysis and minimum requirements for the evaluation concept in this regard. The conditions for this must also be improved by politicians, for example with regard to access to the necessary data. "Anyone planning a research and innovation policy measure must also ensure that the data required for a causal analysis can be collected," says Cantner. The bodies responsible for implementing a measure should systematically make all documents associated with a measure available to the evaluating organizations. The Expert Commission also considers it necessary for all commissioned evaluation studies to be published, regardless of their results. Both positive and negative results of evaluation studies should be equally valued as progress in knowledge.