Making impact measurement more flexible

While the Swiss Agency for Development and Cooperation (SDC) and the State Secretariat for Economic Affairs (SECO) like to emphasise their successes, Parliament and the media regularly use flashpoints like Afghanistan as an occasion for criticising the inadequate impact of international cooperation (IC). But how is the effectiveness of IC even measured, and does the current measurement method make sense? The Control Committee (CC) of the Council of States has also raised this latter question. It commissioned the Parliamentary Control of the Administration (PCA) to examine the tools used to gauge the impact of IC, with the study focusing on the tool most frequently used – that of evaluations. The report of the PCA is now public, and makes one thing abundantly clear: while evaluations are useful as a management tool, they are rather ill-suited to impact measurement.

The effectiveness of international cooperation is reported to Parliament on the basis of success rates, with both SDC and SECO showing above-average success rates of over 80 per cent. These success rates are based on the consolidation of external, project-specific assessments. As the PCA shows, this is problematic for various reasons. The quality of individual assessments varies and there is no standard methodology; most evaluations take place during the life of projects and therefore provide no indication as to their long-term impact; the recommendations of individual evaluations are classified as inadequate and there is not always follow up by the SDC, SECO and the FDFA’s Peace and Human Rights Division (PHRD); moreover, the individual evaluations make scant reference to the overarching aims of international cooperation.

But, as the Control Committee of the Council of States also notes, it would be wrong to conclude from these findings that international cooperation is ineffective. It primarily recognises that Switzerland does achieve many of its international cooperation goals and also implements useful projects. Yet, it is critical of "the Federal Council's practice hitherto of using dubious success rates to account for the effectiveness of international cooperation." But neither does the CC wish to abolish evaluations as such or to declare them to be pointless, as they can undoubtedly be useful as internal management mechanisms if they are meaningfully designed and in fact serve internally to manage or, in other words, to adapt projects.

The trend towards evidence-based approaches and impact analyses

Besides the critical assessment of current impact measurement practices, there are growing calls in Switzerland for evidence-based approaches and impact analyses. On the one hand, this means that scientific evidence will be increasingly factored into the design and conception of new projects, and on the other, that more scientific impact analyses will be undertaken. These in turn refer primarily to "randomised control trials" (RCTs), which have gained appreciable impetus in recent years thanks to the work of Nobel Prize laureates Esther Duflo and Abhijit Banerjee. The principle is simple: the project design entails randomly forming two groups – one that benefits from the development project and another that does not. For example, a number of schools in Kenya are randomly selected – textbooks are distributed to the children in half of the schools, while the children in the control group receive none. Both school attendance and grades are recorded for all the children, before and after the books are distributed. A year later, the same data is again collected. If the group that received textbooks actually shows better school attendance and grades, it may be concluded that the project has been successful and can be replicated in other settings. That, at least, is the theory.

In practice, however, various questions and dilemmas arise:

1. Most IC projects today are considerably more complex; they are not merely about distributing schoolbooks or medicines. In fragile contexts, in particular, numerous factors interact and contexts change quickly, requiring projects to be swiftly adapted. This is hardly compatible with the experimental logic of scientific impact analyses.
2. Modern international cooperation embraces criteria such as participation and localisation. Many of today's IC projects are therefore implemented by local entities, which also help develop the projects. At best, project beneficiaries should also have a say. This, too, does not square with the logic of impact analyses, in which people are viewed more as objects of study than as active individuals.
3. Following on from point 2, randomised field studies also raise ethical questions, as many people affected by poverty and discrimination are deliberately included in experiments without benefiting from them.

The case for a rethink

What, then, is the solution? Taxpayers, development agencies and people affected by poverty all have an interest in seeing international cooperation work. But does that really call for more and more figures and statistics? Often based on rigid bureaucracies, planning instruments and evaluations, current practice offers little indication of the actual added value of IC. And at best, randomised field studies are suited to a small proportion of IC projects.

Parliament and the public deserve one thing above all: an honest debate on international cooperation, including both the successes and the challenges. Switzerland has scored many international cooperation successes, as borne out repeatedly by individual projects and scientific studies. But it often takes time for impacts to be felt. When it comes to the rule of law or the strengthening of local civil society – both of which are fundamental to sustainable development – an immediate impact is not always clearly achievable. Besides, as the case of Afghanistan shows us, achievements can be quickly wiped out, especially in times of crisis.

Apart from improved communication and outreach work, both the practice and impact of IC can be enhanced by better harnessing existing scientific studies and encouraging stakeholders to conduct their own studies – especially in the realm of thematic and country strategies. But project work itself requires more flexibility than rigidity, and it is important for all projects to be clearly managed for results. In concrete terms, this means working with local partners to establish aims that are aligned with those prescribed by law, i. e., assisting in the alleviation of need and poverty in the world and promoting respect for human rights and democracy, the peaceful coexistence of peoples as well as the conservation of natural resources (Art. 54.2 of the Federal Constitution), and also the specific goals laid out in the IC strategy. Rather than relying on rigid logframes for project implementation, it should be possible, at any time, to adjust measures (and goals if need be), should the planned measures prove no longer expedient or if the context changes. This calls for constant monitoring, which could well be done by the implementing partners, especially as local partners usually know best when and which adjustments are needed. Moreover, post-project evaluations can also be useful in determining whether and how the set goals have been achieved. However, as the PCA report also states, these evaluations should ideally be interdepartmental in nature and guided by clear criteria.