6. Germany’s results, evaluation and learning

German development policy and the BMZ 2030 reform strategy recognise the importance of achieving the 2030 Agenda and the SDGs (BMZ, 2020[1]). The reform strategy clearly points out the contribution that the five core areas and ten initiative areas are expected to make to achieve the SDGs. In addition, Germany’s refocusing of development co-operation with Africa aims to contribute to the African Union’s Agenda 2063.

Partner country priorities inform German development co-operation. Country programmes outline the contribution German interventions make to priorities articulated in partners’ national development strategies and plans.

Targets in strategies will focus as closely as possible to the SDGs. Recently approved guidelines on strategies for core and initiative themes, and country strategies requires alignment with the SDGs. As developing countries increasingly apply SDG indicators in their results frameworks, the SDGs will increasingly form a common framework for results (OECD, 2019[2]).

German implementing organisations have long experience with results-based management at project level. Results matrices — generally logical frameworks — are used by KfW and GIZ in developing projects and programmes, monitoring their progress, and steering and reporting on them (OECD, 2015[3]). This is particularly useful for projects and in field settings. The Development Effectiveness Rating (DERa) system,1 introduced in 2017 by KfW subsidiary, Deutsche Investitions- und Entwicklungsgesellschaft mbH (DEG), is used throughout the project cycle to measure contributions to development based on five outcome areas (DEG, n.d.[4]). It is being taken up by other development finance institutions such as the Development Bank of Austria (OECD, 2020[5]).

The joint procedural reform introduced intervention logic to bilateral programmes and links project outcomes to portfolio impacts. All programmes and projects are now required to use a results matrix and intervention logic, enabling implementing agencies to better design projects as part of bilateral portfolios and facilitating adaptation to challenges during implementation. This will contribute to BMZ’s ability to manage country and sector portfolios. While special initiative interventions that form part of a country portfolio apply this approach, it is not yet applied comprehensively to all special initiative activities (Federal Government, 2020[6]).

Germany’s overall objectives are not always articulated in ways that can be measured and assessed. In its latest development strategy paper, Development Policy 2030:/ New Challenges, New Solutions, BMZ describes the significant global challenges facing the world and undertakes to respond to these (BMZ, 2018[7]). However, the strategy is largely input-based, describing actions Germany and its partners will take rather than articulating the results to which these efforts will contribute.

BMZ does not systematically articulate the results it aims for in special initiatives and country, regional and global programmes. The ministry’s description of the Special Initiative on Training and Job Creation2 outlines the results Germany aims to achieve, as it does in the Strategy for Interlinkages between Water, the Environment and Climate Change (BMZ, 2018[8]). However, BMZ is yet to do this systematically for the majority of its strategies and special initiatives.3 The BMZ 2030 reform strategy envisages a more systematic approach. As it develops thematic, sectoral, country, regional and global strategies and updates or creates special initiatives, BMZ should clearly articulate the results Germany seeks to achieve and how it will contribute to the SDGs in line with the DAC Guiding Principles on managing for sustainable development results (OECD DAC, 2019[9]). Staff will require clear guidance on how to do this.

Enhancement of its integrated data management system, and a broader set of quantitative and qualitative indicators, will help Germany better implement results-based management. Current work within BMZ is focused on complementing binding quality standards with standard indicators, and the ministry plans to enhance its data management system (Federal Government, 2020[6]). While development of quantitative indicators is progressing, introducing qualitative indicators that track progress — including with quality criteria such as human rights, gender, environment and climate — is proving more difficult. Given that its staff are mostly generalists, BMZ might consider recruiting staff with specialist skills in results-based management to help roll out a culture of results, investing more in training staff to manage for results and putting the necessary incentives in place to achieve this.

Results information is used by BMZ for accountability and communication, but not for strategic direction and learning. Aggregated results reporting enables Germany to record the results of engagement by GIZ and KfW in ten areas.4 Since 2017, this information has been used to improve communication and accountability, though using it for steering and learning would greatly benefit German development co-operation.

Germany has strengthened its contribution to evaluation within the international community. BMZ and DEval are active participants in the DAC Network on Development Evaluation — DEval’s Director is one of two vice-chairs — and played an active role in the revision of the DAC evaluation criteria. In addition, DEval is represented in the core management group of the COVID-19 Global Evaluation Coalition and is active in European and international research and evaluation networks.

The approach to evaluation across Germany’s development co-operation system continues to improve. BMZ is currently developing guidelines for evaluating development co-operation. GIZ and KfW continue to build upon their strong in-house evaluation capability. DEval’s focus is on development policies, objectives and instruments. Its institutional capacity has grown since the 2015 peer review, and its standing within the German system has improved. However, it continues to face a number of challenges, including staff retention. DEval’s long-term ambition is to establish itself as a knowledge institute for German development co-operation.

Significant resources are invested by GIZ and KfW in evaluating development projects and programmes. Each year GIZ’s evaluation unit steers evaluations of a sample of BMZ-funded activities. Corporate strategic evaluations, cross-section evaluations and evaluative studies are also undertaken along with evaluations of co-financing and those contracted by external or internal commissioning parties. KfW conducts ex-post evaluations of projects, including those funded by BMZ and BMU, cross-sectional evaluations and is increasingly undertaking thematic evaluations. Both implementing organisations publish reports summing up learning (GIZ, 2020[10]) (KfW Group, n.d.[11]).

There is room to consider how best to allocate resources for evaluation across the German development co-operation system. Mechanisms exist for co-ordination and collaboration amongst the entities engaged in evaluation. Nevertheless gaps remain. There is a need for systematic evaluation of special initiatives and country, regional and global programmes. There is also need to determine whether thematic and sectoral approaches, across all German development actors, are achieving more than the sum of the parts. Noting that portfolio management frequently focuses on individual priority areas that are often planned and implemented independently and without systematic consideration of interactions between them, DEval has proposed addressing this issue by undertaking country portfolio reviews. While a good start, systematic evaluation of country (and other) programmes is/ more likely to generate evidence for Germany to use in improving its development co-operation programmes (Hartmann and Vorwerk, 2019[12]).

Additional considerations could enhance the quality of evaluations. DEval policy briefs, in addition to sharing methods and standards used in its own evaluations (German Institute for Development Evaluation, 2018[13]), offer valuable suggestions for Germany and others to improve the quality of development evaluations as do the evaluation units of implementing organisations.5

Decisions about topics for evaluations are independent of policy-making and delivery functions within the German development co-operation system. In GIZ and KfW, the evaluation functions remain independent of programming departments, reporting directly to the management of both organisations. GIZ reformed its approach to evaluation in 2018 with a view to supporting evidence-based decisions, ensuring transparency and accountability, and contributing to organisation learning (GIZ Evaluation Unit, 2018[14]). KfW’s evaluation unit focuses on ex-post evaluations of the impact of projects and programmes; it conducts the evaluations itself, and also assigns either KfW employees who have never been involved in the project and/or programme or external evaluators.6

Concerns about DEval’s independence have proved unwarranted. While BMZ provides DEval’s steadily increasing budget and approves its annual programme of evaluations, DEval is ultimately responsible for the programme. Internal and external proposals for strategic evaluation topics are collated by DEval and ranked using specific criteria. A draft programme is sent to DEval’s Advisory Board and to BMZ and this is either adopted or declined in full; to date, the programme has never been declined.

More detailed management responses would enhance the value of evaluations for German development co-operation. BMZ’s responses to findings and recommendations of DEval evaluations are quite generic, welcoming findings and describing actions that may be taken by way of response. These published management responses would be more useful if they also included specific action to be taken in response to each recommendation, who will take the action and the deadline for doing so, as is the case for more detailed implementation plans.7

GIZ and KfW could improve their approach to evaluation in partner countries. Both organisations could do more to facilitate local participation in the evaluation process, including through the use of participatory processes in evaluation.

DEval and GIZ invest in strengthening evaluation capacity. BMZ and DEval participate in the Global Evaluation Initiative.8 DEval invests in evaluation capacity development in Latin America (Box 6.1) but might consider expanding this to other regions given Germany’s much stronger presence in Asia and Africa. GIZ’s evaluation teams include international and national evaluators. However, it could contribute better to building institutional evaluation capacity in partner countries, as it has done in Latin America by facilitating greater participation in its evaluations by counterparts.

Germany benefits from strong research capacity. The German Development Institute draws on domestic and international expertise in development research and is committed to working with emerging economies from the global South. It initiated a T20 Africa Standing Group as part of Think T20 during Germany’s G20 presidency in 2017.9

Germany has systems in place for managing sectoral and thematic knowledge and learning across its development co-operation system. Thematic issues are fed in to GIZ’s decades-old sector network communities of practice and disseminated to partners, civil society institutions and project managers. GIZ sector specialists play a key role in contributing learning during the design of new initiatives. KfW and GIZ have competence centres and exchange platforms that staff draw on and that enable knowledge sharing, including within regions, as seen in Tunisia (Annex C). Frequent reflection sessions are held with BMZ’s regional and sector units, including on evaluation findings. However, more could be done to broaden these, including by using digital means of communication. KfW recently rolled out an application enabling its database to be searched by country and sector. This allows staff to directly access all lessons learnt from past evaluations when designing new interventions, tailored to the specific case.

Results information, evaluation findings and lessons are not yet disseminated systematically within the German development co-operation system. Evaluation reports are published online by GIZ, KfW and DEval, offering the opportunity for findings and lessons to be drawn upon within and across institutions. Improvements in results-based management, particularly at portfolio, thematic and country levels, should enable Germany to better capture, store and disseminate results information; improvements in information management, data and information technology systems will be key to achieving this. Nevertheless, extending this from individual institutions to the system as a whole remains a challenge. BMZ and DEval are piloting a process to monitor the implementation of evaluation recommendations, specifically how intended steps for transferring recommendations into specific actions have actually been implemented.

References

[19] BMZ (2021), “Initiative area One Health in development cooperation”, BMZ Strategies Paper, No. 1, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, https://www.bmz.de/en/publications/topics/health/Strategiepapier550_one_health_en.pdf (accessed on 8 February 2021).

[1] BMZ (2020), BMZ 2030 Reform Strategy. New Thinking, New Direction, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, http://www.bmz.de/en/publications/type_of_publication/information_flyer/information_brochures/Materilie520_reform_strategy.pdf.

[20] BMZ (2020), Helping Refugees Build a Future: The Special Initiative on Displacement, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, https://www.bmz.de/en/publications/type_of_publication/information_flyer/information_brochures/Materialie315_flucht.pdf.

[8] BMZ (2018), BMZ Strategy for Interlinkages Between Water, the Environment and Climate Change: Synergies and Conflicts Between Goals, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, http://www.bmz.de/en/zentrales_downloadarchiv/web-apps/wasser/Strategiepapier435_01_2018.pdf (accessed on 8 February 2021).

[7] BMZ (2018), Development Policy 2030: New Challenges, New Solutions, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, https://www.bmz.de/en/publications/type_of_publication/strategies/Strategiepapier452_10_2018.PDF (accessed on 3 February 2021).

[4] DEG (n.d.), Monitoring Development: Development Effectiveness Rating (DERa) - Brief Description, Deutsche Investitions- und Entwicklungsgesellschaft mbH (DEG), Cologne, http://www.deginvest.de/DEG-Documents-in-English/About-us/What-is-our-impact/Policy-brief_EN.pdf (accessed on 9 February 2021).

[6] Federal Government (2020), Memorandum for the DAC peer review of Germany 2020/2021 (unpublished).

[13] German Institute for Development Evaluation (2018), “DEval Methods and Standards 2018: Standards for DEval Evaluations”, DEval Policy Brief, No. 6/2018, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policy%20Brief_6.18_Public_Opinion_Agenda2030_EN.pdf (accessed on 9 February 2021).

[10] GIZ (2020), Evaluation Report 2020: Using Knowledge, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), Bonn, https://www.giz.de/en/downloads/giz2021_en_GIZ-Evaluation%20report%202020_USING%20KNOWLEDGE_.pdf (accessed on 30 March 2021).

[14] GIZ Evaluation Unit (2018), GIZ’s Evaluation System: General Description, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), Bonn, https://www.giz.de/en/downloads/GIZ_EVAL_EN_general%20description.pdf (accessed on 9 February 2021).

[18] Hartmann, C., F. Gaisbauer and K. Vorwerk (2017), Evaluation of the DeveloPPP.de Programme, German Institute for Development Evaluation, Bonn, http://www.deval.org/files/content/Dateien/Evaluierung/Berichte/DEval_develoPPP_Bericht_EN_web_final.pdf (accessed on 23 February 2021).

[12] Hartmann, C. and K. Vorwerk (2019), “Country Portfolio Reviews: A Tool for Strategic Portfolio Analysis in German Development Cooperation”, DEval Policy Brief, No. 4/2019, German Institute for Development Evaluation, Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/2019/DEval_Policy%20Brief_4.19_CPR.pdf (accessed on 9 February 2021).

[11] KfW Group (n.d.), 15th Evaulation Report, 2017-2018: For greater impact in small towns and cities, KfW Group, https://www.kfw-entwicklungsbank.de/PDF/Download-Center/Dokumente-Evaluierung/15_Englisch.pdf.

[17] Krapp, S. and E. Geuder-Jilg (2018), “Evaluation capacity development: A systematic project approach by DEval in Latin America”, DEval Policy Brief, No. 7/2018, German Institute for Development Evaluation, Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policy%20Brief_7.18_Foceval_EN_web.pdf (accessed on 14 February 2021).

[16] Krapp, S. and S. Klier (2016), “Evaluation standards for Latin America and the Caribbean”, DEval Policy Brief, No. 06/2016, German Institute for Development Evaluation, Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policybrief_6-2016_2-seiter_EN_WEB.pdf (accessed on 14 February 2021).

[5] OECD (2020), OECD Development Co-operation Peer Reviews: Austria 2020, OECD Publishing, Paris, http://dx.doi.org/doi.org/10.1787/03b626d5-en.

[2] OECD (2019), Sustainable Results in Development: Using the SDGs for Shared Results and Impact, OECD Publishing, Paris, http://dx.doi.org/dx.doi.org/10.1787/368cf8b4-en.

[3] OECD (2015), OECD Development Co-operation Peer Reviews: Germany 2015, OECD Development Co-operation Peer Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264246133-en.

[9] OECD DAC (2019), Managing for Sustainable Development Results: Guiding Principles, OECD Publishing, Paris, http://dx.doi.org/10.1787/44a288bc-en.

[15] Peréz-Yarahuán, G. (2020), National Monitoring and Evaluation Systems: Experiences from Latin America, Center for Learning on Evaluation and Results, Latin America and the Caribbean (CLEAR) and Centro de Investigación y Docencia Económicas (CIDE), https://clear-lac.org/3d-flip-book/national-monitoring-and-evaluation-systems/.

Notes

← 1. DEG uses a theory of change that tracks client activity, client outputs, the desired development effects or societal outcome, and societal impacts. The DERa tool uses five outcome categories: decent jobs, local income, market and sector development, environmental stewardship, and community benefits. The first three categories assess what was achieved; the other two assess how these effects were achieved.

← 2. The Special Initiative on Training and Job Creation aims to create 100 000 jobs and 30 000 training places in Africa, improve local working conditions, and promote sustainable economic growth. For further information, see www.bmz.de/en/issues/sonderinitiative_ausbildung_beschaeftigung/index.html.

← 3. The One Health initiative contains four output indicators (e.g. number of instances where Germany has supported development of One Health strategies and programmes, number of people trained and number of people reached) and outlines action areas (BMZ, 2021[19]). The Special Initiative on Displacement reports on results (BMZ, 2020[20]) and the Strategy on Transitional Development Assistance requires results matrices for funded projects; however, neither articulates expected results.

← 4. The areas covered are drinking water supply; primary and secondary education and vocational training; basic health; energy; reducing the causes of flight and assisting refugees; annual savings in greenhouse gas emissions and adaptation to climate change; employment; rural development, agriculture and food security; social security; and political participation and public administration services.

← 5. DEval policy briefs examine a range of considerations, including how to evaluate sustainability and the use of causal mechanisms, geodata impact evaluation, and text mining. For more information, see www.deval.org/en/policy-briefs.html. The KfW evaluation unit is working with Agence Français de Développement on the use of geodata for evaluation in development co-operation. See www.mapme-initiative.org.

← 6. For details of KfW’s financial co-operation Evaluation Department, see www.kfw-entwicklungsbank.de/International-financing/KfW-Development-Bank/Evaluations/Principles/. Assigning staff from programming departments is a key knowledge management tool for KfW and ensures feedback of lessons learnt into those departments.

← 7. By way of example, the DEval evaluation of the develoPPP.de programme, authored by Hartmann, Gaisbauer and Vorwerk (2017[18]) contained 36 recommendations; it is available at www.deval.org/files/content/Dateien/Evaluierung/Berichte/DEval_develoPPP_Bericht_EN_web_final.pdf. BMZ’s response was three pages long: https://www.bmz.de/en/zentrales_downloadarchiv/erfolg/BMZ_response_to_the_DEval_evaluation_developpp.pdf. BMZ publishes its responses to DEval evaluations at https://www.bmz.de/en/ministry/evaluation/Evaluation/evaluierungsberichte-stellungnahmen/index.html.

← 8. The Global Evaluation Initiative aims to develop “country-owned, sustainable monitoring and evaluation frameworks and capacities to promote the use of evidence in public decision-making, enhance accountability, and achieve better results”. For additional information, see www.globalevaluationinitiative.org/about-gei.

← 9. For details about the T20 Africa Standing Group, see https://www.die-gdi.de/en/t20africastandinggroup/. For additional information about the German Development Institute, see www.die-gdi.de/en/.

Metadata, Legal and Rights

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.

© OECD 2021

The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.