copy the linklink copied!6. Austria’s results, evaluation and learning

Austria’s results system could focus more on its overall contribution to development

Austria has not yet fully realised its intention to enhance management for development results.1 Development co-operation objectives are set for each federal ministry in federal budget documents, but there is no comprehensive, whole-of-government statement outlining the expected results of Austria’s official development assistance (ODA) (Chapter 2). In addition, success at the ministry level is measured using input-related indicators rather than by assessing outcomes or impact.

In funding Austrian Development Cooperation (ADC), the Federal Ministry of Europe, Integration and Foreign Affairs (Ministry of Foreign Affairs, MFA) is expected to achieve sustainable poverty reduction; consolidation of peace and human security; preservation of the environment; and gender equality, including addressing the needs of children and people with disability. However, rather than assessing the outcomes or impacts in these areas, the ministry is required to report on input-related corporate indicators, such as the share of projects and programmes that promote gender equality or that meet ADC quality standards (MOF, 2018[1]).2

The ODA matrix in the three-year programmes on Austrian development policy for 2013-2015 (MFA, 2012[2]) and 2016-2018 (MFA, 2016[3]) described the outcomes Austrian Development Cooperation sought to achieve. However, neither the expected outcomes of multilateral co-operation, nor of the broader bilateral efforts of other Austrian stakeholders were described. This results in a partial picture of Austria’s overall contribution to the SDGs and to sustainable development in its partner countries and territories. The 2016-2018 document and its 2017 update (MFA, 2017[4]) did, nevertheless, include in the ODA matrix a list of additional Austrian stakeholders supporting priority themes in countries, territories and regions supported by ADC.

This fairly comprehensive approach to outlining measurable results has not been carried forward to the 2019-2021 three-year programme document, which does not contain an ODA matrix. The document does, nevertheless, link Austria’s five thematic priorities to the 2030 Agenda for Sustainable Development, its focus on leaving no one behind, and the Sustainable Development Goals (SDGs). This provides Austria with the basis for using the SDGs as a shared framework for results (Zwart, 2018[5]). Such an approach would define outcome indicators that are aligned to the SDGs and ensure synchronisation with partner governments’ results frameworks.

Individual Austrian institutions have improved their results-based management

In 2017, the Austrian Development Agency (ADA) enhanced its approach to managing project and programme results, introducing a results-focused monitoring and tracking system into its electronic aid management system. This includes a selection of SDG indicators, many of which are disaggregated by gender, age and level of vulnerability. Information about disability is obtained via the use of a policy marker.

Country strategies now include results matrices showing the link between Austria’s support, partner countries’ sustainable development objectives and the SDGs. While Austria aims to use partner countries’ data and systems where possible, the approach taken to programming and choice of implementing partners constrains its ability to do so (Chapter 5). At the project level, logical framework matrices are systematically included in project design documents, and reported against, enabling ADC to monitor the chain of expected results from outputs to eventual impact. This is good practice.

The Development Bank of Austria (OeEB) has recently introduced a development effectiveness rating tool (DERa) developed by Deutsche Investitions-und Entwicklungsgesellschaft mbH (DEG),3 which allows the bank to measure the development effects of its interventions.

Austria joins other donors in encouraging multilateral and regional institutions to apply results-based management in their operations and draws on their reporting. The Federal Ministry of Finance, for example, draws on performance reporting against the multilateral development banks’ corporate scorecards and results frameworks.

Austria could be clearer about how its activities contribute to sustainable development

Austrian institutions are using more readily available information about results to improve their programme management. However, it could make greater use of this information to communicate with the Austrian Parliament, public and partner governments about Austria’s contribution to the SDGs and to sustainable development in partner countries. Annual updates of the ODA matrix, as envisaged in the 2016-18 three-year programme, have not eventuated, constraining parliament’s ability to exercise its oversight role.

A new evaluation policy will apply to the five key development actors

Austria is finalising an inter-ministerial evaluation policy which will apply to ADA, OeEB and the ministries of finance, foreign affairs, and sustainability and tourism, and other development actors that choose to adopt it (MFA, 2019[6]). This is a very positive step towards achieving evaluation coverage of the entire aid system, as recommended in the 2015 peer review (OECD, 2015[7]).

The draft policy is comprehensive. It draws on evaluation principles, criteria and standards developed by the Development Assistance Committee (DAC), the German-speaking Evaluation Society (DeGEval),4 the United Nations Evaluation Group5 and the multilateral development banks’ Evaluation Cooperation Group.6 It outlines ten evaluation principles,7 including impartiality and independence from policy-making and programming functions. In evaluating its projects and programmes, Austria will look at the criteria of relevance, effectiveness, efficiency, coherence, impact, and sustainability, in line with the DAC evaluation criteria.8 Humanitarian evaluations will also consider connectedness, coverage and co-ordination. It will not be mandatory to apply all criteria. Instead, criteria will be selected based on their appropriateness to each evaluation. This is good practice. Roles and responsibilities are defined for each of the five development actors.

In ADA the evaluation function is kept independent from policy, programming and delivery.9 However, while the evaluation functions in OeEB10 and the Ministry Foreign Affairs11 are separate from programming and delivery, they sit alongside policy and strategy, which does not allow for sufficient independence. The ministries of finance and foreign affairs rely on evaluations undertaken by their multilateral partners, which is good practice.

Evaluations are prioritised, but could be more selective

ADA is responsible for evaluating Austrian Development Cooperation activities, and requires all projects to be evaluated.12 In addition, one or two strategic evaluations are done each year. By way of example, the 2019-20 Evaluation Plan, which is developed by the Ministry of Foreign Affairs in close collaboration with ADA, envisages that two country strategies will be evaluated in 2019 and that strategic evaluations in 2020 will focus on the human rights approach, a systematic review of results of the whole-of-government approach in ADC, and the evaluability of gender activities (ADC, 2019[8]). Implementation of the recommendations of strategic evaluations are monitored on a regular basis and reported to ADA’s Supervisory Board on an annual basis. This establishes a direct feedback loop to an oversight body in an effort to increase commitment to follow-up on recommendations, and promote institutional learning and use of evaluation findings at all levels.

While this approach is commendable, ADA could make better use of limited resources. Rather than evaluating every project it could be more selective about what is evaluated and draw more on the results of monitoring by implementing partners, as seen in Kosovo (Annex C). This point was raised in the report of a recent meta-evaluation of project and programme evaluations undertaken by ADA between 2016 and 2018 (Silvestrini and Bäthge, 2019[9]). Additional issues for the Ministry of Foreign Affairs and ADA to consider from this report include:

• ensuring that budgets are sufficient to achieve the intended scope of evaluations

• ensuring that terms of reference are adequate

• ensuring a proper inception phase

• ensuring evaluation results are disseminated and discussed.

Evaluations could assess challenges facing Austria’s development system

While the draft evaluation policy tasks each institution with evaluating its own development co-operation efforts, it does not explore the extent to which Austria achieves synergies across its development cooperation system as a whole. This is something that the Ministry of Foreign Affairs might consider as part of its responsibility for co-ordinating Austrian development policy. Resources might usefully be reallocated to increasing the number of strategic evaluations, including looking at challenges across Austria’s development system as a whole (Chapter 4).

Institutional learning remains a challenge within the Austrian system

The fragmented nature of Austria’s development co-operation system makes institutional learning a particular challenge. Informal sharing of information does occur amongst Austrian stakeholders – for example ADA invites other stakeholders to attend learning events where it presents projects, evaluations and results. However, this approach is limited in its ability to disseminate lessons to a wide range of staff across Austrian development institutions and co-ordination offices, let alone to partners.

ADA has improved its dissemination of evaluation results. Evaluation reports are now systematically published on ADA’s website,13 and since 2017 these have been accompanied by evaluation briefs. Recommendations are followed up systematically by management and findings inform policy, as seen in the work by the Ministry of Foreign Affairs and ADA to prepare the latest round of country strategies. ADA’s knowledge management unit maintains a repository of information which is disseminated through newsletters and “good to know” messages. It also organises an annual conference, which serves as a retreat for ADA staff and the MFA’s Directorate General for Development Co-operation, facilitating mutual reflection, knowledge exchange and learning on questions of importance for global development co-operation and ADA as an organisation. While internal lessons are available, the system does not yet capture lessons from implementing partners. The Development Bank of Austria posts summaries of evaluations on its website,14 but it could follow ADA’s example and publish the full reports, including methods and findings.