3. Monitoring the Results of the Strategic Plan of Nuevo León

A whole-of-government legal framework exists for monitoring the Strategic Plan 2015-2030 and the State Development Plan

There is a solid legal basis for monitoring the Strategic Plan 2015-2030 (SP) and the State Development Plan (SDP) in Nuevo León. Article 18 of the Strategic Planning Law mandates monitoring and evaluation to be carried out for both planning instruments (State of Nuevo León, 2014[2]). Moreover, monitoring of the SDP is also embedded in the guidelines. The “General Guidelines of the Executive Power of the state of Nuevo León for the Consolidation of the Results-based Budget and Performance Assessment System” stipulate that “the agencies, entities and administrative tribunals, shall send quarterly reports to the Secretariat, in accordance with the provisions issued by the latter, reports on the performance of the budgetary exercise and programmatic progress.” (State of Nuevo Leon, 2017[4]), which have to be aligned to the objectives of the SDP (article 9 of the guidelines). However, the quality of these quarterly reports is not always high enough to help dependencies track spending decisions along budgetary programmes.

Some OECD countries have similarly adopted clear legal frameworks for performance monitoring. In the United States, for example, the Government Performance and Results Modernization Act of 2010 mandates the government to define government-wide performance goals, as well as each agency to define sectoral goals. In Canada, the Management Accountability Framework was implemented in 2003 to hold heads of departments and agencies accountable for performance management, and to continuously improve performance management (see Box 3.1).

Having a clear legal framework, in the form of a primary law accompanied by guidelines, would be useful in order to underline the importance that the state of Nuevo León attributes to this practice, within the council and across government. Nevertheless, the presence of a legal-basis for monitoring alone is not enough. A robust monitoring system needs to specify the actors involved, their mandates, the timeline, the methodology and tools for monitoring. In the case of Nuevo León, such a monitoring system would need to clarify the articulation of the monitoring for the SP and the SDP, as was done in the US for the 2010 Government Performance and Results Modernisation Act for instance (see Box 3.2).

Box 3.2. The United States Government Performance and Results Modernization Act

The United States undertook reforms to improve the central government’s performance management system, to foster a dialogue on performance across government, and to deliver targeted improvements on high-priority cross-government initiatives. In 2010, the Government Performance and Results Modernization Act (GPRAMA) was adopted to strengthen the efficiency of the system for government agencies to report their progress. The GPRAMA provides enhanced performance planning, management and reporting tools that can help decision makers to address current challenges.

A main objective of this act is to demonstrate the value of performance information and its usefulness in management decisions. A key element to drive such an effort has been to establish leadership roles and performance improvement responsibilities for senior management, making it their job to engage the workforce in a performance-based discussion as well as assuming accountability for agency performance and results.

The Modernisation Act requires every agency to identify two to eight (usually about five) Agency Priority Goals (APGs), which inform the setting of the APG Action Plan. The AGPs are set every two years and are subject to quarterly performance reviews by the chief operating officer (COO) (usually deputy) and the performance improvement officer (PIO).

In addition to agency-level priority goals, federal cross-agency priority goals are set every four years, with performance assessed via quarterly reviews by the Office of Management and Budget (OMB) Director and a Performance Improvement Council. Finally, agency-level strategic goals and objectives, which inform the setting of the AGPs, are subject to annual strategic reviews by agencies and the OMB (see figure below).

Strategic reviews are annual assessments that synthesise the available performance information and evidence to inform budget, legislative and management decisions. Initial results of the first round of reviews are promising. Many agencies were able to identify strategic objectives with relatively weak evidence and thus identify areas for improving metrics. Moreover, agencies reported that the strategic reviews reduced inconsistencies by bringing programmes together to discuss cross-cutting, strategic issues. Agencies also reported new initiatives to begin directly aligning activities with strategic goals and objectives. Furthermore, according to the agencies, the majority of agencies’ performance staff were interested and engaged in finding value from strategic reviews.

Figure 3.1. United States’ performance framework

Sources: Based on information from the United States Office of Management and Budget; Implementing Strategic Reviews: A component of the GPRA Modernization Act of 2010, Mark Bussow, Performance Team Lead at U.S. Office of Management and Budget (OMB), presented at the 10th Annual meeting of the OECD Senior Budget Officials Performance and Results Network

The monitoring methodology and tools for the SP are not well-defined

Moreover, Nuevo León does not have macro-level guidance (document, policy) in place on when and how to carry out monitoring, despite the fact that article 19 of the law stipulates that the monitoring and evaluation report should be produced annually. For example, it does not elaborate on how this report ought to be practically prepared. By creating clear obligations without proper guidance, this blurs the institutional framework.

More importantly, the law and its supporting regulation do not outline the existence of a monitoring set-up beyond the production of an annual report. In particular, the following elements are missing:

• the expected frequency of data collection for each level of indicator (SP impact/ outcome indicators or SDP outcome/output indicators);

• the governance and calendar of the performance dialogue between line ministries, the CoG and the council, that is what actors meet to discuss what issues at what frequency and with what intent;

• and the criteria according to which certain issues need to be escalated to a higher level of decision-making (for instance, from line ministry level to CoG level), as well as how to follow-up on decisions.

Despite the lack of a legally defined monitoring process, an informal process has been developed, both for the preparation of the annual report and for the infra-annual performance dialogue between the council and the state public administration. This process has been carried out in full once since the adoption of the Strategic Plan.

The council spearheads the monitoring report process; the commissions receive and review the relevant information from relevant ministries and generate recommendations accordingly. The CoG coordinates the relationship between different ministries and the council (and its commissions) by requesting and collecting information from the ministries on behalf of the council. The ministries respond to the council’s recommendations in an official document delivered to the legislative branch.

Figure 3.2 outlines a schematic view of this process.

Figure 3.2. Process for elaborating the annual report on the Strategic Plan

Note: This is a schematic view of this process. The process has not always been entirely followed in the past. For instance in 2018, the Council did not receive an answer from the CoG regarding the evaluation.

Source: Authors.

Before 2018, there was also no standard procedure for preparing the monitoring report across commissions; it was only during the second evaluation that the council attempted to harmonise procedures by standardising the questionnaires sent to ministries and the crosschecking procedures.

Likewise, an informal set-up has been developed whereby the state public administration reports to the council – and to each thematic commission in particular – on an infra-annual basis. In this exercise, working methods across commissions have been very heterogeneous, with each commission deciding, more or less collaboratively with the state public administration, on the number of meetings per year, the level and type of information they seek to receive and the potential outputs of these meetings. For example, the Education sub-commission did not have any meetings in 2018.

As a result of this informal set-up, the quality of exchanges between the state public administration and the council has often rested on the existence of trusting personal relationships between the Secretaries and the members of the commissions, rather than on sound and clear working methods and mandates. Moreover, the lack of standardised tools for data collection and analysis has meant that the intended output of the monitoring sessions between the commissions and the secretaries is not only heterogeneous, but also sometimes unclear. This may lead to a lack of incentive to collaborate, especially from the line ministries who do not have an explicit mandate and allocated resources to do so.

A clarification of the monitoring set-up, methodology and tools may allow for greater cooperation and coordination between the council and the state public administration, and the use of monitoring results.

Monitoring evidence can be used to pursue three main objectives (OECD, 2019[3]):

• it contributes to operational decision making, by providing evidence to measure performance and raising specific questions in order to identify implementation delays or bottlenecks;

• it can also strengthen accountability related to the use of resources, the efficiency of internal management processes, or the outputs of a given policy initiative;

• it contributes to transparency, providing citizens and stakeholders with information on whether the efforts carried out by the government are producing the expected results.

Each of these goals implies using monitoring data in a distinct manner. Firstly, for monitoring evidence to serve as a management tool, it must be embedded in a performance dialogue that is conducted regularly and frequently enough that it allows practitioners and decision-makers to identify implementation issues, determine resource constraints and adapt their efforts/ resources in order to solve them. Such an exercise is closely tied to policy implementation and management. In the case of Nuevo Léon, this exercise should be conducted within the state public administration, ideally between the highest levels of the centre of government and the operational levels of the dependencies.

Nuevo León has both a Strategic Plan and a State Development Plan. Given the plurality of strategic frameworks, it is important to ensure that the monitoring processes are streamlined and aligned to minimise the additional burden occurring with each strategy (Vági and Rimkute, 2018[11]). As a result, it is important to clarify that the state public administration should conduct this performance dialogue regarding both the SP and the SDP simultaneously. Given the more strategic nature of the Strategic Plan, the performance dialogue for the Strategic Plan could be held on a biennial basis, depending on the theory of change of every strategic objective and information availability. Two main functions can be identified in the context of this performance dialogue: a coordination and promotion function, which can be naturally conducted by the centre of government, and a data collection, analysis and use function, which can be the responsibility of the line ministries.

This performance dialogue could be centred around the action plans – as recommended in chapter 2 – to monitor the lines of action (processes) and strategies (outputs) which are common to the SP and the SDP, at the level of line ministries, on a quarterly basis1. This dialogue would create an incentive for line ministries to resolve implementation issues at the technical or sectorial level through a gradual escalation process. If the problem is still unresolved after two quarters, it could be referred to the CoG twice a year for decision. Finally, any implementation issues that would require cross-ministerial coordination and mobilisation of additional resources may be referred to cabinet meetings on a yearly basis (see Figure 3.3).

Figure 3.3. Articulation of different levels of indicator dashboards

Source: Authors

These cabinet meetings may also be an opportunity to update and analyse the outcome/impact indicators included in the SP and the SDP. Indeed, it is important to link process and output indicators to the progress of outcome and impact indicators to actually track results and not just activities (The World Bank, 2017[12]). Figure 3.4 demonstrates this process over a one-year period. The next section of this chapter will focus on the relevance and quality of indicators.

Figure 3.4. Proposed performance dialogue between the Line ministries, the CoG and the cabinet meetings

Source: Authors

As previously mentioned, monitoring evidence can also serve as a tool for remaining accountable to stakeholders in the implementation of the Strategic Plan. In order to do so, in preparation of the annual cabinet meeting on the SDP and the SP, the state public administration and the council could conduct a collaborative review of the plans’ indicators, which would be presented to the cabinet. This collaborative review, which could build on the four performance dialogue sessions held every year, could be conducted jointly by the state secretariats and the council’s commissions to review the data to be presented to cabinet, as well as give the opportunity to the council to serve as a sounding board for the administration. Indeed, while attempting to interpret the data and understand implementation gaps, the administration could seek the advice of the council. This review would be an opportunity for the council to offer the state public administration different insights on implementation gaps identified during the performance dialogue. Finally, given the council’s multi-stakeholder composition, it could also offer implementation support to the cabinet where relevant and possible.

In order to carry on the performance dialogue and contribute to better accountability, the state public administration could consider uploading and updating the indicators of the SDP and the SP in a dashboard. This dashboard would include information concerning both the implementation of the action plan recommended in chapter 2, as well as that of the SP and the SDP outcome/impact indicators, but only the action plan indicators will be updated quarterly – the indicators from the SDP and the SP may only be updated annually or when new information is available. Monitoring dashboards are intended as management and decision-making tools, and therefore offer a performance narrative that allows for the identification and analysis of implementation gaps and potential solutions. To develop these dashboards, the state public administration could draw on a range of good practices. For instance, the current citizen councillors who belong to the private sector could share their best practices with the public servants in regard to preparing dashboards and monitoring strategies.

Moreover, the commissions may want to clarify their working methods for conducting this annual review. In particular, the council could create a guide for commissioners and for their technical secretariats in order to define the agenda for review sessions, the role of the commissions, the tools they may use, and how decisions should be taken in regard to this annual review (see chapters 4 and 5 for a more general discussion of commissions’ working methods).

Finally, monitoring evidence serves to foster transparency, in particular to communicate to citizens. The centre of government could choose to produce a communication report on the SP. This communication report, which could be updated every three years, would replace the current “hybrid” monitoring/evaluation report conducted by the council on an annual basis. It would be designed primarily as a communication tool (see section on use of monitoring evidence for a further discussion of this leaflet). Moreover, it is important to communicate more regularly with citizens, notably through an up-to-date website showcasing the monitoring information of the SDP and the SP (see section on promoting the use of monitoring for a more detailed discussion of this recommendation).

The lack of a systematic framework which would link indicators and opportunity areas in the SP hinders its monitoring

Developing performance indicators, their baseline and targets is an important stage in the strategy development process. Although article 9 of the Strategic Planning Law sets among the council’s responsibilities the establishment of criteria for the monitoring and evaluation indicators, there does not seem to be an explicit and systematic framework for their design.

Firstly, there is a lack of systematic linkage between each layer of the Plan and the indicators making it hard for stakeholders to monitor the progress in terms of aspirations/opportunity areas/strategic lines (see Table 3.3). While most commissions have identified a set of indicators, they are not presented in a way that clearly indicates their connection with elements of the Plan (opportunity areas, strategic lines or initiatives). All indicators for the commission/sub-commission are grouped together in a table at the end of the section related to that thematic area. Some sub-commissions2 have introduced indicators that cannot be directly or indirectly linked to either an opportunity area or a strategic line/initiative.

Moreover, not all commissions have defined indicators. For instance, four sub-commissions3 of the Human Development Commission have not defined any indicators, suggesting that there is no way for their members to monitor and evaluate the achievement of the policy goals set out in that part of the Strategic Plan. Similarly, the Economic Development Commission is the only one that has defined indicators for both priority opportunity areas and strategic lines: 40% of opportunity areas have no indicators relevant to them.

Therefore, explicitly linking each indicator to an opportunity area (strategic objective as recommended in chapter 2), first and foremost visually in the strategy document, will be essential to clarify the monitoring structure of the Plan. This exercise should be undertaken by the council.

Such an analysis also suggests that some output indicators may have been linked to outcome-level objectives. It would be preferable to maintain, to a feasible extent, a focus on outcomes in a performance-oriented framework (see example of Scotland in Box 3.4).

Both the Strategic plan and the action plans would benefit from identifying sound and policy actionable indicators

Monitoring a policy, programme or project – such as the strategic plan 2015-2030 or the action plans – implies identifying indicators that are:

• sound, meaning that they are methodologically robust;

• policy actionable, meaning that they correspond to an observable variable that captures the intended direction of change and is under the remit of the actor in charge of implementation.

Firstly, the indicators in the strategic plan 2015-2030 are not always sound. In order for indicators to provide decision makers information on what course of action to take in order to achieve the intended policy objectives, they should be accompanied by information that allows for their appropriate interpretation. That is why, regardless of their typology, all indicators should be presented in a way that provides enough information:

• Description of the indicator: name, unit of measurement, data source and formula.

• Responsibility for the indicator: institution, department or authority responsible for gathering the data;

• Frequency of data collection and update of the indicator;

• Baseline that serves as a starting point to measure progress;

• Target or expected result.

For the most part, the indicators in the strategic plan are explicitly stated, as reflected by the fact that the commissions have chosen to present the indicators in a table often including information on the data source, the baseline and the target values. Nevertheless, some indicators do not have an existing baseline or source responsible for collecting data. The indicators also do not make explicit the institution or person responsible for collecting the data and updating the indicator.

Moreover, the indicators chosen for the Strategic Plan do not all fit the RACER approach (see Box 3.5 for an explanation of this criteria).

For instance, not all indicators are from public or reliable sources. The Strategic Plan also contains indicators that are largely based on international and national statistical data (see Box 3.6 for a detailed description of types of monitoring data). Yet, this type of data is best suited to long-term impact and context indicators, for which the underlying changes are less certain than for outcome indicators (DG NEAR, 2016_[70]). Administrative data and data collected as part of an intervention’s implementation are usually better suited for process, output and intermediate outcome indicators, such as those found in the State Development Plan.

Likewise, the frequency at which some indicators can be collected is not relevant to the yearly monitoring mandate of the council. This is the case for the sub-commission on Social Development that proposes an indicator for “Mujeres en situación de violencia de pareja” using data from the “Encuesta Nacional sobre la Dinámica de las Relaciones en los Hogares (ENDIREH, INEGI)”, which is only collected every five years.4 As a result, 23% of indicators could not be collected in 2018.

Moreover, the current indicators identified for the Strategic Plan are not sufficiently policy actionable, meaning that they do not provide relevant and timely information for decision-making related to the implementation of the Plan. Some indicators in the plan measure actions that fall outside the responsibilities and control of the executive, meaning that they are not actionable or relevant. Indeed, if the Strategic Plan is to be implemented by the executive, its specific actions should therefore be under the remit of this branch of government. Nevertheless, the commission on Security and Justice included opportunity areas and indicators (e.g. “Duración de casos no judicializados”) that fall under the responsibility of the judicial branch.

Similarly, some indicators proposed in the Plan do not have a state-level scope. The sub-commission on health, for example, uses OECD health data for its indicators, which are collected at a national level and therefore cannot directly reflect Nuevo León’s efforts in this area. Nevertheless, in some cases, proxy indicators may be of use when state-level indicators do not exist. This is the case, for instance, for metropolitan area indicators, which cover 90% of Nuevo León’s population.

The council could consider identifying a mix of indicators in order to best monitor the strategic plan

The Council of Nuevo León could consider identifying indicators for the Strategic Plan that are a mix of impact and context indicators, that would be available at state level, calculated on the basis of international and national statistical data – for instance in collaboration with INEGI – as well as outcome indicators calculated on the basis of administrative data or even ad hoc perception survey data. For example, these indicators could be taken from the national survey on government impact and quality prepared by INEGI every other year (INEGI, 2019[19]) (See Box 3.7). Indeed, while output indicators should not be used to measure outcomes and impacts, context and impact indicators say little about the policy levers to achieve the long-term goals associated with the strategic objectives. Instead, identifying intermediate outcomes, which respond fairly rapidly to changes in policy systems but can be linked to longer term objectives, may be beneficial in monitoring the plan (Acquah, Lisek and Jacobzone, 2019[20]).

The council could consider applying the RACER approach in order to assess the extent to which the indicator could support decision-making5. It should include representatives of the different stakeholders, which during implementation have the responsibility of collecting, analysing and reporting on data (DG NEAR, 2016[15]). Each indicator, qualitative as well as quantitative, must correspond to an existing source, be it a statistical source or an administrative one.

Furthermore, in order to monitor the action plans, the state public administration could consider identifying process indicators mostly through administrative sources. Indeed, for this type of information/monitoring, authorities should tend towards regular flows of information, ideally at minimum administrative cost, automatically. The state public administration could consider submitting these indicators for review and comment by the Council of Nuevo León, in order to benefit from the opinion and expertise of a wider range of stakeholders.

The current annual report is not well targeted to its audience

Although crucial to monitoring, collecting data is not sufficient in and of itself; in order to support decision-making and serve as a communication tool to the public, data needs to be analysed (Zall, Ray and Rist, 2004[13]). Indeed, the analysis of the data which serves as the backbone of the monitoring exercise needs to be tailored to the user, focused and relevant (OECD, 2018[22]).

In fact, it is important to identify the types of use for the monitoring information (demonstrate, convince, show accountability, involve stakeholders, etc.). The analysis of data cannot be targeted efficiently without a clear definition of its audience(s) and the types of questions that audience is likely to raise about findings. There are typically two main types of users of monitoring and reporting information:

• External users: citizens, media, NGOs, professional bodies, practitioners, academia, financial donors, etc. They usually seek user-friendly information, which is provided using a simple structure and language, concise text and as visually as possible.

• Internal users: congress, government, ministers, managers and operational staff, and in Nuevo León, the council. They are more interested in strategic information related to overall progress against the objectives and targets, and will seek to understand the challenges encountered in their implementation, as well as requiring indications of action required in response to data findings. Furthermore, the higher the level of decision-making, the more information should be aggregate and outcome-oriented (OECD, 2018[22]).

Currently, these types of users are not clearly differentiated in the annual monitoring report as the report is meant to serve as a transparency tool to communicate vis-à-vis external users (it is published on the website) and as an accountability and decision-making tool for internal users (for instance, the report is to be sent to Congress).

Clearly differentiating between the three different types of monitoring exercises will provide for more fit-for-purpose information

Clearly differentiating between the three monitoring exercises (performance dialogue, the joint review, and the annual report on the SDP) should allow for more fit-for-purpose analyses in each situation. The following section on promoting the use of monitoring evidence will discuss in further detail how to best communicate monitoring information to external users.

While the performance dialogue and the joint review are exercises aimed at an internal audience, they still need to present monitoring evidence in a manner that supports decision-making (i.e. by focusing on strategic implementation challenges, explaining them through analysis and proposing solutions). A robust monitoring dashboard could include the department or agency in charge of updating it, the frequency at which the relevant indicator(s) should be updated, as well as the analysis and action plan (or proposed solutions) in case implementation gaps can be identified. Finally, the dashboard could identify whether or not the issue must be brought to a higher level of decision-making (see Figure 3.3 for the proposed performance dialogue process). Figure 3.5 showcases an example of such a dashboard.

Figure 3.5. Creating user-friendly dashboards to monitor the SP and the SDP

Source: Authors.

More generally, the information included in the dashboards should be analysed and presented in a way that is (OECD, 2018[22]):

• User-friendly, in that the information is provided using a simple structure and language, with visual information;

• Focused, in that the performance information provided is limited to the most important aspects, such as objectives, targets and activities, and is linked to the achievement of results (outcome/impacts). It is therefore important that reports on the implementation of the action plan include both strategic (outcome/impact) and operational (process/ output) information, even if the main object of the meeting is the monitoring of the action plan.

• Relevant, in that only the points requiring decisions or encountering implementation bottlenecks should be discussed, in order to avoid overloading the reader with non-critical information. Only information that helps decision-makers manage the implementation of the strategy and key policy decisions should be provided. Where possible, the reports should include explanations about poor outcomes and identify steps taken or planned to resolve problems.

The state public administration needs to upgrade its capacity to monitor the Plan

Establishing a monitoring set-up that can produce credible, timely and relevant data and analysis requires skills and capacities, which can be defined as “the totality of the strengths and resources available within the machinery of government. It refers to the organisational, structural and technical systems, as well as individual competencies that create and implement policies in response to the needs of the public, consistent with political direction” (OECD, 2008[23]). In particular, two types of skills are crucial:

• Analytical skills to identify indicators, review and analyse data and formulate judgements or conclusions (Zall, Ray and Rist, 2004[13]).

• Communication skills to structure monitoring reports and dashboards in a way that is attractive to the end user (OECD, 2018[22]).

In order to have the necessary skills and capacities, monitoring set-ups require a critical mass of technically trained staff and managers, such as staff trained in data analytics, data sciences, communication, and in at least basic information technology (Zall, Ray and Rist, 2004[13]). All three of these elements rest on the availability of dedicated resources (human and financial) for the monitoring function. These have been framed in a number of relevant cases as “delivery units” or “policy implementation units” and are often located at the centre of government. Yet, currently, the state public administration does not have such resources for monitoring of the Strategic Plan.

A first step in strengthening the capacities of the state public administration may be the creation of dedicated reporting units in each of the line ministries, who would be in charge of collecting and analysing the data in the context of the performance dialogue. Moreover, the centre of government may wish to strengthen its analysis and communication skills with a dedicated delivery unit in order to provide methodological support and direction to the line ministries in preparing impactful monitoring reports.

Quality assurance mechanisms could be developed alongside current quality control

Currently, the data collected for the monitoring report are subject to an audit by a private accounting organisation. This form of quality control (i.e. looking at the end product) serves to provide assurance that the data used for monitoring are valid and reliable, that is to say that the data collection system is consistent across time and space (validity), and that it measures actual and intended performance levels (reliability).

However, quality assurance (looking at the process) is also important. Some countries have developed mechanisms to ensure that monitoring is properly conducted, that is to say that the process of collecting and analysing respects certain quality criteria. In order to do so, countries have developed guidelines, which serve to impose a certain uniformity in this process (Zall, Ray and Rist, 2004[13]). Box 3.8 explores in more detail the difference between quality assurance and quality control.

In Nuevo León, two types of guidelines may be of use. The state public administration could issue guidelines in order to clarify the working methods and tools that will support the performance dialogue to be conducted. In particular, these guidelines could specify quality assurance processes in order to strengthen the quality of the data that are collected in the context of this monitoring exercise, to be applied by every line ministry. Similarly, these guidelines may seek to clarify the criteria for escalating issues from the line ministry level to the CoG level, in order to harmonise this process across government.

Keeping the annual monitoring report public and making data available on the Avanza NL platforms is important to maintain public accountability and engage with citizens

Greater publicity of monitoring results can increase the pressure on decision-makers for implementation and for a more systematic follow-up of recommendations, while providing accountability to citizens concerning the impact of public policies and the use of public funds.

The monitoring report for the Strategic Plan is made public every year on the council online platform (www.conl.mx/evaluacion).Similarly, the government report on the State Development Plan can be found on the government of Nuevo León website, which constitutes a first step in providing information to the public. However, the current monitoring report is not user friendly due to its length. The report includes an executive summary, which is a useful tool for diffusing key messages to citizens. Nevertheless, the executive summary is over ten pages long, separates the main findings from their associated recommendations and does not give a description of major issues encountered in the implementation of the Plan. A shorter and punchier executive summary may be more impactful in reaching stakeholders and the general public alike. Moreover, contrarily to the annual “Informe de gobierno”, the monitoring report is not very visual.

Overall, the format and timeline of the communication report on the advancement of the Strategic Plan could be revised to better fit its primary intended use: to reach a wider audience and inform a variety of stakeholders about the progress of the implementation of the Plan based on tangible and possibly outcome indicators. To that extent, the centre of government could choose to produce a communication report on the Strategic Plan, which could be updated at mid-term and at the end of every gubernatorial mandate, i.e. every three years. This report would replace the current monitoring/evaluation report conducted by the council and would be designed primarily as a communication tool, so it should have a clear summary and be accompanied by clear messages for the press.

The council has also created an online platform, which is currently under construction, which seeks to provide up-to-date information on the evaluation of the Strategic Plan’s indicators. In order to ensure timely updates and minimal use of extra resources, the platform could be directly updated by the state public administration. The platform could focus on the indicators from the Strategic Plan and the State Development Plan in order to ensure more focused communication of the objectives that concern and affect citizens. Importantly, the indicators defined for the State Development Plan should be aligned with those of the Strategic Plan and reflect the underlying theory of change. This could be shown on platforms similar to those created by other Mexican states, such as MIDE Jalisco (see Box 3.9). Avanza NL should thereby seek to remain as simple to use and up-to-date as possible.

The state public administration and the council could consider developing communication strategies in order to increase public awareness and engagement in the Plan

Besides the communication report and the Avanza NL platform, the council and the state public administration could develop other communication tools in order to reach a wide audience. These could include social media strategies, newsletters with focuses on council actions and editorials in local newspapers (World Bank, 2017_[68]), such as the newsletter, social media outlets and editorial column that the Council has on the local media Verificado. It could be useful to this end for the council to allocate a portion of its yearly budget towards communication.

Another way to increase public engagement in the Plan and its monitoring would be to collect periodic feedback on the Strategic Plan and its implementation through a survey of the general population, business leaders and civil society organisations. This survey could be conducted in collaboration with INEGI and the National Survey on Government Quality and Impact (INEGI, 2019[19]), which has experience in conducting opinion surveys on similar topics. It would be good to provide incentives to some of the NGOS to ensure that this could be published.

Increasing the impact of monitoring results will require the development of a performance narrative focused on addressing inconsistencies in implementation

While preparing fit-for-purpose monitoring analysis is important, as it gives users quick and easy access to clear monitoring results, they do not systematically translate to better uptake of the outcomes in decision-making. In fact, it is pivotal that monitoring evidence be presented in a way that is compelling to its audience. Monitoring dashboards should include a narrative on performance, interpreting and using the results (The World Bank, 2017[12]) to understand implementation gaps and propose corrective policy action in a way that creates a coherent and impactful narrative. In order to create such a narrative, it may be necessary to filter the relevant data and to focus the information presented on the most pressing bottlenecks or the reforms with the biggest potential impact. Key messages, takeaways and suggested courses of action should accompany any raw data (i.e. indicators) (Zall, Ray and Rist, 2004[13]).

Increasing stakeholder engagement in the development of the Plan’s indicators matters for monitoring and implementation

Moreover, involving stakeholders in the design of the monitoring set-up can increase the legitimacy of the resulting evidence and ultimately lead to greater impact in decision-making. In fact, cross country evidence shows that it is important to create ownership and manage change in both the public and private sectors when seeking to implement such an ambitious transformation programme (The World Bank, 2017[12]). Involving internal and external stakeholders in the definition of indicators may improve their quality as stakeholders may sometimes be better placed in order to identify which dimensions of change should garner the most attention (DG NEAR, 2016[15]), in order words what output indicators best measure the progression of the causal chain (outcome/ impact).

The council itself is meant to involve stakeholders. Yet, the oversight of the council in the definition of the Plan indicators could have been stronger6. Revising the Plan could be an opportunity for the council to revise the indicators, in order to create buy-in among the implementers, within and outside the administration (through the composition of the council itself). Similarly, the state public administration (each ministry or entity) could consider asking the council for advice when selecting the indicators for the implementation action plan, in order to ensure that there is a common vision for how to operationalize the Strategic Plan and the State Development Plan.

Monitoring for results implies creating feedback loops to institutionalise the use of monitoring results

Formal organisations and institutional mechanisms constitute a sound foundation for use of evidence in policy and decision-making (Results for America, 2017[18]). Mechanisms that enable the creation of feedback loops between monitoring and implementation of policies can be incorporated either:

• in the monitoring process itself, such as through the performance cycle (whereby performance evidence is discussed either at the level of the individual line ministry or at the CoG);

• through the incorporation of performance findings into other processes, for instance the policy-making cycle, the annual performance assessment of senior public sector executives, the budget cycle or discussions in congress.

As discussed in the section clarifying the monitoring set-up, the creation of a performance dialogue could allow practitioners and decision-makers to use monitoring evidence to identify implementation issues, constraints and adapt their efforts/resources in order to solve them. In particular, linking the strategic objectives with individual performance objectives is key to creating incentives for results in the state public administration, particularly at the level of senior public sector executives and leadership. There is a need to ensure the participation of government officials, such as heads of agencies or departments and to ensure that their organisation is contributing to the achievement of high-priority cross-government outcomes such as the Strategic Plan. In Chile, for example, both collective and individual incentives have been used in order to promote public sector performance in line with strategic objectives (see Box 3.10 for a more detailed explanation of this system).

Other uses of monitoring results also include using the data for evaluations, supporting strategic and long-term planning efforts by providing baseline information and communicating to the public (see discussion above). Moreover, monitoring information produced through the performance dialogue could be used to feed into the budget cycle. Given that the performance dialogue should be linked to the monitoring of budget programmes (see previous section on clarifying the monitoring set-up and how these are linked through the action plan), the evidence produced through this exercise could provide useful information to the congress about the efficiency and effectiveness of budgetary spending through spending reviews. According to data from the budgeting and public expenditures survey (OECD, 2007[30]), spending reviews are a widely used tool in OECD countries as part of the budget cycle, which can be informed by monitoring data (Robinson, 2014[31]) (Smismans, 2015[32]) (The World Bank, 2017[12]). Finally, monitoring evidence may be used in the budgetary cycle through performance budgeting practices, as can be seen in Box 3.3.