copy the linklink copied!13. Reality check

This chapter focuses on the fact checking derived from institutional intervention. It presents the principle and international practices to evaluate outcomes, describes OEFA’s view in the designing performance indicators and presents an assessment and recommendations aimed at helping OEFA be at the performance level expected by stakeholders.

Institutions in charge of inspection and enforcement, and the regulatory enforcement and inspection system as a whole, should deliver the levels of performance expected from them – in terms of stakeholder satisfaction, efficiency (benefits/costs), and overall effectiveness (safety, health, environmental protection amongst others,) (OECD, 2014[1]), (OECD, 2018[2]).

Key indicators of performance should be regularly tracked: satisfaction and trust (among regulated subjects, citizens, consumers, amongst others,), efficiency (costs to the budget, burden to regulated subjects) and effectiveness (safety, health, environmental protection amongst others). While there are significant challenges and costs involved in data collection, performance tracking is indispensable for the good functioning of the system and its improvement.

The level of stakeholder satisfaction and trust should be stable or improving. Satisfaction and trust from regulated subjects is to be balanced by the perspective of those that are expecting protection from regulation (citizens, consumers, workers amongst others,) in terms of effectiveness. The level of stakeholder satisfaction should not be the only indicator of success since a lack of information among stakeholders or their bias might play a role when expressing their satisfaction.

Performance in terms of safeguarding social well-being and/or controlling risks should be stable or improving (correcting for possible external shocks). Inspections and enforcement have only an indirect and limited influence on the goals they seek to protect (e.g. safety, health), thus their performance should be assessed in terms of trends and correcting for external shocks. It is also crucial to correct for the quality of data and ad hoc studies when the existing ones are insufficient.

Efficiency in terms of social well-being balanced with costs for the state and burden for regulated entities, should be stable or improving. Inspections and enforcement create costs for the state and burden for regulated subjects. These costs should to be tracked regularly in order to balance them against performance in terms of effectiveness.

copy the linklink copied!Proliferation of indicators and data points

OEFA has set a number of indicators – in particular in OEFA’s Institutional Strategic and Operational Plans, as well as in OEFA’s PLANEFA – aimed at assessing the efficiency and effectiveness of the agency, as well as stakeholders’ satisfaction. Indicators are rather numerous, which may lead to misperception as there are many indicators in different documents, some documents overlap (e.g. PEIs for 2017-2019 and 2019-2022).

Results currently included t in Institutional Memories of OEFA reflect outputs – e.g. number of complaints, types of violations, number of fines, number of administrative measures imposed, non-compliances remedied. Publication of the number of sanctions and other measures imposed contribute to transparency and accountability, but does not reflect the agency’s approach in terms of public welfare, and OEFA has asserted that this is not their purpose. A similar situation applies to most of the results for 2018 contained in the Assessment Report of PEI 2017-2019 – in particular, percentage of business entities inspected, percentage of violations confirmed in the second instance – as well as other indicators, for instance those set in the POI for 2018 and in the PEI for 2019-2022.

Similarly, OEFA’s bulletin (OEFA in Figures) provides statistics and data on the agency’s activities on a quarterly basis, but currently mostly in terms of outputs – not outcome indicators. In addition, the results of measurement are not always easy to find, and it is often unclear to which indicator(s) relate the data that are made available to the public (for instance in OEFA in Figures).

The former is mainly because most indicators are recent introductions (e.g. those set in OEFA’s PLANEFA for calendar year 2019) and have not been measured yet. Furthermore, for the moment, OEFA’s quarterly statistical reports do not differentiate among different types of “results”. OEFA’s yearly and quarterly reports do not only report on indicators – such as the compliance rate – but also give information needed e.g. for accountability purposes (on the activities of the agency, such as the number of comments received during consultation processes).

While the effort made by OEFA to ensure transparency concerning its activities, actions amongst others, is remarkable, the proliferation of numbers, graphs and various information creates some confusion when looking for meaningful data related to OEFA’s performance. OEFA has reported to have acknowledged the need to “clean up”, or better categorise, the information provided in quarterly and annual reports. Results of measurements reflecting outcomes of the agency should start being available at the end of this calendar year. Moreover, in the process for the definition of indicators, OEFA could clarify where and when relevant measurements can be found for each set of indicators.

copy the linklink copied!Newly-introduced performance indicators

Several indicators have been set to measure OEFA’s performance in terms of safeguarding social well-being and/or controlling risks.

PLANEFAs prepared for all EFAs for fiscal/calendar years 2019 and 2020 aim at introducing indicators developed based on a theory of change, as shown in Figure 13.1, Figure 13.2 and Figure 13.3. Workshops on the preparation of these indicators have been/are being delivered.

Figure 13.1. Theory of change used to design the PLANEFA indicators

Figure 13.3. The framework for regulatory policy evaluation

OEFA has developed the methodology to design indicators in PLANEFAs based of a series of sources,1 including the OECD’s Framework for Regulatory Policy Evaluation (2014), as reproduced in Figure 13.3. PLANEFA Guidelines establish that EFAs must have indicators that allow measuring the effectiveness of the performance of environmental inspections and enforcement, such as the reduction of risks and damage prevention with regard to the environmental issue that has been identified, in order to increase social welfare.2

OEFA’s current PLANEFA has impact indicators. Examples of these are: decrease in concentration of contaminants (%), improvement in the population’s perception in the area of OEFAs’ influence, number of hectares with achievement of compliance, number of people directly benefitted by OEFA’s intervention, amongst others, (see Chapter 2 on Evidence-based enforcement and OEFA’s PLANEFA for 2019, p. 36). Indicators in terms of ‘results’ (or outcomes), often in terms of compliance achieved are also included. Then specific indicators are established for each sector of responsibility of OEFA, based on the same method.

We can see in OEFA’s Institutional Strategic Plan a real effort in establishing indicators, based on the four strategic objectives set for the 2019-2022 period. As said in Chapter 6 on Long-term vision, such objectives are 1) strengthening OEFA’s performance; 2) enhance compliance with regulatory requirements among regulated subjects; 3) “modernise” OEFA’s institutional management; and 4) reduce OEFA’s vulnerability with regard to risk of disaster.

The indicator set for strategic objective (2) is the average percentage of compliance with regulatory requirements checked in inspected operators. Then each indicator is combined with several institutional strategic actions to support the expected outcome. To continue with this example, the relevant institutional strategic actions are the following:

(1) Percentage of preventive and responsive 3 environmental assessments serving as technical input to environmental enforcement actions
(2) Percentage of administrative measures imposed at the inspections’ stage complied with by the regulated subjects
(3) Percentage of recommendations in inspections’ reports confirmed by OEFA’s directorate in charge of enforcement and incentives imposition
(4.1) Percentage of decisions determining administrative responsibility confirmed upheld at first instance
(4.2) Percentage of infractions upheld at second instance
(5) Percentage of commitments made by OEFA in dialogue platforms that have been met

This is a useful and well-constructed set of indicators, which aspire to measure compliance, although it does not necessarily measure actual environmental protection.4

However, as already mentioned in Chapter 6 on Long-term vision, such objectives and indicators could be taken further, considering that two main Sectoral strategical objectives set by the Ministry of Environment are a) “improving the conditions of the quality of the environment to protect peoples’ health and ecosystems”, and b) ‘strengthening environmental awareness, culture and governance’. While PLANEFA’s sectoral indicators take more systematically into account fundamental elements such as performance in terms of protection of the environment and the health of the people, the PEI (Institutional Strategic Plan) somewhat fails at getting to the bottom. Percentage of resources devoted to a) preventive and b) reactive actions (with the objective of reducing the latter) should be included.

According to CEPLAN’s Guidebook for Institutional Planning, OEFA should limit itself to objectives that relate, at most, to ‘initial results’ of public policy. From this perspective, a number of objectives, and indicators, set by OEFA in their PEI and PLANEFA go beyond these instructions. The CEPLAN guidelines seem to limit the capacity of OEFA to continue and expand the efforts started in the design and adoption of objectives and indicators in terms of outputs, to focus and the effectiveness of the agency in terms of protection of the regulatory goal. Indicators and data used to assess the performance of the activities of inspection and enforcement authorities should be focused on outcomes and impact – i.e. reducing risks and the number of accidents and other incidents, protecting objects of value. They should help to understand whether activities carried out are useful in terms of regulatory goals. Measuring outputs is not meaningful. It is a circular indicator, which shows the achievement of certain activities. On this topic, see Box 13.1.

copy the linklink copied!

Box 13.1. Measuring outputs and compliance

Measuring outputs

Many regulatory delivery institutions use outputs, and particularly inspection numbers, as a performance indicator. It is, however, a circular reference: increasing the volume of work will automatically lead to assuming performance has increased if the two are equated, and will miss the purpose to assess whether the work was effective or not.

In fact, output measures provide no information as to the timeliness of an enforcement intervention. Measuring only outputs means that the more an agency produces ‘activity’, the higher its ‘rating’ will be, even if it completely fails at its mission. However, this often happens in regulatory delivery, based on an assumption among many that ‘more’ automatically means ‘better’, that more inspections and sanctions will invariably create higher compliance and better outcomes. This is anchored in purely deterrence-based models of compliance, and in the assumption that better compliance is identical with better outcomes (e.g. safety). Moreover, giving incentives to inspectorates to inspect (and use enforcement measures) as much as possible distracts them from considering actual outcomes – both in terms of the public welfare goals they are supposed to help achieve, and of the social and economic impact of their activities.

Measuring compliance

Measuring compliance levels, and using their variations to assess regulatory delivery performance, would appear to relate directly to regulatory mandates and the aim of delivery activities. However, it can be problematic.

First, there is the imperfect link between compliance and desired results: only ‘prescriptive’ technical norms would seem to give the certainty that what is required from the business corresponds to the intent of the regulation. However, these norms leave business uncertain as to how to reach the desired result and put inspectors in a difficult situation due to time-lags or third-party effects.

The second problem is the difficulty of obtaining reliable compliance data. Businesses have an imperfect understanding of what full compliance would be, and are reluctant to report fraud and violations. Inspectors have a number of incentives to report compliance levels that may differ from reality, and of course never have a full view of the level of compliance in any given business, even one that they inspect. A raw compliance rate is not enough in any case, as it should be corrected by the level of risk or harm created by different violations.

A third reason for caution is that maybe full compliance with all regulations is not always a desirable goal. Though it may theoretically be so, achieving full compliance may lead to excessively high enforcement and compliance costs.

Source: Russell and Hodges (2019[5]), Regulatory Delivery, Hart Publishing.

Operational and strategic documents of OEFA also reflect an attempt to set indicators to measure the efficiency of the agency.

The Institutional Memory of OEFA shows a significant decrease in the duration of the preparation of inspection reports – from 274 days on average in 2016 to 57 days in 2018 – as well as in the number of pending reports – from 11 185 in September 2016 to 49 in December 2017. However, there is no indicator reflecting the performance in terms of social well-being balanced with costs for the state and burden for regulated entities. Also, the there are no indicators whether these tasks contribute to actual positive outcomes for regulated entities.

Interesting indicators have been set in the PEI in term of strengthening capacities within OEFA, and gender-oriented human resources’ management. These are to be measured starting in 2019.

copy the linklink copied!Measurement of stakeholder satisfaction

The level of stakeholder (businesses, civil society) satisfaction and trust is being measured to some extent. A first survey on "Perception the Performance of Environmental Supervision in Prioritized Socio-environmental Conflict Areas" carried out on 2017 showed that 80% of respondents – out of 723 respondents who knew about OEFA – did not trust the agency (OEFA, n.d.[6]). The survey was conducted soon after the new management of OEFA took over, and is to be conducted again at the end of 2019. In parallel the level of satisfaction with regard to citizens’ attention services has also been tracked – as an indicator established in the PEI – through surveys carried out by the staff in charge of Assistance to Citizens in OEFA and in local delegations of OEFA. The proportion of respondents satisfied was of 95.94% in 2018, somewhat higher than that of 2017 (94.68%). However, to increase the reliability of the data, such surveys should be administered by others instead of the staff for which they are a performance indicator.

These results suggest that OEFA is seen as responsive to citizens’ queries, but that trust in its activities is being challenged in socio-environmental conflict areas. What is missing is measurement of trust by stakeholders also in areas that are not sensitive from a social and political perspective, as well as satisfaction and trust among inspected subjects.

copy the linklink copied!Assessment

OEFA has set a number of indicators aimed at assessing the efficiency and effectiveness of the agency, as well as stakeholders’ satisfaction. Indicators on trust and satisfaction could also include inspected entities’ feedback – and OEFA needs to pay attention to the quality of collected data.

Significant efforts have been made to ensure that the performance of OEFA is measured though a number of indicators. Available results mostly show measurements mostly in terms of outputs, as most ‘effectiveness’ indicators in terms of outcomes have been set recently, and have not yet been measured.

“Effectiveness” indicators should aim at measuring the impact of the agency’s operations in terms of environmental situation to understand whether activities carried out are useful in terms of regulatory goals, and how to adjust/improve them. OEFA’s scope to do so is somewhat limited by the fact that it is MINAM who has the responsibility to define and report on effectiveness indicators related to environmental protection, and this is something that needs to be reconsidered by the Peruvian government (and this applies to all inspection and enforcement authorities).

Indicators aimed at measuring the efficiency of the agency should be further improved, to effectively consider the performance in terms of social well-being balanced with costs for the state and burden for the private sector.

copy the linklink copied!Recommendations

There is a variety of indicators of OEFA’s performance, as well as a multitude of data on OEFA’s activities (including within the agency). There is a need for OEFA to differentiate and classify better the reported indicators, as well as to clearly communicate to the public when and where these can be found. OEFA may also want to consider informing the public and stakeholders about the different types of indicators, why and in which document they are set out (e.g. by displaying this information clearly on their website, by means of a section within the annual report amongst others).
It is indispensable to strengthen the focus on actual results indicators (effectiveness). A clear difference between indicators of volume of activities and outputs and effectiveness indicators should be made.
The MINAM should consider allowing OEFA to develop performance indicators more oriented to measure effectives of their activities on the environment and other key policy objectives. “Effectiveness” indicators should aim at measuring the impact of the agency’s operations in terms of environmental situation so as to understand whether activities carried out are useful in terms of regulatory goals, and how to adjust/improve them. Based on this, it is recommended that OEFA refine their current indicators and develops new ones to ensure that they focus on what is meaningful – number of accidents and other incidents, amount of illegal discharges, percentage of resources devoted to a) preventive and b) reactive actions, amongst others,
OEFA should reconsider the method used to measure the level of satisfaction with regard to citizens’ services, by including the measurement of trust from stakeholders in general, as well as the satisfaction and trust among inspected subjects.
Indicators aimed at measuring the efficiency of the agency should effectively consider the performance in terms of social well-being balanced with costs for the state and burden for the private sector.
The way information is communicated in quarterly/annual reports need to be improved to ensure that performance indicators are clearly distinguished from indicators that relate to activities and outputs.
Clarify whether reactions of stakeholders for processes different from ex ante consultation of OEFA’s documents and regulations are being assessed and taken into account. If this is not done, develop such a practice.

References

[2] OECD (2018), OECD Regulatory Enforcement and Inspections Toolkit, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264303959-en.

[4] OECD (2014), OECD Framework for Regulatory Policy Evaluation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264214453-en.

[1] OECD (2014), Regulatory Enforcement and Inspections, OECD Best Practice Principles for Regulatory Policy, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264208117-en.

[6] OEFA (n.d.), Estudio de percepción en zonas de conflictividad socioambiental priorizadas por el OEFA, https://griveram8.wixsite.com/oefa-percepcion (accessed on 15 October 2019).

[3] OEFA (n.d.), Organismo de Evaluación y Fiscalización Ambiental, https://www.gob.pe/minam/oefa (accessed on 14 October 2019).

[5] Russell, G. and C. Hodges (2019), Regulatory Delivery, Hart Publishing.

Notes

← 1. OEFA has established this methodology by using different sources: Banco Mundial (2011) La evaluación de impacto en la práctica; CEPAL (2005) Manual para la evaluación de impacto; IVALUA (2009) Colección Ivalua de guías prácticas sobre evaluación de políticas públicas; JPAL (2017) Implementing randomized evaluations in government.

← 2. See Art. 11 of the relevant Guidelines.

← 3. Responsive environmental impact assessments are carried out following the occurrence of a negative impact on the environment to establish the responsibility of the accident/incident/amongst others.

← 4. Compliance cannot be equated with results in terms of environmental protection, because norms are inherently imperfect and can never entirely predict environmental (or other) results.

Metadata, Legal and Rights

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.

https://doi.org/10.1787/54253639-en

The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.

╳

: Regulatory Enforcement and Inspections in the Environmental Sector of Peru:

Continue to open the publication page on OECD iLibrary to find:

All available formats

Buy option for print or PDF copies

All publication details?

Cancel