Annex B. Methodology

This Annex summarises the methodology developed by the OECD to assess regulatory authorities’ governance arrangements, drivers of performance as well as their performance measurement matrices. The methodology was prepared based on the experience of regulators participating in the OECD Network of Economic Regulators. The framework was applied to 12 regulatory bodies, and the present report is the first application of a follow-up review. The reviews up to date, spanning a number of sectors and countries, include: Colombia’s Communications Regulation Commission (OECD, 2015[1]), Latvia’s Public Utilities Commission (OECD, 2016[2]), Mexico’s three energy regulators (OECD, 2017[3]); (OECD, 2017[4]); (OECD, 2017[5]); (OECD, 2017[6]), Ireland’s Commission for Regulation of Utilities (OECD, 2018[7]); Peru’s Energy and Mining Regulator (OECD, 2019[8]); Peru’s Telecommunications Regulator (OECD, 2019[9]), Peru’s Transport Infrastructure Regulator (OECD, 2020[10]), Ireland’s Environmental Protection Agency (OECD, 2020[11]), and Portugal’s Energy Services Regulatory Authority (OECD, 2021[12]). The methodology has been adapted since its first application to learnings throughout the review process and is adjusted to take into account specific needs and contextual characteristics of each regulator, sector and jurisdiction.

The analytical framework that informs this review draws on the work conducted by the OECD on measuring regulatory performance and the governance of economic regulators. OECD countries and regulators have recognised the need for measuring regulatory performance. Information on regulatory performance is necessary to better target scarce resources and to improve the overall performance of regulatory policies and regulators. However, measuring regulatory performance can prove challenging. Some of these challenges include:

  • What to measure: evaluation systems require an assessment of how inputs have influenced outputs and outcomes. In the case of regulatory policy, the inputs can focus on: i) overall programmes intended to promote a systemic improvement of regulatory quality; ii) the application of specific practices intended to improve regulation, or, iii) changes in the design of specific regulations.

  • Confounding factors: there is a myriad of contingent issues that have an impact on the outcomes in society which regulation is intended to affect. These issues can be as simple as a change in the weather, or as complicated as the last financial crisis. Accordingly, it is difficult to establish a direct causal relationship between the adoption of better regulation practices and specific improvements to the welfare outcomes that are sought in the economy.

  • Lack of data and information: countries tend to lack data and methodologies to identify whether regulatory practices are being undertaken correctly and what impact these practices may be having on the real economy.

The OECD (2014[13]) Framework for Regulatory Policy Evaluation starts addressing these challenges through an input-process-output-outcome logic, which breaks down the regulatory process into a sequence of discrete steps. The input-process-output-outcome logic is flexible and can be applied both to evaluate practices to improve regulatory policy in general, and also to evaluate regulatory policy in specific sectors, based on the identification of relevant strategic objectives. It can be tailored to economic regulators by taking into consideration the conditions that support the performance of economic regulators (Box A B.1).

The OECD Best Practice Principles for Regulatory Policy: The Governance of Regulators (OECD, 2014[14]) identifies some of the conditions that support the performance of economic regulators. They recognise the importance of assessing how a regulator is directed, controlled, resourced and held to account, in order to improve the overall effectiveness of regulators and promote growth and investment, including by supporting competition. Moreover, they acknowledge the positive impact of the regulator’s own internal process on outcomes (i.e. how the regulator manages resources and what processes the regulator puts in place to regulate a given sector or market) (Figure A B.1).

The two frameworks are brought together into a Performance Assessment Framework for Economic Regulators that structures the drivers of performance along the input-process-output-outcome framework (Table A B.1).

For regulators, performance indicators need to fit the purpose of performance assessment, which is a systematic, analytical evaluation of the regulator’s activities, with the purpose of seeking reliability and usability of the regulator’s activities. Performance assessment is neither an audit, which judges how employees and managers complete their mission, nor a control, which puts emphasis on compliance with standards (OECD, 2004[15]).

Accordingly, performance indicators need to assess the efficient and effective use of a regulator’s inputs, the quality of regulatory processes, and identify outputs and some direct outcomes that can be attributed to the regulator’s interventions. Wider outcomes should serve as a “watchtower”, which provides the information the regulator can use to identify problem areas, orient decisions and identify priorities (Figure A B.2).

The analytical framework presented above informed the data collection and the analysis presented in the report. The present report looks at the internal and external governance arrangements of Latvia’s Public Utilities Commission (PUC) in the following areas:

  • Role and objectives: to identify the existence of a set of clearly identified objectives, targets, or goals that are aligned with the regulator’s functions and powers, which can inform the development of actionable performance indicators;

  • Input: to determine the extent to which the regulator’s funding and staffing are aligned with the regulator’s objectives, targets or goals, and the regulator’s ability to manage financial and human resources autonomously and effectively;

  • Process: to assess the extent to which processes and the organisational management support the regulator’s performance;

  • Output and outcome: to identify the existence of a systematic assessment of the performance of the regulated entities, the impact of the regulator’s decisions and activities, and the extent to which these measurements are used appropriately.

Data informing the analysis presented in the report was collected via a desk review and a fact-finding mission:

  • Questionnaire and desk review: PUC completed a detailed questionnaire which informed a desk review by the OECD Secretariat. The Secretariat reviewed existing legislation and PUC documents to update its understanding on the de jure functioning of the regulator, and to inform the fact-finding mission. This questionnaire was tailored to PUC for a follow-up review process of the PUC’s progress, based on the findings of the OECD 2016 PAFER report (OECD, 2016[16]), methodology already applied by the OECD to other regulators since 2015 and on the participation of PUC to former OECD data collection exercises such as the 2018 Indicators on the Governance of Sector Regulators.

  • Fact-finding mission: the fact-finding mission focused on meeting PUC internal teams as well as external stakeholders. The mission took place between 7 and 16 December 2020, and it was a key tool to collect and complete the de jure information obtained through the questionnaire with the de facto state of play and progress on recommendations. The work of the fact-finding mission tailored the PAFER methodology to PUC features. Information collected was completed and checked with PUC for accuracy. The mission took place via videoconference due to COVID-19 travel restrictions.

During the fact-finding mission, the team met with PUC’s leadership team as well as a number of staff from across the institution. In addition, the team met with government institutions and external stakeholders, including:

  • Ministry of Finance of the Republic of Latvia

  • Ministry of Economics of the Republic of Latvia

  • Economic, Agricultural, Environmental and Regional Policy Committee of the Saeima

  • Competition Council of Latvia

  • State Audit Office of the Republic of Latvia Consumer Rights Protection Centre (PTAC)

  • Meetings with industry representatives - Augstsprieguma tīkls (electricity TSO) and Conexus Baltic Grid (gas TSO/storage operator)

  • Latvian Chamber of Commerce and Industry (representative holding the position of Deputy Head of the PUC Advisory Council).

References

[12] OECD (2021), Driving Performance at Portugal’s Energy Services Regulatory Authority, OECD, https://doi.org/10.1787/05fb2fae-en.

[11] OECD (2020), Driving Performance at Ireland’s Environmental Protection Agency, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/009a0785-en.

[10] OECD (2020), Driving Performance at Peru’s Transport Infrastructure Regulator, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/d4ddab52-en.

[8] OECD (2019), Driving Performance at Peru’s Energy and Mining Regulator, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264310865-en.

[9] OECD (2019), Driving Performance at Peru’s Telecommunications Regulator, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264310506-en.

[7] OECD (2018), Driving Performance at Ireland’s Commission for Regulation of Utilities, The Governance of Regulators, OECD Publishing, Paris, https://doi.org/10.1787/9789264190061-en.

[6] OECD (2017), Driving Performance at Mexico’s Agency for Safety, Energy and Environment, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280458-en.

[4] OECD (2017), Driving Performance at Mexico’s Energy Regulatory Commission, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280830-en.

[5] OECD (2017), Driving Performance at Mexico’s National Hydrocarbons Commission, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264280748-en.

[3] OECD (2017), Driving Performance of Mexico’s Energy Regulators, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264267848-en.

[2] OECD (2016), Driving Performance at Latvia’s Public Utilities Commission, The Governance of Regulators, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264257962-en.

[16] OECD (2016), Driving Performance at Latvia’s Public Utilities Commission, OECD, https://doi.org/10.1787/9789264257962-en.

[1] OECD (2015), Driving Performance at Colombia’s Communications Regulator, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264232945-en.

[13] OECD (2014), OECD Framework for Regulatory Policy Evaluation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264214453-en.

[14] OECD (2014), The Governance of Regulators, OECD Best Practice Principles for Regulatory Policy, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264209015-en.

[15] OECD (2004), The choice of tools for enhancing policy impact: Evaluation and review, OECD, Paris, http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=gov/pgc(2004)4&doclanguage=en (accessed on 16 November 2018).

Metadata, Legal and Rights

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.

© OECD 2021

The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.