2. Towards evidence-based audit selection in Brazil’s decentralised policy areas

Risk assessment helps focus resources on key concerns

The audit selection process is a way to narrow down the possible audits in a structured way, in order to select audits within the existing capacity of the audit body that will have the highest impact. “Having impact” refers to whether the audits are likely to significantly improve the conduct of government operations and programmes, e.g. by lowering costs and simplifying administration, enhancing the quality and volume of services, or improving their effectiveness or the benefits to society (INTOSAI, 2016[2]).

Audit programmes can be implemented through a robust, risk-based process that considers issues of key concern to society, as well as their materiality (OECD, 2016[3]). International standards for selection of audits all, in one way or another, involve some type of risk- or problem assessment as (see Box 2.1).

Risk-based audit selection allows directing audit capacity and efforts to risk areas that are key among the many alternatives, thus optimising allocation of resources and addressing main issues. For audit bodies with limited resources, the risk-based approach is highly valuable for achieving the greatest impact. A risk map with identified risks and risk ratings can provide a good overview of the risks in the audit universe and thus facilitate the selection of audits (OECD, 2018[5]) (see Box 2.2 for further definition).

Sound risk assessment requires sound evidence

SAIs commonly apply some sort of risk or problem assessment to guide the audit selection process, drawing on their accumulated knowledge. The assessment is usually based on the auditor’s field knowledge and professional judgement. As a result, there will be certain subjective elements involved in selection (Put and Turksema, 2011[8]). Objectivity is one of the core ethical values of SAIs, and one of the main principles of professional auditing (INTOSAI, 2016[9]). SAIs should reduce the subjectivity of the audit selection process in order to select the topics that will maximise the audit’s impact for the citizens and maintain the trust of society.

SAIs can demonstrate accountability by explaining to stakeholders why the topics were selected. A risk identification process that is well documented and repeatable, with validated interpretation and not dependent on the individual auditor, is especially useful here. SAIs could also try to increase the objectivity of the risk analysis undertaken for audit selection by making their selection evidence-based. This can include the use of quantitative and qualitative information to ascertain which auditable programmes or entities pose the greatest risk to the achievement of objectives (see Box 2.3 for the Canadian approaches to audit selection).

Guidance can be taken from ISSAI GUID 3920, which states that “In performance auditing, risks may involve areas of potential poor performance that concerns citizens or have a great impact on specific groups of citizens”. By considering areas that could well perform poorly, the scope of the risks is substantially reduced and the targeting of high-risk areas to audit becomes manageable. Data that may provide evidence for identification of these performance risks are very often collected by the government and available as indicators to be used for monitoring progress towards policy objectives. The evidence-based audit selection process would ultimately produce a consolidated and objective picture of the main outcome risks for decentralised policies, guiding the external auditors in selecting and aligning audits.

Risk-based audit selection in Brazil is an evolving practice

The OECD surveyed the courts of accounts in Brazil to find out how they select audits and received 28 responses. The most common inputs used by the courts of accounts (Tribunais de Contas, TCs) for informing their audit selection process are laws and regulations that specify what TCs should audit and the frequency of those audits. The second most common factor considered is the internal budget and resources of auditees, which reflect a risk-based audit selection process focused on inputs rather than on outputs or outcomes. For instance, some TCs consider information on the percentage of an auditee’s expenditure in relation to the municipality or state’s budget, or in relation to the greatest expenditure of the municipality or state as “indicators of materiality.” Only 10 –- of the 28 –- TCs reported using audit reports or findings from other TCs to inform audit selection, indicating a limited level of collaboration or sharing of insights among TCs when selecting audits.

Figure 2.1. Key inputs used to inform the audit programming of the Brazilian courts of accounts

Which of the following inputs does your institution use to inform audit programming?

Note: N=28. More than one choice allowed per respondent; in this question, “audit programming” is referring to the audit selection process as opposed to planning of individual audits.

Source: OECD Survey (2020).

A number of TCs apply external indicators across the audit cycle (including for the audit selection), such as the Municipal Management Effectiveness Index (Índice de Efetividade da Gestão Municipal), the Human Development Index and demographic information. Some also consider information such as the number of irregularities found in previous audit reports, and assessment of the internal control of the auditees.

The majority of TCs in Brazil have a written policy for prioritising and selecting audits based on the assessment of risks (see Figure 2.2), but some TCs do not apply the policy regularly. According to officials from the TCs, some of the factors preventing them from regular application of the policy include:

• institutional and internal obstacles (e.g. internal organisation that prevents joint application of the policy at the state and municipal levels, lack of a risk assessment system, limited resources)

• external pressure (e.g. media, population, legislature, other control bodies) that may prevail over the policy

• issues related to the policy itself (e.g. outdated policy, its limited applicability, quality of the policy).

Figure 2.2. Risk-based audit selection policy in Brazil

Does your institution have a written policy or strategy for prioritising and selecting audits based on an assessment of risks?

Note: N=29.

Source: OECD Survey (2020).

Some TCs also maintain a risk registry for tracking the risks related to auditees or other policy-related risks in specific sectors. For example, TCE-Rio Grande do Norte uses software developed by the Ministry of Planning to support the monitoring of risks: the Ágatha System –- Integrity, Risk and Controls Management System. Auditees’ internal control units are responsible for updating this tool with information about auditees, and the TC uses system-generated reports for audit programming.

Approaches of the TCs towards the application of risk analysis in audit selection vary considerably. The use of indicators is not universal among TCs. Some apply advanced methods, such as analysis of budget data as described above, and others do not make any use of indicators. Co-ordination and collaboration in audit selection occurs on an ad-hoc basis, if at all, and are only the practice of those TCs that have the capacity for it. Developing a common and shared approach to risk-based audit selection provides an opportunity for TCs, particularly those with little or no experience with co-ordination or risk assessments, to modernise their practices towards greater alignment with international standards. A harmonised risk-based approach to audit selection, based on improved exchange of insights and evidence, can lead to more coherent and outcome-driven auditing of decentralised policies. Each policy area has its own context and set of challenges. To narrow the scope, TCs decided to pilot new approaches for risk-based audit selection and co-ordination in the field of education.

Education is a complex policy field for TCs to monitor and audit

Whereas conducting performance audits to assess outputs and policy outcomes is a challenge regardless of the sector, the challenge is all the greater with education. The multiplicity and diversity of actors involved in governing and providing educational services to citizens in Brazil exacerbates this challenge for TCs. As shown in Table 2.1, the 5 570 municipalities in Brazil are responsible for providing crèche services, pre-school and primary education (initial years of fundamental education) to a majority of children enrolled in these levels of education. Moreover, municipalities vary greatly in size, internal organisation, resources and administrative capacity. In lower secondary education (final years of fundamental education), enrolment in public schools is distributed equally between municipal and state school networks, which operate with limited co-ordination between them – adding another dimension of complexity.

In this landscape, responsibility for auditing the use of public resources in education – and for assessing the degree to which policy objectives are being achieved – is shared between TCU and both state and municipal courts of accounts in each federal entity. The extent to which the different TCs actively conduct audits in the field of education depends, in part, on their internal priorities and capacities in the field. The multiplicity of actors involved in decision-making, policy delivery and oversight can complicate the identification of risks and their root causes.

In health policy measuring the outcomes of patients is largely uncontested, and many aspects of medical practice have internationally recognised standards of best practice. The same cannot be said of measuring the learning outcomes of students. There is no rigid set of practice standards for teaching and learning. For this reason, many models of quality assurance in education rely primarily on qualitative assessments of school activities and performance. Nonetheless, policy makers in Brazil and indeed worldwide routinely use a range of quantitative indicators to assess the performance of the school systems for which they are responsible.

Brazil has high-quality data on education

Brazil has a well-established national system for collecting harmonised data about key aspects of the education system. The Anísio Teixeira National Institute for Educational Studies and Research (Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, INEP), an agency under the responsibility of the federal Ministry of Education (MEC), collects data on student enrolment, teaching staff and other variables for individual schools in its “School Census” (censo escolar) (INEP, 2019[17]). Data are published for municipal and state school networks for each municipality, while microdata exist for every public school.

Moreover, Brazil has a highly developed system for assessing the learning outcomes of school students through the Evaluation System for Basic Education (Sistema de Avaliação da Educação Básica, -SAEB), also co-ordinated by INEP. As part of SAEB, INEP designs and oversees the implementation of standardised national tests of students at three points in their educational pathway: in the last years of primary, lower secondary and upper secondary education4. The SAEB assessments (known as Prova Brasil) at each level assess students’ competence in Portuguese and mathematics. Alongside the results of the learning outcomes tests that students take, the SAEB assessments also generate contextual information about students and schools, gathered through student and school questionnaires administered at the same time as the tests themselves. The data for SAEB results are published for each municipality and microdata are used by INEP and external researchers to analyse educational outcomes.

Largely because of its role as a statistical and evaluation agency for education, INEP has also been tasked with developing indicators to monitor progress towards achievement of the goals in the country’s National Education Plan (PNE) and with conducting regular progress monitoring exercises (INEP, 2018[18]). The indicators used by INEP to monitor the PNE are the most complete set of national indicators of educational performance in Brazil.

TCs can use these data to build a methodology for shared risk-based audit selection

This wealth of statistical data on education in Brazil provides a strong basis to gain a broad overview of the characteristics and certain aspects of basic education performance in Brazil. Existing national data on the education system could be exploited to identify strengths and weakness in the performance of education policy in Brazil and to inform the planning, selection and implementation of audits in the field. At present, however, available data are not systematically used for this purpose in the country’s external audit community.

As recommended in Chapter 1, TCs are in a position to leverage the audit capacity of each external audit body individually and collectively by sharing insights and analysing indicators to take a more strategic and systematic approach to audit selection. To do this, TCs can develop a common risk assessment methodology that is structured around the rich data sources in Brazil and a logic model that accounts for policy outcomes, as described below. This assessment can help TCs develop a harmonised language and map risks related to shared policy goals and national priorities in the field of education (i.e. the PNE), while accounting for each entity’s unique context and mandate. As the process develops, TCs could also consider applying the methodology to other sectors and policy issues with similar challenges for auditing in a decentralised context, such as health and infrastructure.

A logic model of education

The OECD developed a basic logic model for the educational activities that education policy seeks to steer. The objective is to identify the key elements (inputs, processes, outputs and outcomes) in the educational process and illustrate basic assumptions about the relationships among these elements, based on a broad review of educational theory and relevant research literature.

Figure 2.4. A logic model for the education sector

The logic model shown in Figure 2.4 can, in principle, be applied to any level of basic education. It takes an outcome-oriented approach and assumes that the ultimate objective of education is to equip the target population of students with a satisfactory level of knowledge, skills and attitudes (learning outcomes). Outputs are implicit in the outcomes in the logic model. The inputs and processes lead to students having received an education (output), through which they have acquired knowledge and skills (learning outcomes) that are relevant in everyday/working life (outcome). This approach is consistent with a student-centred approach to education policy and practice, as used in international learning assessments such as PISA (ETS/OECD, 2019[21]) and in national educational assessment systems, including the system in Brazil (INEP, 2019[23]). These assessment systems seek to measure students’ learning outcomes in an objective manner. Although assessment of young children in pre-primary and primary education is theoretically possible, such assessments at a young age are rare in OECD countries. As a result, more indirect measures of “outcomes” need to be used to gauge the effectiveness of educational policies at this level.

For early childhood education and care and basic education, the “target population” of students encompasses all children in the relevant age cohorts in the jurisdiction in question. The intended outcome of education policy for these levels of education is thus for all children to acquire a satisfactory level of learning outcomes. Access to primary and lower secondary education has been universal in most OECD countries for many decades, although international assessments such as PISA show considerable variation in the learning outcomes students achieve. Governments in many OECD countries have also aimed to “universalise” early childhood education and care and upper secondary education, although participation rates continue to vary among OECD member and partner countries. Measurement of educational outcomes at different levels of education and in different jurisdictions must take into account the proportion of the targeted population that is actually reached by the education system (the “coverage” of the system). Information on education “output”, such as enrolment and completion rates for target age cohorts, must be considered alongside any available measures of student learning outcomes.

The logic model assumes that the outcomes achieved by the education system (in terms of both coverage and learning) are achieved through the interaction between students and teachers in the school environment. Schools (and notably their teachers) and students lie at the heart of the educational process. A wide range of contextual factors influences the characteristics and behaviour of all three groups. These factors are shaped by a combination of inputs (e.g. student characteristics, teachers and their characteristics or school infrastructure) and school-level processes (e.g. curriculum and related teaching methods, support services and guidance).

The ability of students to acquire intended learning outcomes is affected in particular by the students’ own background and abilities and by the capacity of their teachers to support their learning effectively. The home background of students has been widely shown to have a major influence on their initial cognitive and socio-emotional development (OECD, 2015[24]) and their subsequent performance at school. The initial and ongoing training teachers receive, alongside their own aptitude for the job, plays a critical role in their ability to support the learning of students from different backgrounds (OECD, 2018[25]).

The wider environment in which the teaching-learning process occurs further conditions teachers’ ability to work effectively and students’ ability to learn and develop. Within individual schools, this environment includes the quality and relevance of the educational content (curriculum) specified for students at different levels, the effectiveness of school leadership and the quality of the physical infrastructure and resources available for teaching and related activities. Ancillary services not directly related to the educational process, such as social services and support and school transport, also impact on students and their ability to learn.

Many of the inputs and processes surrounding schools and students identified in the logic model require financial resources. Teachers need salaries, school infrastructure and equipment need to be paid for, funds are required for teacher training, for financial support programmes for students and their families and for support and guidance services. When these financial resources come from public sources, they need to be mobilised and allocated through political decisions and policy design. As illustrated in Figure 2.4, governance and policy affect education not only by providing resources through direct financial transfers and public funding programmes, but also through regulation, guidelines and target-setting, all of which influence the behaviour of schools (and, to some extent, students). For TCs, key elements when analysing the policy and governance environment for education include:

1. 1. Governance arrangements: the overall governance architecture, including the entities responsible for designing and implementing education policy and the distribution of responsibilities among them. This dimension encompasses the degree of centralised control or school-level autonomy within a system.

2. 2. Education policy and programmes: the formulation and implementation of specific policies that guide and affect the way schools do their work, which may include policies on teacher training, school staffing, curriculum guidelines, admissions rules for students, and external tests and examinations, as well as policies that determine the level of resources schools (and teachers) receive. Some ancillary services, such as school transport, may fall under the category of education policy and programmes.

3. 3. Other relevant policies and programmes: the formulation and implementation of policies and programmes not directly related to provision of education, but which directly support the goals of the education system.

4. 4. Targets, evaluation and monitoring: the goal-setting and broader set of evaluation and monitoring exercises – including reviews by audit bodies – that inform the formulation and implementation of education and other relevant policies and programmes. It is helpful to single out these activities when analysing governance and policy as they typically occur at arm’s length from detailed policy formulation and implementation – either at the strategic political level (as in goal setting) or through independent or semi-independent bodies (as in monitoring and evaluation).

Applying the logic model to education in Brazil

The logic model for education discussed above is designed to be sufficiently generic to allow application to any level of basic education in any jurisdiction. However, when the model is applied in Brazil to map the relationship between the policy and governance context, financial inputs, the characteristics of schools and students, and outcomes, the following considerations are particularly important:

1. 1. Some of the elements mapped in the generic model will be more important in some countries than in others. For example, enrolment and completion rates in upper secondary education will be more of a concern in Brazil where access to this level of education is not yet universal than in countries where universal access and participation is taken for granted.

2. 2. The role of social programmes in promoting school attendance, or organised school transport, will be more important in Brazil, where demonstrable low family income and limited transport options are frequent barriers to school attendance. For example, the Bolsa Família programme, which makes social welfare payments conditional on children attending school, has been instrumental in increasing enrolment rates in education – a precondition for achieving good learning outcomes. Similarly, providing school transport for students in rural and remote regions to increase school attendance has been a policy priority for federal government.

3. 3. As discussed further below, the availability of relevant information and data relating to the different components in the logic model will affect the extent to which the model can be applied in practice. While the model describes which elements would be helpful for analysing education policy, some of these elements are difficult to measure - and even those that are measurable may not be measured at all. Particularly in Brazil, with significant social and regional variation in income levels and educational outcomes, information and data disaggregated by locality and social group are important for gaining a nuanced picture of education policy performance in different parts of the country and for different populations when applying the model.

As mentioned, Brazil has a well-developed national system of learning outcomes assessment in the school sector - the Evaluation System for Basic Education (SAEB) (INEP, 2019[23]) – which means comparable data are available on key outcomes achieved by public schools and school networks across the country. This greatly facilitates the practical application of the logic model. In contrast, data on other factors, some of which may be relevant in Brazil, such as enrolment rates for specific age cohorts and access to school transport services, are not readily available.

Key considerations for selecting indicators

The logic model serves as a tool for TCs to conceptualise the main variables that interact to influence the outcomes of a particular policy. This model then provides a basis on which TCs can identify the most relevant indicators for policy monitoring and audit selection in the Brazilian context. Once the conceptual model is in place, the next step is to identify which of the variables in the model can be measured reliably using indicators, taking into account evidence about the strengths and limitations of these indicators.

Annex 2.A of this chapter contains an analysis of the theoretical relevance of each variable in the logic model for the education sector presented above, as well as a short assessment of the main strengths and weaknesses of the indicators commonly used in OECD countries and beyond to capture and measure the variables in question. Based on a list of potential indicators, TCs should narrow down the set of usable indicators to avoid creating a process that becomes burdensome in attempting to measure too much.

When selecting indicators that can be used for policy monitoring and selection of audits, these key criteria can be considered (Schumann, 2016[26]):

1. 1. Relevance: do the available data (and indicators that define them) provide information that helps to measure the performance of policy systems and identify policy problems?

2. 2. Availability and timeliness: are data available for relevant units of analysis (for example, in education, school networks, municipalities, states) for the whole country (or a large part of it)? How frequently are the data collected and updated? Does this frequency fit the purpose?

3. 3. Reliability: are the available data from a reliable source that uses robust methods of data collection and management?

In addition to these, there are other factors that TCs can take into account when defining indicators. Some issues are common regardless of context and are typical of indicator-based assessments. For instance, in general, data on outputs are more likely to be available than data on outcomes. Monitoring basic compliance can be done with output indicators (e.g. “Was the number of schools built in accordance with the agreed plans?”), whereas monitoring performance or effectiveness requires outcome indicators, even if they are imperfect (e.g. ”What proportion of students would not have graduated from high school if the new schools had not been built?”).

When it comes to audit selection of decentralised policies, TCs should also take into account indicators that represent the experience of both national and subnational levels. Depending on existing conditions, the same policies can cause different outcomes in different regions, and in these cases national indicators may be of limited value. The choice of the sub-national level should be determined by the degree of regional variation in the effects of a policy. If for instance there are big differences in outcomes between municipalities or even school networks, an indicator showing this level of data should be considered above a national indicator. Alternatively, if the local variation is small, a national indicator may suffice, but won’t have much added value as an instrument to evaluate regional variation in outcome. Co-ordination gaps that occur on sub-national levels – a weakness of multi-level governance common in Brazil and elsewhere – could be captured by outcome indicators that measure objectives at intermediate levels rather than at local or national levels.

TCs may also consider scoreboard indicators to monitor outcomes on sub-national levels, while exercising care not to generalise results. Scoreboard indicators are the highest level of outcome indicators, primarily used to give a general overview of an entire policy area. The smaller the sub-national unit to which an indicator refers – for instance an indicator on school level – the more strong it is affected by random fluctuations. Every sub-national unit should be large enough to be representative of the entire country with respect to the outcome in question. (Schumann, 2016[26])

Finally, when interpreting quantitative information in any policy field, two factors are vital for TCs to consider:

1. 1. Many important things cannot be – or are not – measured. By definition, quantitative indicators cannot capture variables and factors that cannot be quantified – or for which quantitative data are not collected and computed because it would be too burdensome or methodologically complex to do so.

2. 2. Examination of quantitative data alone will not explain why observed values are as they are. Although regression analysis and other statistical techniques can be used to identify relationships between the values of different variables and points to lines of causality, policy evaluation and performance audit always need to complement analysis of quantitative indicators with qualitative investigation of explanatory factors on the ground.

A two tiered approach of selecting indicators for audit-programming in Brazil

The logic model and considerations for indicator selection described above provide a road map for TCs to select education indicators for audit programming. There are essentially two tiers of indicators that the TCs can consider. The first are indicators that measure the desired outcomes for different levels of education. These indicators can be taken from the nationally adopted education policy (PNE); they include “coverage” (the proportion of the target population in education) and learning outcomes (the skills and competencies acquired).

Key first-tier indicators include enrolment rates by age cohorts and, for educational levels where these exist, data on students learning outcomes obtained from the country’s system of national standardised assessments (the aforementioned SAEB). Examining data for these indicators in conjunction with established benchmark values makes it possible to identify schools, school networks, municipalities or groups of municipalities in each state where outcomes fall below expected standards. Given the broad range of educational performance that exists in Brazil, the benchmarks used may need to be adapted to the circumstances of each state.

Second-tier indicators represent data on selected inputs and processes that are likely to affect the observed outcomes. TCs can use these indicators to identify patterns that may help to explain poor observed outcomes and potentially identify more specific issues that could warrant audit. If, for example, data on the qualification level of teachers in a particular state are lower than in other states in Brazil – and learning outcomes are also poorer than relevant benchmarks – there may be a prima facie case to examine that state’s system of initial teacher training or continuous education for teachers. If the level of teacher qualifications is low and combined with poor learning outcomes only in selected municipalities, it may be worthwhile to investigate why this is the case and whether it is linked to recruitment practices or staff conditions in the school networks concerned.

To ensure national coverage, comparability and reliability, TCs can select indicators from official national data sources, including the national education evaluation agency INEP and the national statistical office (IGBE). In collaboration with the TCs, the OECD has developed an initial indicator panel that includes the following:

• For learning outcomes – Disaggregated data from SAEB on learning outcomes in Portuguese and mathematics are available for students for years 5 and 9 and the final year of upper secondary education.

• For coverage – Data on the number and basic characteristics of children and students enrolled in education from the INEP School Census are combined with estimated populations by age cohort. It is necessary to use population estimates based on an established projection methodology as Brazil lacks reliable population registers and instead relies on decennial censuses, which rapidly become outdated.

• For selected inputs and processes – Basic data on municipal and state school networks; on numbers of classes; and on the number, age, qualification level and contractual state of teachers are taken from the INEP’s School Census. Some more qualitative data on the perceptions of school leaders, teachers and students are collected through the questionnaires that are completed at the same time as the SAEB tests, while limited data on financial inputs at school and school network level are obtained from SIOPE (Sistema de Informações sobre Orçamentos Públicos em Educação) and the Ministry of Education’s FNDE (Fundo Nacional de Desenvolvimento da Educação).

These indicators are not static. As policy objectives, challenges and data availability or quality change, so to do the opportunities, limitations or rationale for using a specific set of indicators. Those indicators offer a starting point for TCs to ground future analysis and risk assessments in the logic model and criteria discussed above.

The application of risk scenarios in audit selection

Risk scenarios help the organisation and prioritisation of the risk universe in an analytical way to steer risk identification and analysis. They serve as hypotheses that can be checked and analysed using indicator data and the auditors’ field knowledge, thus identifying the scenarios with the highest probability and negative impact on policy outcomes. This is what happens in the risk analysis and audit selection phase.

Six consecutive analytical steps connect all previous recommendations in this chapter into one process for risk-based audit selection (Figure 2.9). Each step is guided by a key question. The assumption is that all outcome indicators of the logic model have underlying risk scenarios that hypothetically explain possible weak outcomes, but the steps may also be applied for a single outcome indicator.

Figure 2.9. Steps in risk based audit selection

Information technology can facilitate the risk assessment

The auditors doing the risk-based audit selection (described in the previous section) need to have access to the relevant indicator data and the risk scenarios that have been developed. In case of collaboration among TCs, it would be useful if auditors could document and share the results of their own risk assessment with other TCs to create a common knowledge base such as a shared risk map.

The main purpose of such an IT-tool is to facilitate data collection, exchanges of the data and execution of the analytical steps that are linked to the evaluation of indicators. In combining the risk scenarios, the tool may enable TCs to carry out risk assessments individually, as well as collectively.

Through a Geographic Information System (GIS) user interface for visualisation, auditors could apply the tool to identify key performance challenges and risk scenarios by policy area and by geographical area.

Moreover, it may prove a valuable instrument for harmonising assessment procedures and assuring the quality and comparability across the different collaborating TCs. The tool might also support the sharing of audit reports and facilitate the mapping of audit coverage and results, thus further contributing to collaboration (see Box 2.9 for examples of IT-supported risk assessment tools).

Concerning the evaluation of indicators to facilitate audit selection, the basic functionality of the tool would be to allow Step 1 outcome analysis and subsequently, Step 2 analysis of contributing factors that constitute the risk scenarios. The tool could also have additional functionality, such as access to data on audit coverage, access to qualitative analytical information associated with the risk scenarios, a reporting and documenting unit, a risk map interface, etc.

This indicator-based and scenario-driven approach to audit selection requires strong leadership for implementation, not only because it is innovative but also because it requires multiple TCs to collaborate intensively. The governance, organisation and working methods that are needed are all part of the proposed collaboration mechanism that is the topic of Chapter 4. Within this mechanism, TCs should establish a permanent working group that prepares, supervises and executes the steps of the risk assessment, as described in the previous section. This group should work in an iterative way, developing the practical steps of the procedure and evaluating continuously.

The working group should also consider developing an information technology solution that supports the analysis and sharing of information, as described above. Box 2.10 proposes the basic functionality of this tool. An IT tool may not provide TCs with automated results (i.e. the exact definition of the auditee, or the specific issues of a policy that need to be audited), but it can provide an initial basis for the identification of possible risks by topic, sector and geographical location.

Box 2.10. Required basic functionality of an IT tool to support audit selection in Brazil

Step 1 – Where do observed outcomes diverge from policy objectives and targets?

To answer this question the tool could provide an automated evaluation of all outcome indicators at the lowest level of analysis possible, using a simple decision tree; more decision trees would be needed to calculate the aggregated result. The GIS user interface then would allow the auditors to contemplate combinations of geographical level of analysis and policy objectives, establishing not only the main policy areas at risk, but also geographical variations. They can then focus their next analytical step not only on the policy outcomes that are most critical, but also on those municipalities that show the weakest indicator scores, narrowing down the audit universe.

Figure 2.10. Example of a simple decision tree in education

Step 2 – Do the associated scenario input and process indicators diverge from established targets or highlight specific problems that may lead to poor outcomes?

To answer this question the tool could provide an automated evaluation of all the indicators belonging to the risk scenarios that are associated with the selected policy objective, thus providing the auditors insight into the elements of the scenarios that have indicators and that are critical. In addition, the auditors can evaluate geographical variation, thus establishing if this is a local or more general problem.

The audit selection in itself is not part of the risk assessment, because audit selection requires different criteria, such as strategic, technical or resource related criteria. Audit selection in collaboration with other external audit bodies will be discussed in Chapter 4: Improving collaboration in the public sector audit system in Brazil.