Chapter 5. Creating a stronger framework to monitor and evaluate national progress in education

Reporting against national goals supports accountability

Public reporting of progress against national goals enables the public to hold government accountable. However, the public frequently lacks the time and information to undertake this role, and tends to be driven by individual or constituency interests rather than broad national concerns (House of Commons, 2011[9]). This means that objective and expert bodies like national auditing bodies, parliamentary committees and the research community play a vital role in digesting government reporting and helping to hold the government to account.

An important vehicle for public reporting is an annual report on the education system (OECD, 2013[2]). In many OECD countries, such a report is now complemented by open data. If open data is to support accountability and transparency, it must be useful and accessible. Many OECD countries use simple infographics to present complex information in a format that the general public can understand. Open data should also be provided in a form that is re-usable, i.e. other users can download and use it in different ways, so that the wider evaluation community like researchers and non-governmental bodies can analyse data to generate new insights (OECD, 2018[10]).

National goals are a strong lever for governments to direct the education system

Governments can use national goals to give coherent direction to education reform across central government, sub-national governance bodies and individual schools. For this to happen, goals should be specific, measurable, feasible and above all, relevant to the education system. Having a clear sense of direction is particularly important in the education sector, given the scale, multiplicity of actors and the difficulty in retaining focus in the long-term process of achieving change. In an education system that is well-aligned, national goals are embedded centrally in key reference frameworks, encouraging all actors to work towards their achievement. For example, national goals that all students reach minimum achievement standards or that teaching and learning foster students’ creativity are reflected in standards for school evaluation and teacher appraisal. Through the evaluation and assessment framework, actors are held accountable for progress against these objectives.

Figure 5.1. System evaluation

Administrative data about students, teachers and schools are held in central information systems

In most OECD countries, data such as student demographic information, attendance and performance, teacher data and school characteristics are held in a comprehensive data system, commonly referred to as an Education Management Information System (EMIS). Data are collected according to national and international standardised definitions, enabling data to be collected once, used across the national education system and reported internationally. An effective EMIS also allows users to analyse data and helps disseminate information about education inputs, processes and outcomes (Abdul-Hamid, 2014[11]).

National and international assessments provide reliable data on learning outcomes

Over the past two decades, there has been a major expansion in the number of countries using standardised assessments. The vast majority of OECD countries (30), and an increasing number of non-member countries, have regular national assessments of student achievement for at least one level of the school system (OECD, 2015[12]). This reflects the global trend towards greater demand for outcomes data to monitor government effectiveness, as well as a greater appreciation of the economic importance of all students mastering essential skills.

The primary purpose of a national assessment is to provide reliable data on student learning outcomes that are comparative across different groups of students and over time (OECD, 2013[2]). Assessments can also serve other purposes such as providing information to teachers, schools and students to enhance learning and supporting school accountability frameworks. Unlike national examinations, they do not have an impact on students’ progression through grades. When accompanied by background questionnaires, assessments provide insights into the factors influencing learning at the national level and across specific groups. While the design of national assessments varies considerably across OECD countries, there is consensus that having regular, reliable national data on student learning is essential for both system accountability and improvement.

An increasing number of countries also participate in international assessments like the OECD Programme for International Student Assessment (PISA) and the two programmes of the International Energy Agency (IEA), Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS). These assessments provide countries with periodic information to compare learning against international benchmarks as a complement to national data.

Administrative data collection does not always follow standard definitions and unified procedures

The State Statistical Office (SSO) has started to align collection with international standards set by the joint United Nations Educational, Scientific and Cultural Organization (UNESCO), OECD and the European Statistical Office (Eurostat) data collection. However, international reporting of North Macedonia’s data reveals key gaps, notably on education expenditure, suggesting incomplete national administrative data. In comparison with other countries in the Western Balkans, North Macedonia has more data gaps than others, signalling significant challenges around the quality and availability of data.

In 2010, North Macedonia launched its EMIS, which holds data about students (enrolment, attendance), teachers (employment history, professional development) and schools (maintenance, funding). EMIS does not hold data on the national examination - the state matura - that are stored separately by the National Examination Centre (NEC). Schools are responsible for entering their own data into EMIS. Currently, roughly 1 000 people in the country are authorised to access EMIS. These individuals include school staff who input the data and government officials who might need to view the data.

The quality of data stored in EMIS is sometimes an issue. For example, the review team was told that, in the past, unique student identification numbers were not generated according to the agreed upon format. Instead, schools generated the numbers randomly, making retrieving data difficult and inaccurate. In addition, despite EMIS’s official status as a central source of education data, parallel data collections exist. The information collected by the SSO and reported internationally is requested directly from schools, and it was reported to the review team that MoES staff often bypass EMIS and conduct their own data collections. The parallel data collections do not always follow nationally agreed definitions, creating inconsistencies. For example, how a satellite school is classified (as an individual school or not) according to the SSO and EMIS might differ. This not only leads to problems regarding data accuracy, but also creates an unnecessary administrative burden for schools.

Public reporting of education data are limited

The SSO website offers electronic access to education indicators, which can be downloaded, analysed and re-used. However, the public has little access to EMIS data, as it cannot access and retrieve data from EMIS. Currently, the only front-end portal that allows the public to view portions of EMIS data is e-dnevnik, an electronic gradebook service that mirrors EMIS data and presents student marks. Authorised EMIS users can access EMIS through a back-end interface, but the review team was told that this interface is not user-friendly and even authorised users have difficulties finding data.

A new national assessment is being developed

National assessments in North Macedonia were first introduced in 2001 in the form of a large-scale, sample-based test. The purpose was to identify how students performed compared to established performance standards prescribed in the subject curricula at the end of the grades 4 and 8 in mother language (Macedonian and Albanian) and mathematics (UNICEF, 2017[18]). Sciences and humanities (grade 4) and civic education (grade 6) were also assessed in grade 4. The assessment was implemented until 2006, when it was ended by a new administration.

From 2013 to 2017, a new assessment was administered annually in every grade in randomly selected subjects from grade 4 until the end of secondary school. The purpose of the assessment was to compare teachers’ internal classroom marks with student results on the assessment. Teachers were supposed to be ranked based upon how closely their internal marks corresponded to students’ assessment results (see Chapter 3). The initial intent was that those who ranked highly would receive a financial bonus, while teachers at the bottom would lose some of their salary. However, this reward system was never implemented, and the assessment was abolished, largely on the grounds that it placed too much pressure on teachers and had a negative impact on teachers’ classroom activities.

Currently, North Macedonia is not administering a national assessment, but plans to introduce a new assessment for system monitoring soon. The MoES has established an independent group to develop the assessment, however final decisions such as the subjects and grades to be assessed had not been made at the time of the review team’s visit. According to the newest draft Law on Primary Education, currently under discussion, the assessment will be a sample-based national assessment, with no stakes for student progression or teachers. However, it is suggested that results will be used to rank and reward participating schools.

North Macedonia participates intermittently in international assessments

North Macedonia has participated in TIMSS (1999, 2003, 2011 and 2019), PIRLS (2001 and 2006) and PISA (2000, 2015 and 2018). The country has produced national analysis and reports on the results – most recently in 2016, which was developed by the NEC in collaboration with the World Bank. – but they have not been shared with the public. This kind of national analysis provides the opportunity to exploit the rich datasets that international assessments offer. The experience of administering international assessments has also helped build national expertise that the country can draw on as it develops its own national assessment.

However, the ministry could make greater use of results by communicating them more widely and using them for national goal setting to help galvanise change. It will also be important to ensure more consistent participation to ensure reliable trend data.

There is no annual report on the education system

The MoES does not publish regular reports about the education system. However, the SSO has recently started to release statistical education information at the beginning and end of academic years (State Statistical Office, 2017[19]; State Statistical Office, 2017[20]). On occasion, the MoES also sends data to the SSO for reporting. For example, in 2016 the SSO has published an ad hoc report about the condition of schools in the country (State Statistical Office, 2016[21]). However, the SSO reports do not provide disaggregated data, for example by students’ ethnicity.

Some information from school evaluation is made available for system evaluation

The SEI produces annually a “Report on the quality of the education process in schools” that aggregates information from all integral evaluations (State Education Inspectorate, 2017[22]). This report identifies general trends gathered through school external evaluations, such as what common school needs are and the state of facilities, but the report does not focus strongly on student learning or system-level factors (e.g. what might be associated with common challenges). The review team was told that this report is rarely made public.

As part of integral school evaluations, satisfaction and perception surveys are administered to the students, parents, teachers and support staff. However, the data are not entered into EMIS and thus cannot be accessed by individuals outside of SEI. Consequently, these data, which contain valuable information about the conditions of schools and the attitudes of students and parents, cannot be used for system evaluation purposes.

Donors and non-governmental organisations have undertaken valuable analysis

Donors and non-governmental organisations (NGOs) have, on occasion, provided valuable analysis that has contributed to system evaluation. For example, in 2016 Step-by-Step, an NGO undertook the Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) in grades 2 and 3, providing reliable information about learning outcomes in the early grades (see Chapter 1). In 2015, the World Bank undertook analysis of North Macedonia’s PISA data, as part of a regional review. While the work of external actors can provide important insights, it can also direct national capacity away from national bodies and focus on priorities determined by external actors.

There is little oversight of how municipalities use resources and no evaluation of how they set and achieve education goals

Municipal governments are responsible for allocating funds from the central government to schools and for overseeing the hiring of school staff. However, there are no systematic mechanisms for reviewing or auditing these activities. Little oversight of school funding at the municipal level means that it is not possible to ensure that resources are used efficiently. Despite their role in education delivery, municipal governments are not expected to set objectives or evaluate their performance.

Raise the prominence of EMIS by positioning it closer to central leadership

Administration of EMIS is currently the responsibility of a small team of two individuals in the Department of Informatics in the ministry. These individuals are not involved in the policy-making processes or in systems to regularly report and monitor education data. That EMIS does not have a prominent role within the organisational structure of the ministry likely contributes to its under-utilisation by policy makers and other actors.

The ministry should consider making EMIS more prominent by moving it to the research unit that this review recommends North Macedonia establish (see Recommendation 5.3.1). A stronger institutional position for EMIS would give it greater authority to mandate who can collect data from schools and deter ministry staff from its bypassing rules. Furthermore, the director of EMIS, or the director of the agency responsible for EMIS, should be involved in policy-making processes, which would help solidify the relationship between data and its use in policy making.

Improve staff capacity

A staff of two individuals is likely insufficient to manage a fully functioning EMIS and does not convey a position of organisational significance. In Georgia, for example, EMIS employs five statisticians solely for responding to data and research requests, in addition to department leadership, administrative support and software developers who manage the system. In Fiji, a far less developed country, EMIS employs eight full-time staff and two part-time staff who are responsible for producing training materials and procedural manuals (World Bank, 2017[23]). North Macedonia’s EMIS would be well served by employing additional staff who could help the current two individuals develop the system, respond to data analysis requests and systematise rules and procedures.

Specific capacities that would have to be recruited or developed include software development for maintaining and improving EMIS and quantitative analysis skills for processing data and creating thematic reports. EMIS would also be well served by having permanent leadership that liaises between EMIS and other departments and agencies within ministry. Having more and better-trained staff work in EMIS would help communicate the message that EMIS is an important entity that should be used properly and relied upon.

Establish protocols for data definition, collection and retrieval from schools

Many countries have established strict protocols regarding the definition of data points and who can retrieve data from schools. In the United States, for instance, each state has developed and oversees its own education database and schools are required to enter information directly into this database. To ensure consistency for national-level reporting and analysis, the United States Department of Education has created the Common Education Data Standards that defines education data around the country (Department of Education, n.d.[24]). By implementing common data standards, national education policy makers can be confident that data from different states have the same meaning and can be relied upon to inform federal decision making.

In addition to commonly defining data, the United States also regulates who can collect data from schools. For example, if government parties wish to contact schools to collect information, they must undergo a rigorous screening process that is regulated by data sharing legislation (US Department of Education, 2018[25]). These procedures help restrict outside access to school information, funnel data retrieval to the education database and limit direct collection from schools to data that cannot be found in the education database (e.g. interviews with teachers or students).

In North Macedonia, such data definition and collection protocols have not been created. The result is that schools might have different definitions for indicators or data points (e.g. how student identification numbers are created). They are also forced to exercise discretion about to whom they provide information. While schools could deny third party requests, if a government body contacts a school for information, school leaders might not feel they have the mandate to refuse, though the data they supply might not even match a common definition. A formal data dictionary and sharing protocol would provide schools with guidance on how to define data and give them the mandate to reject external requests, thus encouraging the requestors to turn to EMIS for their desired information. Ensuring that data definitions are consistent with international definitions would help to fill the gaps in North Macedonia’s internationally reported data.

Standardise the collection of data across agencies and link those data to EMIS

Storing different types of data in different places, as is the case in North Macedonia between EMIS and the NEC, without a common linkage also presents problems. While some countries do hold data in different locations, these data are easily linked by a common variable, usually unique identifications numbers for students, teachers and schools (Abdul-Hamid, 2017[26]). This allows for seamless integration and analysis of data across several sources, such as student demographic information vis-à-vis their assessment results. In North Macedonia, such integration is currently not possible because not only are student’s demographic data stored in EMIS while test results are stored in NEC, but student identification numbers are not consistent across the systems. In-depth analysis using these two valuable sources of data, therefore, is also not possible.

Most EMIS systems do not create student identifiers, but instead use the students’ national/civil identification numbers (Abdul-Hamid, 2014[11]). Using this identification has several advantages. It is inherently standardised and therefore will follow a standard structure across all education databases. Moreover, because it exists at the national level, it can be used to conduct research across different sectors (e.g. if one wishes to study education outcomes and labour market success). Finally, by using this identifier, much student information can be retrieved automatically into EMIS by linking EMIS with the national registry, which greatly improves data quality and reduces the data entry burden on schools. Box 5.1 describes how EMIS in Georgia identifies students, regulates data entry procedures and makes data accessible.

Data collection in North Macedonia exists across several databases. In addition to EMIS, the NEC collects data from the state matura and SEI inspectors use forms to collect information during integral evaluations. These systems, however, are not interoperable. The unique identification of each student in EMIS is their national/civil identification number, but NEC’s database identifies students differently. SEI inspection forms are currently organised and filed, but their data are not always entered into a digital format. When they are, the data are stored locally at the SEI. Data from these different systems, therefore, cannot be easily retrieved or analysed in conjunction with each other.

Standardising government data collection would allow for greater interoperability between databases. Two key actions to enable this will be to use the students’ national identification in all databases, perhaps by passing a regulation, and to ensure that all data are digitised.

Establish quality assurance procedures to verify the accuracy of data that is entered

Internationally, countries implement strict data validation and auditing procedures to ensure that data are of the highest quality (Abdul-Hamid, 2014[11]). In North Macedonia, these types of quality assurance procedures are not fully developed. While ad hoc validation, such as validating student identification numbers has occurred, these mechanisms have not been systematised.

Creating regular quality assurance procedures for EMIS data would help to verify its accuracy and encourage more individuals to use the system. Such procedures could include visiting a sample of schools to check if independent data collection aligns with the school’s data collection, and if the school’s data collection aligns with the information they input into EMIS (Mclaughlin et al., 2017[28]). These procedures would not only improve the data found in EMIS, but also increase the level of trust in EMIS. This role should be undertaken by a national or central government body. One option would be for the State Audit Office to take on this role. In the past, the national audit body conducted a performance audit of EMIS, however there are questions on whether it has the capacity and resources to do so on a regular basis. Another would be to create a small team within the SEI, separate from school inspections, with a specific mandate for quality assurance of EMIS data.

Create regular reporting procedures

Reporting is one of the integral features of an EMIS. It is the vehicle through which the system transforms from being a receptacle of data to a provider of information. All EMIS systems have the inherent capability to generate reports using their stored data. It is the responsibility of administrators, however, to instruct EMIS how to process raw data, create reporting templates that display processed data and regularise reporting procedures (Abdul-Hamid, 2014[11]).

In North Macedonia, EMIS administrators have created some data processing instructions and reporting templates. However, they have not created regular reporting procedures, such as an annual statistical report at the end of each school or calendar year. Instead, EMIS reporting occurs in an ad hoc manner, mainly in response to individual requests. This system can be inefficient as requests for information tend to be submitted at similar times (around reporting deadlines) and thus require time to fulfil. This also limits the use of information, since it requires that users know what data are contained in EMIS and take the initiative to request it.

It would be helpful if EMIS administrators identified the most commonly used templates (e.g. data related to participation and completion), created a timetable for regular production of the reports and made those reports publically available. Interested parties can then retrieve the data instantly without needing to request the information from EMIS. After the most commonly used templates start to be reported regularly, EMIS staff can then turn their attention to producing regular reports on specific themes of national interest, such as the education of linguistic minorities (see Recommendation 5.3.3). These procedures would further encourage policy makers to rely on EMIS as a data source and make fuller use of the data in setting goals and designing policy interventions.

Develop a user-friendly portal to quickly retrieve contextual data

In addition to regular reporting, real-time access to data through a web portal is a common method of extracting information from EMIS and presenting it in an accessible manner. At the most fundamental level, users will be able to learn how many students attend a school and how they perform on a national assessment. More sophisticated systems aid research and analysis by facilitating comparison across schools, aggregation at different levels (e.g. regional or national) and providing a set of data visualisation tools (Abdul-Hamid, Mintz and Saraogi, 2017[30]). Box 5.2 explains the functionality of the Florida’s PK-20 Education Information Portal, an online EMIS portal from the Florida Department of Education, United States.

In North Macedonia, the e-dnvenik service was created to allow teachers, parents and students to view, through an online portal, relevant data that is stored in EMIS in addition to student grades. However, e-dnevnik is primarily used as a student monitoring service, not to access EMIS data for broader, analytic purposes. Users of e-dnevnik cannot, for instance, look at schools across the country and filter those schools by certain characteristics. Furthermore, access to e-dnevnik is limited to individuals with direct interaction in the education system, such as parents of students and school staff. Research organisations or higher education institutions cannot access e-dnevnik.

The ministry should create an online platform that allows public access to EMIS data through a dashboard interface. All users of the platform would be able to browse national education data and select schools and municipalities for comparison based upon chosen criteria (for example, location or language of instruction). The platform should also contain features to create dynamically generated charts and figures and export data for further analysis. Importantly, the online platform must be user-friendly such that members of the public can easily navigate it and use these features. Creating such a platform would help schools benchmark their performance in a contextualised manner, assist researchers in analysing system information and help policy makers base their decisions on stronger evidence.

Clarify national goals and create measureable targets

The Comprehensive Education Strategy 2018-25 is focused on achieving outputs, such as curriculum reform and textbook usage. It does not however focus on outcomes, notably improvement in student learning. Internationally, countries use national goals and targets to give visibility to national priorities and direct the education system towards their achievement. Given the evidence that the majority of students in North Macedonia do not master basic competencies, setting an ambitious target to raise learning outcomes would help to ensure that the education system and society in general, recognises this as a national and urgent priority.

This chapter strongly recommends that the government establish specific goals for improving student achievement and associates those goals with measurable, achievable targets. Since the national assessment is still in development, using data from international assessments such as PISA to monitor student performance over time would be an effective method to track changes in student learning. For example, reducing the share of low performers in PISA to below 15% by 2020 in line with European Union (EU) targets (European Commission, 2018[32]). The government can also consider setting interim benchmarks to ensure that the country is progressing towards the long-term goal. Given the evidence of inequity in learning outcomes, such as the gap between rural and urban students, or between students of Macedonian, Albanian and other ethnicities, other goals to improve equity might also be included. For example, goals might be set to reduce the performance difference between different groups of students by 10 score points in all three core domains. North Macedonia could then use these targets to steer attention and accountability towards student learning.

To support the achievement of these student-learning goals, the ministry will need to make sure that it develops clear plans that set out how its goals will be supported. Critically, this should include plans for establishing a new national assessment and prioritise consistent participation in international assessments to provide trend data.

Develop a national indicator framework and use it to co-ordinate data collection and reporting procedures

The absence of a national indicator framework is inhibiting systematic data collection, reporting and monitoring of student outcomes in North Macedonia. A national indicator framework not only specifies the measurable targets associated with goals, but also the data sources that will be used to measure progress and the frequency of reporting around the indicator. Without this valuable document, system evaluation in North Macedonia loses co-ordination around what data points to pay attention to, which results in a general loss of systematic direction and fragmented goal setting.

Developing an indicator framework would not only support accountability vis-à-vis national learning goals, but would also help orient the future development of EMIS. Through the national indicator framework, data gaps can be easily identified. If, for example, a target is to improve the retention of students from ethnic minority groups, the national indicator framework would indicate that EMIS is the data source to be used to monitor this indicator, and that it would need to collect data about students’ ethnicities. If EMIS currently does not hold such data, or if such data are poorly collected, EMIS staff would prioritise developing capacity and data collection procedures to support the monitoring of this indicator. Reporting against indicators from the framework in an education report would also support public accountability and create pressure to ensure that any data gaps are addressed.

Establish a steering committee to define the purpose of the assessment

Carefully defining the purpose of the national assessment to reflect the country’s teaching and learning needs is critical. It will also be important that the MoES take steps to achieve systemic buy-in, so that the results are trusted and used. Towards this end, this review recommends establishing a steering committee comprising a diverse group of stakeholders representing different backgrounds, ethnicities and interests nationally. This should include, for example, representatives from the Vocational Education and Training Centre (VETC). The steering group should also include technical expertise on the development and use of national assessments.

The steering committee will need to take into account not only the goals of the education community, but those of the political administration and reconcile these aspirations with what is technically feasible in the North Macedonian context. International experts can be enlisted to lend a global perspective to the steering committee’s deliberations.

The MoES has recently created an independent group to review Slovenian’s National Assessment of Knowledge (NAK), with a view to incorporating elements into North Macedonia’s national assessment (see Box 5.3). If practical, the steering committee can be formed by building upon the membership of this group. However, it will be important that the mandate of the current group be clearly documented and that transparency around its activities increased, if it is to become an official steering committee that guides the development of the national assessment. During the OECD review visit, many important stakeholders remained unclear as to the purpose of group, creating some concern and potential mistrust in the process.

Determine who will be responsible for the national assessment

One of the first decisions for the steering committee is who will be responsible for the national assessment. In the past, responsibility for national assessments has moved between the BDE and the NEC. Above all, it is important that the responsible organisation has the technical competence and resources to undertake the assessment.

This review recommends that, as the NEC has experience in administering the matura and international assessments, it is best positioned to be responsible for the new national assessment. It already has test administration, marking infrastructure and staff who are familiar with these processes. In order to take on this new task, the NEC will need sustained and adequate funding. In the short term, this may require drawing on an external source. One possible source discussed with the review team is the European Commission. This could provide essential support for the assessment’s design and development in the early stages. However, it will be important that North Macedonia plans for the assessment’s sustainability well before such external funding ends.

Consider making formative feedback to educators the primary function of the assessment

This review recommends that North Macedonia’s national assessment serve primarily a formative function. In other words, it should provide detailed information on how students are performing vis-à-vis the nation’s learning standards and this information should be used by the education system to improve student learning.

Using the national assessment in this way would help to address key teaching and learning challenges in North Macedonia. International data suggests that national outcomes are low and not improving (see Chapter 1). The review team’s interviews suggested that teachers’ classroom assessments are not always an accurate indication of what students know and can do, which is essential to improve learning (see Chapter 2).

At the same time, North Macedonia is progressively implementing a more competency-based curriculum. International experience shows that teachers require significant guidance to adapt teaching to this approach, so that students are assessed in ways that are valid and reliable according to new benchmarks for learning. In this context, the country will require meaningful assessment results about student learning that can help teachers better understand where students are in their learning and tailor teaching to students’ individual needs.

If the ministry decides to use the assessment for primarily formative purposes, care will need to be taken in terms of how results are used for accountability purposes. While all national assessments provide a broad accountability function for the education system overall – by monitoring learning outcomes nationally against standardised expectations –using results for accountability purposes at the level of individual schools and teachers can encourage schools and teachers to attach stakes to an assessment. This can result in distortive practices like teaching to the test, which involves disproportionally focusing on assessed content or repeated assessment practice (OECD, 2013[2]). Such stakes would also undermine the assessment’s formative purpose that this review recommends. The education system in North Macedonia is even more vulnerable to these tensions, given the recent practice of using the national assessment for teacher accountability. Careful steps will need to be taken to avoid that the assessment is perceived to carry high stakes (Recommendation 5.2.2).

Align the assessment’s design with its primary purpose(s)

Once North Macedonia has agreed on the assessment’s primary purpose(s), this should guide subsequent decisions on key aspects of the assessment’s design. Table 5.2 illustrates several components about national assessments that will need to be decided upon. The suggestions in the discussion below are intended to support a prominent formative function.

Combine census and sample-based testing

National assessments can be census-based, in which all students from the population of interest are tested, or sample-based, in which a representative sample of the population is assessed. A census-based design yields more accurate results, can be used for monitoring individual students and can be aggregated into higher-level results, such as school and municipality. Given concerns around teaching and learning in North Macedonia, a census assessment would be particularly valuable since it could provide formative information to help all teachers adapt instruction to their students’ needs.

However, census assessments can easily acquire high stakes, and this risk needs to be addressed. Census assessments are also considerably more expensive to implement and require more time to ensure that all students are tested and all tests are marked. To manage these costs in the immediate to medium term, North Macedonia might consider implementing a hybrid model in which both census- and sample-based testing designs are used in different grades. This review recommends the following configuration:

• Census assessments in grades 3 and 6

  Currently, there is no standardised measure of performance in the early grades of school. Having more information about student learning and school performance at this level would allow for the identification of struggling schools and students, and the provision of more relevant support to them. An assessment at this stage is important since children who do not master basic competencies, like reading in early grades, are more likely to struggle later on (National Research Council, 2015[37]). At the same time, support to address difficulties or learning needs is more effective the earlier it begins. These considerations are especially important in North Macedonia, given the evidence from EGMA and EGRA that suggests that younger students lack essential literacy and numeracy competencies.

  Once the new curriculum is piloted for grades 1 through 3, the MoES could use student performance data at the end of grade 3 to help decide whether, and how, to expand the adoption of the new curriculum. Grades 3 and 6 are also the end of the first and second curriculum cycles, so the assessments would provide information about student performance at key stages.

• Sample assessment in grade 9

  Grade 9 represents the end of the third curriculum cycle and would be a logical point to re-administer the national assessment. Since grade 9 also marks the transition to high school, to avoid confusing the national assessment with an entry examination (which North Macedonia had in the past), this review recommends that the national assessment be administered in grade 9 as a sample.

  As a sample-based assessment is quicker and cheaper to administer, it can also be used for experimental and research purposes. For instance, the MoES could introduce new domains into the grade 9 assessment in order to assess student performance in subjects outside those that appear in the grade 3 and 6 assessments. Sample-based assessments do not provide sufficient information on each school to provide statistically robust and reliable school-level results that are comparable.

Recording student results using the student identity document (ID) would make it possible to link the assessment results to students’ background data in EMIS (such as their mother tongue language, gender, etc.). This would enable initial analysis of how contextual factors impact student-learning outcomes. In the future, once the national assessment is well developed, North Macedonia might consider developing background questionnaires in order to enable further analysis of the contextual factors shaping learning.

Test mother tongue language and mathematics

Among OECD countries with national assessments at the primary level, a third (ten) assess just mathematics, and reading and writing in the national language, which represent core skills (OECD, 2015[12]). Focusing on these two subjects in the assessments at the primary level (grades 3 and 6) would be especially constructive in North Macedonia given the EGRA and EGMA results.

In grade 9 and even perhaps grade 6, additional subjects, e.g. science and/or national history, may be added to the core of language and mathematics. For students whose first language is not Macedonian, Macedonian as a second language could be added given the equity concerns with respect to outcomes by ethnicity (see Chapter 1).

Create criterion-referenced scoring

The vast majority of OECD countries with national assessments use criterion-referenced scoring. For example, among 27 countries with a national assessment in lower secondary, 21 reported using criterion-referenced tests (OECD, 2015[12]). A criterion-referenced test assesses the extent to which students have reached the goals of a set of standards or national curriculum, while a norm-referenced test compares students’ results to each other (OECD, 2011[38]).

Results from criterion-referenced tests are preferred for national assessments because they produce results that are comparable over time. As the purpose of North Macedonia’s national assessment is to understand what students are learning linked to national learning expectations and to monitor progress over time, the assessment should be created as a criterion-referenced test. North Macedonia’s national learning standards represent natural reference points for the assessment.

Develop different item types that are designed to assess student learning

In OECD countries, the most popular types of items that appear on national assessments are multiple-choice responses and closed-format, short answer questions (e.g. true/false, selecting a word of providing a solution to a mathematics problem) (OECD, 2013[2]). These item types are the most common because they are easier and quicker to develop and administer. Moreover, their marking and scoring is more reliable, and therefore test results are more comparable (Hamilton and Koretz, 2002[39]; Anderson and Morgan, 2008[40]). Less frequently used item types include open-ended questions, performing a task, oral questions and oral presentations. Though less common, these types of items are increasing in use due to their ability to assess a broader and more transversal set of skills than closed-ended items (Hamilton and Koretz, 2002[39]).

In interviews with education stakeholders, the review team learnt that one of the primary concerns with the national assessment conducted between 2013 and 2017 was that the questions encouraged memorisation. Therefore, a key consideration for the new national assessment is to ensure that concerns about cost and reliability are balanced with the national need to assess learning in ways that do not encourage memorisation.

Given these trends and considerations, this review recommends that the:

• Grades 3 and 6 assessments consist primarily of multiple-choice and closed-format responses. These are the most common types of questions among OECD countries with assessments at this level of education (OECD, 2015[12]). While there are natural limitations to multiple-choice and closed-format responses, these types of items, when developed well, do have the capacity to assess complex student learning (Anderson and Morgan, 2008[40]). The majority of questions from both PISA and TIMSS represent these two types. Care will need to be taken to ensure that these items are measuring student learning instead of memorisation, and that proper item-writing convention is followed, such as reviewing items for potential bias and varying the placement of distractor choices (Anderson and Morgan, 2008[40]). In grade 3, North Macedonia can also draw on international models for assessing literacy and mathematics in the early grades of school, such as EGRA and EGMA.

• Grade 9 assessment includes multiple-choice questions, closed-format responses and can begin to incorporate more open-format questions. At this age, students are more capable of responding at length, and the sample-based nature of this test produces fewer responses to mark, thus limiting the added costs that these items would create.

Consider computer-based assessment delivery

In most OECD countries, the delivery of the national assessment is through a paper-and-pencil format. Nevertheless, this trend is changing and computer-based administration is becoming more common, particularly in countries that introduced a national assessment relatively recently (OECD, 2013[2]).

Compared to paper-based delivery, computer-based testing has several advantages. It tends to be cheaper to administer (aside from the initial capital investment), less prone to human error in the administrative procedures and the results are delivered more quickly. Given these advantages, and because the ministry in North Macedonia is already dedicated to enhancing the use of technology in education, this review recommends that the national assessment be delivered as computer-based assessments. Importantly, given the intent of the assessment to provide meaningful, formative information to teachers and schools, the faster speed with which results can be delivered through a computer-based test would certainly support this aim.

Avoid decontextualised ranking of individual schools and any judgements on individual teachers

Research shows that concentrating excessively on numerical ranks with respect to students, teachers and schools can have negative consequences on teaching and learning. Especially when coupled with punitive consequences, such a system encourages educators to focus on reporting high marks as opposed to focusing on student progress (Harlen and James, 1997[41]) (OECD, 2013[2]).

The ministry’s intent to hold schools accountable for their assessment results is positive in some respects, since it reflects a desire to encourage schools to focus more on the quality of the teaching and learning environment they provide. Nevertheless, in North Macedonia the history of using assessment data to penalise teachers and principals is fresh in the memory of the education community. Furthermore, international evidence shows that student learning outcomes are influenced by a range of factors that are beyond the control of the individual school or teacher, the most influential of which is a student’s socio-economic background (OECD, 2013[2]). Ranking schools by assessment results alone risks that schools with the greatest concentration of students from more advantaged backgrounds are continually being ranked at the top.

Based on the body of international evidence and the education environment in North Macedonia, this review recommends that schools are not ranked according to their results on the national assessment or any other decontextualised criteria. Such a measure would not provide an accurate judgement of quality or educationally valuable information and would undermine the assessment’s formative function. Instead, information about schools can be presented according to several educational indicators (see Box 5.4). These might include a school’s results on the national assessment, the socio-economic status of a school’s students and their linguistic background. Using this information, the ministry could even publicly recognise schools whose students are performing well in the face of difficult circumstances. Given the recent experience of national assessments, it will be important to communicate clearly that the new assessment will not be used to form a judgement on individual teachers and it will not carry any consequences for them.

Create reporting structures to maximise the formative value of the national assessment

To meet the purpose of providing formative information on all students, the national assessment will test all students in grades 3 and 6. How results are reported, and to which audiences will need to be decided. Adequate financial resources for these reports would need to be considered in central planning, and budgeted and resourced accordingly.

• School-level reports might present the performance of the individual school with benchmarks for comparisons (e.g. a national average or a municipal average). They should also show how schools are doing in relation to their context (e.g. student socio-economic background and language). Individual school reports on assessment results should not be made public. However, this review recommends that school evaluation reports (that are already public) include school outcomes data on national assessments, but as part of a holistic, contextualised quality assessment of the school (see Chapter 4).

• Reports for teachers should contain item-level analysis with information about how their students performed on each item. This information should be presented alongside contextualised comparison groups, such as gender, linguistic minorities and municipalities, as well as the country as a whole. Providing these data to teachers is vital to help them engage with the results in a formative manner to help improve student learning. The results might also analyse common errors that students made, with suggestions on how to improve teaching of the same content in the future.

• The ministry would receive an aggregate report that summarises and analyses the results of the entire country. Results should be disaggregated by several demographic characteristics, such as gender, language of instruction, school type, municipality, student socio-economic status and whether the school is a satellite school. Reporting according to these factors (among many others) would represent the minimum level of analysis that would be required to inform policy making.

  Analysis of individual questions, topics or skills would also be important for the ministry to identify at a national level if students in North Macedonia tend to struggle more with certain areas of knowledge or skills. This information would reveal the need to identify how teaching in certain parts of the curriculum can be improved.

Since the main purpose of the assessment is to provide formative feedback to schools and teachers, care should be taken about how results are provided to students and parents, to avoid the perception that the results carry stakes. Students and parents might be informed about individual student results as part of regular parent-teacher meetings. Teachers might be provided with national guidance on how to provide the results – for example, reporting results within broad categories of meeting or not meeting national expectations, rather than individual scores. In New Zealand, for example, the asTTle (Assessment Tools for Teaching and Learning) allows teachers to identify individual and group strengths and weaknesses, gauge progress, monitor patterns and trends, and to compare these with national standards (Nusche et al., 2012[45])

Reporting assessments results to the public

Making assessment date publicly available promotes transparency and government accountability. However, there are also risks that external bodies and the media may use results to produce decontextualised school rankings, which can encourage that stakes become attached to the assessment. North Macedonia will need to carefully assess the potential risks and benefits of publishing assessment data, especially given how assessment results have been used in the past in the country. Many OECD countries make all results publicly available on request ‒ for research purposes and government accountability ‒ and aggregated school-level data publicly available on the EMIS portal.

North Macedonia should present school “raw” data in a column alongside school contextualised data (e.g. students’ socio-economic background). Including the latter should encourage the media or other bodies that produce a ranking to provide more contextualised information about schools.

Set up a research and analysis unit and clearly define its purpose

The review team understands that the ministry is considering setting up its own research unit, with support from UNICEF. This OECD review strongly supports this proposal. Among the important responsibilities of the research unit would be improving and centralising data access, evaluating the effectiveness of education policy, measuring progress towards strategic goals and promoting the use of evidence to inform policy making. The research unit would not necessarily perform research and evaluation work itself, but could prioritise and help commission it from other bodies and actors. Such an approach would have the value of strengthening demand for evidence and catalysing the development of education research capacity more widely.

These are significant responsibilities, which can be assumed progressively. This review suggests the unit starts with addressing some important gaps in the monitoring and evaluation system. One immediate task would be to work with EMIS to create a national indicator framework. This would provide information for the government to evaluate future major reforms, such as the new curriculum and reshaping the state matura (see Chapter 2). Another task that the research unit would direct is the development of an annual national education report, which could be written by the unit itself (see Recommendation 5.3.3). The unit might initially be staffed by two to three members of staff with experience in quantitative analysis, use of evidence in policy making and delivery of policy.

Ensure the new unit has a prominent role in policy making

For the research unit to achieve its objectives, it must be prominently situated within the ministry for example, by directly supporting and reporting to the minister and being involved in regular meetings with the ministry’s leadership to contribute to policy making. This governance structure would provide the research unit with recognition and prominence to guide different parts of the ministry institutions in a common direction. One of its first priorities might be to ensure that all parts of the ministry use common data systems and adopt data common procedures. The latter would be supported by moving EMIS close to this unit. The unit should be given a dedicated budget line so that it has a predictable, secure finding base.

Clearly define the roles of the specialised agencies and ensure that they have consistent leadership

During its review visit, the OECD was told that the responsibilities of specialised agencies (NEC, BDE and SEI) that are affiliated to, but separate from the ministry can fluctuate greatly. The NEC, for example, has taken on a principal training role, despite being formally tasked with administering national examinations and assessments. Meanwhile, the BDE, despite being explicitly charged with developing curricula, has not been involved in developing the new curriculum that is now being developed by a group formed within the ministry.

The unstable mandate of these agencies is partially due to unstable leadership. At the time of the OECD’s review visit, the NEC did not have a director (though it has had ten in the past ten years). It was reported to the review team that political affiliations are often a prominent consideration in leadership appointments, above professional competence. It is difficult for these organisations to assert themselves and assume responsibility for long-term projects without consistent, professional leadership.

To help these agencies fulfil their duties, the ministry should ensure that their roles are well defined and do not change in response to political considerations. Some ways that this might be achieved include:

• Developing mandates enshrined in legislation. For example, the mandate of the BDE should clearly specify that it has a role in curriculum development, while the NEC is responsible for national examinations and assessments.

• Agreeing a multi-year activity programme and related budget for each agency. This would help to ensure that important activities, like research and evaluation, are valued and undertaken, while protecting these agencies from being expected to undertake activities that are ancillary to their core functions.

• Explicitly setting out the appointment process for directors within each organisation. This should include the technical and professional competencies directors are expected to have, and their appointment length. In the future, a merit-based appointment process should be implemented.

Finally, these bodies need consistent and qualified leadership. Staffing senior management positions in these agencies should be made a priority such that vacancies do not remain unfilled for long periods of time, which can damage their credibility and effectiveness. Having stable leadership would also help fill the resource and capacity gaps that exist in these agencies, such as psychometric and statistical expertise in the NEC and funding for the BDE to provide professional development to teachers.

Formalise the BDE as the research arm of government

The BDE carries several responsibilities, including developing the curriculum, providing professional development to teachers, and undertaking research. This latter role is the least well defined, likely due to the overall lack of focus on system research in North Macedonia and lack of adequate resources. For example, despite having a research responsibility, the BDE does not produce regular analytic reports about education. For system evaluation to become embedded in ministry processes, the research function of the BDE needs to be better articulated and formalised.

Many countries have an organisation that is explicitly responsible for conducting research into the education system. In the United States, for example, the Institute for Education Sciences is tasked with collecting statistics and carrying out rigorous research and evaluation related to education (US Department of Education, n.d.[46]). In some countries, similar organisations have a multi-faceted role that includes research responsibilities and educational development responsibilities. For instance, Slovenia’s National Education Institute is charged with producing educational materials, delivering professional development and conducting research on education in the country (National Education Institute Slovenia, n.d.[47]).

To establish the BDE’s national research role, it must have the necessary capacity. Interviews conducted by the review team suggest that resources and responsibilities are often diverted away from the BDE, such as curriculum development and principal training, which limits what the organisation can do. The ministry should endeavour to bolster the position and capacity of the BDE by promoting merit-based appointments and hiring and ensuring that the organisation is fully staffed, particularly with researchers and individuals with statistical backgrounds, which the review team observed are particularly lacking across central government institutions in North Macedonia. Furthermore, like the other agencies, the BDE’s research function needs to be systematically planned and documented such that it can be held accountable for meeting goals and producing key deliverables. In developing a process for this, the BDE could look to the examples of how similar bodies in other countries do this (see Box 5.5 which describes how Romania’s Institute of Educational Sciences operates and plans its activities).

Define the research and evaluation functions of the NEC and the SEI

Within the institutional architecture recommended by this review, the research unit within the ministry would guide research and evaluation efforts, and the BDE would become the main research arm of the ministry. However, the NEC and the SEI would still have complementary research functions in the evaluation of the education system that need to be defined and supported. Importantly, these agencies independently collect and hold data that are not immediately accessible to external individuals. School inspection and state matura data are critical for comprehensive system evaluation and these two agencies require staff who are capable of working with the research unit, the BDE, and the independent researchers who analyse these data.

The mandates and multi-year plans of both agencies should clearly specify expectations for the research that they are expected to undertake. Developing these plans with the MoES will also help to educate the ministry on the possibilities of further research that the NEC and the SEI could provide. Over time, this should create stronger demand for data and research within the ministry. As discussed above, the multi-year plan for each agency should be accompanied by a multi-year budget to ensure sufficient, protected resources for the work that is expected to be undertaken.

The NEC currently employs staff with research and data analysis capacity. The SEI does not and will need to recruit staff with experience in these areas. With adequately trained staff, the NEC and the SEI both need to clearly delineate the responsibilities of their researchers to ensure that they efficiently support system evaluation efforts without duplication of work. Their tasks would include processing the data, manipulating it to be interoperable with each other and EMIS data, and responding to data requests. SEI researchers would have the added responsibility of digitising its data collection procedures as much information is still contained on paper and stored in filing cabinets.

Annually publish an analytical education report

Most OECD countries regularly publish an analytical report on education (OECD, 2013[2]). The content of this report is guided by national policy goals and priorities. Reports typically report on progress against the national indicator framework (see Recommendation 5.1.3), and explain the strengths and challenges of the system by studying the inputs, processes, outputs and outcomes that are related to the indicators. For example, an analytical report might first describe the overall performance of students on a national assessment, and examine this performance in relation to changes in the teacher profession and school resource allocation. The report might also discuss future policies or activities intended to address certain challenges.

In North Macedonia, some education indicators are reported by the SSO in an annual publication and the SEI produces annual reports based on results from the integral evaluation, but these reports do not offer further analyses. Furthermore, the reports are not prominent documents or an important vehicle for public reporting on government progress or education quality more generally. To support the creation of the new report, the ministry should draw on the research being carried out in external bodies and by independent researchers. The ministry might consider creating the following expectations for the report:

• Prominently reporting against key national goals. For example, the annual report might report on progress against short-and long-term goals for improving learning outcomes.

• The report should be made publicly available. It should be easily accessible for the general public. In the future, the data that is reported should also be downloadable in a format where it can easily be re-used to facilitate secondary analysis by the research community.

• There should be dedicated time for parliamentary debate of the report. The country should consider giving the parliamentary education committee the opportunity to organise committee hearings with the ministry’s senior leadership to discuss the contents. In many OECD countries, these mechanisms provide important means to hold the government accountable.

• Including analysis of progress, and why progress in certain areas may have been quicker or slower than expected. For example, when reporting data on student-learning outcomes, this kind of analysis would help policy makers understand not just how students perform, but why they perform that way and what can be done to improve performance at the national level. These reports can also be used by the wider research community to direct secondary analysis into key issues that affect the system.

The new research and analysis unit might also examine, or commission specific, thematic issues that are important at the national level. For example, is there an association between student learning and double- or triple-shift schools? What is the difference in student performance between schools that instruct in Macedonian and those that instruct in other languages? What can be learnt about the system by studying student results longitudinally, from the grade 3 national assessment, to the grade 6 national assessment to the state matura?

Reinforce processes to embed the use of evidence in the policy-making process

Research on effective policy making emphasises the importance of evidence-based analysis and advice (OECD, 2017[50]). For over a decade, the European Commission has urged member countries to use evidence-based policy and practice and robust evaluation instruments in order to identify the most effective policies and practices (European Commission, 2007[51]). Evidence-based policy making means that before policy and major legislation is introduced, available evidence is studied and possible policy options openly discussed. Governments may also request expert committees that are independent of political affiliation to comprehensively study an issue before making a recommendation to policy makers.

In a legislative context that is heavily politicised like in North Macedonia, it is even more crucial that non-partisan evidence review become an integral part of the policy-making process. For example, the government can establish the guideline that all major changes should first be piloted, and the pilot studied, before full-scale implementation. Major programmes should also be systematically evaluated to determine their effectiveness and inform future reforms.

Having such regulations in place could have benefited previously enacted policies and reforms. For instance, the Cambridge curriculum was neither fully piloted nor evaluated before the new curriculum was developed and set to be introduced. Consequently, it will not be possible to know how effective the Cambridge curriculum is and what benefit the new curriculum in grades 1-3 will provide in comparison. The One Laptop per Child programme has also not undergone systematic evaluation. Having data about this programme, in particular the value for money that it represents, would help inform future policy making regarding technology in the classroom.

Establish clear procedures for municipalities

Municipal governments in North Macedonia are not well resourced for education delivery. Each municipal government has just one or two members of education staff. It was reported to the review team that the municipalities do not come together to share good practices or experiences. In this context, municipal governments need far more central guidance and support. Internationally most sub-national governments are expected to follow clear rules and procedures. For example, while state governments in the United States have considerable autonomy, this is balanced by strong internal control according to federal standards for accounting and reporting. In the Slovak Republic, while regions and municipalities are mostly responsible for the provision of public education services, the central government serves an important regulatory role in areas such as student-learning objectives, levels and terms of funding, requirements for the competence of educational staff, salaries of teachers and the management of the register of schools and school facilities (Santiago et al., 2016[54]).

The government in North Macedonia should consider setting out clearer expectations for how municipalities are expected to perform their role for education delivery. While it is important that the central government does not inhibit local autonomy, setting out the principles for important functions, like school funding or staffing, would help guide municipalities. These principles would also provide the basis for local audit. For example, the ministry could set out the principles that should govern local school funding mechanisms. Many municipalities would also be helped by the ministry providing them with an example of a funding formula that they might adopt if they wish.

In undertaking this work, the government would need to understand more clearly the current flow of funds at the local levels. Research suggests that education funding in North Macedonia is not used efficiently at present (World Bank, n.d.[55]). For example, countries with similar levels of education expenditure achieve much better results in terms of student enrolment in upper secondary and learning outcomes (see Chapter 1). North Macedonia should evaluate education resource allocation to understand how it is being used, what impact is being achieved with the resources that are spent and use the results to inform the development of a more efficient resource allocation policy. Special attention should be paid to small, rural schools, as these have been identified as being some of the least efficient and lowest performing in the country. Questions to consider might include: Are there examples of municipalities where schools are more successful than others? What is their resource allocation scheme? Is there a relationship between how they allocate resources and their students’ outcomes? Can these lessons learnt be generalised to other municipalities in similar situations?

Consider establishing a local audit function to oversee municipal functions related to education

In many OECD countries, where sub-national government has significant responsibilities for public service delivery, there are dedicated institutions or officials to audit their activities. In federal countries like the United States or Mexico, these systems are very developed. For example, in Mexico, the country’s independent governments each have their own State Audit Institution (OECD, 2017[56]). Many unitary countries with decentralised responsibilities have also established frameworks for local audit. In Sweden the local assembly elects local auditors, whose role is regulated by national legislation (World Bank, 2006[53]). In England, while there is no longer a national auditing body for local government, local authorities must follow the national framework for the regulation of auditors of local bodies when appointing their own auditors (House of Commons, 2016[57]).

In the Slovak Republic, municipal auditors audit municipal schools, and there is local transparency of school spending due to the small number of schools per founder and the presence of community members on school boards. Schools and municipalities also publish invoices of their purchases on their websites. Meanwhile, the Slovak State Schools Inspectorate, a body at the national level, is responsible for monitoring schools and school facilities and checking the conditions and results of the education they provide, the quality of their management, the efficiency of their use of resources and their compliance with binding regulations (Santiago et al., 2016[54]).

North Macedonia should consider what mechanisms could be established for auditing the activities of its municipal governments. Given the very small size of each municipality, a separate function within the national auditing body might be established for this purpose. Local audits in North Macedonia should be responsible for ex post compliance and financial audits in the short to medium term. In the longer term, reflecting the changing role of audit bodies internationally, they could also be expected to take on a performance auditing role – in order to hold municipalities accountable for the quality of local education too.