9. Sarajevo Canton

Sarajevo Canton is one of the most experienced education authorities in BiH when it comes to using external, standardised assessments of student learning

Among other jurisdictions in BiH, Sarajevo Canton is one of the most experienced in using external standardised assessments of student learning. Canton authorities piloted a low-stakes standardised assessment in 2018 to support system monitoring but this never materialised into a regular assessment, partly because of student absenteeism and lack of reliable marking procedures (BiH, 2021[7]). Despite this, the canton has a well-established external examination at the end of basic schooling (known as the Grade 9 matura) that assesses students in Bosnian, Croatian or Serbian language and literature, mathematics and the students’ first foreign language. While the canton did not administer the matura in 2020 because of the COVID-19 pandemic, results are typically used as one of the criteria (alongside grade point average) for selecting students into either vocational or general secondary schools.

As the Grade 9 matura has high stakes for students, it influences what students learn and what teachers teach. This can create a “backwash effect” that can be either positive (e.g. by reinforcing curricula and learning outcomes) or negative (e.g. teaching to the test) (OECD, 2013[10]). While there are positive features of Sarajevo Canton’s Grade 9 matura, such as involving subject teachers in developing items, the test instrument consists mainly of multiple-choice questions with some short-answer items and does not include the types of open-ended tasks that can better capture higher-order competences. Moreover, the use of an external exam to sort students into secondary programmes has become less common internationally as policymakers seek to remove barriers to progression and reduce early tracking (Maghnouj, S. et al., 2020[11]). The canton’s lack of external examinations at the end of upper secondary (ISCED 3 or secondary education in BiH) is another notable difference between the way BiH education systems use examinations compared to international peers.

Several good practices that support system-level planning exist but schools lack capacity and support to improve instruction

Sarajevo Canton benefits from its proximity in location to the headquarters of international development partners and universities based in the BiH capital who regularly convene and can conduct research and support other activities to help strengthen the education sector. This context helps explain how the canton has established several good practices that support system-level planning. For example, the canton has a distinct education strategy and the Assembly of Sarajevo Canton aims to discuss reports from the education ministry on a regular basis. Despite some reported limitations (e.g. the inability to collect data on Roma students), the canton has one of the most sophisticated EMIS in the country, which collects and stores school-level data, as well as results from the external examination. Importantly, the ministry reportedly makes adjustments to its EMIS indicator framework when additional information is required. Despite these positive features, Sarajevo Canton struggles to implement education reforms, in part because of resource limitations, high turnover in ministry leadership and more recently, organisational changes to the bodies responsible for education within the canton. For example, since closing its pedagogical institute in 2021, Sarajevo Canton has not been able to carry out school monitoring activities. As school self-evaluation is not mandatory in the canton, the lack of any external or self-monitoring activities presents a major risk to the quality of school education.

Teacher policies are not designed to raise the quality of teaching or motivate teachers to improve their practice

Similar to other BiH jurisdictions, Sarajevo Canton has established a career structure for teachers, as well as systems for recruitment and promotion. However, recruitment procedures prioritise teacher candidates who have been unemployed over those with more teaching experience and promotion decisions are not systematically based on performance (World Bank, 2019[12]). As a result, the canton’s career structure does not motivate teachers to develop their competences, take on new tasks, and demonstrate high performance which could help improve the quality of schooling. This was not always the case: Sarajevo canton previously used a set of country-level teacher standards that were developed through an EU-funded project and covered domains that research recognises as important for quality teaching, such as learning and teaching; monitoring and assessment; co-operating with family and community; and professional development (among others) (British Council, WYG and GIZ, 2017[13]).

These standards underpinned the canton’s external teacher appraisal processes and helped teachers to identify their own strengths and areas for improvement. However, the canton discontinued this practice, partly because teacher unions determined the standards created too much of an administrative burden. Without clear professional standards and appraisal processes, Sarajevo Canton will likely struggle to align its policies related to recruitment, promotion and teacher education, with the incentives and support needed to encourage the student-centred approaches that teachers should use and competences they should help their students develop.

The establishment and use of learning standards is a challenge

Sarajevo Canton started reforming its curriculum in 2016, with a goal to move beyond a narrow focus of recalling knowledge to an approach that is focused on developing competences. In line with the curriculum reform, the canton is developing subject curricula based on learning outcomes, which will help define the knowledge and competences that students should have achieved by the time they complete key stages in their schooling (BiH, 2021[7]). At present, however, teachers, students and schools do not have clear learning outcomes and assessment criteria to guide the educational process and review student performance. This context contributes to important challenges, such as grade inflation and an implementation gap between the goals of the canton’s competence-based curricula and what is taught and learnt in classrooms.

Share the canton’s experience with standardised assessments while ensuring the reliability and validity of the Grade 9 matura

This review recommends that Sarajevo Canton concentrate on sharing its experience with standardised testing by helping to introduce a high-quality external BiH Matura examination that would help improve the reliability and rigour of upper secondary certifications (ISCED 3 or secondary education in BiH) across the country (see Chapter 2). This initiative would also help signal the mastery of students’ core competences to potential employers and help establish more merit-based and equitable selection into tertiary education. In parallel, however, authorities in Sarajevo Canton should take steps to improve the quality of its existing Grade 9 matura so that it can help make fair, trusted and transparent decisions about students’ pathways after basic schooling. The extent to which current external examinations in BiH jurisdictions fulfil their purposes depends upon the strength of test characteristics, which are themselves affected by several decisions related to the test’s design and procedures (see Table 9.2).

As Sarajevo Canton is undertaking major reforms to align its curricula with more competence-based approaches to education, the matura test instruments need to align with the new curricula and a clear set of learning outcomes. This will require clearly defining the competences that students should achieve by the time they complete key stages in their schooling and developing assessment criteria to guide the educational process. The canton’s external Grade 9 matura also has a role to play in either helping to reinforce the competence-based curricula and desired assessment practices or hold back these efforts by encouraging teachers and students to narrowly focus on recalling the knowledge needed for success on more traditional types of examinations. Specifically, using a combination of multiple-choice items and open-ended items in the Grade 9 matura would help improve the validity of this exam. To ensure the Grade 9 matura and other external assessments are valid instruments that have positive backwash effects, canton authorities can take the following steps:

• Develop high-quality item banks. To develop valid exam instruments, canton authorities should develop item banks that reflect the goals and standards of the CCC that the Agency for Pre-school, Primary and Secondary Education (APOSO) has developed as well as criterion-based assessments. Item developers could draw on examples from PISA and other competence-based assessments to introduce more applied item types into local standardised tests. For instead of asking mathematics questions to assess whether students can recall complex procedures to solve logarithms, questions might ask students to formulate, use and interpret mathematical concepts to solve problems in real life contexts (OECD, 2019[15]). Teachers should be involved in writing these items, which can increase their familiarity with assessing higher-order competences.

• Recruit and train technical staff to manage standardised tests. To support local examinations and assessments, the units responsible for developing and managing the Grade 9 matura should recruit and train the technical staff needed to undertake this task. In particular, they will need individuals with expertise and experience in psychometrics and statistics, who will be able to develop and ensure the quality of competency-based items, as well as analyse results at a more granular level to inform instruction. This could, for instance, be done through co-operation with the University of Sarajevo or as one of the mandates for the canton’s new Institute for Development of Pre-tertiary Education.

Given that students in Grade 9 are still young (generally around 14-15 years old), and have 3-4 years of schooling left (depending on their secondary track), it is right to prioritise the validity of the external examinations at this level. However, there are trade-offs associated with the marking of more complex, open-ended test items, as these may require subjective judgements, which in turn makes reliability harder to ensure. Students in Sarajevo Canton take the Grade 9 matura in their schools and results are marked by an external commission of teachers. At present, there are no external moderation procedures to ensure that all students take the exam under the same conditions or that marking is consistent. One way to address this challenge is to conduct random external visits during the administration of the exam to ensure that all schools are following the Grade 9 matura rulebook. Another way is to have qualified staff from the Institute for Development of Pre-tertiary Education (once it is developed) perform spot checks on samples of student work to ensure that the marks granted by the commission of teachers do not deviate from marking guides. Authorities could also move towards computer-based test administration and marking, trends that have accelerated in response to the COVID-19 pandemic (OECD, 2020[16]). Introducing moderation processes would be an effective way for canton authorities to prioritise the validity of the Grade 9 matura while maintaining its reliability and integrity as a trusted instrument for allocating students into different secondary pathways.

In the medium to longer term, canton authorities should also consider investing their resources in implementing a standardised assessment at earlier levels of schooling instead of as a selection examination. Creating a low-stakes standardised test that does not have consequences for student’s future pathways but primarily serves to support their learning could help to better consolidate the foundational numeracy and literacy skills of younger students in basic schooling while also providing a benchmark against local, regional, country-wide or international standards.

Re-introduce the use of professional standards to inform teacher appraisal processes

Sarajevo Canton should re-introduce the use of teacher standards and mandate the regular, standards-based appraisal of teachers’ work for development purposes. This practice should take place within schools and be conducted by the school leadership team or experienced teachers (see Chapter 3). Many OECD countries require that regular teacher appraisals take place annually to help ensure that teachers receive regular feedback, as well as opportunities and support to improve their practice (OECD, 2013[10]). If these standards are to be considered more than a narrow administrative tool, but rather means to reinforce expectations for teachers’ roles in line with the competence-based curriculum, it will be important that the standards align with the canton’s other teacher policies, such as those related to recruitment, promotion and professional development. Participating in the development of a state-level online platform that collects and disseminates resources for using the standards in teachers’ daily work and creating opportunities for job-embedded learning related to the standards are among some of the ways that Sarajevo Canton can ensure the standards serve as a meaningful and relevant reference for strengthening the quality of teaching and learning. Such a platform could be led by APOSO, in partnership with pedagogical institutes.

Strengthen the capacity of schools to drive their own improvement

Sarajevo Canton does not currently conduct external school evaluations, and self-evaluations are not mandatory. Moreover, principal appointments remain vulnerable to politicisation despite changes to selection procedures over the past decade. The canton’s lack of professional standards for principals, as well as appraisal processes and training opportunities also means that principals are often unprepared to make well-informed judgements about how their school is performing in relation to broader system priorities and how to develop school improvement plans. Other jurisdictions in BiH, including Republika Srpska and West Herzegovina Canton, have developed – or are in the process of developing – professional standards and appraisal processes for school principals. This policy action can help strengthen the capacity of principals in Sarajevo Canton to drive improvements in their schools. Investing in school principals and providing schools with support is especially important since the canton does not currently have a separate pedagogical institute to provide expert advice to schools (although they are in the process of forming the Institute for Development of Pre-tertiary Education).

The Ministry of Education can help promote improvements to school performance by working with APOSO to define a core set of five to ten school quality indicators that principal and other actors can use to better understand a school’s level of performance. These indicators can be tailored to Sarajevo Canton’s specific context but indicators should address aspects of schooling that are most important to students’ learning and development, as well as student outcomes (e.g. outcomes related to student learning and progress, equity, etc.). Sarajevo Canton could also leverage its EMIS to provide summary reports on the state of education that describe what schools in the canton are doing well and what they can do to improve, which could inform school-led improvement efforts. In the longer term, the canton should re-introduce external school evaluations and associated supports to raise school performance.