# Annex B. Composite indices of innovation

The analyses reported throughout this book have shown considerable variation in the amount of change in educational practices and thus the potential extent of innovation. In order to provide an overview of change across school and classroom practices and to draw some conclusions about the level of innovation in each country, it may be considered helpful to combine some of this information and look at the extent and focus of innovation within education in different countries.

There may be important differences between practices at different education levels (primary or secondary) or across disciplines. For this reason, broader composite indices have been created to group together practices and represent innovation at the discipline level- maths, science and reading and at the education level- primary and secondary, besides and index for overall educational innovation. Additionally, composite indices for ICT practices and more specific educational practices have been computed. This allows readers and policy makers to identify which aspects of countries’ education system(s) appear to have experienced relatively more innovation, and identifies countries that are innovating throughout the education system.

## Creating the indices

The indices draw from the analysis reported in this book. The approach used is broadly based on the guidance provided in the 2018 OECD handbook on constructing composite indicators. In particular, the indices are derived (as far as possible) from the definition of innovation discussed in the introduction and the process of creating them takes into account the need for appropriate data and imputing missing values.

The indices are based on the effect sizes of changes in responses to specific questions between baseline and endline years. Effect sizes reflect the size and direction of changes seen across two points in time, with a large positive effect size indicating a large increase over time and a large negative effect size indicating a large decrease. Effect sizes give a standardised measure of the change and can thus be easily added together.

Table B.1. Data sources for indices

Study name

Questionnaire used

TIMSS

Principals

Teachers

Students

PIRLS

Principals

Teachers

Students

PISA

Principals

Students

15-year-olds

## Education level, discipline level, and overall indices of innovation

These indices are constructed in order to represent change in practices across different grades, disciplines or throughout the whole education system. Given that both increases and decreases indicate change which can be part of innovation, the absolute value of the effect size has been used to create these indicators. An index that kept the sign of the effect size would make countries that have large changes in both directions appear to have no change at all.

In order to have a fair representation of innovation, different disciplines have been given different weights at different levels. Primary and secondary levels were given equal weights, whereas maths, science, and reading were given different weights defined on the basis of the relative instruction time spent on each one of the disciplines in every respective grade (source: Education at a Glance 2011) For instance, as reading instruction time is roughly twice as large as science instruction time in primary education, change in reading practices was given twice as much weight as change in science practices for this particular level.

## ICT and thematic indices

These indices illustrate change in more specific educational practices. However, it is relevant in this case to not only analyse whether the use of certain practices has met significant change, but also whether the use has more often increased or decreased. Thus, besides the value of composite indices with absolute effect sizes, the graphs for ICT and thematic practices also demonstrate the decomposition of the change into increases and decreases.

The conceptual grouping of these indicators was done to maintain a more or less balanced representation of practices across both grades and across all the disciplines. This allowed us to go ahead with an unweighted average rather than weighting by grades or disciplines.

## Missing values

Variation in the coverage of PISA and TIMSS/PIRLS means that school and classroom change effect sizes are therefore not available for all education systems across all of the questions asked. Furthermore, data are missing when certain questions (or questionnaires) were omitted at the national level at certain points in time. This is not an issue when reporting responses to a single question, but it does pose a potential problem when seeking to combine information across questions. In order to analyse as many countries as possible whilst keeping a wide range of questions in the analysis, it has been necessary to manage the missing data through a combination of deletion and estimations processes.

An iterative process has been used to manage observations (education systems) and variables (questions) with missing data, and some systems/countries and questions have had to be omitted in the construction of an index:

1. 1. Education systems that had effect size data for fewer than 20% of the potential question set were excluded.

2. 2. Following this, questions with high proportions of missing data were dropped. Specifically, those questions with effect size missing for more than 50% of the remaining database were excluded.

3. 3. Education systems with less than 60% valid data on the remaining questions were then excluded from the analysis.

Following the deletion process, some of the remaining education systems still had portions of missing data. Data was typically missing when education system had not participated in one of the surveys. As information for a whole dataset was missing, it was not possible to undertake an imputation at the indicator level. However, it was possible to estimate the effect of a missing dataset on the final index.

The estimation process uses information from countries having all the data points in order to estimate the impact of including a dataset on the index computation. We use this information to adjust the indices of countries missing one dataset. The process goes as follows:

• For education systems with all the information available, a subset of indices was computed, each one of them excluding one of the datasets from the index computation ( ${I}_{-A}$). The index including all the data was also calculated (I). For instance if other countries missed PISA, countries with all the information available will have an index excluding PISA ( ${I}_{-A}$ ) and one with PISA (I).

• The ratios of complete index to sub-indices were calculated for each country ( $I/{I}_{-A}\right)$.

• The cross-country mean ratio of full index to every sub-index was computed, giving us a dataset factor effect for each potential missing data source. ( ${DF}_{A}=Mean\left(I/{I}_{-A}\right)\right)$

• Finally, countries missing data from one source (A) had their index computed with all their information available ( ${I}_{m\left(A\right)}$). This index is then corrected by multiplying it by the dataset factor of the corresponding missing database, giving us the final composite index ( $I={I}_{m\left(A\right)}*{DF}_{A}$).

## Criteria for including questions in the indices

Highly correlated questions may unduly influence an index that seeks to explore the extent to which change occurs over different aspects of education, particularly given the existence of missing data. For this reason, where question effect sizes are highly correlated [0.6 or more using Person’s r] and the wording of the questions is the same across different grades or subjects, only the question with the highest absolute effect size at the OECD level has been included in the classroom, school and overall indices. Where the effect sizes of different questions within a module are correlated, but the wording differs, both questions have been included as separate items within the indicator. Questions have also been retained for indices at subject and grade level where the possibility of correlation is not a problem.

Table B.2. Number of available questions – Main indices

Countries and regions

Overall Index

Primary Index

Secondary Index

Maths Index

Science Index

Australia

125

62

66

28

49

-

Austria

-

59

-

-

-

33

Belgium (Fl.)

-

-

-

-

-

33

-

59

-

-

-

-

110*

62

51*

28

43*

33

110*

62

51*

28

43*

33

Czech Republic

-

62

-

-

-

-

Denmark

-

62

-

-

-

33

France

-

-

-

-

-

33

Germany

-

62

-

-

-

33

Hungary

125

62

66

28

49

33

Israel

100***

-

66

-

-

33

Italy

125

62

63

28

49

33

Japan

91**

-

66

28

49

-

Korea

91**

-

66

-

48

-

Latvia

-

-

-

-

-

33

Lithuania

125

62

66

28

49

33

Netherlands

-

62

-

-

-

33

New Zealand

125

61

-

28

49

33

Norway

125

62

61

28

49

33

Poland

-

59

-

-

-

33

Portugal

-

59

-

-

-

33

Slovak Republic

-

62

-

-

-

33

Slovenia

125

62

66

28

49

33

Spain

-

59

-

-

-

33

Sweden

125

62

66

28

49

33

Turkey

-

-

61

28

48

-

U.K. (England)

110*

62

51*

28

43*

33

United States

125

62

61

28

49

33

U.S. (Massachusetts)

-

-

51

-

-

-

U.S. (Minnesota)

-

-

51

-

-

-

Hong Kong, China

125

62

66

28

49

33

Indonesia

100***

-

61

-

-

33

Russian Federation

125

62

66

28

49

33

Singapore

125

62

56

28

43*

33

South Africa

-

-

51*

-

-

28

Note: * Missing PISA data- database effect estimation applied; ** Missing PIRLS data - database effect estimation applied; ***Missing TIMSS 4th grade data- database effect estimation applied.

Source: PISA, TIMSS and PIRLS databases.

Table B.3. Number of available questions – ICT indices

Countries and regions

ICT availability

ICT use

Australia

7

18

Austria

5

11

5

15

5

15

Chile

6

15

Czech Republic

5

11

Denmark

5

11

Finland

5

11

Hungary

7

18

Ireland

5

11

Italy

7

18

Japan

6

15

Korea

6

15

Lithuania

7

18

Netherlands

5

11

New Zealand

7

18

Norway

5

15

Poland

5

11

Portugal

5

11

Slovak Republic

5

11

Slovenia

7

18

Spain

5

11

Sweden

7

18

Turkey

4

-

U.K. (England)

5

15

United States

5

15

Hong Kong, China

7

18

Russian Federation

7

18

Singapore

7

18

Source: PISA, TIMSS and PIRLS databases.

Table B.4. Number of available questions – Thematic indices

Countries and regions

Assessment Index

Homework Index

Active Learning in Science Index

High order skills index

Knowledge transmission and acquisition index

Learning resource availability index

Rote learning Index

Professional Development- Teacher training index

Professional Development- Peer learning index

External relations and HRM index

Other practices index

Australia

10

8

8

16

8

-

9

16

9

8

-

Austria

-

-

-

-

5

11

-

-

-

-

7

-

-

-

-

5

-

-

-

-

-

-

10

-

6

11

8

12

8

16

9

6

-

10

8

6

11

8

12

8

16

9

6

-

Chile

-

-

-

-

-

-

8

16

-

-

-

Czech Republic

-

-

5

13

5

11

-

-

-

-

7

Denmark

-

-

5

13

5

11

-

-

-

-

7

Finland

-

-

-

-

-

11

-

-

-

-

7

France

-

-

-

11

-

-

-

-

-

-

-

Germany

-

-

5

13

5

-

-

-

-

-

7

Hungary

10

7

8

16

8

14

9

16

9

8

7

Ireland

-

-

5

-

-

-

-

-

-

-

-

Israel

10

8

-

14

5

11

6

-

6

7

7

Italy

10

8

8

16

8

14

9

16

9

-

7

Japan

-

8

8

10

6

-

8

16

9

7

-

Korea

-

8

8

10

6

-

8

16

9

7

-

Latvia

-

-

-

11

-

-

-

-

-

-

7

Lithuania

10

7

8

16

8

14

9

16

9

8

7

Netherlands

-

-

5

13

5

11

-

-

-

-

7

New Zealand

9

-

8

-

8

14

9

16

9

8

7

Norway

10

8

8

16

8

12

9

16

9

8

6

Poland

-

-

5

13

5

11

-

-

-

-

7

Portugal

-

-

-

-

5

11

-

-

-

-

7

Slovak Republic

-

-

5

13

5

11

-

-

-

-

7

Slovenia

10

7

8

16

8

14

9

16

9

8

7

Spain

-

-

5

13

5

11

-

-

-

-

7

Sweden

10

7

8

16

8

14

9

16

9

8

7

Turkey

-

8

-

10

6

-

8

16

9

7

-

U.K. (England)

10

8

6

11

8

12

8

16

9

6

-

United States

10

8

8

16

8

12

9

16

9

8

6

U.S. (Massachusetts)

-

8

-

-

-

-

-

-

-

-

-

U.S. (Minnesota)

-

8

-

-

-

-

-

-

-

-

-

Hong Kong, China

10

8

8

16

8

14

9

16

9

8

7

Indonesia

10

7

-

14

5

-

6

-

6

7

6

Russian Federation

10

7

8

16

8

14

9

16

9

8

7

Singapore

10

8

6

11

8

14

8

16

9

6

-

South Africa

10

-

-

-

5

-

-

-

6

-

-

## Developing and reporting the indices

The indices developed are intended to show the extent of change or innovation in one country when compared with other countries. They can be used to rank countries according to their relative levels of innovation across levels, disciplines and in more specific educational practices.

Discipline, education level and overall innovation indices for each country =100 x (weighted average of absolute effect sizes)

ICT and thematic innovation indices do not accord any weight to values, therefore the composite indices for each country= 100 x (unweighted average of absolute effect sizes)

The number of questions included depends on whether data exist in PISA and/or TIMSS/PIRLS and therefore differs across education systems. It also clearly depends on the indicator itself: up to 33 questions are used in the reading innovation index compared to 49 in science for example. The number of questions included across ICT and thematic indices also varies considerably.

It is possible for the absolute effect sizes to take a value that is greater than one; however in practice they mostly range between 0 and 1; the indices can therefore take values from 0 to positive infinity but in practice they never cross 100 for the broad composite indices. For the ICT and specific composite indices the index itself has the same range as the broader ones but their decomposition shows the negative and positive contributions as well.

## Cautions

### Question inclusion

The indices combine information from a large and diverse pool of questions asked on different surveys. On the assumption that each question can provide additional information about the extent of change and innovation in an education system, the process employed to develop the indices has drawn on as many of the questions as possible and their inclusion has been determined by the availability of valid data. However, a more theoretical approach focusing on the most relevant questions, or a statistical approach to data reduction may provide different results.

### Education system coverage

The indices provide some information about a subset of the education systems discussed in the previous chapters. This subset has been determined by the availability of data. It may be the case that other systems sit at the extremes of the ranking. It should be noted that the inclusion or removal of education systems would also impact on the estimation of missing values. Although it gives a robust synthesis of change covered by our change indicators, the country ranking should not be over-interpreted.

### OECD average

The OECD average is computed for all the education systems for which data are available for all years concerned. In calculating the weights of regions that do not correspond to an entire OECD member the following procedure has been followed. Education systems that are part of a country for which the overall data is available are not considered – this being the case for the different states in the United States. Conversely, education systems that do not have a figure for the whole country they belong to have been given weight equal to 1- this being the case, for example of Ontario and Flanders (Belgium) among others.

### Time periods

The effect size of the change in responses to a particular question is typically calculated across the same two points of time for each country but the two points in time may differ by question. The indices therefore show a tendency to change or innovate across slightly different time periods, rather than the extent of change over a specific time period.

### Interpreting the findings

The indices reported help the reader to consider the benefits of such a composite innovation indicator based on change measures, but may not provide a fully accurate representation of the level of change and innovation within a country. Whilst the indicator is based on many questions and observations, the missing data imputation and correction which were needed to construct the innovation indices invites the reader to be cautious. The innovation indices are mere indicators of innovation, and small differences in levels are almost certainly not meaningful.

A higher score on the indicator suggests that an education system is characterised by more change than other systems. However, there is currently no theory that could be applied to describe the different levels in terms of adequacy of innovation. Similarly, the scale does not provide information about what is necessary to move from one point to another. Additional work could be undertaken to develop qualitative descriptions of different points on the scale, but this should be preceded by improved data collection.

Component indicators of the ICT based and thematic composite indices

Table B.5. Indicators included in the composite index of innovation in computer availability in schools

Practice

Availability of computers (including tablets) to use during maths lessons

Primary and secondary

Availability of computers (including tablets) to use during science lessons

Primary and secondary

Availability of computers (including tablets) to use during reading lessons

Primary

Availability of desktop computers for use at school

Secondary

Availability of portable laptops or notebooks for use at school

Secondary

Table B.6. Indicators included in the composite index of innovation in ICT use in schools

Practice

Practising skills and procedures on computers in maths

Primary and secondary

Practising skills and procedures on computers in science

Primary and secondary

Study natural phenomena through simulations on computers in science

Primary and secondary

Processing and analysing data on computers in maths

Secondary

Processing and analysing data on computers in science

Secondary

Students using computers to write stories and texts in reading

Primary

Using computers to look for information in reading

Primary

Frequency of use of computer or a tablet at school

Primary

Use of digital devices for foreign language learning or mathematics

Secondary

Using digital devices for playing simulations at school

Secondary

Use of school computers for group work and communication with other students

Secondary

Teacher participation in a programme integrate information technology into mathematics

Primary and secondary

Teacher participation in a programme to integrate information technology into science

Primary and secondary

Table B.7. Indicators included in the composite index of innovation in active learning practices in science education

Practice

Students conducting scientific experiments and investigations in science

Primary and secondary

Study natural phenomena through simulations on computers in science

Primary and secondary

Students doing practical experiments in laboratories

Secondary

Students designing and planning science experiments

Primary and secondary

Scope for students to design their own experiments

Secondary

Table B.8. Indicators included in the composite index of innovation in homework practices

Practice

Frequency of homework in maths

Secondary

Frequency of homework in science

Secondary

Monitoring homework completion in maths

Secondary

Monitoring homework completion in science

Secondary

Students correcting their own homework in maths

Secondary

Students correcting their own homework in science

Secondary

Discussion of homework in class in maths

Secondary

Discussion of homework in class in science

Secondary

Table B.9. Indicators included in the composite index of innovation in assessment practices

Practice

Frequency of correction of assignment and feedback in maths

Secondary

Frequency of correction of assignment and feedback in science

Secondary

Importance of classroom tests in maths

Secondary

Importance of classroom tests in science

Secondary

Importance of national or regional achievement tests in maths

Secondary

Importance of national or regional achievement tests in science

Secondary

Primary

Emphasis on classroom test in reading

Primary

Emphasis on national or regional tests in reading

Primary

Primary

Table B.10. Indicators included in the composite index of innovation in fostering higher order skills

Practice

Students explaining their understanding of text in reading

Primary

Students explaining style and structure of text in reading

Primary

Students drawing inferences and generalisations from text in reading

Primary

Students identifying main ideas of text in reading

Primary

Primary

Opportunities for students to explain their ideas in reading

Secondary

Primary

Observing and describing natural phenomena in Primary

Primary and secondary

Students designing and planning science experiments

Primary and secondary

Students drawing conclusions from an experiment in science

Secondary

Teacher explaining relevance of broad science topics

Secondary

Teacher explaining practical application of school science topics

Secondary

Scope for students to design their own experiments

Secondary

Solving problems with no obvious method of solution in maths

Secondary

Table B.11. Indicators included in the composite index of innovation in independent knowledge acquisition

Practice

Primary

Reading textbooks and resource materials in science

Primary and secondary

Using computers to look for information in reading

Primary

Using computers to look up for ideas and information in maths

Primary and secondary

Using computers to look up for ideas and information in science

Primary and secondary

Table B.12. Indicators included in the composite index of innovation in rote learning practices

Practice

Memorising rules, procedures and facts as a pedagogical technique in maths

Primary and secondary

Memorising rules, procedures and facts as a pedagogical technique in science

Primary and secondary

Watching teachers demonstrate an experiment in science

Primary and secondary

Use scientific formulas and laws to solve routine problems

Secondary

Students doing practical experiments in laboratories

Secondary

Teaching new vocabulary systematically in reading

Primary

Table B.13. Indicators included in the composite index of innovation in formal teacher training

Practice

Teacher participation in mathematics content

Primary and secondary

Teacher participation in science content

Primary and secondary

Teacher participation in a program on maths pedagogy/instruction

Primary and secondary

Teacher participation in a program on science pedagogy/instruction

Primary and secondary

Teacher participation in a program on maths curriculum

Primary and secondary

Teacher participation in a program on science curriculum

Primary and secondary

Teacher participation in a program on mathematics assessments

Primary and secondary

Teacher participation in a program on science assessments

Primary and secondary

Table B.14. Indicators included in the composite index of innovation in teachers’ peer learning

Practice

Collaborating in planning and preparing instructional material

Primary and secondary

Primary and secondary

Discussing how to teach a particular topic

Primary and secondary

Note: In secondary education, these questions were asked to both maths and science teachers; in primary education, no distinction was made on the basis of disciplines.

Table B.15. Indicators included in the composite index of innovation in availability of school learning resources

Practice

Availability of a school library for students

Primary

Availability of a library or a reading corner in the classroom

Primary

Allowing students to borrow books from the classroom library

Primary

Students visiting a library other than their classroom library

Primary

Availability of desktop computers for use at school

Secondary

Availability of portable laptops or notebooks for use at school

Secondary

Frequency of use of computer or a tablet at school

Primary

Availability of computers (including tablets) to use during reading lessons

Primary

Availability of computers (including tablets) to use during maths lessons

Primary and secondary

Availability of computers (including tablets) to use during science lessons

Primary and secondary

Availability of a science laboratory for students

Primary and secondary

Table B.16. Indicators included in the composite index of innovation in school external relations and human resource management (HRM) practices

Practice

Primary

Incentives to recruit or retain maths teachers

Secondary

Incentives to recruit or retain science teachers

Secondary

Incentives to recruit or retain teachers other than maths and science

Secondary

Degree of parental involvement in school activities

Primary and secondary

Public posting of school achievement data (e.g. in the media)

Secondary

Tracking achievement data over time by an administrative authority

Secondary

Table B.17. Indicators included in the composite index of innovation in other miscellaneous educational practices

Practice

Teaching strategies for decoding sounds and words in reading

Primary

Same class-ability groups in reading classes

Primary

Primary

Primary

Use of school computers for group work and communication with other students

Secondary

Student grouping by ability into different classes

Secondary

Student grouping by ability within classes

Secondary

End of the section – Back to iLibrary publication page