# Annex A. Data sources and methods

## Data sources: overview

This publication reports the results of secondary analyses of data from several sources collected in surveys of students, teachers and principals. These data are drawn from PISA (Programme on International Student Assessment), TIMSS (Trends in International Mathematics and Science Study) and PIRLS (Progress in International Reading Literacy Study). PISA, TIMSS and PIRLS have been created to look at student achievements in maths and science (PISA and TIMSS) and text understanding (PISA and PIRLS). Background questionnaires provide relevant information about classroom or school practices which have been used to identify the extent to which they have changed over time. All these surveys are cross-sectional.

## Coverage of the statistics

PISA is designed to assess learning outcomes of 15-year-old students and make comparisons over time. PISA focuses on the extent to which students can apply the knowledge and skills they have learnt and practised at school when confronted with situations and challenges for which that knowledge may be relevant.

PISA uses questionnaires to collect background information from students and data on various aspects of organisation and educational provision in schools from school principals.

The target population of PISA is 15-year-old students in grade 7 or higher who attend educational institutions, including those enrolled part-time and those in vocational training programmes. It is important to note that the sample is not designed to be representative of schools or classrooms and has not been reweighted. Results should be read as “the percentage of 15-year-old students who report …..”

TIMSS and PIRLS are designed to measure student achievement around the world and make comparisons over time. TIMSS has two target populations—all students enrolled at the 4th grade and all students enrolled at the 8th grade, although countries may choose to assess either or both student populations. Fourth and eighth grade represent four and eight years of schooling respectively, counting from the first year of ISCED Level 1, providing the mean age at the time of testing is at least 9.5 years/13.5 years.

The target population for PIRLS is all students enrolled at the 4th grade. All schools of all educational sub-systems that have students learning full-time in the target grade are part of the international target population, including schools that are not under the authority of the national Ministry of Education or its equivalent.

TIMSS and PIRLS are designed to pay particular attention to students’ curricular and instructional experiences and therefore sample intact classes of students. However, as with PISA, TIMSS and PIRLS are not designed to be representative of schools or classrooms and data have not been reweighted. Results should be read as “the percentage of 4th /8th grade students who report…..”

## Country coverage

This publication incorporates information from 47 education systems or countries within the OECD, and 6 partner countries.

• 36 education systems within the OECD participated in PISA 2015, 34 in 2009 and 32 in 2006.

• 29 education systems within the OECD participated in TIMSS 2015, 38 in 2011 and 27 in 2007.

• 31 education systems within the OECD participated in PIRLS 2016, 29 in 2011 and 27 in 2006.

## Sample sizes

Table A.1. TIMSS sample sizes: Principals

OECD countries

2007

2011

2015

2007

2011

2015

Australia

229

280

287

228

277

285

Austria

196

158

Belgium Flemish

142

153

441

276

146

143

145

186

190

121

170

189

122

188

146

151

176

143

138

Chile

200

179

193

171

Colombia

142

148

Czech Republic

144

177

159

147

Denmark

137

216

193

Finland

145

158

145

France

164

Germany

246

197

204

Hungary

144

149

144

144

146

144

Ireland

150

149

149

Israel

146

151

200

Italy

170

202

164

170

197

161

Japan

148

149

148

146

138

147

Korea

150

149

150

150

150

Lithuania

156

154

225

145

141

208

Netherlands

141

128

129

New Zealand

220

180

174

158

145

Norway

145

119

140

139

134

143

Poland

150

150

Portugal

147

217

Slovak Republic

184

197

198

Slovenia

148

195

148

148

186

148

Spain

151

358

Sweden

155

152

144

159

153

150

Turkey

257

242

146

239

218

U.K. (England)

143

154

147

137

118

143

U.K. (Northern Ireland)

136

118

United States

257

369

250

239

501

246

U.S. (Massachusetts)

47

48

56

U.S. (Minnesota)

50

49

55

Non OECD countries

Hong Kong

126

136

132

120

117

133

Indonesia

230

149

153

Russian Federation

206

202

208

210

210

204

Singapore

177

176

179

164

165

167

South Africa

297

285

292

Table A.2. TIMSS sample sizes: Teachers

OECD countries

2007

2011

2015

2007

2011

2015

2007

2011

2015

Australia

360

594

584

251

802

824

496

1049

909

Austria

356

296

Belgium Flemish

268

295

807

384

278

252

235

222

234

308

300

195

226

265

165

192

323

167

279

362

309

214

244

202

219

245

96

Colombia

214

149

149

Chile

200

261

194

172

194

191

Czech Republic

253

291

347

212

845

Denmark

246

341

305

Finland

310

400

264

827

France

310

Germany

373

312

307

Hungary

255

324

307

289

280

232

987

1005

516

Ireland

220

214

516

352

Israel

394

514

596

270

282

347

Italy

323

314

328

287

205

21

287

205

228

Japan

250

265

292

216

181

231

178

151

169

Korea

168

226

243

376

310

181

202

215

Lithuania

283

282

301

209

222

264

596

617

905

Netherlands

218

210

223

New Zealand

609

494

499

354

435

265

329

Norway

280

280

270

175

220

264

171

80

Poland

257

390

Portugal

240

322

Slovak Republic

343

422

404

Slovenia

340

245

256

503

523

352

779

901

527

Spain

200

517

Sweden

396

369

233

491

405

198

680

540

210

Turkey

263

251

146

240

220

146

240

218

U.K. (England)

250

261

238

235

212

210

615

751

775

U.K. (Northern Ireland)

187

154

United States

904

767

540

532

559

429

687

931

517

U.S. (Massachusetts)

156

103

105

114

107

U.S. (Minnesota)

168

104

110

116

147

Non OECD countries

Hong Kong

282

267

279

145

148

173

123

124

145

Indonesia

378

149

170

276

259

Russian Federation

268

218

226

273

239

221

1083

916

748

Singapore

508

515

538

357

330

324

429

330

318

South Africa

325

305

Table A.3. TIMSS sample sizes: Students

OECD countries

2007

2011

2015

2007

2011

2015

Australia

4108

6146

10338

4069

7556

10338

Austria

4859

4668

Belgium Flemish

4849

8757

8757

4037

3645

3885

4235

3950

3956

6149

3950

3496

4570

4520

3448

4756

4520

Chile

5585

4849

5835

4849

Czech Republic

4235

4578

4845

Denmark

3519

3987

Finland

4638

4266

France

Germany

5200

3995

Hungary

4048

5204

4893

4111

5178

4893

Ireland

4560

4704

4704

Israel

5512

3294

4699

5512

Italy

4470

4200

4481

4408

3979

4481

Japan

4487

4411

4745

4312

4414

4745

Korea

4334

5309

4240

5166

5309

Lithuania

3980

4688

4347

3991

4747

4347

Netherlands

3349

3229

New Zealand

4940

5572

8142

5336

8142

Norway

4108

3121

4697

4627

3862

4697

Poland

5027

Portugal

4042

Slovak Republic

4963

5616

Slovenia

4351

4492

4257

4043

4415

4257

Spain

4183

Sweden

4676

4663

4090

5215

5573

4090

Turkey

7479

6079

4498

6928

6079

U.K. (England)

4316

3397

4814

4025

3842

4814

U.K. (Northern Ireland)

3571

United States

7896

12569

10221

7377

10477

10221

U.S. (Massachusetts)

1897

2075

U.S. (Minnesota)

1777

2500

Non OECD countries

Colombia

4801

4873

Hong Kong

3791

3957

4155

3470

4015

4155

Indonesia

4203

5795

Russian Federation

4464

4467

4780

4472

4893

4780

Singapore

5041

6368

6116

4599

5927

6116

South Africa

12514

11969

12514

Table A.4. PIRLS sample sizes: Principals, teachers and students

Principals

Teachers

Students

OECD countries

2006

2011

2016

2006

2011

2016

2006

2011

2016

Australia

280

286

513

531

6126

6341

Austria

158

158

150

263

284

259

5067

4670

4360

Belgium Flemish

137

148

237

277

4479

5198

Belgium French

150

127

158

277

217

254

4552

3727

4623

1111

926

1393

1119

23206

18245

150

145

233

218

4243

3789

185

190

127

210

217

166

3748

4244

3179

180

189

188

200

275

251

3988

4561

4270

Colombia

150

151

3966

Chile

154

154

4294

Czech Republic

0

177

157

235

270

4556

5537

Denmark

145

232

185

216

236

186

4001

4594

3508

Finland

0

145

151

285

295

4640

4896

France

169

174

163

261

276

284

4404

4438

4767

Germany

405

197

208

418

222

227

7899

4000

3959

Hungary

149

149

149

194

245

206

4068

5204

4623

Iceland

128

239

3673

Ireland

0

151

148

221

219

4524

4607

Israel

149

152

159

149

165

159

3908

4186

4041

Italy

150

202

149

198

239

217

3581

4189

3940

Latvia

145

150

213

216

4162

4157

Lithuania

144

154

195

270

277

243

4701

4661

4317

Luxembourg

178

363

5101

Netherlands

139

138

132

207

207

226

4156

3995

4206

New Zealand

243

192

188

509

434

411

6256

5644

5646

Norway

135

120

150

227

190

211

3837

3190

4232

Poland

148

150

148

250

257

214

4854

5005

4413

Portugal

0

148

218

242

318

4085

4642

Slovak Republic

167

197

220

263

314

333

5380

5630

5451

Slovenia

145

195

160

315

243

253

5337

4512

4499

Spain

152

312

629

193

402

678

4094

8580

14595

Spain (Andalusia)

0

149

0

197

188

0

4333

Sweden

147

152

154

255

254

214

4394

4622

4525

U.K. (England)

148

129

170

186

182

210

4036

3927

5095

U.K. (Northern Ireland)

136

134

184

161

3586

3693

United States

183

370

158

253

606

208

5190

12726

4425

Non OECD countries

Hong Kong

144

132

138

144

138

150

4712

3875

3349

Indonesia

168

158

168

163

4774

4791

Russian Federation

232

202

206

232

209

213

4720

4461

4577

Singapore

178

176

177

356

355

354

6390

6367

6488

South Africa

397

341

293

403

111

14657

3515

12810

Table A.5. PISA sample sizes: Principals and students

Principals

Students

OECD countries

2006

2009

2015

2006

2009

2015

Australia

350

345

758

14170

14251

14530

Austria

197

280

269

4927

6590

7007

Belgium

269

275

288

8857

8501

9651

861

908

759

22646

23207

20058

Chile

173

199

227

5233

5669

7053

Colombia

165

275

372

4478

7 921

11795

Czech Republic

244

260

344

5932

6064

6894

Denmark

209

285

333

4532

5924

7161

Estonia

169

175

206

4865

4727

5587

Finland

155

203

168

4714

5810

5882

France

179

166

252

4716

4298

6108

Germany

225

226

256

4891

4979

6522

Greece

189

183

211

4873

4969

5532

Hungary

189

187

245

4490

4605

5658

Iceland

135

129

124

3789

3646

3374

Ireland

164

141

167

4585

3937

5741

Israel

149

176

173

4584

5761

6598

Italy

796

1095

474

21773

30905

11583

Japan

181

185

198

5952

6088

6647

Korea

154

157

168

5176

4989

5581

Latvia

176

184

250

4719

4 502

4869

Lithuania

197

196

311

4744

4 528

6525

Luxembourg

31

39

44

4567

4622

5299

Mexico

1128

1531

275

30971

38250

7568

Netherlands

183

185

187

4871

4760

5385

New Zealand

170

161

183

4823

4643

4520

Norway

203

197

229

4692

4660

5456

Poland

221

179

169

5547

4917

4478

Portugal

172

212

246

5109

6298

7325

Slovak Republic

188

189

290

4731

4555

6350

Slovenia

356

337

333

6595

6155

6406

Spain

686

888

201

19604

25887

6736

Sweden

197

189

202

4443

4567

5458

Switzerland

509

425

227

12192

11812

5860

Turkey

160

170

187

4942

4996

5895

United Kingdom

494

481

550

13152

12179

14157

United States

166

160

177

5611

5233

5712

Non OECD

Brazil

625

947

841

9295

20 127

23141

Hong Kong

146

151

138

4645

4 837

5359

Indonesia

352

183

236

10647

5 136

6513

Russian Federation

209

213

210

5799

5 308

6036

Singapore

171

177

5 283

6115

## Year coverage

This publication focuses on change across time and therefore requires data from the same questions asked in different years. There are many such questions in the datasets employed, but it should be noted that the years in which they were answered varies.

Where possible, analysis focuses on change between 2006 and 2016, although data from TIMSS presents change between 2007 and 2015, and PISA data between 2006 and 2015 or 2009 and 2015. The years included in the analyses are indicated in the chapters.

In some cases, data are also available for an additional year between the two end points. In this case, the data from all three data collection exercises are represented in figures but only the end points are discussed in the text.

## Calculation of cross-country means and totals

Given the range of education systems covered in each chapter, cross-country means may not always incorporate the same countries or the same number of education systems. Where practical, the average cross-country statistics have been calculated using data for OECD countries (as in PISA, TIMSS, and PIRLS). In each indicator in TIMSS, PIRLS and PISA, the OECD average (unweighted) is computed taking into account the subset of OECD education systems with data available for all years concerned.

## Calculation of effect sizes

Effect sizes are presented for all analyses in addition to tests of statistical significance. Tests of significance allow the reader to determine whether the difference between the two percentages reported could have happened by chance if the actual difference is zero and thus consider the quality of the instrument used for measurement. However, statistical significance is dependent on the sample size (the larger the sample and the more confident the reader can be that even small differences wouldn’t have happened by chance) and can, in principle, be improved simply by increasing the number of observations. Yet this does not tell the reader anything about how meaningful the observed effects are in real-world terms. For example, a change in classroom practice could be statistically significant but only amount to a few percentage points of relative change with no practical meaning.

The effect size provides important information about the size of the relationship between two statistics. The main difference between effect size and significance is that change is normalized by the standard deviation as opposed to standard error, which means that the result no longer depends on sample size. The precise form of calculation depends on the type of question asked, but is typically calculated as:

$\mathrm{E}=\frac{{\mathrm{X}}_{2}-{\mathrm{X}}_{1}}{{\sigma }_{21}}$

i.e. as the change between a treatment and control group (or any two subgroups of a sample; or – as in our case - two different years), divided by a “pooled” standard deviation:

${\sigma }_{21}=\sqrt{\frac{{{\sigma }_{1}}^{2}-{{\sigma }_{2}}^{2}}{2}}$

Sometimes, the control group standard deviation or more complicated forms of pooled standard deviations are used instead of the one displayed. This book looks at effect sizes in two ways. One approach is to calculate country level effect sizes. Here, means and standard deviations refer to the individual country samples. The effect size calculation provides information about how much, in terms of their own standard deviation, a country has moved up (or down) over time. For country level effect sizes, and are estimated via σ =SE*√n (with n being the sample sizes), which provides a conservative (lower) estimate for the effect size (as n could potentially be overestimated by including invalid observations).

A second way of looking at effect size is required for questions that evaluate proportions, i.e. those that deal with categorical variables and ask, for example, “How often do you do this activity in class? Daily? At least weekly? At least monthly? Rarely or never?”. In this case, Cohen h is applied to carry out an arcsin-transformation, whereby h=2(arcsin √P1-arcsin √P2).

In accordance with common practices, effect sizes are assessed at three different levels. Effect sizes of less than 0.2 are considered negligible to very small, between 0.2 and 0.5 are come under small to modest, between 0.5 and 0.8, are large, and effect sizes above 0.8 are considered to be very large. While the usefulness of such cut-offs is debatable, this convention is followed by adding a colour coding in three different shades of blue when displaying effect sizes. The reader should interpret the colour coding with care as there is little practical difference between an effect size of 0.18 and 0.22, even if the colour coding is different.

## Further resources

The publication uses the OECD StatLinks service. Below each table and Figure is a URL that leads to a corresponding Excel workbook containing the underlying data for that indicator. These URLs are stable and will remain unchanged over time. In addition, readers of the electronic version of this publication (the e-book) will be able to click directly on the links and the relevant workbook will open in a separate window. The tables in the Excel files contain additional information and computations that could not be presented in the paper version.

End of the section – Back to iLibrary publication page