copy the linklink copied! 4. Reporting the results of the Survey of Adult Skills (PIAAC)

copy the linklink copied! picture

This chapter examines the proficiency levels used to report the results of the Survey of Adult Skills (PIAAC). It provides information on the languages used and how results were reported in countries/economies that conducted the survey in more than one language.

    

This chapter describes how the results from the Survey of Adult Skills (PIAAC) are reported. It shows how the literacy, numeracy and problem-solving items used in the assessment are categorised according to their difficulty, the cognitive strategies required of adults to answer the questions, the real-life contexts in which such problems/questions may arise, and the medium used to deliver the item to the respondent. The chapter also shows how the proficiency levels for each of the three domains are related to the scores, and describes in detail what adults can do at each of the proficiency levels. The chapter concludes with information about the languages in which the test was conducted and the approach to reporting in countries/economies where the assessment was delivered in more than one language.

copy the linklink copied! The proficiency scales

In each of the three domains assessed, proficiency is considered as a continuum of ability involving the mastery of information-processing tasks of increasing complexity. The results are represented on a 500-point scale. At each point on the scale, an individual with a proficiency score of that particular value has a 67% chance of successfully completing test items located at that point.1 This individual will also be able to complete more difficult items (those with higher values on the scale) with a lower probability of success and easier items (those with lower values on the scale) with a greater chance of success.

To illustrate this point, Table 4.1 shows the probability with which a person with a proficiency score of 300 on the literacy scale can successfully complete items of greater and lesser difficulty. As can be seen, a person with a proficiency score of 300 will successfully complete items of this level of difficulty 67% of the time, items with a difficulty value of 250, 95% of the time, and items with a difficulty value of 350, 28% of the time.

copy the linklink copied!
Table 4.1. Probability of successfully completing items of varying difficulty for a person scoring 300 on the literacy scale

Difficulty score (literacy scale)

200

250

300

350

Probability of success

0.97

0.95

0.67

0.28

copy the linklink copied! Proficiency levels

The proficiency scale in each of the domains assessed can be described in relation to the items that are located at the different points on the scale according to their difficulty. Tables 4.2, 4.3 and 4.4 present the location of the test items used in the Survey of Adult Skills on the difficult scales in the three domains assessed. In addition to the difficulty score, unit name and ID, a description of the key features of the item is provided in relation to the relevant measurement framework.

To help interpret the results, the reporting scales have been divided into “proficiency levels” defined by particular score-point ranges. Six proficiency levels are defined for literacy and numeracy (Levels 1 through 5 plus below Level 1) and four for problem solving in technology-rich environments (Levels 1 through 3 plus below Level 1). These descriptors provide a summary of the characteristics of the types of tasks that can be successfully completed by adults with proficiency scores in a particular range. In other words, they offer a summary of what adults with particular proficiency scores in a particular skill domain can do.

With the exception of the lowest level (below Level 1), tasks located at a particular level can be successfully completed approximately 50% of the time by a person with a proficiency score at the bottom of the range defining the level. In other words, a person with a score at the bottom of Level 2 would score close to 50% in a test made up of items of Level 2 difficulty. A person at the top of the level will get items located at that level correct most of the time. The “average” individual with a proficiency score in the range defining a level will successfully complete items located at that level approximately two-thirds of the time.

copy the linklink copied!
Table 4.2. [1/2] Literacy item map

CBA/PBA Design

Difficulty score

Unit name

Item ID

Cognitive strategies

Context

Medium

Format

376

Library Search

C323P005

Evaluate and reflect

Education and training

Digital

Multiple

374

Work-related Stress

C329P003

Integrate and interpret

Work-related

Digital

Multiple

372

CANCO

C306B111

Access and identify

Work-related

Print

Continuous

371

Baltic Stock Market

C308A116

Access and identify

Personal

Print

Mixed

359

Apples

P317P001

Integrate and interpret

Personal

Print

Continuous

350

Summer Streets

C327P004

Evaluate and reflect

Community

Digital

Mixed

349

Work-related Stress

C329P002

Evaluate and reflect

Work-related

Digital

Multiple

348

Library Search

C323P002

Integrate and interpret

Education and training

Digital

Multiple

347

Milk Label

P324P002

Integrate and interpret

Personal

Print

Mixed

337

Baltic Stock Market

C308A118

Access and identify

Personal

Print

Mixed

329

Generic Medicines

C309A322

Integrate and interpret

Personal

Print

Mixed

329

Library Search

C323P004

Evaluate and reflect

Education and training

Digital

Multiple

324

International Calls

C313A410

Access and identify

Personal

Print

Mixed

320

Summer Streets

C327P003

Integrate and interpret

Community

Digital

Mixed

318

Distances-Mexican Cities

C315B512

Integrate and interpret

Community

Print

Non-continuous

316

Civil Engineering

C318P003

Integrate and interpret

Education and training

Digital

Mixed

315

International Calls

C313A411

Access and identify

Personal

Print

Mixed

312

Memory Training

C310A407

Integrate and interpret

Personal

Print

Continuous

312

Milk Label

P324P003

Access and identify

Personal

Print

Mixed

309

TMN Anti-Theft

C305A218

Integrate and interpret

Community

Print

Continuous

306

Summer Streets

C327P002

Evaluate and reflect

Community

Digital

Mixed

304

Contact Employer

C304B711

Integrate and interpret

Work-related

Print

Continuous

303

Civil Engineering

C318P001

Access and identify

Education and training

Digital

Mixed

298

Summer Streets

C327P001

Integrate and interpret

Community

Digital

Mixed

297

Baltic Stock Market

C308A119

Access and identify

Personal

Print

Mixed

294

Lakeside Fun Run

C322P003

Access and identify

Personal

Digital

Mixed

293

Lakeside Fun Run

C322P004

Access and identify

Personal

Digital

Mixed

289

Library Search

C323P003

Access and identify

Education and training

Digital

Multiple

288

MEDCO Aspirin

C307B402

Access and identify

Personal

Print

Continuous

286

Contact Employer

C304B710

Access and identify

Work-related

Print

Continuous

286

International Calls

C313A413

Access and identify

Personal

Print

Mixed

286

Discussion forum

C320P003

Evaluate and reflect

Work-related

Digital

Multiple

285

Discussion forum

C320P004

Evaluate and reflect

Work-related

Digital

Multiple

283

Lakeside Fun Run

C322P001

Integrate and interpret

Personal

Digital

Mixed

281

Discussion forum

C320P001

Integrate and interpret

Work-related

Digital

Multiple

279

Baltic Stock Market

C308A121

Access and identify

Personal

Print

Mixed

272

Generic Medicines

C309A319

Access and identify

Personal

Print

Mixed

272

Memory Training

C310A406

Access and identify

Personal

Print

Continuous

272

International Calls

C313A414

Access and identify

Personal

Print

Mixed

265

Apples

P317P003

Evaluate and reflect

Personal

Print

Continuous

262

Apples

P317P002

Integrate and interpret

Personal

Print

Continuous

260

TMN Anti-theft

C305A215

Access and identify

Community

Print

Continuous

257

International Calls

C313A412

Access and identify

Personal

Print

Mixed

254

Baltic Stock Market

C308A120

Access and identify

Personal

Print

Mixed

251

Internet Poll

C321P001

Integrate and interpret

Community

Digital

Multiple

244

CANCO

C306B110

Access and identify

Work-related

Print

Continuous

244

Lakeside Fun Run

C322P005

Access and identify

Personal

Digital

Mixed

240

Lakeside Fun Run

C322P002

Evaluate and reflect

Personal

Digital

Mixed

239

Baltic Stock Market

C308A117

Access and identify

Personal

Print

Mixed

239

Generic Medicines

C309A320

Access and identify

Personal

Print

Mixed

238

Internet Poll

C321P002

Access and identify

Community

Digital

Multiple

219

Generic Medicines

C309A321

Integrate and interpret

Personal

Print

Mixed

207

Guadeloupe

P330P001

Access and identify

Community

Print

Mixed

201

Dutch Women

C311B701

Access and identify

Community

Print

Mixed

169

MEDCO Aspirin

C30B7401

Access and identify

Personal

Print

Continuous

162

Election Results

C302BC02

Access and identify

Community

Print

Mixed

136

Employment Ad

C300AC02

Access and identify

Work-related

Print

Continuous

75

SGIH

C301AC05

Access and identify

Community

Print

Non-continuous

Note: CBA = computer-based assessment; PBA = paper-based assessment.

copy the linklink copied!
Table 4.2. [2/2] Literacy item map

PBA ONLY (Indonesia [Jakarta])

Difficulty score

Unit name

Item ID

Cognitive strategies

Context

Medium

Format

385

CANCO

P306B111

Integrate and interpret

Work-related

Digital

Multiple

374

Water Chlorination

P340A426

Access and identify

Community

Print

Continuous

347

Milk Label

P324P002

Evaluate and reflect

Education

Digital

Multiple

337

Baltic Stock Market

P308A118

Access and identify

Personal

Print

Mixed

331

Water Chlorination

P340A424

Integrate

Community

Print

Continuous

329

Apples

P317P001

Integrate and interpret

Community

Print

Non-continuous

329

Generic Medicine

P309A322

Evaluate and reflect

Community

Digital

Mixed

320

International Calls

P313A411

Access and identify

Community

Print

Mixed

318

Mexican Cities

P315B512

Integrate and interpret

Community

Print

Non-continuous

317

International Calls

P313A410

Integrate and interpret

Personal

Print

Mixed

312

Memory Training

P310A407

Integrate and interpret

Personal

Print

Continuous

312

Milk Label

P324P003

Evaluate and reflect

Education

Digital

Multiple

301

Water Chlorination

P340A422

Access and identify

Community

Print

Continuous

297

Baltic Stock Market

P308A119

Integrate and interpret

Personal

Print

Mixed

295

Baltic Stock Market

P308A116

Access and identify

Personal

Print

Mixed

291

International Calls

P313A413

Access and identify

Personal

Print

Mixed

288

MEDCO Aspirin

P307B402

Access and identify

Personal

Print

Mixed

281

Preschool Rules

P303A103

Access and identify

Personal

Print

Mixed

281

Generic Medicine

P309A319

Access and identify

Personal

Digital

Mixed

267

Generic Medicine

P309A321

Evaluate and reflect

Education and training

Digital

Multiple

265

Exercise Equipment

P312A315

Access and identify

Personal

Digital

Mixed

265

Apples

P317P003

Integrate and interpret

Community

Print

Non-continuous

262

Apples

P317P002

Integrate and interpret

Community

Print

Non-continuous

260

TMN AntiTheft

P305A215

Access and identify

Personal

Print

Mixed

255

Swimmer Completes

P341B502

Access and identify

Community

Print

Continuous

255

Problem Solver

P314B101

Access and identify

Personal

Digital

Mixed

254

Baltic Stock Market

P308A120

Access and identify

Personal

Print

Continuous

253

International Calls

P313A412

Access and identify

Personal

Print

Mixed

250

TMN AntiTheft

P305A218

Access and identify

Personal

Print

Mixed

247

International Calls

P313A414

Access and identify

Personal

Print

Mixed

246

Memory Training

P310A406

Evaluate and reflect

Work-related

Digital

Multiple

245

Exercise Equipment

P312A318

Access and identify

Personal

Digital

Mixed

239

Baltic Stock Market

P308A117

Access and identify

Personal

Print

Mixed

235

CANCO

P306B110

Access and identify

Education and training

Digital

Mixed

234

Problem Solver

P314B102

Access and identify

Personal

Digital

Mixed

231

Preschool Rules

P303A102

Access and identify

Personal

Print

Continuous

219

Generic Medicine

P309A320

Access and identify

Education and training

Digital

Multiple

169

MEDCO Aspirin

P307B401

Access and identify

Work-related

Print

Continuous

162

Election Results

P302BC02

Integrate and interpret

Personal

Print

Continuous

149

Swimmer Completes

P341B501

Access and identify

Community

Print

Continuous

140

Guadeloupe

P330P001

Integrate and interpret

Work-related

Digital

Multiple

136

Employment Ad.

P300AC02

Integrate and interpret

Personal

Print

Mixed

75

SGIH

P301AC05

Access and identify

Personal

Print

Continuous

Note: PBA = paper-based assessment.

copy the linklink copied!
Table 4.3. [1/2] Numeracy item map

CBA/PBA Design

Difficulty score

Unit name

Item ID

Content

Cognitive strategies

Context

375

Dioxin (MOD)

C612A518

Pattern, relationships, change

Interpret, evaluate

Community and society

354

Educational Level

C632P001

Data and chance

Interpret, evaluate

Community and society

348

Compound Interest

P610A515

Pattern, relationships, change

Act upon, use

Education and training

341

Wine

P623A618

Data and chance

Interpret, evaluate

Community and society

332

Weight history

C660P004

Pattern, relationships, change

Act upon, use

Community and society

326

Cooper test

C665P002

Pattern, relationships, change

Act upon, use

Everyday life

324

Amoeba

C641P001

Pattern, relationships, change

Act upon, use

Education and training

320

BMI

C624A620

Pattern, relationships, change

Act upon, use

Everyday life

318

Peanuts

C634P002

Pattern, relationships, change

Act upon, use

Everyday life

317

NZ Exports

C644P002

Pattern, relationships, change

Act upon, use

Community and society

315

Study fees

C661P002

Data and chance

Interpret, evaluate

Community and society

315

Package

C657P001

Dimension and shape

Interpret, evaluate

Work-related

314

Fertilizer

C651P002

Pattern, relationships, change

Interpret, evaluate

Work-related

308

Study fees

C661P001

Data and chance

Interpret, evaluate

Community and society

308

Inflation

C620A612

Data and chance

Act upon, use

Community and society

307

Orchestra tickets

C664P001

Pattern, relationships, change

Act upon, use

Work-related

305

Peanuts

C634P001

Pattern, relationships, change

Act upon, use

Everyday life

303

Map

C617A605

Dimension and shape

Interpret, evaluate

Work-related

301

Classified

C622A615

Pattern, relationships, change

Act upon, use

Work-related

297

SixPack1

C618A608

Quantity and number

Act upon, use

Education and training

296

Temp Scale

C611A517

Dimension and shape

Interpret, evaluate

Community and society

294

Lab Report

C636P001

Quantity and number

Interpret, evaluate

Everyday life

287

Map

C617A606

Dimension and shape

Act upon, use

Work-related

282

Tiles

C619A609

Dimension and shape

Act upon, use

Everyday life

276

Wine

C623A617

Quantity and number

Act upon, use

Community and society

276

Weight history

C660P003

Data and chance

Interpret, evaluate

Everyday life

273

Solution

C606A509

Dimension and shape

Act upon, use

Work-related

267

Inflation

C620A610

Data and chance

Identify, locate or access

Community and society

266

Educational Level

C632P002

Data and chance

Interpret, evaluate

Community and society

261

Temp Scale

C611A516

Dimension and shape

Interpret, evaluate

Community and society

260

Urban Population

C650P001

Data and chance

Interpret, evaluate

Community and society

260

Tree

C608A513

Dimension and shape

Act upon, use

Everyday life

259

Photo

C605A506

Dimension and shape

Act upon, use

Everyday life

259

Price Tag

C602A503

Quantity and number

Act upon, use

Everyday life

258

Wine

C623A616

Quantity and number

Act upon, use

Community and society

256

Rug Production

C646P002

Data and chance

Act upon, use

Community and society

250

Logbook

C613A520

Pattern, relationships, change

Act upon, use

Work-related

249

Path

C655P001

Pattern, relationships, change

Act upon, use

Everyday life

242

Photo

C605A507

Dimension and shape

Interpret, evaluate

Everyday life

240

Rope

P666P001

Dimension and shape

Act upon, use

Work-related

239

TV

C607A510

Pattern, relationships, change

Act upon, use

Everyday life

238

Price Tag

C602A502

Quantity and number

Act upon, use

Everyday life

234

Cooper test

C665P001

Data and chance

Interpret, evaluate

Everyday life

231

Candles

C615A603

Dimension and shape

Act upon, use

Work-related

231

Airport Timetable

C645P001

Dimension and shape

Act upon, use

Work-related

228

Gas Gauge

C604A505

Quantity and number

Act upon, use

Everyday life

227

Photo

C605A508

Quantity and number

Act upon, use

Everyday life

221

BMI

C624A619

Data and chance

Identify, locate or access

Everyday life

221

Candles

C615A602

Dimension and shape

Interpret, evaluate

Education and training

217

SixPack1

C618A607

Quantity and number

Act upon, use

Everyday life

195

Odometer

P640P001

Dimension and shape

Act upon, use

Everyday life

185

Watch

C614A601

Quantity and number

Interpret, evaluate

Everyday life

179

Parking Map

C635P001

Dimension and shape

Identify, locate or access

Work-related

168

Price Tag

C602A501

Quantity and number

Act upon, use

Everyday life

155

Election results

C600AC04

Quantity and number

Act upon, use

Work-related

129

Bottles

C601AC06

Dimension and Shape

Interpret, evaluate

Everyday life

136

Employment Ad

C300AC02

Access and identify

Work-related

Print

75

SGIH

C301AC05

Access and identify

Community

Print

Note: CBA = computer-based assessment; PBA = paper-based assessment.

copy the linklink copied!
Table 4.3. [2/2] Numeracy item map

PBA ONLY (Indonesia [Jakarta])

Difficulty score

Unit name

Item ID

Content

Cognitive strategies

Context

348

CompoundInteres

P610A515

Pattern, relationships, change

Act upon, use

Further learning

341

Wine

P623A618

Data and chance

Interpret, evaluate

Community and society

332

Weighthistory

P660P004

Pattern, relationships, change

Act upon, use

Everyday life

328

TempScale

P611A517

Dimension and Shape

Interpret, evaluate

Community and society

320

BMI

P624A620

Pattern, relationships, change

Act upon, use

Everyday life

318

Peanuts

P634P002

Pattern, relationships, change

Act upon, use

Everyday life

314

Fertilizer

P651P002

Pattern, relationships, change

Interpret, evaluate

Workplace

308

Inflation

P620A612

Data and chance

Act upon, use

Community and society

305

Peanuts

P634P001

Pattern, relationships, change

Act upon, use

Everyday life

301

Classified

P622A615

Pattern, relationships, change

Act upon, use

Workplace

301

Orchestraticket

P664P001

Pattern, relationships, change

Act upon, use

Workplace

294

LabReport

P636P001

Quantity and number

Interpret, evaluate

Everyday life

286

Package

P657P001

Dimension and Shape

Interpret, evaluate

Workplace

282

Tiles

P619A609

Dimension and Shape

Act upon, use

Everyday life

281

TempScale

P611A516

Dimension and Shape

Interpret, evaluate

Community and society

276

Wine

P623A617

Quantity and number

Act upon, use

Community and society

276

Weighthistory

P660P003

Data and chance

Interpret, evaluate

Everyday life

273

Solution

P606A509

Dimension and Shape

Act upon, use

Workplace

267

Inflation

P620A610

Data and chance

Identify, locate or access

Community and society

260

UrbanPopulation

P650P001

Data and chance

Interpret, evaluate

Community and society

260

Tree

P608A513

Dimension and Shape

Act upon, use

Everyday life

258

Wine

P623A616

Quantity and number

Act upon, use

Community and society

258

SixPack1

P618A608

Quantity and number

Act upon, use

Further learning

252

PriceTag

P602A503

Quantity and number

Act upon, use

Everyday life

250

Logbook

P613A520

Pattern, relationships, change

Act upon, use

Workplace

248

Parking Time

P616A604

Dimension and Shape

Interpret, evaluate

Workplace

241

Path

P655P001

Pattern, relationships, change

Act upon, use

Everyday life

240

Rope

P666P001

Dimension and Shape

Act upon, use

Workplace

239

TV

P607A510

Pattern, relationships, change

Act upon, use

Everyday life

231

Candles

P615A603

Dimension and Shape

Act upon, use

Workplace

228

GasGauge

P604A505

Quantity and number

Act upon, use

Everyday life

226

PriceTag

P602A502

Quantity and number

Act upon, use

Everyday life

223

Airport Timetab

P645P001

Dimension and Shape

Act upon, use

Everyday life

221

Craft

P621A613

Quantity and number

Act upon, use

Further learning

221

BMI

P624A619

Data and chance

Identify, locate or access

Everyday life

221

Candles

P615A602

Dimension and Shape

Interpret, evaluate

Further learning

197

Raincoat

P603A504

Pattern, relationships, change

Act upon, use

Everyday life

195

Odometer

P640P001

Dimension and Shape

Act upon, use

Everyday life

178

Watch

P614A601

Quantity and number

Interpret, evaluate

Everyday life

168

PriceTag

P602A501

Quantity and number

Act upon, use

Everyday life

165

SixPack1

P618A607

Quantity and number

Act upon, use

Everyday life

155

Election Result

P600AC04

Quantity and number

Act upon, use

Everyday life

127

Bottles

P601AC06

Dimension and Shape

Interpret, evaluate

Everyday life

Note: PBA = paper-based assessment.

copy the linklink copied!
Table 4.4. [1/2] Problem solving in technology-rich environments item map

Difficulty score

Item name

Item ID

Content

Cognitive strategies

Context

Description

Technology

Task

374

Class Attendance

U04A

Spread-sheet, e-mail

  • Multiple steps

  • Single constraint

  • Explicit problem statement

  • Goal setting and progress monitoring

  • Planning, self-organising

  • Acquiring and evaluating information

  • Making use of information

Work-related

Using information embedded in an e-mail message, establish and apply the criteria to transform the e-mail information to a spreadsheet. Monitor the progress of correctly organising information to perform computations through novel built-in functions.

355

Locate E-mail – File 3 e-mails

U11B

E-mail

  • Single step

  • Single constraint

  • Implicit problem statement

  • Goal setting and progress monitoring

  • Planning, self-organising

  • Acquiring and evaluating information

  • Making use of information

Personal

Infer the proper folder destination in order to transfer a subset of incoming e-mail messages based on the subject header and the specific contents of each message.

346

Meeting Rooms

U02

E-mail, Internet

  • Multiple steps

  • Multiple constraints

  • Implicit problem statement

  • Goal setting and progress monitoring

  • Planning, self-organising

  • Acquiring and evaluating information

  • Making use of information

Work-related

Using information from a novel Internet application and several e-mail messages, establish and apply criteria to solve a scheduling problem where an impasse must be resolved, and communicate the outcome.

342

Sprained Ankle – Site Evaluation Table

U06A

Internet

  • Single step

  • Single constraint

  • Explicit problem statement

  • Acquiring and evaluating information

Personal

Evaluate several entries in a search engine results page given an explicit set of separate reliability criteria.

325

Sprained Ankle – Reliable/Trustworthy Site

U06B

Internet

  • Multiple steps

  • Single constraint

  • Explicit problem statement

  • Goal setting and progress monitoring

  • Acquiring and evaluating information Making use of information

Personal

Apply evaluation criteria and then navigate through multiple websites to infer the most reliable and trustworthy site. Monitoring throughout the process is required.

320

Tickets

U21

Internet

  • Multiple steps

  • Multiple constraints

  • Explicit problem statement

  • Goal setting and progress monitoring

  • Planning, self-organising

  • Acquiring and evaluating information

Personal

Use a novel Internet-based application involving multiple tools to complete an order based on a combination of explicit criteria.

321

Lamp Return

U23

Internet, e-mail

  • Multiple steps

  • Single constraint

  • Explicit problem statement

  • Goal setting and progress monitoring

  • Planning, self-organising

  • Acquiring and evaluating information

Personal

Enact a plan to navigate through a website to complete an explicitly specified consumer transaction. Monitor the progress of submitting a request, retrieving an e-mail message, and filling out a novel online form.

316

CD Tally

U03A

Internet, spreadsheet

  • Single step

  • Single constraint

  • Implicit problem statement

  • Goal setting and progress monitoring Planning, self-organising

  • Making use of information

Work-related

Organise large amounts of information in a multiple column spreadsheet and determine a value based on a single explicit criterion; use a drop-down menu in a novel Internet application to communicate the result.

305

Digital Photography Book Purchase

U07

Internet

  • Multiple steps

  • Multiple constraints

  • Implicit problem statement

  • Goal setting and progress monitoring

  • Acquiring and evaluating information

Work-related

Choose an item on a web page that best matches a set of given criteria from a search engine results page; the information can be made available only by clicking on links and navigating through several web pages; based on a search engine results page, navigate through several Internet sites in order to choose an item on a web page that best matches a set of given criteria.

299

Party Invitations Accommo-dations

U01B

E-mail

  • Single step

  • Multiple constraints

  • Implicit problem statement

  • Planning, self-organising

  • Making use of information

Personal

Categorise a small number of messages in an e-mail application by creating a new folder; evaluate the contents of the entries based on one criterion in order to file them in the proper folder.

296

Club Membership – Eligibility for Club President

U19B

Spreadsheet, e-mail

  • Single step

  • Multiple constraints

  • Implicit problem statement

  • Goal setting and progress monitoring

  • Planning, self-organising

  • Acquiring and evaluating information

  • Making use of information

Society-community

Organise large amounts of information in a multiple-column spreadsheet using multiple explicit criteria; locate and mark relevant entries.

286

Party Invitations – Can/Cannot Come

U01A

E-mail

  • Single step

  • Single constraint

  • Implicit problem statement

  • Planning, self-organising

  • Making use of information

Personal

Categorise a small number of messages in an e-mail application into existing folders according to one explicit criterion.

286

Reply All

U16

E-mail

  • Single step

  • Single constraint

  • Explicit problem statement

  • Acquiring and evaluating information

  • Planning, self-organising

Personal

With a defined goal and explicit criteria, use e-mail and send information to three people.

268

Club Membership – Member ID

U19A

Spreadsheet, e-mail

  • Single step

  • Single constraint

  • Implicit problem statement

  • Planning, self-organising

  • Acquiring and evaluating information

Society-community

Locate an item within a large amount of information in a multiple-column spreadsheet based on a single explicit criterion; use e-mail to communicate the result.

Literacy and numeracy

Six proficiency levels are defined for the domains of literacy and numeracy. The score-point ranges defining each level and the descriptors of the characteristics of tasks located at each of the levels can be found in Table 4.5. In the case of literacy and numeracy, the score-point ranges associated with each proficiency level are the same as those that apply in the International Adult Literacy Survey (IALS) and the Adult Literacy and Life Skills Survey (ALL) for document and prose literacy and in ALL for numeracy. However, the descriptors that apply to the proficiency levels in the domains of literacy and numeracy differ between the Survey of Adult Skills (PIAAC) and IALS and ALL.

copy the linklink copied!
Table 4.5. [1/2] Proficiency levels: Literacy and numeracy

Level

Score range

Literacy

Numeracy

Below Level 1

Below 176 points

The tasks at this level require the respondent to read brief texts on familiar topics to locate a single piece of specific information. There is seldom any competing information in the text and the requested information is identical in form to information in the question or directive. The respondent may be required to locate information in short continuous texts. However, in this case, the information can be located as if the text was non-continuous in format. Only basic vocabulary knowledge is required, and the reader is not required to understand the structure of sentences or paragraphs or make use of other text features. Tasks below Level 1 do not make use of any features specific to digital texts.

Tasks at this level require the respondents to carry out simple processes such as counting, sorting, performing basic arithmetic operations with whole numbers or money, or recognising common spatial representations in concrete, familiar contexts where the mathematical content is explicit with little or no text or distractors.

1

176 to less than 226 points

Most of the tasks at this level require the respondent to read relatively short digital or print continuous, non-continuous, or mixed texts to locate a single piece of information that is identical to or synonymous with the information given in the question or directive. Some tasks, such as those involving non-continuous texts, may require the respondent to enter personal information onto a document. Little, if any, competing information is present. Some tasks may require simple cycling through more than one piece of information. Knowledge and skill in recognising basic vocabulary determining the meaning of sentences, and reading paragraphs of text is expected.

Tasks at this level require the respondent to carry out basic mathematical processes in common, concrete contexts where the mathematical content is explicit with little text and minimal distractors. Tasks usually require one-step or simple processes involving counting; sorting; performing basic arithmetic operations; understanding simple percentages such as 50%; and locating and identifying elements of simple or common graphical or spatial representations.

2

226 to less than 276 points

At this level, the medium of texts may be digital or printed, and texts may comprise continuous, non-continuous, or mixed types. Tasks at this level require respondents to make matches between the text and information, and may require paraphrasing or low-level inferences. Some competing pieces of information may be present. Some tasks require the respondent to

  • cycle through or integrate two or more pieces of information based on criteria;

  • compare and contrast or reason about information requested in the question; or

  • navigate within digital texts to access-and-identify information from various parts of a document.

Tasks at this level require the respondent to identify and act on mathematical information and ideas embedded in a range of common contexts where the mathematical content is fairly explicit or visual with relatively few distractors. Tasks tend to require the application of two or more steps or processes involving calculation with whole numbers and common decimals, percentages and fractions; simple measurement and spatial representation; estimation; and interpretation of relatively simple data and statistics in texts, tables and graphs.

3

276 to less than 326 points

Texts at this level are often dense or lengthy, and include continuous, non-continuous, mixed, or multiple pages of text. Understanding text and rhetorical structures become more central to successfully completing tasks, especially navigating complex digital texts. Tasks require the respondent to identify, interpret, or evaluate one or more pieces of information, and often require varying levels of inference. Many tasks require the respondent to construct meaning across larger chunks of text or perform multi-step operations in order to identify and formulate responses. Often tasks also demand that the respondent disregard irrelevant or inappropriate content to answer accurately. Competing information is often present, but it is not more prominent than the correct information.

Tasks at this level require the respondent to understand mathematical information that may be less explicit, embedded in contexts that are not always familiar and represented in more complex ways. Tasks require several steps and may involve the choice of problem-solving strategies and relevant processes. Tasks tend to require the application of number sense and spatial sense; recognising and working with mathematical relationships, patterns, and proportions expressed in verbal or numerical form; and interpretation and basic analysis of data and statistics in texts, tables and graphs.

4

326 to less than 376 points

Tasks at this level often require respondents to perform multiple-step operations to integrate, interpret, or synthesise information from complex or lengthy continuous, non-continuous, mixed, or multiple type texts. Complex inferences and application of background knowledge may be needed to perform the task successfully. Many tasks require identifying and understanding one or more specific, non-central idea(s) in the text in order to interpret or evaluate subtle evidence-claim or persuasive discourse relationships. Conditional information is frequently present in tasks at this level and must be taken into consideration by the respondent. Competing information is present and sometimes seemingly as prominent as correct information.

Tasks at this level require the respondent to understand a broad range of mathematical information that may be complex, abstract or embedded in unfamiliar contexts. These tasks involve undertaking multiple steps and choosing relevant problem-solving strategies and processes. Tasks tend to require analysis and more complex reasoning about quantities and data; statistics and chance; spatial relationships; and change, proportions and formulas. Tasks at this level may also require understanding arguments or communicating well-reasoned explanations for answers or choices.

5

Equal to or higher than 376 points

At this level, tasks may require the respondent to search for and integrate information across multiple, dense texts; construct syntheses of similar and contrasting ideas or points of view; or evaluate evidence-based arguments. Application and evaluation of logical and conceptual models of ideas may be required to accomplish tasks. Evaluating reliability of evidentiary sources and selecting key information is frequently a requirement. Tasks often require respondents to be aware of subtle, rhetorical cues and to make high-level inferences or use specialised background knowledge.

Tasks at this level require the respondent to understand complex representations and abstract and formal mathematical and statistical ideas, possibly embedded in complex texts. Respondents may have to integrate multiple types of mathematical information where considerable translation or interpretation is required; draw inferences; develop or work with mathematical arguments or models; and justify, evaluate and critically reflect upon solutions or choices.

This is because the domain of literacy in the Survey of Adult Skills replaces the previously separate domains of prose and document literacy used in IALS and ALL, and because the survey defines proficiency levels differently than the other surveys do. An explanation of these changes and their impact is provided in Annex A.

Tables 4.6 and 4.7 show the probability that adults with particular proficiency scores will complete items of different levels of difficulty in the domains of literacy and numeracy. For example, an adult with a proficiency score of 300 in literacy (i.e. the mid-point of Level 3) has a 68% chance of successfully completing items of Level 3 difficulty. He or she has a 29% chance of completing items of Level 4 difficulty and a 90% probability of successfully completing items of Level 2 difficulty.

copy the linklink copied!
Table 4.6. Probability of successfully completing items at different difficulty levels, by proficiency score: Literacy

Item difficulty

Proficiency score

150

175

200

225

250

275

300

325

350

375

400

425

Level 1

0.56

0.68

0.78

0.86

0.92

0.95

0.97

0.98

0.99

0.99

1.00

1.00

Level 2

0.08

0.15

0.27

0.44

0.63

0.80

0.90

0.95

0.98

0.99

0.99

1.00

Level 3

0.01

0.03

0.06

0.13

0.26

0.46

0.68

0.83

0.92

0.96

0.98

0.99

Level 4

0.01

0.01

0.02

0.05

0.09

0.16

0.29

0.47

0.65

0.80

0.90

0.95

copy the linklink copied!
Table 4.7. Probability of successfully completing items at different difficulty levels, by proficiency score: Numeracy

Item difficulty

Proficiency score

150

175

200

225

250

275

300

325

350

375

400

425

Level 1

0.47

0.60

0.72

0.82

0.89

0.93

0.96

0.98

0.99

0.99

1.00

1.00

Level 2

0.11

0.20

0.33

0.49

0.66

0.80

0.89

0.94

0.97

0.98

0.99

1.00

Level 3

0.02

0.04

0.08

0.15

0.26

0.43

0.63

0.80

0.90

0.95

0.98

0.99

Level 4

0.02

0.03

0.05

0.08

0.14

0.24

0.37

0.54

0.69

0.80

0.88

0.93

Problem solving in technology-rich environments

The problem-solving proficiency scale was divided into four levels. The problem solving in technology-rich environments framework (PIAAC Problem Sloving in Technology-Rich Environment, 2009) identifies three main dimensions along which problems vary in quality and complexity. These are (1) the technology dimension, (2) the task dimension and (3) the cognitive dimension. Variations along each of these dimensions contribute to the overall difficulty of a problem. For instance, a problem is likely to be more complex if it involves the combined use of more than one computer application (e.g. e-mail and a spreadsheet); similarly, a problem is more complex if the task is defined in vague terms, as opposed to fully specified. Finally, a problem is likely to be more difficult if the respondent has to generate lots of deductions and inferences than if he or she just has to assemble or match different pieces of explicit information. The relationship between these dimensions and the proficiency levels is presented in Table 4.8. The descriptors of the levels are presented in Table 4.9.

copy the linklink copied!
Table 4.8. Technology, task and cognitive features of problems at each of the three main levels of proficiency

Level

Technology features

Task features

Cognitive processes

1

  • Generic applications

  • Little or no navigation required

  • Relevant information is directly available

  • Use of facilitating tools not required

  • Few steps

  • Single operators

  • Reach a given goal Apply explicit criteria

  • Minimal monitoring demands

  • Simple relevance match

  • Categorical reasoning

  • No integration or transformation

2

  • Both generic and novel applications (e.g. web-based services)

  • Some navigation required to acquire information or perform actions

  • Use of tools facilitates operations

  • Multiple steps

  • Multiple operators

  • Goal may need to be defined

  • Apply explicit criteria

  • Generally higher monitoring demands

  • Generally involves resolving impasses

  • Some evaluation of relevance

  • Some integration or transformation

  • Inferential reasoning

3

  • Generic and novel applications

  • Some navigation required to acquire information or perform actions

  • Use of tools required to efficiently solve the problem

  • Multiple steps

  • Multiple operators

  • Goal may need to be defined

  • Establish and apply criteria

  • Generally high monitoring

  • High inferential reasoning and integration

  • Evaluate relevance and reliability

  • Generally involves resolving impasses

copy the linklink copied!
Table 4.9. Proficiency levels: Problem solving in technology-rich environments

Level

Score range

The types of tasks completed successfully at each level of proficiency

Below Level 1

Below than 241 points

Tasks are based on well-defined problems involving the use of only one function within a generic interface to meet one explicit criterion without any categorical, inferential reasoning or transforming of information. Few steps are required and no sub goal has to be generated.

1

241 to less than 291 points

At this level, tasks typically require the use of widely available and familiar technology applications, such as e-mail software or a web browser. There is little or no navigation required to access the information or commands required to solve the problem. The problem may be solved regardless of the respondent’s awareness and use of specific tools and functions (e.g. a sort function). The tasks involve few steps and a minimal number of operators. At the cognitive level, the respondent can readily infer the goal from the task statement; problem resolution requires the respondent to apply explicit criteria; and there are few monitoring demands (e.g. the respondent does not have to check whether he or she has used the appropriate procedure or made progress towards the solution). Identifying contents and operators can be done through simple match. Only simple forms of reasoning, such as assigning items to categories, are required; there is no need to contrast or integrate information.

2

291 to less than 341 points

At this level, tasks typically require the use of both generic and more specific technology applications. For instance, the respondent may have to make use of a novel online form. Some navigation across pages and applications is required to solve the problem. The use of tools (e.g. a sort function) can facilitate the resolution of the problem. The task may involve multiple steps and operators. The goal of the problem may have to be defined by the respondent, though the criteria to be met are explicit. There are higher monitoring demands. Some unexpected outcomes or impasses may appear. The task may require evaluating the relevance of a set of items to discard distractors. Some integration and inferential reasoning may be needed.

3

Equal to or higher than 341 points

At this level, tasks typically require the use of both generic and more specific technology applications. Some navigation across pages and applications is required to solve the problem. The use of tools (e.g. a sort function) is required to make progress towards the solution. The task may involve multiple steps and operators. The goal of the problem may have to be defined by the respondent, and the criteria to be met may or may not be explicit. There are typically high monitoring demands. Unexpected outcomes and impasses are likely to occur. The task may require evaluating the relevance and reliability of information in order to discard distractors. Integration and inferential reasoning may be needed to a large extent.

Table 4.10 shows the probability of adults with particular proficiency in problem solving in technology-rich environments completing problem solving items of different levels of difficulty.

copy the linklink copied!
Table 4.10. Probability of successfully completing items at different difficulty levels by proficiency score: Problem solving in technology-rich environments

Item difficulty

Proficiency score

190

215

240

265

290

315

340

365

390

415

Level 1

0.02

0.06

0.17

0.40

0.69

0.87

0.95

0.98

0.99

1.00

Level 2

0.03

0.05

0.10

0.19

0.35

0.56

0.76

0.88

0.94

0.97

Level 3

0.00

0.01

0.02

0.05

0.13

0.29

0.49

0.67

0.80

0.87

A note about the reporting of problem solving in technology-rich environments

The populations for whom proficiency scores for problem solving in technology-rich environments are reported are not identical across countries/economies. Proficiency scores relate only to the proportion of the target population in each participating country that was able to undertake the computer-based version of the assessment, and thus meets the preconditions for displaying competency in this domain.

Four groups of respondents did not take the computer-based assessment,2 those who:

  • indicated in completing the background questionnaire that they had never used a computer (group 1)

  • had some experience with computers but who “failed” the ICT core assessment (see Chapter 3) designed to determine whether a respondent had the basic computer skills necessary to undertake the computer-based assessment (group 2)

  • had some experience with computers but opted not to take the computer-based assessment (group 3)

  • did not attempt the ICT core for literacy-related reasons (group 4).

By definition, a minimum level of competency in the use of computer tools and applications and a minimum level of proficiency in literacy and numeracy is required in order to display proficiency in problem solving in technology-rich environments. Individuals in groups 1 and 2 are, thus, treated as not meeting the necessary preconditions for displaying proficiency and have no proficiency score in the domain of problem solving in technology-rich environments.

Respondents who did not attempt the ICT core for literacy-related reasons (group 4) have not been attributed a problem-solving score due to lack of sufficient information.

Respondents who opted not to take the computer-based assessment (group 3), however, represent a different category. They are individuals who, on their own initiative, decided to take the paper-and-pencil version of the assessment without going through the process designed to direct respondents to the computer-based or paper pathways of the assessment. As a result, it is not known whether or not they possessed the computer skills necessary to complete the computer-based assessment.

Three options for how to treat this group were considered: imputing their proficiency in problem solving on the basis of their proficiency in literacy and numeracy and their background characteristics; treating them as non-respondents; or reporting them as a separate category of the group that could not display competency. The latter option was adopted. Imputation was rejected on the grounds that refusals appeared to have different characteristics to respondents taking the computer-based assessment pathway. In fact, they appeared to be more similar to the respondents who did not have computer skills than to those who took the computer-based assessment. The option of treating them as non-respondents was rejected for similar reasons.

In reporting the results concerning problem solving in technology-rich environments, the following approach was adopted:

  • When reporting proficiency in problem solving in technology-rich environments on the continuous scale at the country level, the proportion of the population displaying proficiency is reported in conjunction with country-level statistics (e.g. means, standard deviations, etc).

  • When reporting distributions of the population by proficiency levels, information is presented for the entire adult population as a whole (i.e. those displaying proficiency plus those not displaying proficiency). The number or proportion of the population not displaying proficiency is always reported when results are presented by proficiency level.

copy the linklink copied! Test languages and reporting

In each participating country/economy, the Survey of Adult Skills was administered in the official national language(s) of the country and, in some cases, in a widely used language in addition to the national language(s). A small number of countries/economies administered the cognitive assessments in the national language only but administered the background questionnaire in the national language and a widely spoken language. The objective there was to minimise the number of respondents who failed to provide information for language-related reasons. Table 4.11 shows the languages in which the survey was administered.

copy the linklink copied!
Table 4.11. Test languages by country

OECD countries and economies

Language(s) of the cognitive assessment

Language(s) of the background questionnaire

Australia

English

English

Austria

German

German, Bosnian/Serbian/Croatian, Turkish

Canada

English, French

English, French

Chile

Spanish

Spanish

Czech Republic

Czech

Czech

Denmark

Danish

Danish

England (UK)

English

English

Estonia

Estonian, Russian

Estonian, Russian

Finland

Finnish, Swedish

Finnish, Swedish

Flanders (Belgium)

Dutch

Dutch

France

French

French

Germany

German

German

Greece

Greek

Greek

Hungary

Hungarian

Hungarian

Ireland

English

English

Israel

Hebrew, Arabic, Russian

Hebrew, Arabic, Russian

Italy

Italian

Italian

Japan

Japanese

Japanese

Korea

Korean

Korean

Lithuania

Lithuanian

Lithuanian

Mexico

Spanish

Spanish

Netherlands

Dutch

Dutch

New Zealand

English

English

Northern Ireland (UK)

English

English

Norway

Norwegian

Norwegian, English

Poland

Polish

Polish

Slovak Republic

Slovak, Hungarian

Slovak, Hungarian

Slovenia

Slovenian

Slovenian

Spain

Castilian, Catalan, Basque, Galician, Valencian

Castilian, Catalan, Basque, Galician, Valencian

Sweden

Swedish

Swedish

Turkey

Turkish

Turkish

United States 2012/2014

English

English, Spanish

United States 2017

English

English, Spanish

Partners

Cyprus1

Greek

Greek

Ecuador

Spanish

Spanish

Jakarta (Indonesia)

Indonesian

Indonesian

Kazakshtan

Kazakh, Russian

Kazakh, Russian

Peru

Spanish

Spanish

Russian Federation2

Russian

Russian

Singapore

English, Chinese

English

1. Note by Turkey: The information in this document with reference to “Cyprus” relates to the southern part of the Island. There is no single authority representing both Turkish and Greek Cypriot people on the Island. Turkey recognises the Turkish Republic of Northern Cyprus (TRNC). Until a lasting and equitable solution is found within the context of the United Nations, Turkey shall preserve its position concerning the “Cyprus issue”.

Note by all the European Union Member States of the OECD and the European Union: The Republic of Cyprus is recognised by all members of the United Nations with the exception of Turkey. The information in this document relates to the area under the effective control of the Government of the Republic of Cyprus.

2. See note at the end of this chapter.

Countries and economies are ranked in alphabetical order.

For those countries/economies that tested in more than one language, results are presented as a single proficiency score. In other words, the mean proficiency score for literacy in Estonia, for example, is the mean proficiency of Estonian adults in reading in either Estonian or Russian. In only one country, Canada, was the sample designed to allow for reliable proficiency estimates in each of the languages in which the test was administered (in this case, English and French). However, as is the case for all other countries in which the test was administered in more than one language, Canadian results are presented in the international report in the form of a single proficiency estimate rather than as separate estimates for English and French speakers.

The Survey of Adult Skills was designed to assess the proficiency of the adult population in reading, in working with numbers, and in solving problems in the language(s) that are most relevant to and/or commonly used in the economic and civic life (e.g. in interaction with public bodies and institutions, in educational institutions) of a participating country. Therefore, poor performance in the test language(s) among non-native speakers of those languages, such as immigrants and their children, is not necessarily indicative of poor performance, as such. In the case of non-native speakers of the test language(s), low proficiency cannot be assumed to indicate low proficiency in their native language. A Turkish immigrant in Germany, for example, may display poor skills in the test language (German) but be a proficient reader and have good problem-solving skills when working in Turkish.

copy the linklink copied!
A note regarding the Russian Federation

The sample for the Russian Federation does not include the population of the Moscow municipal area. The data published, therefore, do not represent the entire resident population aged 16-65 in the Russian Federation but rather the population of the Russian Federation excluding the population residing in the Moscow municipal area.

More detailed information regarding the data from the Russian Federation as well as that of other countries can be found in the Technical Report of the Survey of Adult Skills, Third Edition (OECD, 2019).

References

OECD (2012), Literacy, Numeracy and Problem Solving in Technology-Rich Environments: Framework for the OECD Survey of Adult Skills, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264128859-en.

PIAAC Expert Group in Problem Solving in Technology-Rich Environments (2009), “PIAAC Problem Solving in Technology-Rich Environments: A Conceptual Framework”, OECD Education Working Papers, No. 36, OECD Publishing, Paris, http://dx.doi.org/10.1787/220262483674.

Notes

← 1. This differs from the approach used in IALS and ALL in which a value of 0.80 was used to locate items and test takers on the relevant scales. Further information on the change in approach and its impact is provided in Annex A.

← 2. Defined as taking, at a minimum, the core literacy and numeracy assessments on the computer.

Metadata, Legal and Rights

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.

https://doi.org/10.1787/f70238c7-en

© OECD 2019

The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.

4. Reporting the results of the Survey of Adult Skills (PIAAC)