copy the linklink copied! 1. What the Survey of Adult Skills (PIAAC) measures

An assessment of key information-processing competencies

The skills assessed in the Survey of Adult Skills are conceived as “key information-processing competencies”.1 They represent skills essential for accessing, understanding, analysing and using text-based information and, in the case of some mathematical information, information in the form of representations (e.g. pictures, graphs). These texts and representations may exist in the form of printed material or screen-based displays.

They are considered to be “key information-processing skills” in that they are:

• Necessary for fully integrating and participating in the labour market, education and training, and social and civic life.

• Highly transferable, in that they are relevant to many social contexts and work situations.

• “Learnable” and, therefore, subject to the influence of policy.

At the most fundamental level, literacy and numeracy constitute a foundation for developing higher-order cognitive skills, such as analytic reasoning, and are essential for gaining access to and understanding specific domains of knowledge. In addition, these skills are relevant across the range of life contexts, from education through work to home life and interaction with public authorities. In information-rich societies, in which information in text format (whether print-based or digital) is ubiquitous, a capacity to read and respond to text-based information is essential, whether that means understanding the user information on a packet of medicine, reacting appropriately to a memo from a colleague or superior at work, or enrolling a child at school. Similarly, numeracy skills are essential in most areas of life, from buying and selling goods, to understanding pension entitlements, to planning one’s working day.

In addition, the capacity to manage information and solve problems in technology-rich environments – that is, to access, evaluate, analyse and communicate information – is becoming as important as understanding and interpreting text-based information and being able to handle mathematical content. Information and communication technology (ICT) applications have become a feature in most workplaces, in education, and in everyday life.

A use-oriented conception of competency

Literacy, numeracy and problem solving are competencies that are essential for functioning in the modern world, for realising the myriad tasks adults must undertake in the various life contexts. Adults read, deal with situations involving mathematical content and representations, and try to solve problems in order to do things and achieve certain objectives in a range of contexts. Consequently, the focus of the Survey of Adult Skills is less on the mastery of certain content (e.g. vocabulary or arithmetical operations) and a set of cognitive strategies than on the ability to draw on this content and these strategies to successfully perform information-processing tasks in a variety of real-world situations.

Proficiency as a continuum

The competencies assessed in the Survey of Adult Skills are understood as involving a continuum of proficiency. Individuals are considered to be proficient to a greater or a lesser degree in the competency in question as opposed to being either “proficient” or “not proficient”. In other words, there is no threshold that separates those who have the competency in question from those who do not. The measurement scales describe gradations in the complexity of the information-processing tasks in the domains of literacy, numeracy and problem solving in technology-rich environments. In each domain, this complexity is seen as a function of a small number of factors, such as the type of cognitive operations required by the task, the presence of distracting information, and the nature of information and knowledge required to successfully complete a task.

At the lower end of the proficiency scale, individuals have skills that allow them to undertake tasks of limited complexity, such as locating single pieces of information in short texts in the absence of other distracting information, or performing simple mathematical operations involving a single step, such as counting or ordering. At the highest level of proficiency, adults can undertake tasks that involve integrating information across multiple dense texts, reasoning by inference, working with mathematical arguments and models, and solving complex problems using information technologies that require navigation and the use of multiple tools.

Literacy and numeracy are often described as “basic” skills, in that they provide a “foundation” on which the development of other competencies rests. This description is unfortunate in that it can give the impression that such skills are less complex than certain other “higher-order” skills, or that the policy interest in such skills lies in ensuring that the population possesses an acceptable minimum or basic level of proficiency in these skills. It is important to emphasise that the objective of the Survey of Adult Skills is to see how the adult population is distributed over the entire spectrum of proficiency in each of the domains assessed, not to assess whether adults have achieved a basic level of skills.

The importance of contextual information

In addition to estimating the level and distribution of proficiency in the population, the Survey of Adult Skills seeks to provide information that will enable policy makers and others to better understand the relationship between the measured skills and economic and social outcomes, and the factors related to acquiring, maintaining, developing and losing skills. The assessment of literacy, numeracy and problem solving in technology-rich environments is, thus, complemented by information on the use of the measured cognitive skills and certain generic skills (see Chapter 2 for more information). This information includes details about respondents’ literacy and numeracy practices and their use of information and communication technologies (ICT) at work and in other contexts. It also encompasses the extent to which individuals are required to use a range of generic skills in their work, including interpersonal skills, such as collaboration and influencing, learning skills, organising, including both self-organisation and delegating tasks, and physical skills. Respondents also report on how and whether their skills and qualifications match the requirements of their jobs.

Definition

In the Survey of Adult Skills, literacy is defined as “understanding, evaluating, using and engaging with written texts to participate in society, to achieve one’s goals, and to develop one’s knowledge and potential”.

Key to this definition is the fact that literacy is defined in terms of the reading of written texts and does not involve either the comprehension or production of spoken language or the production of text (writing). While literacy is commonly seen as encompassing the ability to write as well as read (UNESCO, 2005), the dimension of writing is not part of the construct measured in the Survey of Adult Skills. This is largely because of the difficulty of assessing writing in a reliable and valid way in an international comparative assessment. In addition, literacy is conceived as a skill that involves constructing meaning, and evaluating and using texts to achieve a range of possible goals in a variety of contexts. In other words, in the Survey of Adult Skills, literacy extends well beyond the skills of decoding or comprehending texts to using them appropriately in context.

Content

The corpus of texts to which adults are required to respond are classified along two principle axes: medium and format. Medium refers to the nature of the support in which a text is instantiated or displayed. Format refers to the organisational and structural features of texts, whether digital or print-based.

In terms of medium, texts are classified as either digital or print-based. Digital texts are texts that are stored as digital information (a series of 1s and 0s) and accessed in the form of screen-based displays on devices such as computers and smart phones. Print-based texts are texts printed on paper or other material supports; these include newspapers, books, pamphlets and road signs. Digital texts have a range of features, in addition to being displayed on screens, that distinguishes them from print-based texts. These include hypertext links to other documents, specific navigation features (e.g. scroll bars, use of menus) and interactivity. The Survey of Adult Skills is the first international assessment of adult skills to incorporate the reading of digital texts as part of the construct of (reading) literacy.3

In terms of format, texts are categorised in the following way:

• Continuous texts, which are made up of sentences organised in paragraphs that incorporate a range of rhetorical stances, such as description, narration, instruction and argumentation.

• Non-continuous texts, which are organised in a matrix format or around graphic features. Several different organising structures are identified, including simple and complex lists, graphic documents (e.g. graphs, diagrams), locative documents (e.g. maps) and entry documents (e.g. forms).

• Mixed texts, which involve combinations of continuous and non-continuous elements (e.g. a newspaper article or a webpage that includes text and graphics).

• Multiple texts, which consist of juxtaposing or linking independently generated elements, such as an e-mail that contains a record of the separate messages that constitute an exchange over a period of time, or a blog post that contains an initial text and a string of related texts consisting of comments in response to the initial text and comments in response to other comments.

Cognitive strategies

Readers generally use three broad cognitive strategies when responding to written texts:

• access and identify

• integrate and interpret

• evaluate and reflect.

Accessing and identifying involves locating information in a text. At one extreme, this can be a relatively simple operation when the information sought is clearly identified. At the other, it can be a complicated operation requiring inferential reasoning and an understanding of rhetorical strategies.

Integrating and interpreting involves understanding the relationships between different parts of a text to construct meaning and draw inferences from the text as a whole.

Evaluating and reflecting requires the reader to relate the information in the text to other information, knowledge and experiences, for example, to assess the relevance or credibility of a text.

Contexts

Adults read materials in a variety of contexts that affect the types of texts they encounter, the nature of the content, the motivation to read, and the manner in which texts are interpreted. The texts selected for the literacy assessment are related to four broad contexts:

• work-related

• personal

• society and community

• education and training.

Texts related to work and occupation include materials that discuss job search, wages, salaries and other benefits, and the experience of work.

Materials in the area of personal include texts concerning the home and family (e.g. interpersonal relationships, personal finances, housing and insurance); health and safety (e.g. drugs and alcohol, disease prevention and treatment, safety and accident prevention, first aid, emergencies, and lifestyle); consumer economics (banking, savings, advertising, prices); and leisure and recreation (travel, recreational activities).

Texts related to society and community includes materials that deal with public services, government, community groups and activities, and current events. Materials related to education and training cover text which refer to learning opportunities for adults or others.

Distribution of test items by task characteristics

Tables 1.2, 1.3 and 1.4 below show the distribution of the literacy assessment items in the Survey of Adult Skills by task characteristics. The final selection of items was determined taking into account the following factors: the performance of items in the field test, the need to cover the main dimensions of literacy as defined by the assessment frameworks, the need to include sufficient items that had been used in previous surveys to ensure comparability of the results, and the constraints imposed by the assessment design.4

copy the linklink copied!

Literacy sample items

Two examples of the literacy items used in the Survey of Adult Skills (PIAAC) are presented below. Both use print-based stimuli. The sample problem-solving items presented further below give an idea of the type of “digital” stimulus material used.

The items are presented in the form delivered by the computer-based version of the assessment. To answer the questions, respondents highlighted words and phrases or clicked on the appropriate location on the screen using a mouse.

Sample item 1: Preschool rules

“Preschool Rules” represents an easy item and focuses on the following aspects of the literacy construct:

copy the linklink copied!

Medium	Print
Context	Personal
Cognitive strategy	Access and identify

copy the linklink copied!

Sample items 2 and 3: Physical exercise equipment

In many cases, several questions are associated with the same stimulus material. In the case of the stimulus relating to physical exercise equipment, there are two associated questions or test items.

The first item represents a relatively easy item and focuses on the following aspects of the literacy construct:

copy the linklink copied!

Medium	Print
Context	Personal
Cognitive strategy	Access and identify

Respondents answer the question by clicking on the cell in the chart that contains information about exercise equipment. Each of the cells and all of the images can be highlighted by clicking on them and multiple cells can be selected.

copy the linklink copied!

The second item represents a relatively easy item and focuses on the following aspects of the literacy construct:

copy the linklink copied!

Medium	Print
Context	Personal
Cognitive strategy	Access and identify

copy the linklink copied!

Reading components

To provide more detailed information about adults with poor literacy skills, the survey’s literacy assessment is complemented by a test of “reading components” skills. Reading components are the basic set of decoding skills that are essential for extracting meaning from written texts: knowledge of vocabulary (word recognition), the ability to process meaning at the level of the sentence, and fluency in reading passages of text. Skilled readers are able to undertake these types of operations automatically. To assess this skill, the time taken by respondents to complete the tasks was recorded.

copy the linklink copied!

Examples of reading component items

Print vocabulary

Items testing print vocabulary consist of a picture of an object and four printed words, one of which refers to the pictured object. Respondents are asked to circle the word that matches the picture.

copy the linklink copied!

copy the linklink copied!

ear

egg

lip

jar

Sentence processing

The sentence-processing items require the respondent to assess whether a sentence makes sense in terms of the properties of the real world or the internal logic of the sentence. The respondent reads the sentence and circles YES if the sentence makes sense, or NO if the sentence does not make sense.

copy the linklink copied!

Three girls ate the song.	YES	NO
The man drove the green car.	YES	NO
The lightest balloon floated in the bright sky.	YES	NO
A comfortable pillow is soft and rocky.	YES	NO
A person who is twenty years old is older than a person who is thirty years old.	YES	NO

Passage comprehension

In items assessing passage comprehension, respondents are asked to read a passage in which they are required, at certain points, to select the word from the two alternatives provided that makes the most sense.

To the editor: Yesterday, it was announced that the cost of riding the bus will increase. The price will go up by twenty percent starting next wife / month. As someone who rides the bus every day, I am upset by this foot / increase. I understand that the cost of gasoline / student has risen. I also understand that riders have to pay a fair price / snake for bus service. I am willing to pay a little more because I rely on the bus to get to object / work. But an increase / uncle of twenty percent is too much.

This increase is especially difficult to accept when you see the city’s plans to build a new sports stadium. The government will spend millions on this project even though we already have a science / stadium. If we delay the stadium, some of that money can be used to offset the increase in bus fares / views. Then, in a few years, we can decide if we really do need a new sports cloth / arena. Please let the city council know you care about this issue by attending the next public meeting / frames.

Definition

The Survey of Adult Skills defines numeracy as “the ability to access, use, interpret and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life” (OECD, 2012). Numeracy is further defined in terms of the concept of “numerate behaviour” that involves managing a situation or solving a problem in a real context by responding to mathematical information and content represented in various ways.

It is recognised that literacy skills such as reading and writing enable numerate behaviour, and that when mathematical representations involve text, performance on numeracy tasks is, in part, dependent on the ability to read and understand text. However, numeracy in the Survey of Adult Skills involves more than applying arithmetical skills to information embedded in text. In particular, numeracy relates to a wide range of skills and knowledge, not just arithmetic knowledge and computation, a range of responses that may involve more than numbers, and responses to a range of representations, not just numbers in texts.

Content

The survey covers four areas of mathematical content, information and ideas:

• quantity and number

• dimension and shape

• pattern, relationships and change

• data and chance.

Quantity encompasses attributes such as the number of features or items, prices, size (e.g. length, area and volume), temperature, humidity, atmospheric pressure, populations and growth rates, revenues and profit, etc. Number is fundamental to quantification. Numbers (whether whole numbers or fractions, decimals or percentages) serve as counters or estimators, indicate parts or comparisons. Positive and negative numbers can also serve as directional indicators. In calculations, operations (i.e. the four main operations of +, –, x, ÷ and others, such as squaring) are performed on quantities and numbers.

Dimension covers the description of “things” in space, such as projections, lengths, perimeters, areas, planes, surfaces, location, etc. Shape involves a category describing real images and entities that can be visualised in two or three dimensions (e.g. houses and buildings, designs in art and craft, safety signs, packaging, snowflakes, knots, crystals, shadows and plants).

Pattern covers regularities encountered in the world, such as those in musical forms, nature, traffic, etc. Relationships and change relate to the mathematics of how things in the world are associated with one another or develop over time.

Data and chance encompass two separate but related topics. Data covers the “big ideas” related to variability, sampling, error, prediction and statistical topics, such as data collection, data displays and graphs. Chance covers the “big ideas” related to probability and relevant statistical methods.

Representations of mathematical information

In the Survey of Adult Skills, mathematical information may be represented in the form of:

• objects and pictures

• numbers and symbols

• visual displays; texts

• technology-based displays.

Objects (physical entities) can be counted and measured. Pictures (e.g. photographs, paintings, videos) also represent mathematical information such as number, size, volume or location. Numbers and symbols include numerals, letters, and operation or relationship signs and formulae. Visual displays cover graphic presentations of mathematical information, such as diagrams or charts, graphs and tables (used to display aggregate statistical or quantitative information through objects, counting data, etc.) or maps (e.g. of a city or a project plan). Two different kinds of text may be encountered in numeracy tasks. The first involves representing mathematical information in textual form, i.e. as words or phrases that carry mathematical meaning. The second involves expressing mathematical information in mathematical notations or symbols (e.g. numbers, plus or minus signs, symbols for units of measure, etc.) that are surrounded by text that provides additional information and context.

Cognitive strategies

Four processes define the dimension of cognitive strategies:

• identify, locate, or access

• act upon or use

• interpret, evaluate/analyse

• communicate.

In virtually all situations, people have to identify, locate or access some mathematical information relevant to their purpose or goal. In isolation, this response type often requires a low level of mathematical understanding or the application of simple arithmetic skills. However, this response type is usually combined with the other types of responses listed below.

Acting upon or using involves the use of known mathematical procedures and rules, such as counting and making calculations. It may also call for ordering or sorting, estimating or using various measuring devices, or for using (or developing) a formula that serves as a model of a situation or a process.

Interpretation involves evaluating the meaning and implications of mathematical or statistical information (e.g. a graph showing variation in an exchange rate) and developing an opinion about the information. Evaluation/analysis is in part an extension of interpretation. It involves analysing a problem, evaluating the quality of the solution against some criteria or contextual demands and, if necessary, reviewing the interpretation, analysis and evaluation stages.

While defined as a cognitive process forming part of this dimension of the numeracy framework, the ability to communicate numerical and mathematical content is not assessed in the Survey of Adult Skills.

Contexts

The items selected for the numeracy assessment are related to four contexts:

• work-related

• personal

• society and community

• education and training.

Representative tasks related to work situations include: completing purchase orders; totalling receipts; calculating change; managing schedules, budgets and project resources; using spreadsheets; organising and packing goods of different shapes; completing and interpreting control charts; making and recording measurements; reading blueprints; tracking expenditures; predicting costs; and applying formulas.

Representative tasks related to the context of personal life include: handling money and budgets; shopping and managing personal time; planning travel; playing games of chance; understanding sports scoring and statistics; reading maps; and using measurements in home situations, such as cooking, doing home repairs or pursuing hobbies.

Adults need to have an awareness of what is occurring in the society, the economy and the environment (e.g. trends in crime, health, wages, pollution), and may have to take part in social events or community action. This requires a capacity to read and interpret quantitative information presented in the media, including statistical messages and graphs. Adults also have to manage a variety of situations, such as raising funds for a football club or interpreting the results of a study on a medical condition.

Competency in numeracy may enable a person to participate in education and training, whether for academic purposes or as part of vocational training. In either case, it is important to know some of the more formal aspects of mathematics that involve symbols, rules and formulae and to understand some of the conventions used to apply mathematical rules and principles.

Distribution of test items by task characteristics

Tables 1.5, 1.6 and 1.7 below show the distribution of the numeracy assessment items included in the Survey of Adult Skills by task characteristics. As in the case of literacy, the final selection of items reflected the performance of items in the field test, the need to cover the main dimensions of literacy as defined by the assessment frameworks, the need to include sufficient items that had been used in previous surveys to ensure comparability of the results, and the constraints imposed by the assessment design.

copy the linklink copied!

Numeracy sample items

Sample item 1: Births in the United States

The items are presented in the form delivered by the computer-based version of the assessment. To answer the questions, respondents clicked in the appropriate box or typed figures in the spaces provided.

This item (of medium difficulty) focuses on the following aspects of the numeracy construct:

copy the linklink copied!

Content	Data and chance
Process	Interpret, evaluate
Context	Society and community

Respondents were asked to respond by clicking on one or more of the time periods provided in the left pane on the screen.

copy the linklink copied!

Sample item 2: Thermometer

This item (of low difficulty) focuses on the following aspects of the numeracy construct:

copy the linklink copied!

Content	Dimension and shape
Process	Act upon, use (measure)
Context	Personal or work-related

Respondents were asked to type in a numerical response based on the graphic provided.

copy the linklink copied!

Sample item 3: Wind power stations

This sample item (of medium difficulty) focuses on the following aspects of the numeracy construct:

copy the linklink copied!

Content	Quantity and number
Process	Act upon, use (compute)
Context	Society and community

copy the linklink copied!

Definition

In the Survey of Adult Skills, problem solving in technology-rich environments is defined as “using digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks”. The first cycle of the Survey of Adult Skills focuses on “the abilities to solve problems for personal, work and civic purposes by setting up appropriate goals and plans, and accessing and making use of information through computers and computer networks” (OECD, 2012).

The problem solving in technology-rich environments domain covers the specific types of problems people deal with when using ICT. These problems share the following characteristics:

• The problem is primarily a consequence of the availability of new technologies.

• The solution to the problem requires the use of computer-based artefacts (applications, representational formats, computational procedures).

• The problems are related to technology-rich environments themselves (e.g. how to operate a computer, how to fix a settings problem, how to use an Internet browser).

Problem solving in technology-rich environments is a domain of competency that represents the intersection of what are sometimes described as “computer literacy” skills (i.e. the capacity to use ICT tools and applications) and the cognitive skills required to solve problems. Some basic knowledge regarding the use of ICT input devices (e.g. use of a keyboard and mouse and screen displays), file management tools, applications (word processing, e-mail), and graphic interfaces is essential for performing assessment tasks. However, the objective is not to test the use of ICT tools and applications in isolation, but rather to assess the capacity of adults to use these tools to access, process, evaluate and analyse information effectively.

Content

The content of the assessment encompasses two areas: technology and tasks.

Technology refers to the devices, applications and functionalities through which problem solving is conducted. It encompasses digital devices such as computers, mobile phones, GPS devices, software applications and the commands, functions and representations of information on which these applications depend. In the first cycle of the survey, only laptop computers with a limited number of simulated software applications – including e-mail, word processing, spreadsheets and websites – were used. For operational reasons, sound, animations and videos were not used.

Tasks are the circumstances that trigger a person’s awareness and understanding of the problem and determine the actions needed to be taken in order to solve the problem. Ordinarily, a wide range of conditions can initiate problem solving. Tasks are defined in terms of intrinsic complexity and the explicitness of the problem statement. The intrinsic complexity of a problem is determined by:

• the minimum number of steps required to solve the problem

• the number of options or alternatives at various stages in the solution path

• the diversity of operators required to be used, and the complexity of computation/transformation

• the likelihood of impasses or unexpected outcomes

• the number of requirements that have to be satisfied to arrive at a solution

• the amount of transformation required to communicate a solution.

The explicitness of the problem statement relates to the extent to which the problem is ill-defined (the task is implicit and its components are largely unspecified) or well-defined (the task is explicit and its components are described in detail).

Cognitive strategies

The process aspect of the assessment relates to the mental structures and processes involved when a person solves a problem. These include setting goals and monitoring progress; planning; locating, selecting and evaluating information; and organising and transforming information.

Setting goals and monitoring progress involves identifying objectives in the context of the constraints (explicit and implicit) of a situation; establishing and applying criteria for respecting constraints and arriving at a solution; monitoring progress; and detecting and interpreting unexpected events, impasses and breakdowns as one proceeds along the path to a solution.

Planning and self-organisation covers the processes of setting up adequate plans, procedures and strategies (operators) and selecting appropriate devices, tools or categories of information.

Acquiring and evaluating information involves orienting and focusing attention; selecting information; assessing the reliability, relevance, adequacy and comprehensibility of information; and reasoning about sources and contents.

Using information involves organising information; integrating information drawn from different and possibly inconsistent texts and from different formats; making informed decisions; transforming information through rewriting, from text to table, from table to graph, etc.; and communicating with relevant parties.

Contexts

The contexts are those of personal life, work-related and society and community.

Distribution of test items by task characteristics

Tables from 1.8 to 1.13 show the distribution of the problem-solving assessment items included in the Survey of Adult Skills by task characteristics.

In total 16 items were administered in the assessment of problem solving in technology environments. Items consisted of scenarios that involved a number of sub-tasks such as searching through simulated websites for relevant information or transferring information from one application to another. The time taken to complete the problem-solving tasks was considerably longer than that in either literacy or numeracy.

copy the linklink copied!

Problem-solving sample item

An example of a problem-solving item is provided below. This item involves a scenario in which the respondent assumes the role of a job-seeker. Respondents access and evaluate information relating to job search in a simulated web environment. This environment includes tools and functionalities similar to those found in real-life applications. Users are able to:

• click on links on both the results page and associated web pages

• navigate, using the back and forward arrows or the Home icon

• bookmark web pages and view or change those bookmarks.

copy the linklink copied!

The first stimulus accessed by respondents is the results page of the search-engine application, which lists five employment agency websites. To complete the task successfully, respondents have to search through the pages of the listed websites to identify whether registration or the payment of a fee is required in order to gain further information about available jobs. Respondents can click on the links on the search page to be directed to the websites identified. For example, by clicking on the “Work Links” link, the respondent is directed to the home page of “Work Links”.

copy the linklink copied!

In order to discover whether access to the information on available jobs requires registration with the organisation or payment of a fee, the respondent must click the “Learn More” button which opens the following page. The respondent must then return to the search results page to continue evaluating the sites in terms of the specified criteria, using the back arrows without bookmarking the page (correct answer) or having bookmarked the page (incorrect answer).

copy the linklink copied!