copy the linklink copied! 3. The methodology of the Survey of Adult Skills (PIAAC) and the quality of data

copy the linklink copied! Assessment design

The Survey of Adult Skills involved the direct assessment of literacy, numeracy and problem solving in technology-rich environments. While conceived primarily as a computer-based assessment (CBA), the option of taking the literacy and numeracy components of the assessment in paper-based format (PBA) had to be provided for those adults who had insufficient experience with computers to take the assessment in CBA mode. This necessitated a relatively complex design, which is presented graphically in Figure 3.1.

In Jakarta (Indonesia), the assessment was delivered in paper-based format only due to the low rate of familiarity with computers among the target population. This was a version of the paper-based assessment used in other countries that included additional items.

Pathways through the cognitive assessments in the Survey of Adult Skills (PIAAC): Computer-based assessment

As can be seen, there are several pathways through the computer-based assessment. Respondents with no experience in using computers, as indicated by their response to the relevant questions in the background questionnaire, were directed to the paper-based version of the assessment. Respondents with some experience of computer use were directed to the computer-based assessment where they took a short test of their ability to use the basic features of the test application (use of a mouse, typing, use of highlighting, and drag and drop functionality) – the CBA core Stage 1. Those who “failed” this component were directed to the paper pathway.

copy the linklink copied!

Figure 3.1. Pathways through the cognitive assessments in the Survey of Adult Skills (PIAAC): Computer-based assessment

Notes: The percentages presented in this diagram are based on the average of the OECD countries and economies participating in the Survey of adult Skills (PIAAC)

Respondents taking the computer path then took a short test (the CBA core Stage 2) composed of three literacy and three numeracy items of low difficulty to determine whether or not they should continue with the full assessment. Those who “failed” this module were directed to the reading components assessment. Respondents who passed this module continued on to take the full test and were randomly assigned to a first module of literacy, numeracy or problem-solving items. Following completion of the first module, respondents who had completed a literacy module were randomly assigned to a numeracy or problem-solving module, respondents who had completed a numeracy module were randomly assigned to a literacy or problem-solving module, and respondents who had completed a problem-solving module were randomly assigned to a literacy, a numeracy or a second problem-solving module.

The assessment design assumed that the respondents taking the PBA path would be either those who had no prior experience with computers (as assessed on the basis of responses to the relevant questions in the background questionnaire) or those who failed the ICT core. It was, however, possible for respondents with some computer experience to take the PBA pathway if they insisted. Respondents with some computer experience who opted to take the paper-based pathway without attempting the CBA core represented 9.5% of all respondents in Rounds 1 and 2.

Respondents taking the paper path first took a “core” test of four simple literacy and four simple numeracy items. Those who passed this test were randomly assigned to a module of either 20 literacy tasks or 20 numeracy tasks. Once the module was completed, respondents were given the reading-components test. Respondents who failed the initial “core” test proceeded directly to the reading-components test.

copy the linklink copied!

Table 3.3. Participation in the cognitive-assessment modules
OECD countries and economies	Literacy and numeracy	Problem solving in technology-rich environments	Reading components
Australia	Yes	Yes	Yes
Austria	Yes	Yes	Yes
Canada	Yes	Yes	Yes
Chile	Yes	Yes	Yes
Czech Republic	Yes	Yes	Yes
Denmark	Yes	Yes	Yes
England (UK)	Yes	Yes	Yes
Estonia	Yes	Yes	Yes
Finland	Yes	Yes	No
Flanders (Belgium)	Yes	Yes	Yes
France	Yes	No	No
Germany	Yes	Yes	Yes
Greece	Yes	Yes	Yes
Ireland	Yes	Yes	Yes
Israel	Yes	Yes	Yes
Italy	Yes	No	Yes
Hungary	Yes	Yes	Yes
Japan	Yes	Yes	No
Korea	Yes	Yes	Yes
Mexico	Yes	Yes	Yes
Lithuania	Yes	Yes	Yes
Netherlands	Yes	Yes	Yes
New Zealand	Yes	Yes	Yes
Northern Ireland (UK)	Yes	Yes	Yes
Norway	Yes	Yes	Yes
Poland	Yes	Yes	Yes
Slovak Republic	Yes	Yes	Yes
Slovenia	Yes	Yes	Yes
Spain	Yes	No	Yes
Sweden	Yes	Yes	Yes
Turkey	Yes	Yes	Yes
United States 2012/2014	Yes	Yes	Yes
United States 2017	Yes	Yes	Yes
Partners
Cyprus1	Yes	No	Yes
Ecuador	Yes	Yes	Yes
Jakarta (Indonesia)	Yes	No	Yes
Kazakshtan	Yes	Yes	Yes
Peru	Yes	Yes	Yes
Russian Federation2	Yes	Yes	No
Singapore	Yes	Yes	Yes
← 1. See note 1 under Table 3.1.
← 2. See note at the end of this chapter.Countries and economies are ranked in alphabetical order.

Some 76.6% of respondents attempted the CBA core Stage 1. In total, 72.4% of respondents took the CBA core Stage 2 and 71.6% of the sample went on to the CBA literacy, numeracy or problem solving assessment with 0.8% being directed to the reading components assessment.

Some 23.8% of respondents took the PBA assessment core and 21.6% went on to complete the paper-based literacy and numeracy assessment modules and the reading components assessment. A total of 2.9% of respondents completed the reading components assessment only. There was a small proportion of respondents who did not complete the cognitive assessment.

The Survey of Adult Skills was designed to provide accurate estimates of proficiency in the three domains across the adult population and its major subgroups, rather than at the level of individuals. Each respondent was given a subset of the test items used in the assessment. No individual took modules from all the domains assessed. As can be seen from Figure 3.1, respondents following the CBA path took two assessment modules in either one or two of the three assessment domains.1 Respondents following the PBA path took either a literacy or a numeracy module.

In the CBA mode, the literacy and numeracy assessments had an adaptive design. Respondents were directed to different blocks of items on the basis of their estimated ability. Individuals who were estimated to have greater proficiency were more likely to be directed to groups of more difficult items than those who were estimated to be less proficient. Each of the literacy and numeracy modules was composed of two stages containing testlets (groups of items) of varying difficulty. Stage 1 contained three testlets and Stage 2, four. Respondents’ chances of being assigned to testlets of a certain difficulty depended on their level of educational attainment, whether their native language was the same as the test language, their score on the literacy/numeracy core and, if relevant, their score on a Stage 1 testlet.2

All participating countries/economies were required to administer the literacy and numeracy components of the assessments. Administration of the problem solving in technology-rich environments and the reading-components assessments was optional. All but four countries administered the problem-solving assessment, and all but three administered the reading components assessment. Table 3.3 provides details of participation in each of the cognitive assessments.

copy the linklink copied! Sampling

To maximise the comparability of results, countries/economies participating in the Survey of Adult Skills were expected to meet stringent standards relating to the target population, sample design, sample selection response rates, and non-response bias analysis.

copy the linklink copied! Translation and adaptation of instruments

Participating countries/economies were responsible for translating the assessment instruments and the background questionnaire. Any national adaptations of either the instruments or the questionnaire was subject to strict guidelines, and to review and approval by the international consortium. The recommended translation procedure was for a double translation from the English source version by two independent translators, followed by reconciliation by a third translator.

All national versions of the instruments were subject to a full verification before the field test, which involved:

• A sentence-by-sentence check of linguistic correctness, equivalence to the source version, and appropriateness of national adaptations.

• A final optical check to verify the final layout of the instruments, the equivalence of computer and paper forms, and the correct implementation of changes recommended by the verifiers.

All national version materials revised following the field test were subject to partial verification before the main study. Edits made between the field test and the main study were checked for their compliance with the PIAAC translation and adaptation guidelines and for correct implementation.

copy the linklink copied! Survey administration

The Survey of Adult Skills was administered under the supervision of trained interviewers either in the respondent’s home or in a location agreed between the respondent and the interviewer. After the sampled person was identified, the survey was administered in two stages: completion of the background questionnaire and completion of the cognitive assessment.

The background questionnaire, which was the first part of the assessment, was administered in Computer-Aided Personal Interview format by the interviewer. Respondents were able to seek assistance from others in the household in completing the questionnaire, for example, in translating questions and answers. Proxy respondents were not permitted.

Following completion of the background questionnaire, the respondent undertook the cognitive assessment either using the computer provided by the interviewer or, by completing printed test booklets in the event that the respondent had limited computer skills, was estimated to have very low proficiency in literacy and numeracy, or opted not to take the test on the computer. Respondents were permitted to use technical aids such as an electronic calculator, a ruler (which were provided by interviewers) and to take notes or undertake calculations using a pen and pad during the assessment.

Respondents were not allowed to seek assistance from others in completing the cognitive assessment. However, the interviewer could intervene if the respondent had problems with the computer application or had questions on how to proceed with the assessment.

The direct-assessment component of the survey was not designed as a timed test; respondents could take as much or as little time as needed to complete it. However, interviewers were trained to encourage respondents to move to another section of the assessment if they were having difficulties. Respondents who started the cognitive assessment tended to finish it. The time taken to complete the cognitive assessment varied between 41 and 50 minutes on average depending on the country/language version.

The survey (background questionnaire plus cognitive assessment) was normally undertaken in one session. However, in exceptional circumstances, a respondent could take the questionnaire in one session and the cognitive assessment in another. The cognitive assessment was required to be completed in one session. Respondents who did not complete the assessment within a single session for whatever reason were not permitted to finish it at a later time.

Data collection in Round 1 of the Survey of Adult Skills took place from 1 August 2011 to 31 March 2012 in most participating countries/economies. In Canada, data collection took place from November 2011 to June 2012 and France collected data from September to November 2012. Data collection for Round 2 of the Survey of Adult Skills took place from 1 April 2014 to 31 March 2015. Data collection for Round 3 of the Survey of Adult Skills took place from 1 August 2017 to 30 April 2018 with the exception of the United States where data collection took place between March 2017 and September 2017.

Interviewers administering the survey were required to be trained according to common standards. These covered the timing and duration of training, its format and its content. A full set of training materials was provided to countries. The persons responsible for organising training nationally attended training sessions organised by the international consortium.

copy the linklink copied! Response rates and non-response bias analysis

A major threat to the quality of the data produced by the Survey of Adult Skills was low response rates. The PIAAC Technical Standards and Guidelines (PIAAC, 2014) required that countries/economies put in place a range of strategies to reduce the incidence and effects of non-response, to adjust for it when it occurred, and to evaluate the effectiveness of any weighting adjustments implemented to reduce non-response bias. In particular, countries/economies were expected to establish procedures during data collection to minimise non-response. These included pre-collection publicity, selecting high-quality interviewers, delivering training on methods to reduce and convert refusals, and monitoring data collection closely to identify problem areas or groups and directing resources to these particular groups. At least seven attempts were to be made to contact a selected individual or household before it could be classed as a non-contact. The overall rate of non-contact was to be kept below 3%.

Response rates were calculated for each stage of the assessment: screener (only for countries/economies that need to sample households before selecting respondents); background questionnaire and Job Requirement Approach module; assessment (without reading components); and reading components.

The overall response rate was calculated as the product of the response rates (complete cases/eligible cases) for the relevant stages of the assessment. For countries/economies with a screener questionnaire, the overall response rate was the product of the response rates for the screener, background questionnaire/Job Requirement Approach module and assessment; for countries/economies without a screener, it was the product of the response rates for the questionnaire/module and the assessment.

The computations at each stage are hierarchical in that they depend on the response status from the previous data collection stage. A completed case thus involved completing the screener (if applicable), the background questionnaire, and the cognitive assessment. In the case of the questionnaire, a completed case was defined as having provided responses to key background questions, including age, gender, highest level of schooling and employment status or responses to age and gender for literacy-related non-respondents. For the cognitive assessment, a completed case was defined as having completed the “core” module, and a literacy/numeracy core module, or a case in which the core module was not completed for a literacy-related reason, for example, because of a language difficulty or because the respondent was unable to read or write in any of a country’s test languages or because of learning or mental disability.

As noted above, countries/economies using population register-based sampling frames were able to treat some or all of the individuals in their samples who were untraceable as exclusions (i.e. as outside the target population) and exclude them from the numerator and denominator of the response-rate calculation (provided that the 5% threshold for exclusions was not exceeded).

The survey’s Technical Standards and Guidelines set a goal of a 70% unit response rate. Ten countries achieved this goal, five in Round 1, two in Round 2, and 3 in Round 3. For the most part, response rates were in the range of 50%-60%. Response rates by country/economy are presented in Table 3.9.

Countries/economies worked to reduce non-response bias to the greatest extent possible before, during, and after data collection. Before data collection, countries implemented field procedures with the goal of obtaining a high level of co-operation. Most countries followed the PIAAC required sample monitoring activities to reduce bias to the lowest level possible during data collection. Finally, countries gathered and used auxiliary data to reduce bias in the outcome statistics through non-response adjustment weighting.

All countries/economies were required to conduct a basic non-response bias analysis (NRBA) and report the results. The basic analysis was used to evaluate the potential for bias and to select variables for non-response adjustment weighting. In addition, countries were required to conduct and report the results of a more extensive NRBA if the overall response rate was below 70%, or if any stage of data collection (screener, background questionnaire, or the assessment) response rate was below 80%. A NRBA was required for any BQ item with response rate below 85%.

Australia, Indonesia (Jakarta), Kazakhstan, Korea, Mexico, Peru, Turkey and the United States achieved an overall response rate of 70% or greater. As their response rates for each stage were greater than 80%, they did not require the extended NRBA. Cyprus4 and Ireland also achieved overall response rates of 70% or greater, but they achieved a lower than 80% response rate for one stage of their sample. The remaining countries achieved response rates lower than 70%.

The main purpose of the extended analysis was to assess the potential for remaining bias in the final weighted proficiency estimates after adjusting for non-response. As the proficiency levels of non-respondents are unknown, the NRBA is carried out by making assumptions about non-respondents. Multiple analyses were, therefore, undertaken to assess the potential for bias as each individual analysis has limitations due to the particular assumptions made about non-respondents. The extended NRBA included seven analyses (as listed below). Together, they were used to assess the patterns and potential for bias in each country data.

1. 1 Comparison of estimates before and after weighting adjustments

2. 2 Comparison of weighted estimates to external totals

3. 3 Correlations of auxiliary variables and proficiency estimates

4. 4 Comparison of estimates from alternative weighting adjustments

5. 5 Analysis of variables collected during data collection

6. 6 Level-of-effort analysis

7. 7 Calculation of the range of potential bias

Cyprus¹ and Ireland were required to do only a subset of the analyses since their overall response rate was higher than 70%.

Table 3.10 summarises the results of the NRBA for countries/economies with response rates lower than 70%. The overall conclusion was that, on the balance of evidence, the level of non-response bias was in the range of minimal to low in countries required to undertake the extended analysis available. The results for England/Northern Ireland (United Kingdom) were, however, inconclusive because many of the analyses were either incomplete or not conducted. Data users should be aware that the analyses are all based on various assumptions about non-respondents. Multiple analyses, with different assumptions, were included in the NRBA to protect against misleading results. However, the lower the response rate, the higher is the risk of hidden biases that are undetectable through non-response bias analysis even when multiple analyses are involved.

copy the linklink copied! Literacy-related non-response

In most participating countries/economies a proportion of respondents were unable to undertake the assessment for literacy-related reasons, such as being unable to speak or read the test language(s), having difficulty reading or writing, or having a learning or mental disability. Some of these respondents completed the background questionnaire, or key parts of it, presumably with the assistance of an interviewer who spoke the respondent’s language, a family member or another person.

The available background information regarding these respondents was used to impute proficiency scores in literacy and numeracy. Scores were not, however, imputed in problem solving in technology-rich environments domain, as these respondents did not undertake the ICT core assessment. Other respondents were able to provide only very limited background information as there was no one present (either the interviewer or another person) to translate into the language of the respondent or answer on behalf of the respondent. For most of these respondents, the only information collected was their age, gender and, in some cases, highest educational attainment. As a result, proficiency scores were not estimated for these respondents in any domain; however, they have been included as part of the weighted population totals and are included in the charts and tables in OECD Skills Outlook 2013 (OECD, 2013) and Skills Matter: Further Results from the Survey of Adult Skills (OECD, 2016), and Skills Matter: Additional Results from the Survey of Adult Skills (2019b) under the category of literacy-related non-response (missing).The proportions of respondents who did not undertake the cognitive assessment and (a) received imputed scores and (b) did not receive imputed scores are presented in Table 3.11. Flanders (Belgium) and Cyprus¹ each stand out as having a high proportion of respondents who did not receive imputed scores due to having relatively high proportions of respondents for whom limited background information was available.

copy the linklink copied! Scoring

For the large majority of respondents who took the assessment in its CBA format, scoring was done automatically. Manual scoring was necessary in the case of respondents taking the PBA version. Participating countries/economies were required to undertake within-country reliability studies during both the field test and main survey to check the consistency of scoring. This required a second scorer to re-score a pre-defined number of cognitive paper-and-pencil assessments.5 The level of agreement between the two scorers was expected to be at least 95%.

In addition, a cross-country reliability study was conducted to identify the presence of systematic scoring bias across countries. At least two bilingual scorers (fluent in the national language and English) scored English-language international anchor booklets to ensure the equivalence of scoring across countries. These scores were compared and evaluated against the master scores for accuracy.

The levels of agreement achieved in the within-country and between-country studies of scoring reliability are presented in Table 3.12.

copy the linklink copied! Overall assessment of data quality

The data from participating countries/economies was subject to a process of “adjudication” to determine whether it was of sufficient quality to be reported and released to the public. The adjudication process used a broad definition of quality – that of “fitness for use”. While countries’ compliance with the requirements of the PIAAC Technical Standards and Guidelines (PIAAC, 2014) was an important component of the quality assessment, the goal was to go beyond compliance to assess whether the data produced were of sufficient quality in terms of their intended uses or applications. In assessing overall quality, the focus was on four key areas:

• sampling

• coverage and non-response bias

• data collection, and

• instrumentation.

In each of the domains identified above, countries/economies were assessed against a set of quality indicators. These indicators reflected the major requirements of the survey’s Technical Standards and Guidelines (PIAAC, 2014) in the domains concerned. All countries/economies either fully met the required quality standards or, if they did not fully meet them, they met them to a degree that was believed not to compromise the overall quality of the data. The data from all participating countries/economies were determined to have met the quality standards required for reporting and public release. The assessments of the quality of participating countries’ data were reviewed by the project’s Technical Advisory Group before being submitted to the Board of Participating Countries.

Over the three data collection rounds of the first cycle of PIAAC, there were specific concerns about some aspects of the quality of data in three countries: the Russian Federation, Greece and Kazakhstan. These concerns and the action taken to rectify them are described below. In the Russian Federation (Round 1), concerns regarding the process of data collection in the Moscow municipal area led to data from this area to be removed from the Russian data file. Thus, the sample for the Russian Federation covers the population of the Russian Federation with the exception of the population of the Moscow municipal area. In Greece (Round 2), a large number of cases (1 032 in total) were collected without complete cognitive data. Proficiency scores in literacy and numeracy have been imputed for these cases. In Kazakhstan, as a result of quality control checks, 70 interviewers were identified who had falsified interviews. A total of 239 falsified cases were identified out of a randomly sampled 1 961 cases for verification among the entire sample of 6 343. The PIAAC Consortium ran quality control checks on all the data and further identified a further 19 interviewers who had duplicated 55 cases. The Technical Advisory Group recommended that both the 239 falsified cases and the 55 duplicated cases be deleted from the final dataset (one case was both falsified and duplicated). Accordingly, a total of 293 unique cases were excluded and Kazakhstan’s remaining data of 6 050 cases received their final weights. Further information can be found in the Technical Report of the Survey of Adult Skills, Third Edition (OECD, 2019a).