-
Digital technologies are revolutionising many aspects of our everyday life, and skills assessments are no exception. The Survey of Adult Skills, a product of the Programme for the International Assessment of Adult Competencies (PIAAC) (hereafter referred to as “PIAAC”) was the first large-scale international assessment fully designed to be primarily delivered on a computer.
-
-
This report describes and analyses the recently released dataset of information extracted from the log files generated during the Survey of Adult Skills cognitive assessment. It explores the potential and shortcomings of these data, as well as pitfalls to avoid when working with them. This chapter explains the value and limitations of log files and discusses the information available from the Survey of Adult Skills log files, with particular focus on timing indicators.
-
This chapter explains what log files are and how they complement traditional proficiency scores. It describes the features of the log files generated in the Survey of Adult Skills cognitive assessment and explores the impact of assessment design on interpretation of information contained in the files.
-
This chapter provides an overview of the indicators extracted from log files, analysing differences across countries, cognitive domains and the socio-demographic characteristics of respondents. Older respondents and more educated adults tended to spend more time on the assessment, and time spent on the assessment was found to be positively correlated with performance. The data also show a positive relationship between timing indicators and literacy proficiency.
-
This chapter analyses disaggregated data at the respondent-item level to illustrate how respondents chose to allocate time to different items. Time spent on items was found to be strongly related to intrinsic characteristics of items, such as difficulty. Respondents devoted considerably less time to items administered in the second half of the assessment. This was accompanied by a decrease in performance (measured by the fraction of items answered correctly) and an increase in the proportion of missing answers. Respondents seem to allocate time to tasks rationally, spending less time on items that are both too difficult and too easy and more time on challenging items for which the probability of success is close to 50%. Spending more time on an item appears to increase the probability of giving a correct answer, although at declining rates.
-
This chapter uses timing indicators to estimate and analyse disengagement with the Survey of Adult Skills assessment. The analysis shows that the incidence of disengagement varies substantially across countries. Respondents with low levels of education and low familiarity with information and communications technology (ICT) are more likely to be disengaged, and respondents are more likely to be disengaged with items that appear in the second module of the assessment. Disengagement strongly reduces the probability of giving a correct answer, which results in disengaged individuals performing worse in the assessment. This relationship holds at both individual and country levels.
-
Log files for 18 countries that participated in the first round of the Survey of Adult Skills, a product of the Programme for the International Assessment of Adult Competencies (PIAAC) (hereafter referred to as “PIAAC”) in 2011/12 have recently been released and can be downloaded from the German Social Science Infrastructure Services (GESIS) Data Catalogue (OECD, 2017[1]). More information and documentation is available at www.oecd.org/skills/piaac/log-file/. The files have been fully anonymised to prevent identification of individual respondents. The records can nevertheless be matched with information already available in the PIAAC Public Use File, which contains the individual answers to the background questionnaire, as well as the performance of test-takers in the PIAAC assessment.