Beyond Proficiency
Using Log Files to Understand Respondent Behaviour in the Survey of Adult Skills
Computer-based administration of large-scale assessments makes it possible to collect a rich set of information on test takers, through analysis of the log files recording interactions between the computer interface and the server. This report examines timing and engagement indicators from the Survey of Adult Skills, a product of the Programme for the International Assessment of Adult Competencies (PIAAC), both of which indicate large differences across countries and socio-demographic groups, in the amount of time spent by respondents and their levels of disengagement, which reduce the probability of giving a correct answer and consequently reduces measured performance. Such insights can help policy makers, researchers and educators to better understand respondents’ cognitive strategies and the underlying causes of low and high performance. This, in turn, can help improve the design of assessments and lead to more effective training and learning programmes.
The PIAAC LogDataAnalyzer
Log files for 18 countries that participated in the first round of the Survey of Adult Skills, a product of the Programme for the International Assessment of Adult Competencies (PIAAC) (hereafter referred to as “PIAAC”) in 2011/12 have recently been released and can be downloaded from the German Social Science Infrastructure Services (GESIS) Data Catalogue (OECD, 2017[1]). More information and documentation is available at www.oecd.org/skills/piaac/log-file/. The files have been fully anonymised to prevent identification of individual respondents. The records can nevertheless be matched with information already available in the PIAAC Public Use File, which contains the individual answers to the background questionnaire, as well as the performance of test-takers in the PIAAC assessment.