1996-3777 (online)
1990-8539 (print)
Hide / Show Abstract

A series of reports on the OECD’s Programme for International Student Assessment’s (PISA) periodic testing program on student performance. The reports generally compare student (15 year olds) academic performance across countries, or discuss the methodology used to gather the data.

Also available in French, German
PISA Computer-Based Assessment of Student Skills in Science

PISA Computer-Based Assessment of Student Skills in Science You or your institution have access to this content

Click to Access: 
  • PDF
  • http://www.keepeek.com/Digital-Asset-Management/oecd/education/pisa-computer-based-assessment-of-student-skills-in-science_9789264082038-en
  • READ
25 Aug 2010
9789264082038 (PDF) ;9789264082021(print)

Hide / Show Abstract

This report documents the initial step towards an electronically-delivered Programme for International Student Assessment (PISA) test pioneered by Denmark, Iceland and Korea. In 2006, the PISA assessment of science included for the first time a computer-based test. The results discussed in this report highlight numerous challenges and encourage countries to take the work further.

PISA Computer-Based Assessment of Student Skills in Science describes how the 2006 survey was administered, presents 15-year-olds’ achievement scores in science and explains the impact of information communication technologies on both males’ and females’ science skills. While males outperformed females on the computer-based test in all three countries, females in Iceland and males in Denmark performed better than their counterparts on the paper-and-pencil test. The evidence shows that, overall, males are more confident and use computers more frequently. While females tend to use the Internet more for social networking activities, males tend to browse the Internet, play games and download software.

Readers will also learn how students reacted to the electronic questionnaire and how it compared with pencil-and-paper tests. In general, there were no group differences across test methods buts students enjoyed the computer-based test more than the paper-and-pencil test.

loader image

Expand / Collapse Hide / Show all Abstracts Table of Contents

  • Mark Click to Access
  • Foreword
    The capacity of countries – both the world’s most advanced economies as well those experiencing rapid development – to compete in the global knowledge economy increasingly depends on whether they can meet a fast-growing demand for high-level skills. This hinges on significant improvements in the quality of schooling outcomes and a more equitable distribution in learning opportunities.
  • Introduction
    Computer-based assessments are becoming more and more prevalent in the educational testing domain and the computer-based assessment of science in PISA was envisioned to test a number of important aspects of this methodology in comparison to paper-and-pencil methods. Only three of the PISA countries, Denmark, Iceland and Korea, finished this project and those results are reported here.
  • Administration of the CBAS test
    The computer-based assessment of science was field tested in thirteen PISA countries in 2005 and the main study was carried out in three of them in 2006, Denmark, Iceland and Korea. Rigorous sampling was used in all countries and all students also took the regular PISA test. The computer-based test had appropriate test characteristics and sampled a number of different aspects of science knowledge and skills.
  • Students' achievement in science
    Overall achievement within countries did not change from one test modality to the next. Yet, there was a tendency for Denmark’s performance to decrease on the computer-based test. Korean students outperformed Danish and Icelandic students in the computer-based test just as they did in the paper-and-pencil test. In the computer based test, male performance increased in Iceland and Korea while female performance decreased. Males outperformed females on the computer-based test in all three countries. Females outperformed males on the paper-and-pencil test of science in Iceland whereas there was a gender difference in favour of males in the paper-and-pencil results for Denmark. The association between reading literacy and achievement on the science test was weaker for the computer-based items than for the paper-and-pencil items.
  • Use of Information Communication Technologies (ICT) across genders and impact on achievement

    The vast majority of students have computers and Internet connections at home. Not having a computer at home is associated with poorer performance on the CBAS test. Overall, males score higher than females on the frequency of use of scales. Perhaps as a result of this, they also score higher on the confidence in using ICT scales, particularly in Iceland.

    Females tend to use the Internet more for social networking activities such as chatting and email and their confidence is consequently higher for these activities. Males tend to browse the Internet, play games and download software a lot more than females and they perform advanced computer activities more frequently. Overall, males have much more confidence than females in most ICT activities, which is also found in ICT PISA 2003 results.

  • CBAS questionnaire results
    Male responses to questionnaire items were more polarised than female replies. They usually strongly disagreed or strongly agreed with statements more than females, who tended towards more neutral categories. Motivation for the computer-based test was higher than for the paper-and-pencil test across all countries. Students enjoyed the computer-based test more than the paper-andpencil test. Most students prefered to do a computer-based test than a paperand- pencil test. Most students reported that they put the same amount of effort into both tests. Test enjoyment and motivation seemed to have little to do with achievement. Test preference and relative effort reports showed no association with test performance.
  • Features of the computer-based items and performance

    The high reading load on paper-and-pencil science items appeared as a disadvantage for males. However, males did better than females on more difficult paper-and-pencil items, as well as computer-based items.

    While the domain coverage across test modalities was similar, males outperformed females in all domains on the computer-based items. There were more computer-based items that were easier for males than females.

  • Summary
    Paper-and-pencil science vs. CBAS difference: In all participating countries, there was no evidence to suggest that overall group performance was affected by the method of test presentation (computer-based or paper-and-pencil). However, there was a slight trend in Denmark for the scores on the computerbased test to drop.
  • References
  • Annex A. Sampling information
    The decision to impute plausible values on the CBAS scale for students in CBAS participating schools that participated only in PISA paper-and-pencil test was made by data analysts of the Consortium (Westat and ACER). The quality of imputing missing values was evaluated and the ISL CBAS situation approximated as closely as possible: effective sample size = 400, correlation between SCIE and CBAS r=0.9, correlation between probability to respond and CBAS performance p = 0.1 (it is low, because for most students the non-response is random, so for that group p = 0) and non-response rate = 0.8. In the following simulation, only scores on one other dimension are used (therefore background information, reading and mathematics performance are not taken into account).
  • Annex B. Frequency of performing ICT activities by country and gender
  • Add to Marked List
Visit the OECD web site