The OECD’s Programme for International Student Assessment (PISA) surveys, which take place every three years, have been designed to collect information about 15-year-old students in participating countries. PISA examines how well students are prepared to meet the challenges of the future, rather than how well they master particular curricula. The data collected during each PISA cycle are an extremely valuable source of information for researchers, policy makers, educators, parents and students. It is now recognised that the future economic and social well-being of countries is closely linked to the knowledge and skills of their populations. The internationally comparable information provided by PISA allows countries to assess how well their 15-year-old students are prepared for life in a larger context and to compare their relative strengths and weaknesses.
Programme for International Student Assessment:
The OECD Programme for International Student Assessment (PISA) is a collaborative effort among OECD member countries to measure how well 15-year-old students approaching the end of compulsory schooling are prepared to meet the challenges of today’s knowledge societies. The assessment is forward-looking: rather than focusing on the extent to which these students have mastered a specific school curriculum, it looks at their ability to use their knowledge and skills to meet real-life challenges. This orientation reflects a change in curricular goals and objectives, which are increasingly concerned with what students can do with what they learn at school.
Test Design and Test Development
This chapter describes the test design for PISA 2009 and the processes by which the PISA Consortium, led by ACER, developed the PISA 2009 paper-and-pencil tests for reading, mathematics and science. It also describes the design and development of the computer-based assessment of reading, the digital reading assessment, an innovation in PISA 2009. In the following discussion, the term "reading" generally refers to the core, paper-based reading assessment. The computerbased assessment is referred to as the "digital reading assessment".
The Development of the PISA Context Questionnaires
In its Call for Tender for PISA 2009, the PISA Governing Board (PGB) established the main policy issues it sought to address in the fourth cycle of PISA. In particular, the PGB required PISA 2009 to collect a set of basic demographic data as a core component that replicated key questions from the previous cycles. In addition, PISA 2009 needed to address issues related to important aspects of the affective domain, information about students’ experience with reading in and out of school (e.g. experience of different approaches to the teaching of reading, preferred ways of learning), motivation, interest in reading and engagement in reading. At the school level, PISA 2009 needed to explore curriculum, teaching and learning in the area of reading, including aspects of the teachers’ careers and qualifications concerning the test language. Since the impact of out-of-school factors was considered of particular interest in a PISA survey where reading literacy was the major domain, the PGB recommended the inclusion of a parent questionnaire as an optional instrument.
The desired base PISA target population in each country consisted of 15-year-old students attending educational institutions in grades 7 and higher. This meant that countries were to include:
15-year-olds enrolled full-time in educational institutions;
15-year-olds enrolled in educational institutions who attended only on a part-time basis;
students in vocational training programmes, or any other related type of educational programmes; and
students attending foreign schools within the country (as well as students from other countries attending any of the programmes in the first three categories).
Translation and Verification of the Test and Survey Material
One of the important responsibilities of PISA is to ensure that the instruments used in all participating countries to assess students’ performance provide reliable and fully comparable information. In order to achieve this, PISA implemented strict verification procedures for translation/adaptation and verification procedures.
PISA was co-ordinated in each country by a National Project Manager (NPM) who implemented the procedures prepared by the Consortium. Each NPM typically had several assistants, working from a base location that is referred to throughout this report as a national centre. For the school level operations the NPM coordinated activities with school level staff, referred to in PISA as School Co-ordinators (SCs). Trained Test Administrators (TAs) administered the PISA assessment in schools.
PISA data collection activities are undertaken in accordance with strict quality assurance procedures. The quality assurance that ensures the PISA 2009 data are fit for use consists of two components. The first is to develop and document procedures for data collection and the second is to monitor and record the implementation of those procedures.
Survey Weighting and the Calculation of Sampling Variance
Survey weights are required to analyse PISA data, to calculate appropriate estimates of sampling error and to make valid estimates and inferences of the population. The PISA Consortium calculated survey weights for all assessed, ineligible and excluded students, and provided variables in the data that permit users to make approximately unbiased estimates of standard errors, conduct significance tests and create confidence intervals appropriately, given the complex sample design for PISA in each individual participating country.
Scaling PISA Cognitive Data
The mixed coefficients multinomial logit model as described by Adams, Wilson and Wang (1997) was used to scale the PISA data, and implemented by ConQuest® software (Wu, Adams and Wilson, 1997).
Data Management Procedures
The PISA assessment establishes standard data collection requirements that are common to all PISA participants. Test instruments include the same test items in all participating countries, and data collection procedures are applied in a common and consistent way amongst all participants to help ensure data quality. Test development is described in Chapter 2, and the data collection procedures are described in this chapter.
This chapter reports on PISA sampling outcomes. Details of the sample design are provided in Chapter 4. Table 11.1 shows the quality indicators for population coverage and the various pieces of information used to derive them. The following notes explain the meaning of each coverage index and how the data in each column of the table were used.
This chapter describes the application of Item Response Theory (IRT) scaling and plausible value methodology to the PISA 2009 assessment data.
Coding Reliability Studies
A substantial proportion of the PISA 2009 items were open-ended and required coding by trained personnel. It was important therefore that PISA implemented procedures which maximised the validity and consistency (both within and between countries) of this coding. Each country coded items on the basis of coding guides prepared by the Consortium (see Chapter 2) using the design described in Chapter 6. Training sessions to train coders from different countries on the use of the coding guides were held prior to both the field trial and the main survey.
This chapter describes the process used to adjudicate the implementation of PISA 2009 in each of the adjudicated entity (i.e. the participating countries, economies and adjudicated regions) and it gives the outcomes of the data adjudication which are mainly based on the following aspects:
the extent to which each adjudicated entity met PISA sampling standards;
the outcomes of the adaptation, translation and verification process;
the outcomes of the national centre and PISA quality monitoring visits;
the quality and completeness of the submitted data; and
the outcomes of the international coding review.
Proficiency Scale Construction
PISA reports student performance not just as scores, but also in terms of content, by describing what students who achieve a given level on a PISA scale typically know and can do. This chapter explains how these "described proficiency scales" are developed, and also how the results are reported and how they can be interpreted.
Scaling Procedures and Construct Validation of Context Questionnaire Data
The PISA 2009 context questionnaires included numerous items on student characteristics, student family background, student perceptions, school characteristics and perceptions of school principals. In 14 countries the optional parent questionnaires were administered to the parents of the tested students.
Digital Reading Assessment
PISA 2009 included an assessment of digital reading which was known during the cycle as the Digital Reading Assessment (DRA). Chapter 2 dealt with the associated test development activities, test design and framework coverage of the DRA. This chapter focuses on the technicalities and functionality of the delivery system and the various supporting systems.
The PISA 2009 international database consists of five data files: three with student responses, one with school responses and one with parent responses. All are provided in text (or ASCII format) with the corresponding SAS® and SPSS® control files.
Add to Marked List