The recognition of the Programme for International Student Assessment (PISA) in many countries has fostered an interest in the tests the students take. This publication examines the link between the PISA test requirements and student performance. Focus is placed on the proportions of students who answer questions correctly across the range of difficulty from easy, to moderately difficult to difficult. The questions are classified by content, competencies, context and format and analysed to see what connections exist.
The present study affords an opportunity to view 15-year-old students’ capabilities internationally through the lens of mathematical literacy as defined by the PISA 2003 mathematics framework and the resulting assessment. The framework (Chapter 2), the focus on the actual items (Chapter 3), students’ performance by mathematical subtopic areas and competency clusters (Chapter 4), the influence of item format and reading level on item difficulty (Chapter 5), and the assessment and interpretation of student problem solving (Chapter 6) present an interesting view of mathematical literacy and instruction in an international context.
Main Features of the PISA Mathematics Theoretical Framework
This chapter provides a detailed description of the PISA 2003 assessment framework (OECD, 2003). It explains in detail the constructs of the mathematics assessment in PISA and lays out the context for the examples and further analysis presented in subsequent chapters.
A Question of Difficulty
This chapter illustrates the PISA 2003 assessment with released assessment items and links to different levels of mathematical literacy proficiency. Actual assessment items can be found in this chapter, along with a discussion of students’ performance on each of them.
Comparison of Country Level Results
This chapter focuses on differences in the patterns of student performance by aspects of mathematical content contained within PISA 2003 assessment items’ expectations. In participating countries, by the age of 15, students have been taught different subtopics from the broad mathematics curriculum. The subtopics vary in how they are presented to the students depending on the instructional traditions of the country.
The Roles of Language and Item Formats
This chapter focuses on factors other than the three Cs (mathematical content, competencies and context) that influence students’ performances. Just as countries differ, students’ experiences differ by their individual capabilities, the instructional practices they have experienced, and their everyday lives. The chapter examines some of these differences in the patterns of performance by focusing on three factors accessible through data from PISA 2003: language structure within PISA 2003 assessment items, item format, and student omission rates related to items.
Mathematical Problem Solving and Differences in Students' Understanding
This chapter concentrates on problem solving methods and differences in students’ mathematical thinking. It discusses the processes involved in what is referred to as the "mathematisation" cycle. The chapter provides two case studies, explaining how the elements required in the different stages of mathematisation are implemented in PISA items.
Note that all the publicly released questions that were used in the PISA 2003 mathematics assessment are presented in Chapter 3. Note as well that the data in these graphs are from the compendium (available at pisa2003.acer.edu.au/downloads.php) and that the percentages refer to students that reached the question (the percentage of students who did not reach the question plus the percentage of students that attempted to answer it will therefore add up to more than 100%).
This annex includes all PISA mathematics questions that were released before the PISA 2003 main testing was conducted.
This annex provides technical details for the analysis of variance performed for the language demand section of Chapter 5. In all cases a full factorial ANOVA was performed in SPSS using a univariate GLM procedure. The dependent variable in all cases was defined as item difficulty in logits centered at 0 for each country. The two factors in all cases were the same: country and categorized word count (see Chapter 5 for the detailed description of word-count). In the first case no other factors were added (see Table 5.1), in the subsequent cases one of the 3Cs were added (Tables 5.2 through Table 5.5) as a third factor (context, competency and content in the form of traditional topics). The third factor was added to find whether there are interactions between the word-count and features discussed in Chapter 5. Please note that as in Chapter 5, the content is categorized by traditional topics such as Algebra, Data, Geometry, Measurement, and Number. In addition to the F statistics, that show significance of main effects and interactions, partial ..2 is also provided to assess for the percent of total variance in the dependent variable accounted for by the variance between categories (groups) formed by the independent variable(s).
This annex provides technical details for the analysis of variance performed for the item format section of Chapter 5. In all cases a Full Factorial ANOVA was performed in SPSS using a univariate GLM procedure. The dependent variable in all cases was defined as item difficulty in logits centered at 0 for each country. The two factors in all cases were the same: country and item-format. In the first case no other factors were added (see Table A5.1), in the subsequent cases one of the following factors were added (Table A5.2-Competency, Table A5.3-Context, and Table A5.4-Word-Count). Table A5.5 presents the distribution of items by item format and traditional topic groupings. Please note that interactions between topics and item format were not considered due to the very small number of items in each cell (see Table A5.5). Table A5.6 presents post hoc comparisons for item format mean difficulties using the Bonferroni adjustment for multiple comparisons. In addition to the F statistics, that shows significance of main effects and interactions, partial ..2 is also provided to assess for the percent of total variance in the dependent variable accounted for by the variance between categories (groups) formed by the independent variable(s).
Add to Marked List