Educational Research and Innovation

Centre for Educational Research and Innovation

2076-9679 (online)
2076-9660 (print)
Hide / Show Abstract

This series of books from the OECD's Centre for Educational Research and Innovations provides the results of OECD work on innovation in education.

Also available in French
The Nature of Problem Solving

The Nature of Problem Solving

Using Research to Inspire 21st Century Learning You do not have access to this content

Centre for Educational Research and Innovation

Click to Access:
  • PDF
  • READ
11 Apr 2017
9789264273955 (PDF) ;9789264273894(print)

Hide / Show Abstract

Solving non-routine problems is a key competence in a world full of changes, uncertainty and surprise where we strive to achieve so many ambitious goals. But the world is also full of solutions because of the extraordinary competences of humans who search for and find them. We must explore the world around us in a thoughtful way, acquire knowledge about unknown situations efficiently, and apply new and existing knowledge creatively.

The Nature of Problem Solving presents the background and the main ideas behind the development of the PISA 2012 assessment of problem solving, as well as results from research collaborations that originated within the group of experts who guided the development of this assessment. It illustrates the past, present and future of problem-solving research and how this research is helping educators prepare students to navigate an increasingly uncertain, volatile and ambiguous world.

loader image

Expand / Collapse Hide / Show all Abstracts Table of Contents

  • Mark Click to Access
  • Foreword and Acknowledgements

    The demands on learners and thus education systems are evolving fast. In the past, education was about teaching people something. Now, it’s about making sure that students develop a reliable compass and the navigation skills to find their own way through an increasingly uncertain, volatile and ambiguous world. These days, we no longer know exactly how things will unfold, often we are surprised and need to learn from the extraordinary, and sometimes we make mistakes along the way. And it will often be the mistakes and failures, when properly understood, that create the context for learning and growth. A generation ago, teachers could expect that what they taught would last a lifetime for their students. Today, teachers need to prepare students for more rapid economic and social change than ever before, for jobs that have not yet been created, to use technologies that have not yet been invented, and to solve social problems that we don’t yet know will arise.

  • Executive summary

    Problem solving is one of the key competencies humans need in a world full of changes, uncertainty and surprise. It is needed in all those situations where we have no routine response at hand. Problem solving requires the intelligent exploration of the world around us, it requires strategies for efficient knowledge acquisition about unknown situations, and it requires creative application of the knowledge available or that can be gathered during the process. The world is full of problems because we strive for so many ambitious goals – but the world is also full of solutions because of the extraordinary competencies of humans who search for and find them.

  • Add to Marked List
  • Expand / Collapse Hide / Show all Abstracts Problem solving: Overview of the domain

    • Mark Click to Access
    • The development and assessment of problem solving in 21st-century schools

      The skills considered most essential in our modern societies are often called 21stcentury skills. Problem solving is clearly one of them. Students will be expected to work in novel environments, face problems they have never seen and apply domain-general reasoning skills that are not tied to specific contents. Computerised dynamic problem solving can be used to create just such an interactive problem situation in order to assess these skills. It may therefore form the basis for a type of assessment which helps answer the question of how well schools are preparing their students for an unknown future. This chapter shows how education systems may benefit from such an assessment. It reviews educational methods that have aimed at developing students’ higher-order thinking skills and indicates how experiences with these approaches can be used to improve problem solving, from direct teaching, through content-based methods, to innovative classroom processes. It outlines the evolution of large-scale assessment programmes, shows how assessing problem solving adds value and, finally, identifies some directions for further research.

    • Analytical problem solving: Potentials and manifestations

      Being able to solve problems is a crucially important outcome of almost all domains of education, as well as a requirement for future learning and acheivement. This chapter summarises the results from the cross-curricular analytical problem solving element of the 2003 Programme for International Student Assessment (PISA). The results from Germany indicated that schools may not be sufficiently exploiting students’ cognitive potential to develop their subject-specific competencies. To investigate this hypothesis and find ways to make use of this potential, the chapter describes research on the cognitive components and the structure of analytical problem-solving competence and its relation to mathematical competence. It concludes with results from two experimental studies aiming to foster students’ mathematical competence by training in analytical problemsolving competence.

    • Problem solving: Understanding complexity as uncertainty

      Understanding complexity has dominated the domain of problem solving. There needs to be a cohesive idea about what makes problem solving hard in contexts broadly described as complex, and what are the sources of errors in problem solving in situations thought to be complex. This chapter argues that our understanding of complexity requires an overhaul, and perhaps a more encouraging and fruitful way forward in research on complex problem solving, is to reconceptualise “finding solutions to complex problems” as instead “findings ways of reducing uncertainty”. This entails redefining the domain of “complex dynamic problem solving” into “decision making under uncertainty”. This enables a Bayesian approach to understanding what makes a situation controllable, or appear to be controllable to a problem solver, and hence what cognitive and psychological factors they bring to bear on the situation.

    • Problem solving from a mathematical standpoint

      Problem solving has always been at the heart of both pure and applied mathematics. This chapter examines the evolution of the concept of what constitutes a problem, and the models and findings concerning problem solving from research into the learning and teaching of mathematics. It then considers the emerging understanding of the role played by metacognition in current conceptions of problem solving in mathematics and its learning.

    • Add to Marked List
  • Expand / Collapse Hide / Show all Abstracts Dynamic problem solving as a new perspective

    • Mark Click to Access
    • The PISA 2012 assessment of problem solving

      In 2012, the OECD’s Programme for International Student Assessment (PISA) featured a computer-based test of problem-solving competency. Computer delivery enabled the assessment to include for the first time in such a large-scale survey, problems that require the solver to interact with the problem situation to uncover information that is not explicitly disclosed from the outset. These types of problems occur frequently in everyday life when dealing with technological devices such as mobile telephones and vending machines. This chapter presents the PISA 2012 definition of problem solving, outlines the development of the assessment framework and discusses its key organising elements: the problem context, the nature of the problem situation and the four problemsolving process groups. It then presents sample items with commentary.

    • Interactive problem solving: Exploring the potential of minimal complex systems

      Interactive problem solving (IPS) is considered an increasingly important skill in professional and everyday life as it mirrors the interaction of a human user with dynamic technical and non-technical devices. Here, IPS is defined as the ability to identify the unknown structure of artefacts in dynamic, mostly technology-rich, environments to reach certain goals. Two overarching processes, the acquisition of knowledge and its application, can be theoretically distinguished and this chapter presents two measurement approaches assessing these processes: MicroDYN and MicroFIN, both of which rely on the idea of minimal complex systems (MICS). Whereas MicroDYN models quantitative connections between elements of a problem (i.e. more and less), MicroFIN models qualitative connections between them (e.g. on/off or white/black). This chapter summarises research on these approaches and discusses the implications for educational assessment.

    • The history of complex problem solving

      Complex problem solving (CPS) is about reaching one’s goals taking into account a large number of highly interrelated aspects. CPS has a rich history in experimental and psychometric research. The chapter highlights some of the most important findings of this research and shows its relationship to interactive problem solving. More specifically, it 1) characterises typical human strategies and shortcomings in coping with complex problems; 2) summarises some of the most influential theories on cognitive aspects of CPS; and 3) outlines the history of including CPS skills and competency in assessment contexts. The last section summarises the current state of play and points out some trends for future research.

    • Add to Marked List
  • Expand / Collapse Hide / Show all Abstracts Empirical results

    • Mark Click to Access
    • Empirical study of computer-based assessment of domain-general complex problem-solving skills

      This study reviews the results of a recent project on problem solving. Taking a developmental and structural perspective, it contrasts static, paper-and-pencil tests with interactive, technology-based tests of thinking skills, with a special reference to reasoning skills including knowledge acquisition, knowledge application, and transfer of knowledge. Hungarian students aged 11 to 17 completed problem-solving tests in static scenarios (assessing domain-specific problem-solving skills from maths and science) and in interactive scenarios (assessing domain-general complex problem-solving skills). The students were also assessed for inductive reasoning and fluid intelligence, both by first generation tests. This chapter uses the results to elicit evidence for the development of dynamic problem solving and for the relationship between those 21st-century skills. Finally, it discusses the possibility of using traditional static tests to predict performance in third generation tests measuring dynamic problem solving.

    • Factors that influence the difficulty of problem-solving items

      This chapter presents a study undertaken by the authors using data from the PISA 2012 computer-based assessment of problem solving. It considered ten characteristics understood to influence the difficulty of items used in problem-solving assessement. Each item was rated to reflect the amount of each characteristic it possessed. The item responses from about 85 000 participants from 44 countries were analysed to obtain item response theory (IRT) estimates of item difficulties. The predictor characteristics were analysed in a number of ways, including a hierarchical cluster analysis, regression analysis with item difficulty as outcome variable and a principal component factor analysis. The main characteristics predicting difficulty seem to be: the complexity and type of reasoning skills involved in solving the problem; the amount of opportunity the solver is given to experiment or uncover hidden facets in a problem scenario (more opportunity to explore and experiment will make a problem easier); and the number and nature of constraints that a solution must satisfy (complex or conflicting constraints will make a problem more difficult).

    • Assessing complex problem solving in the classroom: Meeting challenges and opportunities

      Complex problem solving (CPS) is now established as a key aspect of today’s educational curricula and a central competence for international assessment frameworks. It has become clear that the educational context – particularly among general school-age students – places particular demands on the instruments used to assess CPS, such as computer-based microworlds. This chapter shows how these challenges can successfully be addressed by reviewing recent advancements in the field of complex problem solving. Using the example of the Genetics Lab, a newly developed and psychometrically sound microworld which emphasises usability and acceptance amongst students, this chapter discusses the challenges and opportunities of assessing complex problem solving in the classroom.

    • Add to Marked List
  • Expand / Collapse Hide / Show all Abstracts New indicators

    • Mark Click to Access
    • Log-file data as indicators for problem-solving processes

      This chapter describes a theoretical rationale and an empirical methodology for analysing log-file data recorded from students’ interactions with computer-based problem-solving assessment tasks. These analyses can play an important role in the development and refinement of rules for automatically scoring complicated sequences of process data which describe procedural aspects of problem solving. It presents examples of log files and analyses from an interactive problem-solving task. In complex domains where a variety of skills and dispositions may influence performance, interpreting log-file data to infer how students solve problems could provide valid information to help meet their instructional needs.

    • Educational process mining: New possibilities for understanding students' problem-solving skills

      The assessment of problem-solving skills heavily relies on computer-based assessment (CBA). In CBA, all student interactions with the assessment system are automatically stored. Using the accumulated data, the individual test-taking processes can be reproduced at any time. Going one step further, recorded processes can even be used to extend the problem-solving assessment itself: the test-taking process-related data gives us the opportunity to 1) examine human-computer interactions via traces left in the log file; 2) map students’ response processes to find distinguishable problem-solving strategies; and 3) discover relationships between students’ activities and task performance. This chapter describes how to extract process-related information from event logs, how to use these data in problem-solving assessments and describes methods which help discover novel, useful information based on individual problem-solving behaviour.

    • EcoSphere: A new paradigm for problem solving in complex systems

      The western perspective on the environment has changed in the last decades. Many natural processes, such as hurricanes or tsunamis, and cultural ones, such as nuclear power plants or space missions, hold immense potential to cause problems, so the current state of the world is often seen as a complex system. Dealing with complex systems, particularly unstable ones with autonomously changing (eigendynamic) elements, will be one of the key competences for future generations, which is why measuring and training these competences is so important today. This chapter focuses on measuring competency at problem solving in complex systems (PSCS). It introduces the processes involved in human problem solving – from observation, whether through instruction or active construction, to exploration and finally regulation, and discusses their relations to the environment, which is built up of complex systems. It then presents the new assessment tool EcoSphere, which is a simulation framework for testing and training human behaviour in complex systems. In particular, it allows test takers’ previous knowledge to be taken into account to better measure their problem-solving abilities.

    • Add to Marked List
  • Expand / Collapse Hide / Show all Abstracts Future issues: Collaborative problem solving

    • Mark Click to Access
    • Assessment of collaborative problem-solving processes

      This chapter presents a framework for understanding collaborative problem solving, including both the cognitive and social perspectives, and identifies the construct’s theoretical underpinnings, structure and elements. It describes the circumstances under which collaborative problem solving might best be used, with consequences for the design of tasks to assess the component skills. It highlights the characteristics of problemsolving tasks, interdependence between problem solvers and the asymmetry of stimulus and response, through a focus on task design. The chapter then outlines approaches to measuring collaborative problem solving and illustrate them with examples.

    • Assessing conversation quality, reasoning, and problem solving with computer agents

      Conversational agents have been developed to help students learn in computerlearning environments with collaborative reasoning and problem solving. Conversational agents were used in the 2015 Programme for International Student Assessment (PISA) collaborative problem-solving assessments, where a human interacted with one, two or three agents. This chapter reviews advances in conversational agents and how they can help students learn by engaging in collaborative reasoning and problem solving. Using the example of AutoTutor it demonstrates how dialogues can mimic the approaches of expert human tutors. Conversations with intelligent systems are quite different depending on the number of agents involved. A human interacting with only one computer agent during a dialogue needs to continuously participate in the exchange. In trialogues there are two agents, so there are more options available to the human (including social loafing and vicarious observation) and the conversation patterns can be more complex, illustrated by Operation ARIES!, which uses a number of patterns in teaching students the basics of research methodology. Conversational agent systems use online, continuous, formative assessment of human abilities, achievements, and psychological states, tracked during the course of the conversations. Some of these formative assessment approaches are incorporated in the PISA 2015 assessment of collaborative problem solving. However, this chapter focuses on formative assessment in learning environments rather than on summative assessments.

    • Add to Marked List
  • Expand / Collapse Hide / Show all Abstracts Finale

    • Mark Click to Access
    • Epilogue

      As we finish the work of editing this book and look back at the process behind that effort, we have the feeling that it has simultaneously been one of the most inspiring experiences and one of the most challenging undertakings of our professional careers. The idea of creating a book on the state of the art of problem solving research was conceived in a series of formal and informal meetings among the authors, and it has been shaped in a number of discussions and e-mail exchanges since then.

    • Add to Marked List
Visit the OECD web site