AI and the Future of Skills, Volume 1
Capabilities and Assessments
Artificial intelligence (AI) and robotics are major breakthrough technologies that are transforming the economy and society. The OECD’s Artificial Intelligence and the Future of Skills (AIFS) project is developing a programme to assess the capabilities of AI and robotics, and their impact on education and work.
This volume reports on the first step of the project: identifying which capabilities to assess and which tests to use in the assessment. It builds on an online expert workshop that explored this question from the perspectives of both psychology and computer science. The volume consists of expert contributions that review skills taxonomies and tests in different domains of psychology, and efforts in computer science to assess AI and robotics. It provides extensive discussion on the strengths and weaknesses of different approaches, and outlines directions for the project. The report can therefore be a resource for the research community of multiple fields and policy makers who wish to obtain deeper insight into the complexity of machine capabilities.
Identifying artificial intelligence capabilities: What and how to test
Evaluating the capabilities of artificial intelligence (AI) has enormous implications in many areas, especially for the future of work and education. The context is also changing rapidly: the capabilities of humans and AI co‑evolve, with scenarios of replacement, displacement or enhancement. Beginning with a review of several taxonomies from human evaluation and AI, this chapter presents a 14-ability taxonomy to identify abilities as potentially disassociated clusters to characterise AI systems. It explores a range of human tests used for decades in recruitment and education, contrasting them with the increasing trend towards basing AI evaluation on benchmarks. The chapter reviews the challenges of bringing human tests to evaluate AI, identifying guidelines to devise reliable tests to compare the capabilities of humans and AI.