Abstract

The assessment of highly domain-general problem solving skills is increasingly important as problem solving is increasingly demanded by modern workplaces and increasingly present in international large-scale assessments such as the Programme for International Student Assessment. This thesis is about the computer-based assessment of problem solving skills based on Multiple Complex Systems : The main idea of the MCS approach is to present multiple computer-simulations of “minimally complex” problems in order to reliably assess certain problem solving skills. In each simulation, the problem solver has to interact with a problem in order to find out how to adequately represent the problem, and how to solve the problem. Up to now, two instances of the MCS approach have been proposed: the MicroDYN approach and – more recently, in the second paper of this thesis – the MicroFIN approach. In the current thesis I will elaborate on three research questions regarding the validity of the MCS approach: its content validity with regard to the concept of complex problem solving; the convergent validity of different instances of the MCS approach; the discriminant validity of the interactive problems of the MCS approach with regard to traditional static measures of reasoning and analytic problem solving skills. Each research question will be addressed in one corresponding paper: In a first paper complex problem solving is defined as the goal-oriented control of systems that contain multiple highly interrelated elements. After reviewing some of the major strands of research on complex problem solving a theoretical framework outlining the most important cognitive processes involved in solving complex problems is derived. The theoretical framework highlights both interactive knowledge acquisition and interactive knowledge application as the two major phases in the process of complex problem solving. Both phases are represented in all current instances of the MCS approach. In a second paper the convergent validity of MicroDYN and MicroFIN is investigated in order to demonstrate that both instances address the same kind of problem solving skills. Based on a multitrait-multimethod analysis of a sample of university students it is demonstrated that – in addition to method-specific skills – both instances assess a common set of skills related to representing and solving different kinds of interactive problems. In a regression of science grades on reasoning and the skills assessed by the instances of the MCS approach it is demonstrated that only the method-general representation trait and reasoning have substantial unique contributions. Thus, MicroDYN and MicroFIN seem to address a common set of skills and this set of skills is relevant for explaining school grades in science classes even beyond reasoning. In a third paper the discriminant validity of the interactive MicroDYN test is investigated by relating it to reasoning and traditional static measures of Analytic Problem Solving skills as they were applied in PISA 2003. Besides a common core of problem solving skills addressed by both kinds of tasks Fischer et al. expected to find evidence for additional skills that were related to interactive problems only. Results indicate that MicroDYN shares a lot of variance with APS even after controlling for reasoning in a sample of high-school students and the university student sample. With regard to the explanation of school grades MicroDYN had an incremental value compared to reasoning and APS in the high-school student sample but not significantly so in the university student sample. Basically these findings highlight both potential and limitations of the MicroDYN approach in its current form. Current instances of the MCS approach address a small set of problem solving skills reliably, but it takes more than these skills to competently solve complex problems. Implications for future research on the assessment of problem solving skills are discussed.