Metacognitive confidence judgments and their link to complex problem solving

Exploring the Relation between Time on Task and Ability in Complex Problem Solving

Innovative assessments of cross-curricular competencies such as complex problem solving (CPS) have currently received considerable attention in large-scale educational studies. This study investigated the nature of CPS by applying a state-of-the-art approach to assess CPS in high school. We analyzed whether two processes derived from cognitive psychology, knowledge acquisition and knowledge application, could be measured equally well across grades and how these processes differed between grades. Further, relations between CPS, general mental ability (g), academic achievement, and parental education were explored. Hungarian high school students in Grades 5 to 11 (N 855) completed MicroDYN, which is a computer-based CPS test, and the Culture Fair Test 20-R as a measure of g. Results based on structural equation models showed that empirical modeling of CPS was in line with theories from cognitive psychology such that the two dimensions identified above were found in all grades, and that there was some development of CPS in school, although the Grade 9 students deviated from the general pattern of development. Finally, path analysis showed that CPS was a relevant predictor of academic achievement over and above g. Overall, results of the current study provide support for an understanding of CPS as a cross-curricular skill that is accessible through computer-based assessment and that yields substantial relations to school performance. Thus, the increasing attention CPS has currently received on an international level seems warranted given its high relevance for educational psychologists.

Complex problem solving — More than reasoning?

This thesis analyzes the influence of structural knowledge on the individual level and the influence of knowledge heterogeneity on the group level on complex problem solving (CPS) performance. For the elicitation of structural knowledge, a computer based method, the association structure test (AST), is developed. Through term associations, measurement of thinking times, and through pairwise concept comparisons, the AST elicits a graph for each participant. The AST is tested in the domain of CPS. As complex problems are usually addressed by groups, a group setting is chosen. A curvilinear n-shaped connection between the group''s knowledge heterogeneity and its CPS performance is assumed. In an experiment, 150 participants were divided into dyads. Each participant received a text with seven knowledge elements on the control of a complex scenario. The heterogeneity of knowledge in a dyad was varied (small, medium, and large). After learning, dyad members self-assessed their knowledge. Knowledge similarity was calculated with knowledge management system (the skillMap). The knowledge similarity was also used for performance prediction. A discussion followed, during which dyad members taught each other what they had learned. Their structural knowledge was then assessed with the AST. In the following CPS task, dyads with medium heterogeneity exhibited a significant superior performance in comparison with the other two conditions. Knowledge heterogeneity exhibited a curvilinear relationship with the dyad''s CPS performance. The weighted density of AST-elicited knowledge graphs weakly correlated with CPS performance and explained a small but unique fraction of its variance. The skillMap similarity measure correlated significantly with CPS performance. Computer-based knowledge elicitation tools are thus potentially suited for performance prediction. CPS performance of groups is partially determined by the way in which knowledge is distributed inside the group.

Complex problem analysis and solving are an important evolutionary direction of TRIZ. Identifying core root cause and confirming contradictions are important for the management of complex problem solving. Both Theory of Constraints (TOC) and Fault Tree Analysis (FTA) include tools which can help get root cause from complex problem situation. FTA provides qualitative and quantitative analysis methods to evaluate the weights of root causes. A complex problem solving model integrating tools of TOC and FTA in TRIZ frame is put forward, a case shows the application of this model.

<span lang="EN-US">Complex problem solving (CPS) is a new paradigm in solving problems and is one of the soft skills needed to face the industrial revolution 4.0. Reflective abstraction is associating and modifying pre-existing conceptions into new situations. This article reviews research on CPS and reflective abstraction. This research is needed to know the relationship between reflective abstraction and CPS. The systematic writing of this review was assisted by the Publish or Perish 7 application, Mendeley, and VOSviewer. A literature search was performed through the ScienceDirect and ERIC databases. Based on the search results with the term “complex problem solving” and several exclusion criteria, 58 articles were found, whereas with the word “reflective abstraction” there were 23 articles, totaling 81 papers. Based on the literature review, it was found that there is a relationship between CPS and reflective abstraction by obtaining common ground in the form of prior knowledge. CPS requires prior knowledge from reflective abstraction to integrate the most relevant information. To improve CPS, efforts and special attention can be made to build initial knowledge through reflective abstraction. This article contributes to further research and becomes a study for the themes of CPS and reflective abstraction in learning and education.</span>

A Computational Theory of Complex Problem Solving Using Latent Semantic Analysis Jose Quesada, Walter Kintsch ([quesadaj, wkintsch]@psych.colorado.edu) Institute of Cognitive Science, University of Colorado, Boulder Boulder, CO 80309-0344 USA Emilio Gomez (egomez@ugr.es) Department of Experimental Psychology University of Granada Campus cartuja, S/N, Granada, Spain Abstract Complex Problem Solving (CPS) is a hybrid between field studies and experimental studies. This paper introduces a new, abstract conceptualization of microworlds research based on two innovations: (1) a problem representation, which treats protocols as objects in a feature space and, (2) a similarity metric which is defined in this problem space. Latent Semantic Analysis (LSA) is used to analyze performance in CPS, using actions or states as units instead of words and trials instead of text passages. Basic examples of applications are provided, and advantages and limitations are discussed. Introduction Many real-word decision making and problem solving situations are (1) dynamic, because early actions determine the environment in which subsequent decision must be made, and features of the task environment may change independently of the solver’s actions; (2) time-dependent, because decisions must be made at the correct moment in relation to environmental demands; and (3) complex, in the sense that most variables are not related to each other in one-to-one manner. In these situations, the problem requires not a single decision, but a long series of decisions which are dependent on one another. For a task that is changing continuously, the same action can be successful at moment t1 and useless at moment t2. However, traditional, experimental problem solving research has focused largely on tasks such as anagrams, concept identification, puzzles, etc. that are not representative of the features described above. In Europe, researchers led by Broadbent (e.g., Broadbent, 1977) in the UK and Dorner (e.g., Dorner, 1975) in Germany, were concerned about that fact and started working on a set of computer-based, experimental tasks that are dynamic, time-dependent, and complex, called Microworlds 1 . The study of microworlds is an example of Complex Problem Solving (e.g., French & Funke, 1995). This term sometimes has other meanings. For example, educational applications created to teach physics (Henderson, Klemes, & Eshet, 2000), simulated words in the early AI programs like the block word of SHRDLU, (Winograd, 1972) and static tasks to study decision making (Green, 2001) have been called Compared to traditional Problem Solving, Complex Problem Solving (CPS) radically changed the kind of phenomena reported, the kind of explanations looked for, and even the kind of data that were generated. However, the results obtained to date are far from being integrated and consolidated. This fact led Funke to affirm that ‘Despite 10 years of research in the area, there is neither a clearly formulated specific theory nor is there an agreement on how to proceed with respect to the research philosophy. Even worse, no stable phenomena have been observed’ (Funke, 1992, p. 25). Almost another 10 years after Funke’s argument, although more empirical research has been conducted in the area, we cannot say that the situation has changed drastically. At this moment, there is no theory able to explain even part of the specific effects that have been described or how they can be generalized. A theory of generalization and similarity is as necessary to psychology as Newton's laws are to physics (Shepard, 1987). However, for CPS there is no common, explicit theory to explain why a complex, dynamic situation is similar to any other situation or how two slices of performance taken from a problem solving task can possibly be compared quantitatively (Klein, Orasanu, Calderwood, & Zsambok, 1993). This lack of formalized, analytical models is slowing down the development of theory in the field. At least two problems make it difficult to apply the classical problem solving approach to CPS, one theoretical and one methodological: (1) The utility of state space representation for tasks with inner dynamics is reduced because in most CPS environments it is not possible to undo the actions. For example, imagine that two participants in Firechief (see below) are in an identical situation (system state) when the trial starts. One of them proceeds to make a control fire on the east side of a fire, while the other one is preparing a control fire on the north front of the fire. After these actions, the system state is no longer identical for them. Now they have to cope with rather different problems. Moreover, if the first participant wants to apply the same technique that Microworlds. However, we are concerned here only with tasks that fulfill the conditions described above.

In recent decades, the types of problems encountered in daily life have changed considerably. In particular, the technology-based society of today increasingly requires people to deal with dynamically changing, intransparent, and complex problems that cannot be solved only by applying factual knowledge. Moreover, in education and at work, such complex problems often need to be addressed collaboratively. Problem solving skills belong to the skills needed to successfully master such problems and are therefore considered to be crucial in the twenty-first century, but we can differentiate between problem solving on an individual level and collaborative problem solving. In the first part of this chapter, we provide insight into individual complex problem solving (CPS) research, with a special focus on its assessment. Specifically, we elaborate on different computer-based approaches to the measurement of CPS and their benefits and limitations. On the basis of the results of various empirical studies, we describe to which extent the CPS assessment has an added value in explaining the scholastic achievement and offer insights into a particular development of the assessment of this skill in large-scale educational contexts. Specifically, we elaborate on the assessment of CPS in the most influential large-scale project worldwide – the Programme for International Student Assessment (PISA) – and illustrate the process by describing example tasks. Further, we elaborate on how research on CPS has expanded towards the conceptually related skill of collaborative problem solving (ColPS) and discuss specific challenges in its assessment. With respect to the recent inclusion of ColPS in large-scale assessments, we present the assessment approach of another large-scale worldwide project – The Assessment and Teaching of Twenty-first Century Skills (ATC21S) – and describe how ColPS was assessed in the PISA 2015 cycle. Finally, we outline the importance of problem solving concepts in general and CPS specifically for future research, and discuss open research questions in this field.

Research on complex problem solving (CPS) has reached a stage where certain standards have been achieved, whereas the future development is quite ambiguous. In this situation, the editors of the Journal of Dynamic Decision Making asked a number of representative authors to share their point of view with respect to seven questions about the relevance of (complex) problem solving as a research area, about the contribution of laboratory-based CPS research to solving real life problems, about the roles of knowledge, strategies, and intuition in CPS, and about the existence of expertise in CPS.

Research on complex problem solving (CPS) has reached a stage where certain standards have been achieved, whereas the future development is quite ambiguous. In this situation, the editors of the Journal of Dynamic Decision Making asked a number of representative authors to share their point of view with respect to seven questions about the relevance of (complex) problem solving as a research area, about the contribution of laboratory-based CPS research to solving real life problems, about the roles of knowledge, strategies, and intuition in CPS, and about the existence of expertise in CPS.

This study examined whether experience with different computer-based complex problem situations would improve complex problem solving (CPS) competence in an unknown problem situation. Therefore, university students (N = 110) took part in a control group study. They were trained in five different complex problem situations for up to 7 hr, and their performance was tested in a sixth complex problem situation. The data analyses revealed that the training influenced the CPS process of knowledge acquisition. However, the CPS process of knowledge application was not impacted by experience with other problem situations. Implications for the concept of CPS as a trainable competence as well as the training of CPS in general are discussed.

Influence of curriculum reform: An analysis of student mathematics achievement in Mainland China

Although individual differences in complex problem solving (CPS) are well–established, relatively little is known about the process demands that are common to different dynamic control (CDC) tasks. A prominent example is the VOTAT strategy that describes the separate variation of input variables (“Vary One Thing At a Time”) for analyzing the causal structure of a system. To investigate such comprehensive knowledge elements and strategies, we devised the real-time driven CDC environment Dynamis2 and compared it with the widely used CPS test MicroDYN in a transfer experiment. One hundred sixty five subjects participated in the experiment, which completely combined the role of MicroDYN and Dynamis2 as source or target problem. Figural reasoning was assessed using a variant of the Raven Test. We found the expected substantial correlations among figural reasoning and performance in both CDC tasks. Moreover, MicroDYN and Dynamis2 share 15.4% unique variance controlling for figural reasoning. We found positive transfer from MicroDYN to Dynamis2, but no transfer in the opposite direction. Contrary to our expectation, transfer was not mediated by VOTAT but by an approach that is characterized by setting all input variables to zero after an intervention and waiting a certain time. This strategy (called PULSE strategy) enables the problem solver to observe the eigendynamics of the system. We conclude that for the study of complex problem solving it is important to employ a range of different CDC tasks in order to identify components of CPS. We propose that besides VOTAT and PULSE other comprehensive knowledge elements and strategies, which contribute to successful CPS, should be investigated. The positive transfer from MicroDYN to the more complex and dynamic Dynamis2 suggests an application of MicroDYN as training device.

Granular computing is to imitate human’s multi-granular computing strategy to problem solving in order to endow computers with the same capability. Its final goal is to reduce the computational complexity. To the end, based on the simplicity principle the problem at hand should be represented as simpler as possible. From structural information theory, it’s known that if a problem is represented at different granularities, the hierarchical description of the problem will be a simpler one. The simpler the representation the lower the computational complexity of problem solving should be. We presented a quotient space theory to multi-granular computing. Based on the theory a problem represented by quotient spaces will have a hierarchical structure. Therefore, the quotient space based multi-granular computing can reduce the computational complexity in problem solving. In the talk, we’ll discuss how the hierarchical representation can reduce the computational complexity in problem solving by using some examples.

Complexity manifests itself in many ways when attempting to solve different problems, and different tools are needed to deal with the different dimensions underlying that complexity. Not all complexity is created equal. We find that most treatments of complexity in problem solving within both the statistical and quality literature focus narrowly on technical complexity, which includes complexity of subject matter knowledge as well as complexity in the data access and analysis of that data. The literature lacks an understanding of how political complexity or organizational complexity interferes with good technical solutions when trying to deploy a solution. Therefore, people trained in statistical problem solving are ill-prepared for the situations they are likely to face on real projects. We propose a framework that illustrates examples of complexity from our own experiences, and the literature. This framework highlights the need for more holistic problem-solving approaches and a broader view of complexity. We also propose approaches to successfully navigate complexity.