Two experiments are reported that employed think-aloud methods to test predictions concerning relevance effects and rationalisation processes derivable from Evans' (1996) heuristic-analytic theory of the selection task. Evans' account proposes that card selections are triggered by relevance-determining heuristics, with analytic processing serving merely to rationalise heuristically cued decisions. As such, selected cards should be associated with more references to both their facing and their hidden sides than rejected cards, which are not subjected to analytic rationalisation. Experiment 1 used a (...) standard selection-task paradigm, with negative components permuted through abstract conditional rules. Support was found for all heuristic-analytic predictions. This evidence was shown to be robust in Experiment 2, where "select - don't select" decisions were enforced for all cards. Both experiments also clarify the role played by secondary heuristics in cueing the consideration of hidden card values during rationalisation. We suggest that whilst Evans' heuristic-analytic model and Oaksford and Chater's (e.g., 2003) optimal data selection model can provide compelling accounts of our protocol findings, the mental models theory fares less well as an explanation of our full dataset. (shrink)

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and less mathematical verbal Cognitive Reflection Test (vCRT). Importantly, thinking aloud did not disrupt test performance compared to a (...) control group. Moreover, verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of ‘reflection’ (as deliberate and conscious). (shrink)

Thinking aloud is the concurrent verbalization of thoughts while performing a task. The study of thinking-aloud protocols has a long tradition in cognitive psychology, the field of education, and the industrial-organizational context. It has been used rarely in cultural and cross-cultural psychology. This paper will describe thinking aloud as a useful method in cultural and cross-cultural psychology referring to a few studies in general and one study in particular to show the wide applications of (...) this method. Thinking-aloud protocols can be applied for a) improving the validity of cross-cultural surveys, b) process analysis of thoughts and the analysis of changes over time, c) theory development across cultures, d) the study of cultural meaning systems, and e) individual as well as group level analyses allowing hypothesis testing cross-culturally. Limitations of the thinking-aloud method are also discussed. (shrink)

The use of remember–know judgments to assess subjective experience associated with memory retrieval, or as measures of recollection and familiarity processes, has been controversial. In the current study we had participants think aloud during study and provide verbal reports at test for remember–know and confidence judgments. Results indicated that the vast majority of remember judgments for studied items were associated with recollection from study , but this correspondence was less likely for high-confidence judgments . Instead, high-confidence judgments were more (...) likely than remember judgments to be associated with incorrect recollection and a lack of recollection. Know judgments were typically associated with a lack of recollection , but still included recollection from the study context . Thus, although remember judgments provided fairly accurate assessments of retrieval including contextual details, know judgments did not provide accurate assessments of retrieval lacking contextual details. (shrink)

Two studies examined the use of think aloud (TA) protocol as a means for collecting data of cognitive processes during performance in golf. In study 1, TA was employed to examine if different verbalisation (Level 2 or Level 3 TA) instructions influence performance of high and low skilled golfers. Participants performed 30 putts using TA at either Level 2, Level 3, or no verbalization condition. Although Level 3 verbalization produced a higher volume of verbal data than Level 2, TA (...) at either level 2 or 3 did not impair putting performance compared to no verbalization. Study 2 examined the congruence between data collected via TA at Level 3 and cued retrospective recall of cognitive processes during golf performance. Experienced golfers performed six holes of golf whilst engaging in Level 3 TA. After performance, three semi-structured retrospective interviews were conducted (ten minutes after performance, 24 hours after performance and 48 hours after performance). A comparison of the themes identified large discrepancies between the information reported during TA and at interview, with only 38-41% similarity in variables reported to influence decision making on each hole. Both studies suggest TA is a valuable method for recording cognitive processes of individuals during task performance. TA provides richer verbal data regarding decisions than cued retrospective recall, and TA does not negatively impact performance. (shrink)

It has been widely theorized and empirically proven that self-regulated learning is related to more desired learning outcomes, e.g., higher performance in transfer tests. Research has shifted to understanding the role of SRL during learning, such as the strategies and learning activities, learners employ and engage in the different SRL phases, which contribute to learning achievement. From a methodological perspective, measuring SRL using think-aloud data has been shown to be more insightful than self-report surveys as it helps better in (...) determining the link between SRL activities and learning achievements. Educational process mining on the basis of think-aloud data enables a deeper understanding and more fine-grained analyses of SRL processes. Although students’ SRL is highly contextualized, there are consistent findings of the link between SRL activities and learning outcomes pointing to some consistency of the processes that support learning. However, past studies have utilized differing approaches which make generalization of findings between studies investigating the unfolding of SRL processes during learning a challenge. In the present study with 29 university students, we measured SRL via concurrent think-aloud protocols in a pre-post design using a similar approach from a previous study in an online learning environment during a 45-min learning session, where students learned about three topics and wrote an essay. Results revealed significant learning gain and replication of links between SRL activities and transfer performance, similar to past research. Additionally, temporal structures of successful and less successful students indicated meaningful differences associated with both theoretical assumptions and past research findings. In conclusion, extending prior research by exploring SRL patterns in an online learning setting provides insights to the replicability of previous findings from online learning settings and new findings show that it is important not only to focus on the repertoire of SRL strategies but also on how and when they are used. (shrink)

Data literacy – the ability to analyze, interpret, evaluate, and use data and data visualizations – has become increasingly important for understanding and communicating information in the discipline of history. In the United States, curricular standards and standardized assessments already reflect this importance, but educators lack a clear picture of how students use data visualizations when reasoning about the past. How do students use data visualizations when reasoning about a historical question? To what degree does using data visualizations enhance students’ (...) historical reasoning? What challenges do students face when attempting to use data visualizations? This study addresses these questions, investigating how fifth-grade, eighth-grade, and high school students use data visualizations from typical school textbooks. Twenty-seven students were interviewed, surveyed, and asked to think aloud about a historical question while using a textbook entry that included a data visualization. Results from qualitative and quantitative analyses highlight the potential for data visualizations to enhance historical reasoning. However, results also indicate that students are unlikely to read data visualizations on their own, underscoring the importance of providing targeted data literacy instruction. (shrink)

Einstellung effects designate the phenomenon where established routines can prevent people from finding other, possibly more efficient solutions. Here we investigate the mechanism behi...

The squared paper or graphs are grid-based design representations used in engineering, industrial and craft design practices wherein designs are drawn over symmetrical grids. This paper reports grid-processing strategies undertaken by actors in a native craft practice, viz. Kashmiri carpet-weaving having three task contexts: (1) _design_, wherein designs are drawn on graph sheets and color scheme given by assigning practice-specific symbolic codes to the motifs by designers; (2) _coding_, wherein a cryptic script, called _talim_, is generated from these encoded graphs (...) by talim-writers; and (3) _weaving_, wherein this script is read, decoded and communicated in an equally cryptic trade-language by weavers among their teams. 32 video-recorded think-aloud sessions were conducted with talim-writers to understand their graph-processing and resultant coding strategies. The observations and qualitative analysis of transcripts revealed highly _situated_ strategies undertaken by experts keen on imbibing visuality of design embedded in the grids, in contrast to _embodied_ strategies undertaken by less-experienced coders keen on processing grids’ structural features and getting overwhelmed by grid-clutter in the process. In this landscape, the findings revealed a negotiated nature of errors instead of being deviations from the underlying grid and thereby objectively assessable. The paper makes recommendations for CAD systems to include more of those grid features exploited by experts during their graph processing to facilitate designers. (shrink)

Ethics isn’t a conversation exclusive to philosophers. There is value, then, in not only understanding how laypeople think about issues in ethics, but also bringing their judgments into dialogue with those of philosophers in order to make sense of agreement, disagreement, and the consequences of each. Experimental philosophers facilitate this dialogue uniquely by capturing laypeople’s judgments and analyzing them in light of philosophical theory. They have done so almost exclusively by using face valid quantitative surveys about philosophically interesting thought experiments. (...) Based on high participant support for this or that response, researchers conclude that a given theory is more or less intuitive, and in some cases that it is or isn’t true. However, such conclusions can only be drawn from quantitative survey data if one assumes those data accurately reflect laypeople’s thinking on the issues of interest, an assumption that can’t be justified by the quantitative data itself since they are potentially opaque. This is the methodological problem of experimental philosophy: experimental philosophers who use quantitative methods alone can’t sufficiently demonstrate their data to be reflective of relevant judgments/intuitions. To better explore people’s judgments on issues in ethics and to determine whether experimental philosophers are or aren’t getting what they take themselves to be getting in surveys, I recreated part of an experiment by Chituc, Henne, Sinnott-Armstrong, and De Brigard on the Kantian principle that “ought implies can”, or that whenever an agent ought to do something then it is the case that she can, and added a layer of qualitative data to it by having participants think aloud while completing the surveys and conduct a follow-up interview. With similar quantitative results to those from Chituc et al., I argue that my qualitative results 1) cast doubt on Chituc et al.’s data being reflective of many people’s judgments, especially on the more interesting questions relating to moral obligation; 2) reveal unique insights into how people think about OIC; 3) generally speak to the value of experimental philosophers using triangulation of methods in better understanding laypeople’s complex and varied thinking about questions important to ethics and beyond. Supplemental documents: Appendices A, B, C, D, and E are surveys I used in my study. Appendices C and D are derived from Chituc et al.’s publication “Blame, Not Ability, Impacts Moral ‘Ought’ Judgments for Impossible Actions: Toward an Empirical Refutation of ‘Ought’ Implies ‘Can.’” Appendix F is the recruitment flyer I distributed for my study. Appendix H is the quantitative data from my study. (shrink)

This paper considers the piloting of an online learning component of a final-year social work degree ethics module at an inner-city English university. An Action Research approach was used to evaluate this pilot project and the paper illustrates how students were involved in developing and designing the teaching programme as part of the Action Research cycle. The paper explores theoretical aspects of e-learning pedagogy through an analysis of issues emerging during the planning and delivery of this pilot project. The cognitive (...) and affective impact of both face-to-face and electronic learning is discussed. It is suggested that some students may experience e-learning as a safe space for thinking about ethical dilemmas. The challenges for teachers and learners are highlighted and the author argues that in order to support student learning, online teaching needs to be carefully designed and appropriately resourced. (shrink)

I mention the benefits, challenges, and costs of using small group activities to enhance our students’ learning of critical thinking skills in our courses, and then describe ten examples of these groups. Two of these examples are not commonly reported in the literature on small groups, so I describe them in greater detail to facilitate their use in our courses.

Summary Why and how is the Gestalt theorists’ concept of productive thinking particularly suitable for being applied to the educational question of how student motivation can be encouraged, thus providing an important condition for self-regulated, intrinsically motivated learning? An answer to this question has been sought using an approach to the fostering of text comprehension ability, based upon the features specific to productive thinking, originally identified by Wertheimer and Duncker. Firstly, these specific features are dealt with and their (...) educational implications compared with those deriving from the definitions of problem-solving used most frequently in educational research. Secondly, an analysis is made of the process by which the features specific to productive thinking are turned into the conditions for a kind of text analysis suitable for designing an instructional project aimed at enhancing text comprehension ability and, at the same time, encouraging intrinsic motivation and self-regulation on the part of the learner. Thirdly, an educational project centred on the thinking-aloud poor reader is described, where thinking aloud and reflection–response are combined in order to guarantee the maximum level of intrinsic motivation. In the concluding section, the most important features of the project are discussed in relation to reciprocal teaching and scaffolding. (shrink)

This book, based on a critical/collective/auto/ethnographic research project, describes an assemblage of theoretically informed, arts-based methods that aim to promote multiplicity and thinking. It explores multiplicities of knowing, sensing, doing and being, generated by analyzing knowing frames, poetry, reading aloud, fableing, playwriting and other inventive, playful and scholarly ways of working with experiences and stories. By offering engaging and inspiring strategies that can disturb standardizations and interrupt cultural normativities, the book sheds light on the conditions that might be (...) present in cultural contexts that enable diversity and creativity. The research project on which this book is based originated from a contradictory set of conditions characterized on the one hand by a marked interest in creative research methods and novel knowledge practices and, on the other hand, by a widespread concern that we live in increasingly standardized times, featuring systems that specify objectives ahead of time, demand compliance and narrow the possibilities for human action. The book takes readers on an arts-based journey designed to enhance the opportunities for imaginative and ethical professional practice in education, human services and the arts. (shrink)

Soc. …I speak of what I scarcely understand; but the soul when thinking appears to me to be just talking—asking questions of herself and answering them, affirming and denying. And when she has arrived at a decision, either gradually or by a sudden impulse, and has at last agreed, and does not doubt, this is called her opinion. I say, then, that to form an opinion is to speak, and opinion is a word spoken,—I mean, to oneself and in (...) silence, not aloud or to another: What think you? Theaet. I agree. (shrink)

The article reflects on the nature of the political in theatre, assessing the notion that theatre is the last free public space and evaluating the claims to be political of rival, problematic modes of writing—the theatre of fact or verbatim theatre and the allegorical late plays of Bond, Pinter and Churchill, turning to consider the problematic legacy of Brecht, the avatar of the political. The discussion turns to writers often excluded from the political nomenclature, developing the notion of the centrality (...) of critique and offering an argument for the Naturalist writers as propagators of true ‘thinking aloud’, thereby suggesting they provide a model for theatre as such. The piece concludes with a discussion of the author’s own contribution to the genre in the light of these analyses. (shrink)

The digital revolution has widened the array of curriculum materials available to history teachers. Given the variable quality of these new materials and the deeply contextual nature of teaching, educators need better tools for selecting among the vast options available. This study aimed to validate a device designed for that purpose, called the Curriculum Materials Evaluation Tool (CMET). Using a questionnaire and think-aloud interview, the study examined how four social studies teachers evaluated a novel material set for potential classroom (...) use, and how the CMET mediated their analysis. Findings showed that the teachers preferred when materials are readily accessible and adaptable for their specific teaching context, in contrast to “one-size-fits-all” resources that dominate the Web. The CMET appeared to foster a deeper and broader analysis of the given material set, especially regarding its political and cultural orientation. The study suggests that teachers’ role in vetting and adapting digital materials be recognized and embraced. The CMET can contribute to that end through its use in district adoption committees, teacher professional development, and collaborative curriculum research. (shrink)

Experimental philosophers rely almost exclusively on quantitative surveys that potentially misrepresent participants’ multifarious judgments. To assess the efficacy of qualitative methods in experimental philosophy and reveal limitations with quantitative surveys, a study was conducted on the Kantian principle that ‘ought implies can’, which limits moral obligation to actions that agents can do. Specifically, the think aloud method and a follow-up interview were employed in a modified version of a prominent experiment that recorded participants’ judgments of ability, blame, and obligation (...) using quantitative surveys. The modified version produced quantitative results similar to the original experiment along with qualitative data that reveal that the surveys fundamentally misrepresented participants’ judgments. The qualitative transcripts from 40 participants are analyzed to show that ‘ought implies can’ judgments are complex and multifarious, that ‘ought implies can’ judgments are misrepresented by quantitative survey questions, and that the majority of participants uphold or preserve ‘ought implies can.’ The results suggest that experimental philosophers can more accurately capture judgments by using qualitative methods, and that studies which rely on quantitative surveys possibly misrepresent participants’ judgments. (shrink)

Research in the fields of complex problem solving (CPS) and dynamic decision making using microworlds has been mainly conducted in Western industrialized countries. This study analyzes the CPS process by investigating thinking‐aloud protocols in five countries. Participants were 511 students from Brazil, Germany, India, the Philippines, and the United States who worked on two microworlds. On the basis of cultural‐psychological theories, specific cross‐national differences in CPS strategies were hypothesized. Following theories of situatedness of cognition, hypotheses about the specific (...) frequency of problem‐solving strategies in the two microworlds were developed. Results of the verbal protocols showed (a) modification of the theoretical CPS model, (b) task dependence of CPS strategies, and (c) cross‐national differences in CPS strategies. Participants’ CPS processes were particularly influenced by country‐specific problem‐solving strategies. (shrink)

In spite of the wide variety of approaches to ethics training it is still debatable which approach has the highest potential to enhance professionals’ integrity. The current effort assesses a novel curriculum that focuses on metacognitive reasoning strategies researchers use when making sense of day-to-day professional practices that have ethical implications. The evaluated trainings effectiveness was assessed by examining five key sensemaking processes, such as framing, emotion regulation, forecasting, self-reflection, and information integration that experts and novices apply in ethical decision-making. (...) Mental models of trained and untrained graduate students, as well as faculty, working in the field of physical sciences were compared using a think-aloud protocol 6 months following the ethics training. Evaluation and comparison of the mental models of participants provided further validation evidence for sensemaking training. Specifically, it was found that trained students applied metacognitive reasoning strategies learned during training in their ethical decision-making that resulted in complex mental models focused on the objective assessment of the situation. Mental models of faculty and untrained students were externally-driven with a heavy focus on autobiographical processes. The study shows that sensemaking training has a potential to induce shifts in researchers’ mental models by making them more cognitively complex via the use of metacognitive reasoning strategies. Furthermore, field experts may benefit from sensemaking training to improve their ethical decision-making framework in highly complex, novel, and ambiguous situations. (shrink)

This article presents an exploratory analysis of the metaphoric structure of five artistic paintings within “Think aloud” protocols, in which a group of 14 English speakers with a low self-rated level of expertise in art and history of art expertise were asked to verbalize all their thoughts, ideas and impressions of the artworks. The main findings of this study can be summarized as follows: (1) multiple interpretations for the same artwork are possible, (2) the interpretations of the metaphorical structures (...) described by the participants often diverge from those advanced by the researchers. These findings challenge the methods by which metaphor identification and analysis in pictorials is currently approached. As a matter of fact, most of the research in pictorial metaphors tends to reduce stimuli such as artistic paintings to unique metaphoric interpretations generally produced by a single researcher by means of introspection. By addressing this methodological problem in metaphor research, this article contributes to the development of a theoretical and operational participant-based framework that takes into account the role of metaphoric conceptualization within the domain of art and art cognition. (shrink)

In this study, we intend to examine information retrieval behaviors from a psychological point of view using a search engine on the World Wide Web (WWW). We investigated information retrieving behaviors in detail based on both the recorded data of retrievers’ web browsing actions and their thinking processes by the “think aloud” method. We focused on selected keywords for retrieving and compared them between retrievers who had enough knowledge about their task and those who did not. Our goal (...) was to learn about the literacy needed for finding required information efficiently on the WWW. (shrink)

Prison work creates ethical challenges for which a training program was initiated for Dutch prison staff to foster their Moral Craftsmanship (MCS). The concept of MCS is not yet defined and operationalized in literature. This explorative study aims to 1) define MCS, 2) identify conceptual elements of MCS, and 3) develop a measurement tool for MCS. A document and literature study provided input for the definition and selection of conceptual elements related within DCIA policy documents, identifying three conceptual levels of (...) MCS: individual (cognition, attitude, and actions), team, and leadership. This first MCSQ consists of seventy-one items distributed across the domains: values and norms, judgment and reasoning, consultation, implementation, looking back, interactions, and leadership. We tested its usability via ‘think-aloud interviews.’ Validation of the current version of the MCS-Questionnaire and consensus among experts about what to include or exclude in the concept of MCS are needed for further development. (shrink)

Despite the importance of effective observational analysis in the technical aspects of climbing performance, limited research informs this aspect of climbing coach education. Thus, the purpose of the present research was to explore cognitive-perceptual mechanisms underpinning visual search strategies of expert and novice climbing coaches through the novel combination of eye-tracking technology and retrospective think-aloud methodology. Analysis of gaze data revealed expert climbing coaches to demonstrate fewer fixations of greater duration, and fixate on distinctly different areas of the visual (...) display, than their novice counterparts. Retrospective think-aloud interviews further demonstrated differences in the cognitive-perceptual mechanisms underpinning these visual search strategies, with expert coaches more cognisant of their visual search strategy. To expand, the gaze behaviour of expert climbing coaches was underpinned by hierarchical and complex knowledge structures relating to principles of climbing movement. This enabled expert coaches to actively focus on the most relevant aspects of a climber’s performance for analysis. Findings demonstrate the utility of combining eye tracking and retrospective think-aloud interviewing as a new, efficient methodology of capturing the cognitive-perceptual processes of climbing coaches to inform coaching education/strategies. (shrink)

Interest in thought experiments (TEs) derives from the paradox: “How can findings that carry conviction result from a new experiment conducted entirely within the head?” Historical studies have established the importance of TEs in science but have proposed disparate hypotheses concerning the source of knowledge in TEs, ranging from empiricist to rationalist accounts. This article analyzes TEs in think‐aloud protocols of scientifically trained experts to examine more fine‐grained information about their use. Some TEs appear powerful enough to discredit an (...) existing theory—a disconfirmatory purpose. In addition, confirmatory and generative purposes were identified for other TEs. One can also use details in transcript data, including imagery reports and gestures, to provide evidence for a central role played by imagistic simulations in many TEs, and to suggest that these simulations can generate new knowledge using several sources, including the “extended application” of perceptual motor schemas, implicit prior knowledge, and spatial reasoning operations, in contrast to formal arguments. These sources suggest what it means for TEs to be grounded in embodied processes that can begin to explain the paradox above. This leads to a rationalistic view of TEs as using productive internal reasoning, but the view also acknowledges the historical role that experience with the world can play in forming certain schemas used in TEs. Understanding such processes could help provide a foundation for developing a larger model of scientific investigation processes grounded on imagistic simulation (Clement, 2008). (shrink)

One of the most influential studies in all expertise research is de Groot’s (1946) study of chess players, which suggested that pattern recognition, rather than search, was the key determinant of expertise. Many changes have occurred in the chess world since de Groot’s study, leading some authors to argue that the cognitive mechanisms underlying expertise have also changed. We decided to replicate de Groot’s study to empirically test these claims and to examine whether the trends in the data have changed (...) over time. Six Grandmasters, five International Masters, six Experts, and five Class A players completed the think-aloud procedure for two chess positions. Findings indicate that Grandmasters and International Masters search more quickly than Experts and Class A players, and that both groups today search substantially faster than players in previous studies. The findings, however, support de Groot’s overall conclusions and are consistent with predictions made by pattern recognition models. (shrink)

This paper presents a cognitive model of the planning process. The model generalizes the theoretical architecture of the Hearsay‐ll system. Thus, it assumes that planning comprises the activities of a variety of cognitive “specialists.” Each specialist can suggest certain kinds of decisions for incorporation into the plan in progress. These include decisions about: (a) how to approach the planning problem; (b) what knowledge bears on the problem; (c) what kinds of actions to try to plan; (d) what specific actions to (...) plan; and (e) how to allocate cognitive resources during planning. Within each of these categories, different specialists suggest decisions at different levels of abstraction. The activities of the various specialists are not coordinated in any systematic way. Instead, the specialists operate opportunistically, suggesting decisions whenever promising opportunities arise. The paper presents a detailed account of the model and illustrates its assumptions with a “thinking aloud” protocol. It also describes the performance of a computer simulation of the model. The paper contrasts the proposed model with successive refinement models and attempts to resolve apparent differences between the two points of view. (shrink)