Deliberate Practice

Defining Training and Performance Caliber: A Participant Classification Framework

Alannah K.A. McKay, Trent Stellingwerff, Ella S. Smith +5 more · International Journal of Sports Physiology and Performance · 2021 · 2,249 citations

Throughout the sport-science and sports-medicine literature, the term "elite" subjects might be one of the most overused and ill-defined terms. Currently, there is no common perspective or terminology to characterize the caliber and training status of an individual or cohort. This paper presents a 6-tiered Participant Classification Framework whereby all individuals across a spectrum of exercise backgrounds and athletic abilities can be classified. The Participant Classification Framework uses training volume and performance metrics to classify a participant to one of the following: Tier 0: Sedentary; Tier 1: Recreationally Active; Tier 2: Trained/Developmental; Tier 3: Highly Trained/National Level; Tier 4: Elite/International Level; or Tier 5: World Class. We suggest the Participant Classification Framework can be used to classify participants both prospectively (as part of study participant recruitment) and retrospectively (during systematic reviews and/or meta-analyses). Discussion around how the Participant Classification Framework can be tailored toward different sports, athletes, and/or events has occurred, and sport-specific examples provided. Additional nuances such as depth of sport participation, nationality differences, and gender parity within a sport are all discussed. Finally, chronological age with reference to the junior and masters athlete, as well as the Paralympic athlete, and their inclusion within the Participant Classification Framework has also been considered. It is our intention that this framework be widely implemented to systematically classify participants in research featuring exercise, sport, performance, health, and/or fitness outcomes going forward, providing the much-needed uniformity to classification practices.

A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour

Melissa Bond, Hassan Khosravi, Maarten de Laat +6 more · International Journal of Educational Technology in Higher Education · 2024 · 503 citations

This meta-review of 66 systematic reviews on AI in higher education found that most research focuses on adaptive systems and personalisation, but suffers from weak methodology, limited ethical consideration, and a lack of interdisciplinary collaboration — meaning the evidence base for using AI tools in your own learning is currently thin and unreliable.

The Psychology of Entrepreneurship

Michael Fresé, Michael M. Gielnik · Annual Review of Organizational Psychology and Organizational Behavior · 2014 · 901 citations

This meta-analytic review found that personality traits like general self-efficacy and need for achievement, along with entrepreneurial orientation, are strongly associated with both starting a business and achieving business success—but the effect sizes vary dramatically depending on whether you measure business creation (starting) versus business growth (success), meaning the psychological profile for launching a venture differs from the profile for scaling one.

Hepatic encephalopathy in chronic liver disease: 2014 Practice Guideline by the American Association for the Study Of Liver Diseases and the European Association for the Study of the Liver

Hendrik Vilstrup, Piero Amodio, Jasmohan S. Bajaj +5 more · Hepatology · 2014 · 2,025 citations

This joint clinical guideline synthesises the best available evidence on diagnosing, classifying, and treating hepatic encephalopathy (HE) in people with chronic liver disease — a condition that impairs brain function and dramatically reduces quality of life, but which has historically been managed inconsistently due to poor standardisation across studies and clinical settings.

Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data)

Susan Michie, Caroline E Wood, Marie Johnston +3 more · Health Technology Assessment · 2015 · 687 citations

This 3-year project produced a standardised "menu" of 93 named behaviour change techniques (BCTs) — things like goal-setting, self-monitoring, and social support — that anyone designing or analysing a behaviour change intervention can use as a shared vocabulary; knowing which specific techniques an intervention contains is the first step toward figuring out what actually works. ---

Two-Year Randomized Controlled Trial and Follow-up of Dialectical Behavior Therapy vs Therapy by Experts for Suicidal Behaviors and Borderline Personality Disorder

Marsha M. Linehan, Katherine Anne Comtois, Angela M. Murray +7 more · Archives of General Psychiatry · 2006 · 1,993 citations

Dialectical behavior therapy (DBT) cut the risk of suicide attempts in half compared to expert non-behavioral therapy over two years, and kept more people in treatment, suggesting that DBT's structured skills-training approach produces measurably better outcomes than general expert psychotherapy for suicidal individuals with borderline personality disorder.

Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students

Ali M. Fazlollahi, Mohamad Bakhaidar, Ahmad Alsayegh +9 more · JAMA Network Open · 2022 · 236 citations

AI-based audiovisual feedback (the Virtual Operative Assistant) improved surgical skill acquisition by 0.66 points on a -1.00 to 1.00 scale compared to remote expert instruction and by 0.65 points compared to no feedback, with equivalent emotional and cognitive load — suggesting that automated, metric-based feedback can outperform human instruction for procedural learning in simulation.

Large language models encode clinical knowledge

Karan Singhal, Shekoofeh Azizi, Tao Tu +29 more · Nature · 2023 · 2,987 citations

Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality

John Torous, Sandra Bucci, Imogen Bell +7 more · World Psychiatry · 2021 · 1,072 citations

As the COVID-19 pandemic has largely increased the utilization of telehealth, mobile mental health technologies - such as smartphone apps, vir-tual reality, chatbots, and social media - have also gained attention. These digital health technologies offer the potential of accessible and scalable interventions that can augment traditional care. In this paper, we provide a comprehensive update on the overall field of digital psychiatry, covering three areas. First, we outline the relevance of recent technological advances to mental health research and care, by detailing how smartphones, social media, artificial intelligence and virtual reality present new opportunities for "digital phenotyping" and remote intervention. Second, we review the current evidence for the use of these new technological approaches across different mental health contexts, covering their emerging efficacy in self-management of psychological well-being and early intervention, along with more nascent research supporting their use in clinical management of long-term psychiatric conditions - including major depression; anxiety, bipolar and psychotic disorders; and eating and substance use disorders - as well as in child and adolescent mental health care. Third, we discuss the most pressing challenges and opportunities towards real-world implementation, using the Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework to explain how the innovations themselves, the recipients of these innovations, and the context surrounding innovations all must be considered to facilitate their adoption and use in mental health care systems. We conclude that the new technological capabilities of smartphones, artificial intelligence, social media and virtual reality are already changing mental health care in unforeseen and exciting ways, each accompanied by an early but promising evidence base. We point out that further efforts towards strengthening implementation are needed, and detail the key issues at the patient, provider and policy levels which must now be addressed for digital health technologies to truly improve mental health research and treatment in the future.

CONSORT 2010 statement: extension to randomised pilot and feasibility trials

Sandra Eldridge, Claire Chan, Michael J. Campbell +4 more · BMJ · 2016 · 3,389 citations

This paper provides a standardised checklist for reporting pilot and feasibility trials, which helps researchers (including self-experimenters) distinguish between studies that test whether an intervention *can* work from those that test whether it *does* work—preventing premature conclusions from underpowered data.

Setting a Research Agenda for Simulation-Based Healthcare Education

S. Barry Issenberg, Charlotte Ringsted, Doris Østergaard +1 more · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · 2011 · 131 citations

Although the use of simulation as a methodology for learning continues to grow at a rapid pace throughout all of the healthcare professions and disciplines, research in this field is still at an early stage. Increasingly, decision makers and stakeholders must see evidence that the use of such a methodology leads to desired and demonstrable learning outcomes. These include the assurance that simulation may serve as a complement and in some cases a substitute for clinical experience in improving the quality and safety of patient care. Research will be a key factor in advancing the field of simulation to the benefit of patients and healthcare professionals. The simulation community needs an improved understanding of conceptual issues and evidence for their effectiveness to guide simulation use in optimizing the interplay of healthcare professionals, technology, organizational systems, and patients. This recognition extends beyond those educators in the simulation community. In a recent publication summarizing the findings of a task force that identified priorities for medical education research based on their perceived national importance, feasibility, fundability, and amenability for multi-institutional research,1 the no. 1 research issue to emerge was to study the impact of medical school simulation learning on residents' performance. Within the simulation community, there have already been several initiatives involving systematic literature reviews, task forces, committees, and summits whose goal was to identify a research agenda for the use of simulation for learning.2–8 Leadership from the Society in Europe for Simulation Applied to Medicine and Society for Simulation in Healthcare (SSH) sought to further these efforts with a special emphasis to include broad international, multidisciplinary, and interprofessional representation. The Utstein Style Meeting process that has proven successful for catalyzing international, multidisciplinary, and interprofessional research in emergency medicine was adopted for a simulation expert meeting.9,10 The organizers of this meeting recognized that the field of simulation is broad and considered a range of research categories including research about simulation (eg, learning effectiveness and methods, engineering of anatomy and physiology, theoretical frameworks on simulation, and sociological investigation) and research using simulation (eg, human factors oriented investigation, incident analysis, and usability studies).11 While the organizers chose to focus on research related to simulation-based healthcare education, they recognized that in some instances research using simulation to study other factors (eg, using simulation to study the effects of fatigue on human performance) will be used to inform the educational focus. Within the educational domain, the overall goals were to (1) identify the state of the art of educational simulation-based research; (2) identify future directions for educational simulation-based research with headline topics and research questions; and (3) identify methodological issues when conducting educational simulation-based research and provide guidelines on reporting and publishing this research. This effort complements the SSH simulation research summit that took place in January 2011 in conjunction with the International Meeting on Simulation in Healthcare. This report has two sections: a summary of the process to develop a research agenda and a research agenda with proposed research questions. PROCESS TO DEVELOP SIMULATION RESEARCH AGENDA—UTSTEIN STYLE MEETING Selection of Participants For the selection of participants in the meeting, the aim was to build on existing collaborations and experience of simulation research experts with international, multidisciplinary, and multiprofessional representation. The organizers chose individuals with a strong research record, experience in participating in collaborative projects, and who represented key target stakeholder groups, reflecting a diversity of professional (eg, nursing, medicine, and psychology) and geographical backgrounds (eg, Europe, North America). Because the “traditional” size of Utstein Style Meetings is approximately 20 participants, and as this has proven to be a good balance between representativeness and effectiveness, a total of 20 participants (in addition to the four organizers) were invited to take part (Table 1).Table 1: Summary of Participants in the Utstein MeetingPreparation of the Utstein Style Meeting The meeting organizers, consisting of the authors of this article, planned the meeting and its structure. A deliberate decision was made to work with the knowledge and expertise represented in the group by collecting it in an inductive process so that the topics and their priorities would be developed during the meeting. However, it was recognized that collection and synthesis of prior knowledge were important elements to inform the meeting. Consequently, participants were requested to send in recommendations for five references seen as important for the theme of the Utstein Style Meeting (Appendix). Process of the Utstein Style Meeting The meeting took place in June 2010 in Copenhagen, Denmark, and began with an informal evening gathering to build an open, constructive, and trustful working relationship. Day 1—Introduction and Plenum Presentations On day 1, the meeting began with an outline about the idea of the Utstein Style Meetings in general and of this specific one. Two plenum presentations sketched the state-of-the-art and the open questions in simulation-based educational research (noting that much of the research is not grounded in a theory) and presented a model outlining the concepts and challenges in theory-based educational simulation research. The model includes a core—the conceptual, theoretical framework—that is basic to any research approach and a rough clustering of research approaches in four main categories: exploratory studies (qualitative studies, psychometric studies, and descriptive studies), experimental studies (randomized controlled trials and quasi-experimental studies), observational studies (cohort, case-control, and associational studies), and translational studies (reviews and effect studies).12 This framework was used to guide the discussions during the meeting. Day 1—Discussion Rounds The groups were arranged to maximize the variety of expertise and, during each round, were facilitated by one of the authors. Discussion points were recorded on flip charts or using the electronic format and projection. In the first round, discussions followed a modified nominal group process.13 Each participant reflected on general topic areas that “we need to know more about” in regard to education, research, and simulation. In an iterative process, each participant suggested a new topic until no new aspects emerged. The contributions of each group were presented and collected in plenum, allowing for clarifying questions by the other groups. During this session, the organizers independently began grouping each of the topics into different themes as they were discussed. During the subsequent break period, the organizers synthesized their notes and agreed that three major themes emerged: (1) instructional design, (2) outcome measures, and (3) translational research. (Based on the charge to the Utstein Meeting and the agreed relevance of the topic, a fourth group was also formed to specifically address guidelines for reporting research on simulation and education. This report does not include the results of the fourth group as that will appear elsewhere.) In the next round, the groups discussed research problems within the topic in an open group discussion and reported the results back to the plenum. Day 2—Discussion Rounds At the beginning of day 2, the discussion round aimed to present a general outline of research questions for each topic. After that session, members of each group rotated to “critique,” “enlarge,” and “challenge” the findings of another group while two persons of each group stayed. In the following session, the original groups reconvened and refined the research questions along the challenges and newly developed ideas. Those refined research questions were subsequently presented in plenum and each briefly discussed. Day 2—Final Plenum In a final step, the plenum reflected on the implications of the discussions for formulating a general research agenda for educational simulation research and opportunities for greater international collaboration. The authors have synthesized and built upon the literature and the experts' contributions from the Utstein Style Meeting to present a research agenda relating to the three overall themes—Instructional Design, Outcome Measures, and Translational Research (Table 2).Table 2: Research Questions Grouped According to Three Main Themes of Utstein Style Meeting on SimulationTable 2: (Continued)RESEARCH AGENDA WITH PROPOSED RESEARCH QUESTIONS Instructional Design Deliberation by participants resulted in several topic questions they felt would provide focus and priority in the simulation research community. To facilitate grouping of the questions and to illustrate their interdependencies, an hourglass was chosen to represent this conceptually. Research related to learning theories and/or conceptual frameworks was placed at the top. At the next level down (neck of the hourglass), those research questions were placed that relate to resource requirements and systems challenges that often impact the theoretical application of simulation. At the bottom of the hourglass were research questions related to simulation program implementation that take into account the theoretical framework with the local resource challenges within a complex healthcare system. Learning Acquisition, Retention of Skills, and Cognitive Load It is recognized that there is a range of simulation modalities available to choose which may address similar learning outcomes. Studies grounded in context-based learning can provide guidance on the level of authenticity required for a particular competency.14 While there is ample evidence from the literature that the optimal use of a single modality such as mannequin-based simulation may lead to long-term retention of resuscitation skills and central venous catheter insertion skills,15,16 it has yet to be demonstrated whether these and other outcomes can be achieved with different, often less costly and more flexible, simulation modalities such as virtual patients or hybrid task trainers with standardized patients. The intended purpose of researching this topic is not to show superiority of one modality over another. The intent is to provide evidence for a range of options with the expectation that individual simulation programs will possess (or choose) at least a single modality that can be used to achieve a desired outcome. Additional research may identify features unique to a particular modality and its use that impact acquisition and retention of skills (eg, availability of and the two there has been work in and in the of learning that can inform the of simulation systems and in which they and have features related to the learning and of skills as with complex with (eg, benefit from that the and the For a of several be with a range of that would be to be in a of or This is in to a of with (eg, a to for a challenges be to so the is not Research Acquisition, Retention of Skills, and Cognitive Load theories of learning and inform the of simulation (eg, and deliberate theories of inform the and of simulation and based on the of required for to and different simulation modalities and their use and is recognition on the of in simulation and as a there have been a range of reported in the during simulation and discussed in or discussion Research focus on the optimal approach to achieve desired and learning outcomes. In research efforts be guidance to which and Research the of that lead to is the optimal use of of during the (eg, of the and selection of to be the in outcomes between and or to a particular clinical and level (eg, simulation-based research studies include some of the These include their level of knowledge and and However, is about learning and often of the existing studies include or who there will be to no to their overall or based on their performance. In addition to research that includes participants that represent the target group for which acquisition of the task is important for their clinical observational studies that and in a particular to identify elements of these can provide guidance on that for all Research does the acquisition and retention of skills in simulation-based (eg, that from and these specific or to all and on Learning It is recognized that simulation-based research be grounded in a theoretical or conceptual This process of is important to individual studies in a learning theories have been to guide working on simulation in a framework on which to build a complex of and which important to simulation-based These include of of deliberate for the of and and model on skills Simulation has an to these theories in new and However, simulation can also provide a controlled to develop and new theories or about Research on Learning can simulation be used to learning theories in the of theoretical concepts and findings about learning in such as and in healthcare education in a specific task (eg, be achieved with (eg, from on and of A of evidence in the simulation literature when simulation is used the can build and which in the clinical and can have a impact on patient and/or However, it is to identify in the literature the of the and contributions that were important and often to the of the learning In on simulation, the or However, the contributions of the and often and/or of the Utstein Style Meeting and the of an who has some in learning and of the in the experience for the individual or and the to their clinical This is a of simulation-based education, one that is to the of simulation that in simulation systems, and not in the human and provide opportunities for these individuals to develop skills and for their will have a the of their simulation learning program and will see an optimal on their Research of the and that learning and the effects of (eg, on that simulation has adopted within the of education throughout the professions (eg, nursing, medicine, (eg, medicine, and of to identify of to and these Research is in systems to identify elements for healthcare learning education can the work in systems and the conceptual frameworks that have been to guide These findings will guide on those that have the impact on Research (eg, optimal simulation can address issues to that their educational outcomes achieved and systems (eg, in by a of simulation experts lead to maximize the of simulation with requirements of and Simulation simulation programs with issues as an of disciplines, and professions and simulation into their overall a must identify to the at the simulation on those that take of important issue is related to those and/or in which the opportunities for learning in clinical (eg, to more healthcare the field a in clinical for clinical to medical and that for individuals to and that to of in the and quality of learning in the clinical that can be with the simulation learning in the have already for to of clinical learning in a programs have already or will these research be to study different in different to provide guidance to stakeholders at all Research of is to simulation-based learning be with simulation of the (eg, learning in simulation (eg, individual multidisciplinary, Summary on the Research for Instructional Design Research related to instructional needs to beyond of a local its simulation systems and to its studies that simulation to or no is often the in in which the goal is to its use or it can to the field of human learning and to discussions of it or it can a of with challenges the research community to from research that is intended to the effectiveness of educational and research that to the in those This research agenda on outcomes on issues that in prior research and need further The important issue of psychometric to any has been with and will not be in in this However, aspects of and to be any future studies on The for with the topic of outcomes in this to four of the effect of (1) (2) (3) and in the these four aspects as for the of formulating a research with the four aspects one at a in the with a or a learning is and to any It can serve a purpose to stakeholder groups to simulation does not for in the is about on a long-term recent studies that simulation-based learning a of other beyond these the and of is in the of simulation-based is about simulation and the clinical to simulation-based Research the and intended and to simulation-based learning beyond and To which elements in the simulation and at these relate to or impact the three of learning (eg, and in simulation and on future and in healthcare of and can be identified or be developed to broad aspects of to simulation-based results from these inform the design, and of simulation-based learning and of studies report on learning from simulation-based However, controlled experimental studies on of including groups that not the intended target groups (eg, studies on studies on target groups one for future research. to this the questions of prior and professional impact their Three issues important to The of points (eg, when and in learning outcome is The effect of on The of learning outcome. there is an from theories on skills has to be by retention and/or Studies from skills learning have demonstrated that learning outcome can be and in some cases learning outcome the of learning outcome as retention or has implications for studies on the effect of simulation-based and be to points is the effect of on recent studies have demonstrated that which has been in knowledge also to clinical skills Although this topic needs more research into the the of skills it to with and the topic of outcome can be used for learning needs further To that know about and the studies a complex between and The of outcome is related to questions about the basic of simulation-based learning to to achieve a of performance. However, as any education and be seen as a for future in part the skills to be in clinical and need some to the This the of and not the Outcome studies related to simulation have until on of and is about simulation to for future to this is the of deliberate which has some in research on simulation-based However, prior focus has been on the of specific the to to and in and on studies to the focus on process goals outcome goals lead to studies whether learning approaches an over more studies into this topic Research of learning needs is required simulation-based and need to be developed to learning needs is the effect of simulation-based learning on learning outcomes when on a broad in educational (eg, (eg, and (eg, simulation clinical factors skills learning and in can process goals related to simulation-based learning be and in simulation and clinical is the effect of simulation-based learning on for future and can this be and a studies have the effect of simulation-based learning on simulation to a of to guidelines for or However, several studies that with the in is not In this is by retention of the other factors such as of and organizational to application of from the learning into in the healthcare more research is on and factors in the individual and and on the simulation learning can Research the and beyond learning to application of simulation-based learning outcomes in healthcare outcome to the application of to healthcare including the to to different the of at the and organizational level and the between those in their on and about the of simulation-based However, there is a of studies on learning needs and the of simulation-based it is to use simulation to learning needs at the and organizational a studies the effect of simulation-based learning on quality of patient and and this is an that more research. In this there is a need for and of quality issues that can be used to study the effect on healthcare organizational impact on it is that a of related to healthcare (eg, quality that use to often beyond the of simulation will have to be the issue of simulation learning and learning in clinical is another for future Research the needs for simulation-based learning within the and can these be and is the impact of simulation-based learning on healthcare clinical work quality of and patient of and work quality of and patient outcome can be identified or be developed to and educational with healthcare organizational of educational and can be identified or be developed to and educational and organizational Summary of the Research for In there is need for much more research related to all four This includes for a part or new outcome of the outcome a broad of research approaches is often in a broad for conceptual and theoretical frameworks is required for these studies upon several within and the and However, it is that existing theories and new and theories relating to the simulation need to be Translational Research Translational research has been as the process that leads from medicine to for The of has made translational research a priority for The purpose of translational research is to and/or developed in the to the The from the in the to clinical trials with human to the implementation of the new in It is to as the to (Table Two translational 1 and The and in the two place in the experimental and the studies controlled trials on a study studies findings from the groups in to a study and the and the research different research skills and methods, such as implementation and studies and study be to provide to of and a more approach or a be to studies will need with expertise in different such as and a has been to to be and in into Each can new research questions that must be a research that different and a with the research research that may research questions to problems is also of the in Translational Research in Learning as an this article, the of translational research is to healthcare education, and more specifically to simulation-based In this simulation-based learning the and the to facilitate learning and not the basic clinical trials followed by community and based on the synthesis of evidence lead to The lead to and However, one of the challenges is the of and evidence of a it to between different educational simulation-based learning a new more as an the it for to the use of simulation-based in simulation research, the of translational can also be perceived as (1) the effect in the (2) the effect on clinical and (3) the effect on patient the simulation education community also be to from to to inform and further research questions back to the simulation Simulation also have a in needs as it can be used as an and in the Research Research can simulation be used to the interplay between healthcare technology, and can simulation to learning needs and inform the design, and of initiatives to organizational of learning and from simulation-based learning in standardized research into learning in or local can be identified who will benefit from a specific educational simulation be used to identify and learning needs of specific target of learning and from simulation-based learning into learning in clinical and in the of educational can of learning and from simulation-based learning be in for individuals and for in clinical areas (eg, in which simulation-based learning will be a over clinical learning learning and research is still a new it can an important in the of the interplay between technology, and the healthcare Simulation is a complex that in a complex and often a needs to identify learning and at the and to problems in that this of research be based on different and is by a multiprofessional of The Utstein Style Meeting in was with the aim of the of educational simulation research and further the research agenda in simulation-based goal was to identify the future directions for educational simulation research, with central themes and research instructional design, outcome and translational research. The research questions not and they intended to be and will over The participants in this meeting and the authors of this also to the of the topic groups presented at the SSH Research that the 2011 International Meeting on Simulation in Healthcare. These and that will complement each other and aim to provide further guidance to the simulation and healthcare community at improved understanding of conceptual issues and evidence of their effectiveness will guide the and of simulation. these efforts the that research and its publication to advancing the field of simulation for the benefit of healthcare clinical professionals, and patients.

Cognitive Architecture and Instructional Design: 20 Years Later

John Sweller, Jeroen J. G. van Merriënboer, Fred Paas · Educational Psychology Review · 2019 · 1,841 citations

Cognitive load theory was introduced in the 1980s as an instructional design theory based on several uncontroversial aspects of human cognitive architecture. Our knowledge of many of the characteristics of working memory, long-term memory and the relations between them had been well-established for many decades prior to the introduction of the theory. Curiously, this knowledge had had a limited impact on the field of instructional design with most instructional design recommendations proceeding as though working memory and long-term memory did not exist. In contrast, cognitive load theory emphasised that all novel information first is processed by a capacity and duration limited working memory and then stored in an unlimited long-term memory for later use. Once information is stored in long-term memory, the capacity and duration limits of working memory disappear transforming our ability to function. By the late 1990s, sufficient data had been collected using the theory to warrant an extended analysis resulting in the publication of Sweller et al. (Educational Psychology Review, 10, 251–296, 1998). Extensive further theoretical and empirical work have been carried out since that time and this paper is an attempt to summarise the last 20 years of cognitive load theory and to sketch directions for future research.

Text Data Augmentation for Deep Learning

Connor Shorten, Taghi M. Khoshgoftaar, Borko Furht · Journal Of Big Data · 2021 · 1,665 citations

Natural Language Processing (NLP) is one of the most captivating applications of Deep Learning. In this survey, we consider how the Data Augmentation training strategy can aid in its development. We begin with the major motifs of Data Augmentation summarized into strengthening local decision boundaries, brute force training, causality and counterfactual examples, and the distinction between meaning and form. We follow these motifs with a concrete list of augmentation frameworks that have been developed for text data. Deep Learning generally struggles with the measurement of generalization and characterization of overfitting. We highlight studies that cover how augmentations can construct test sets for generalization. NLP is at an early stage in applying Data Augmentation compared to Computer Vision. We highlight the key differences and promising ideas that have yet to be tested in NLP. For the sake of practical implementation, we describe tools that facilitate Data Augmentation such as the use of consistency regularization, controllers, and offline and online augmentation pipelines, to preview a few. Finally, we discuss interesting topics around Data Augmentation in NLP such as task-specific augmentations, the use of prior knowledge in self-supervised learning versus Data Augmentation, intersections with transfer and multi-task learning, and ideas for AI-GAs (AI-Generating Algorithms). We hope this paper inspires further research interest in Text Data Augmentation.

epic3: National Evidence-Based Guidelines for Preventing Healthcare-Associated Infections in NHS Hospitals in England

Heather Loveday, Jennie Wilson, Robert Pratt +6 more · Journal of Hospital Infection · 2013 · 1,603 citations

Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors

Auke Jan Ijspeert, Jun Nakanishi, H. Hoffmann +2 more · Neural Computation · 2012 · 1,568 citations

Nonlinear dynamical systems have been used in many disciplines to model complex behaviors, including biological motor control, robotics, perception, economics, traffic prediction, and neuroscience. While often the unexpected emergent behavior of nonlinear systems is the focus of investigations, it is of equal importance to create goal-directed behavior (e.g., stable locomotion from a system of coupled oscillators under perceptual guidance). Modeling goal-directed behavior with nonlinear systems is, however, rather difficult due to the parameter sensitivity of these systems, their complex phase transitions in response to subtle parameter changes, and the difficulty of analyzing and predicting their long-term behavior; intuition and time-consuming parameter tuning play a major role. This letter presents and reviews dynamical movement primitives, a line of research for modeling attractor behaviors of autonomous nonlinear dynamical systems with the help of statistical learning techniques. The essence of our approach is to start with a simple dynamical system, such as a set of linear differential equations, and transform those into a weakly nonlinear system with prescribed attractor dynamics by means of a learnable autonomous forcing term. Both point attractors and limit cycle attractors of almost arbitrary complexity can be generated. We explain the design principle of our approach and evaluate its properties in several example applications in motor control and robotics.

Human-in-the-loop machine learning: a state of the art

Eduardo Mosqueira-Rey, Elena Hernández-Pereira, David Alonso-Ríos +2 more · Artificial Intelligence Review · 2022 · 794 citations

Abstract Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in which the system remains in control; interactive machine learning, in which there is a closer interaction between users and learning systems; and machine teaching, where human domain experts have control over the learning process. Aside from control, humans can also be involved in the learning process in other ways. In curriculum learning human domain experts try to impose some structure on the examples presented to improve the learning; in explainable AI the focus is on the ability of the model to explain to humans why a given solution was chosen. This collaboration between AI models and humans should not be limited only to the learning process; if we go further, we can see other terms that arise such as Usable and Useful AI. In this paper we review the state of the art of the techniques involved in the new forms of relationship between humans and ML algorithms. Our contribution is not merely listing the different approaches, but to provide definitions clarifying confusing, varied and sometimes contradictory terms; to elucidate and determine the boundaries between the different methods; and to correlate all the techniques searching for the connections and influences between them.

Naturalistic Developmental Behavioral Interventions: Empirically Validated Treatments for Autism Spectrum Disorder

Laura Schreibman, Géraldine Dawson, Aubyn C. Stahmer +10 more · Journal of Autism and Developmental Disorders · 2015 · 1,349 citations

Earlier autism diagnosis, the importance of early intervention, and development of specific interventions for young children have contributed to the emergence of similar, empirically supported, autism interventions that represent the merging of applied behavioral and developmental sciences. "Naturalistic Developmental Behavioral Interventions (NDBI)" are implemented in natural settings, involve shared control between child and therapist, utilize natural contingencies, and use a variety of behavioral strategies to teach developmentally appropriate and prerequisite skills. We describe the development of NDBIs, their theoretical bases, empirical support, requisite characteristics, common features, and suggest future research needs. We wish to bring parsimony to a field that includes interventions with different names but common features thus improving understanding and choice-making among families, service providers and referring agencies.

OpenSim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement

Ajay Seth, Jennifer L. Hicks, Thomas K. Uchida +14 more · PLoS Computational Biology · 2018 · 1,317 citations

Movement is fundamental to human and animal life, emerging through interaction of complex neural, muscular, and skeletal systems. Study of movement draws from and contributes to diverse fields, including biology, neuroscience, mechanics, and robotics. OpenSim unites methods from these fields to create fast and accurate simulations of movement, enabling two fundamental tasks. First, the software can calculate variables that are difficult to measure experimentally, such as the forces generated by muscles and the stretch and recoil of tendons during movement. Second, OpenSim can predict novel movements from models of motor control, such as kinematic adaptations of human gait during loaded or inclined walking. Changes in musculoskeletal dynamics following surgery or due to humandevice interaction can also be simulated; these simulations have played a vital role in several applications, including the design of implantable mechanical devices to improve human grasping in individuals with paralysis. OpenSim is an extensible and user-friendly software package built on decades of knowledge about computational modeling and simulation of biomechanical systems. OpenSim's design enables computational scientists to create new state-of-the-art software tools and empowers others to use these tools in research and clinical applications. OpenSim supports a large and growing community of biomechanics and rehabilitation researchers, facilitating exchange of models and simulations for reproducing and extending discoveries. Examples, tutorials, documentation, and an active user forum support this community. The OpenSim software is covered by the Apache License 2.0, which permits its use for any purpose including both nonprofit and commercial applications. The source code is freely and anonymously accessible on GitHub, where the community is welcomed to make contributions. Platform-specific installers of OpenSim include a GUI and are available on simtk.org.

Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates

Anders Eklund, Thomas E. Nichols, Hans Knutsson · Proceedings of the National Academy of Sciences · 2016 · 3,632 citations

The most widely used task functional magnetic resonance imaging (fMRI) analyses use parametric statistical methods that depend on a variety of assumptions. In this work, we use real resting-state data and a total of 3 million random task group analyses to compute empirical familywise error rates for the fMRI software packages SPM, FSL, and AFNI, as well as a nonparametric permutation method. For a nominal familywise error rate of 5%, the parametric statistical methods are shown to be conservative for voxelwise inference and invalid for clusterwise inference. Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape. By comparison, the nonparametric permutation test is found to produce nominal results for voxelwise as well as clusterwise inference. These findings speak to the need of validating the statistical methods being used in the field of neuroimaging.

Algorithmic management and app‐work in the gig economy: A research agenda for employment relations and HRM

James Duggan, Ultan Sherman, Ronan Carbery +1 more · Human Resource Management Journal · 2019 · 864 citations

Abstract Current understanding of what constitutes work in the growing gig economy is heavily conflated, ranging from conceptualisations of independent contracting to other forms of contingent labour. This article calls for a move away from problematic aggregations by proposing a classification of gig work into three variants, all based strongly upon key technological features: app‐work, crowdwork, and capital platform work. Focusing specifically on the app‐work variant, this article's more delineated focus on the textured dimensions of this work proposes new lines of enquiry into employment relationships and human resource management. Examining the crucial role of algorithmic management, we critically discuss the impact of this novel mediation tool used by gig organisations for the nature of employment relations within app‐work, work assignment processes, and performance management. In so doing, we propose a series of research questions that can serve as a guide for future research in this increasingly important field.

Effects of Cognitive Training Interventions With Older Adults

Karlene Ball, Daniel B. Berch, Karin F. Helmers +10 more · JAMA · 2002 · 2,018 citations

CONTEXT: Cognitive function in older adults is related to independent living and need for care. However, few studies have addressed whether improving cognitive functions might have short- or long-term effects on activities related to living independently. OBJECTIVE: To evaluate whether 3 cognitive training interventions improve mental abilities and daily functioning in older, independent-living adults. DESIGN: Randomized, controlled, single-blind trial with recruitment conducted from March 1998 to October 1999 and 2-year follow-up through December 2001. SETTING AND PARTICIPANTS: Volunteer sample of 2832 persons aged 65 to 94 years recruited from senior housing, community centers, and hospital/clinics in 6 metropolitan areas in the United States. INTERVENTIONS: Participants were randomly assigned to 1 of 4 groups: 10-session group training for memory (verbal episodic memory; n = 711), or reasoning (ability to solve problems that follow a serial pattern; n = 705), or speed of processing (visual search and identification; n = 712); or a no-contact control group (n = 704). For the 3 treatment groups, 4-session booster training was offered to a 60% random sample 11 months later. MAIN OUTCOME MEASURES: Cognitive function and cognitively demanding everyday functioning. RESULTS: Thirty participants were incorrectly randomized and were excluded from the analysis. Each intervention improved the targeted cognitive ability compared with baseline, durable to 2 years (P<.001 for all). Eighty-seven percent of speed-, 74% of reasoning-, and 26% of memory-trained participants demonstrated reliable cognitive improvement immediately after the intervention period. Booster training enhanced training gains in speed (P<.001) and reasoning (P<.001) interventions (speed booster, 92%; no booster, 68%; reasoning booster, 72%; no booster, 49%), which were maintained at 2-year follow-up (P<.001 for both). No training effects on everyday functioning were detected at 2 years. CONCLUSIONS: Results support the effectiveness and durability of the cognitive training interventions in improving targeted cognitive abilities. Training effects were of a magnitude equivalent to the amount of decline expected in elderly persons without dementia over 7- to 14-year intervals. Because of minimal functional decline across all groups, longer follow-up is likely required to observe training effects on everyday function.

Measuring Risk Literacy: The Berlin Numeracy Test

Edward T. Cokely, Mirta Galešić, Eric Schulz +2 more · Judgment and Decision Making · 2012 · 769 citations

Abstract We introduce the Berlin Numeracy Test, a new psychometrically sound instrument that quickly assesses statistical numeracy and risk literacy. We present 21 studies ( n =5336) showing robust psychometric discriminability across 15 countries (e.g., Germany, Pakistan, Japan, USA) and diverse samples (e.g., medical professionals, general populations, Mechanical Turk web panels). Analyses demonstrate desirable patterns of convergent validity (e.g., numeracy, general cognitive abilities), discriminant validity (e.g., personality, motivation), and criterion validity (e.g., numerical and non-numerical questions about risk). The Berlin Numeracy Test was found to be the strongest predictor of comprehension of everyday risks (e.g., evaluating claims about products and treatments; interpreting forecasts), doubling the predictive power of other numeracy instruments and accounting for unique variance beyond other cognitive tests (e.g., cognitive reflection, working memory, intelligence). The Berlin Numeracy Test typically takes about three minutes to complete and is available in multiple languages and formats, including a computer adaptive test that automatically scores and reports data to researchers ( http://www.riskliteracy.org ). The online forum also provides interactive content for public outreach and education, and offers a recommendation system for test format selection. Discussion centers on construct validity of numeracy for risk literacy, underlying cognitive mechanisms, and applications in adaptive decision support.

The Musicality of Non-Musicians: An Index for Assessing Musical Sophistication in the General Population

Daniel Müllensiefen, Bruno Gingras, Jason Musil +1 more · PLoS ONE · 2014 · 1,173 citations

Musical skills and expertise vary greatly in Western societies. Individuals can differ in their repertoire of musical behaviours as well as in the level of skill they display for any single musical behaviour. The types of musical behaviours we refer to here are broad, ranging from performance on an instrument and listening expertise, to the ability to employ music in functional settings or to communicate about music. In this paper, we first describe the concept of 'musical sophistication' which can be used to describe the multi-faceted nature of musical expertise. Next, we develop a novel measurement instrument, the Goldsmiths Musical Sophistication Index (Gold-MSI) to assess self-reported musical skills and behaviours on multiple dimensions in the general population using a large Internet sample (n = 147,636). Thirdly, we report results from several lab studies, demonstrating that the Gold-MSI possesses good psychometric properties, and that self-reported musical sophistication is associated with performance on two listening tasks. Finally, we identify occupation, occupational status, age, gender, and wealth as the main socio-demographic factors associated with musical sophistication. Results are discussed in terms of theoretical accounts of implicit and statistical music learning and with regard to social conditions of sophisticated musical engagement.

The Cognitive Affective Model of Immersive Learning (CAMIL): a Theoretical Research-Based Model of Learning in Immersive Virtual Reality

Guido Makransky, Gustav Bøg Petersen · Educational Psychology Review · 2021 · 1,072 citations

Abstract There has been a surge in interest and implementation of immersive virtual reality (IVR)-based lessons in education and training recently, which has resulted in many studies on the topic. There are recent reviews which summarize this research, but little work has been done that synthesizes the existing findings into a theoretical framework. The Cognitive Affective Model of Immersive Learning (CAMIL) synthesizes existing immersive educational research to describe the process of learning in IVR. The general theoretical framework of the model suggests that instructional methods which are based on evidence from research with less immersive media generalize to learning in IVR. However, the CAMIL builds on evidence that media interacts with method. That is, certain methods which facilitate the affordances of IVR are specifically relevant in this medium. The CAMIL identifies presence and agency as the general psychological affordances of learning in IVR, and describes how immersion, control factors, and representational fidelity facilitate these affordances. The model describes six affective and cognitive factors that can lead to IVR-based learning outcomes including interest, motivation, self-efficacy, embodiment, cognitive load, and self-regulation. The model also describes how these factors lead to factual, conceptual, and procedural knowledge acquisition and knowledge transfer. Implications for future research and instructional design are proposed.

Gaze Perception Triggers Reflexive Visuospatial Orienting

Jon Driver, Greg Davis, Paola Ricciardelli +3 more · Visual Cognition · 1999 · 1,445 citations

This paper seeks to bring together two previously separate research traditions: research on spatial orienting within the visual cueing paradigm and research into social cognition, addressing our tendency to attend in the direction that another person looks. Cueing methodologies from mainstream attention research were adapted to test the automaticity of orienting in the direction of seen gaze. Three studies manipulated the direction of gaze in a computerized face, which appeared centrally in a frontal view during a peripheral letter-discrimination task. Experiments 1 and 2 found faster discrimination of peripheral target letters on the side the computerized face gazed towards, even though the seen gaze did not predict target side, and despite participants being asked to ignore the face. This suggests reflexive covert and/or overt orienting in the direction of seen gaze, arising even when the observer has no motivation to orient in this way. Experiment 3 found faster letter discrimination on the side the computerized face gazed towards even when participants knew that target letters were four times as likely on the opposite side. This suggests that orienting can arise in the direction of seen gaze even when counter to intentions. The experiments illustrate that methods from mainstream attention research can be usefully applied to social cognition, and that studies of spatial attention may profit from considering its social function.

Subject pool recruitment procedures: organizing experiments with ORSEE

Ben Greiner · Journal of the Economic Science Association · 2015 · 2,322 citations

Abstract This paper discusses aspects of recruiting subjects for economic laboratory experiments, and shows how the Online Recruitment System for Economic Experiments can help. The software package provides experimenters with a free, convenient, and very powerful tool to organize their experiments and sessions.

Explicit and Implicit Contributions to Learning in a Sensorimotor Adaptation Task

Jordan A. Taylor, John W. Krakauer, Richard B. Ivry · Journal of Neuroscience · 2014 · 866 citations

Visuomotor adaptation has been thought to be an implicit process that results when a sensory-prediction error signal is used to update a forward model. A striking feature of human competence is the ability to receive verbal instructions and employ strategies to solve tasks; such explicit processes could be used during visuomotor adaptation. Here, we used a novel task design that allowed us to obtain continuous verbal reports of aiming direction while participants learned a visuomotor rotation. We had two main hypotheses: the contribution of explicit learning would be modulated by instruction and the contribution of implicit learning would be modulated by the form of error feedback. By directly assaying aiming direction, we could identify the time course of the explicit component and, via subtraction, isolate the implicit component of learning. There were marked differences in the time courses of explicit and implicit contributions to learning. Explicit learning, driven by target error, was achieved by initially large then smaller explorations of aiming direction biased toward the correct solution. In contrast, implicit learning, driven by a sensory-prediction error, was slow and monotonic. Continuous error feedback reduced the amplitude of explicit learning and increased the contribution of implicit learning. The presence of instruction slightly increased the rate of initial learning and only had a subtle effect on implicit learning. We conclude that visuomotor adaptation, even in the absence of instruction, results from the interplay between explicit learning driven by target error and implicit learning of a forward model driven by prediction error.

A survey of socially interactive robots

Terrence Fong, Illah Nourbakhsh, Kerstin Dautenhahn · Robotics and Autonomous Systems · 2003 · 3,100 citations

The role of deliberate practice in the acquisition of clinical skills

Robbert Duvivier, Jan van Dalen, Arno M. M. Muijtjens +3 more · BMC Medical Education · 2011 · 222 citations

BACKGROUND: The role of deliberate practice in medical students' development from novice to expert was examined for preclinical skill training. METHODS: Students in years 1-3 completed 34 Likert type items, adapted from a questionnaire about the use of deliberate practice in cognitive learning. Exploratory factor analysis and reliability analysis were used to validate the questionnaire. Analysis of variance examined differences between years and regression analysis the relationship between deliberate practice and skill test results. RESULTS: 875 students participated (90%). Factor analysis yielded four factors: planning, concentration/dedication, repetition/revision, study style/self reflection. Student scores on 'Planning' increased over time, score on sub-scale 'repetition/revision' decreased. Student results on the clinical skill test correlated positively with scores on subscales 'planning' and 'concentration/dedication' in years 1 and 3, and with scores on subscale 'repetition/revision' in year 1. CONCLUSIONS: The positive effects on test results suggest that the role of deliberate practice in medical education merits further study. The cross-sectional design is a limitation, the large representative sample a strength of the study. The vanishing effect of repetition/revision may be attributable to inadequate feedback. Deliberate practice advocates sustained practice to address weaknesses, identified by (self-)assessment and stimulated by feedback. Further studies should use a longitudinal prospective design and extend the scope to expertise development during residency and beyond.

Modelling with stakeholders – Next generation

Alexey Voinov, Nagesh Kolagani, Michael K. McCall +5 more · Environmental Modelling & Software · 2015 · 581 citations

Self-awareness, self-regulation, and self-transcendence (S-ART): a framework for understanding the neurobiological mechanisms of mindfulness

David R. Vago, David Silbersweig · Frontiers in Human Neuroscience · 2012 · 1,300 citations

Mindfulness-as a state, trait, process, type of meditation, and intervention has proven to be beneficial across a diverse group of psychological disorders as well as for general stress reduction. Yet, there remains a lack of clarity in the operationalization of this construct, and underlying mechanisms. Here, we provide an integrative theoretical framework and systems-based neurobiological model that explains the mechanisms by which mindfulness reduces biases related to self-processing and creates a sustainable healthy mind. Mindfulness is described through systematic mental training that develops meta-awareness (self-awareness), an ability to effectively modulate one's behavior (self-regulation), and a positive relationship between self and other that transcends self-focused needs and increases prosocial characteristics (self-transcendence). This framework of self-awareness, -regulation, and -transcendence (S-ART) illustrates a method for becoming aware of the conditions that cause (and remove) distortions or biases. The development of S-ART through meditation is proposed to modulate self-specifying and narrative self-networks through an integrative fronto-parietal control network. Relevant perceptual, cognitive, emotional, and behavioral neuropsychological processes are highlighted as supporting mechanisms for S-ART, including intention and motivation, attention regulation, emotion regulation, extinction and reconsolidation, prosociality, non-attachment, and decentering. The S-ART framework and neurobiological model is based on our growing understanding of the mechanisms for neurocognition, empirical literature, and through dismantling the specific meditation practices thought to cultivate mindfulness. The proposed framework will inform future research in the contemplative sciences and target specific areas for development in the treatment of psychological disorders.

The magical number 4 in short-term memory: A reconsideration of mental storage capacity

Nelson Cowan · Behavioral and Brain Sciences · 2001 · 6,745 citations

Miller (1956) summarized evidence that people can remember about seven chunks in short-term memory (STM) tasks. However, that number was meant more as a rough estimate and a rhetorical device than as a real capacity limit. Others have since suggested that there is a more precise capacity limit, but that it is only three to five chunks. The present target article brings together a wide variety of data on capacity limits suggesting that the smaller capacity limit is real. Capacity limits will be useful in analyses of information processing only if the boundary conditions for observing them can be carefully described. Four basic conditions in which chunks can be identified and capacity limits can accordingly be observed are: (1) when information overload limits chunks to individual stimulus items, (2) when other steps are taken specifically to block the recording of stimulus items into larger chunks, (3) in performance discontinuities caused by the capacity limit, and (4) in various indirect effects of the capacity limit. Under these conditions, rehearsal and long-term memory cannot be used to combine stimulus items into chunks of an unknown size; nor can storage mechanisms that are not capacity-limited, such as sensory memory, allow the capacity-limited storage mechanism to be refilled during recall. A single, central capacity limit averaging about four chunks is implicated along with other, noncapacity-limited sources. The pure STM capacity limit expressed in chunks is distinguished from compound STM limits obtained when the number of separately held chunks is unclear. Reasons why pure capacity estimates fall within a narrow range are discussed and a capacity limit for the focus of attention is proposed.

Automation, Algorithms, and Beyond: Why Work Design Matters More Than Ever in a Digital World

Sharon K. Parker, Gudela Grote · Applied Psychology · 2019 · 740 citations

Abstract We propose a central role for work design in understanding the effects of digital technologies. We give examples of how new technologies can—depending on various factors—positively and negatively affect job resources (autonomy/control, skill use, job feedback, relational aspects) and job demands (e.g., performance monitoring), with consequences for employee well‐being, safety, and performance. We identify four intervention strategies. First, work design choices need to be proactively considered during technology implementation, consistent with the sociotechnical systems principle of joint optimization. Second, human‐centred design principles should be explicitly considered in the design and procurement of new technologies. Third, organizationally oriented intervention strategies need to be supported by macro‐level policies. Fourth, there is a need to go beyond a focus on upskilling employees to help them adapt to technology change, to also focus on training employees, as well as other stakeholders, in work design and related topics. Finally, we identify directions for moving the field forward, including new research questions (e.g., job autonomy in the context of machine learning; understanding designers’ work design mindsets; investigating how job crafting applies to technology); a reorientation of methods (e.g., interdisciplinary, intervention studies); and steps for achieving practical impact.

The Knowledge‐Learning‐Instruction Framework: Bridging the Science‐Practice Chasm to Enhance Robust Student Learning

Kenneth R. Koedinger, Albert T. Corbett, Charles A. Perfetti · Cognitive Science · 2012 · 712 citations

Despite the accumulation of substantial cognitive science research relevant to education, there remains confusion and controversy in the application of research to educational practice. In support of a more systematic approach, we describe the Knowledge-Learning-Instruction (KLI) framework. KLI promotes the emergence of instructional principles of high potential for generality, while explicitly identifying constraints of and opportunities for detailed analysis of the knowledge students may acquire in courses. Drawing on research across domains of science, math, and language learning, we illustrate the analyses of knowledge, learning, and instructional events that the KLI framework affords. We present a set of three coordinated taxonomies of knowledge, learning, and instruction. For example, we identify three broad classes of learning events (LEs): (a) memory and fluency processes, (b) induction and refinement processes, and (c) understanding and sense-making processes, and we show how these can lead to different knowledge changes and constraints on optimal instructional choices.

AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking

Michael Gerlich · Societies · 2025 · 514 citations

The proliferation of artificial intelligence (AI) tools has transformed numerous aspects of daily life, yet its impact on critical thinking remains underexplored. This study investigates the relationship between AI tool usage and critical thinking skills, focusing on cognitive offloading as a mediating factor. Utilising a mixed-method approach, we conducted surveys and in-depth interviews with 666 participants across diverse age groups and educational backgrounds. Quantitative data were analysed using ANOVA and correlation analysis, while qualitative insights were obtained through thematic analysis of interview transcripts. The findings revealed a significant negative correlation between frequent AI tool usage and critical thinking abilities, mediated by increased cognitive offloading. Younger participants exhibited higher dependence on AI tools and lower critical thinking scores compared to older participants. Furthermore, higher educational attainment was associated with better critical thinking skills, regardless of AI usage. These results highlight the potential cognitive costs of AI tool reliance, emphasising the need for educational strategies that promote critical engagement with AI technologies. This study contributes to the growing discourse on AI’s cognitive implications, offering practical recommendations for mitigating its adverse effects on critical thinking. The findings underscore the importance of fostering critical thinking in an AI-driven world, making this research essential reading for educators, policymakers, and technologists.

Team-based learning: A practical guide: AMEE Guide No. 65

Dean X. Parmelee, Larry K. Michaelsen, Sandy Cook +1 more · Medical Teacher · 2012 · 666 citations

Team-based learning™ (TBL) is an instructional strategy developed in the business school environment in the early 1990s by Dr Michaelsen who wanted the benefits of small group learning within large classes. In 2001, a US federal granting agency awarded funds for educators in the health sciences to learn about and implement the strategy in their educational programs; TBL was put forward as one such strategy and as a result it is used in over 60 US and international health science professional schools. TBL is very different from problem-based learning (PBL) and other small group approaches in that there is no need for multiple faculty or rooms, students must come prepared to sessions, and individual and small groups of students (teams) are highly accountable for their contributions to team productivity. The instructor must be a content-expert, but need not have any experience or expertise in group process to conduct a successful TBL session. Students do not need any specific instruction in teamwork since they learn how to be collaborative and productive in the process. TBL can replace or complement a lecture-based course or curriculum.

Training and simulation for patient safety

Raj Aggarwal, Oliver Mytton, Miliard Derbrew +9 more · BMJ Quality & Safety · 2010 · 709 citations

BACKGROUND: Simulation-based medical education enables knowledge, skills and attitudes to be acquired for all healthcare professionals in a safe, educationally orientated and efficient manner. Procedure-based skills, communication, leadership and team working can be learnt, be measured and have the potential to be used as a mode of certification to become an independent practitioner. RESULTS: Simulation-based training initially began with life-like manikins and now encompasses an entire range of systems, from synthetic models through to high fidelity simulation suites. These models can also be used for training in new technologies, for the application of existing technologies to new environments and in prototype testing. The level of simulation must be appropriate to the learners' needs and can range from focused tuition to mass trauma scenarios. The development of simulation centres is a global phenomenon which should be encouraged, although the facilities should be used within appropriate curricula that are methodologically sound and cost-effective. DISCUSSION: A review of current techniques reveals that simulation can successfully promote the competencies of medical expert, communicator and collaborator. Further work is required to develop the exact role of simulation as a training mechanism for scholarly skills, professionalism, management and health advocacy.

Adaptive Environmental Assessment and Management

C. S. Holling · Forest Science · 1980 · 3,243 citations

Functional Neural Plasticity and Associated Changes in Positive Affect After Compassion Training

Olga Klimecki, Susanne Leiberg, Claus Lamm +1 more · Cerebral Cortex · 2012 · 633 citations

The development of social emotions such as compassion is crucial for successful social interactions as well as for the maintenance of mental and physical health, especially when confronted with distressing life events. Yet, the neural mechanisms supporting the training of these emotions are poorly understood. To study affective plasticity in healthy adults, we measured functional neural and subjective responses to witnessing the distress of others in a newly developed task (Socio-affective Video Task). Participants' initial empathic responses to the task were accompanied by negative affect and activations in the anterior insula and anterior medial cingulate cortex--a core neural network underlying empathy for pain. Whereas participants reacted with negative affect before training, compassion training increased positive affective experiences, even in response to witnessing others in distress. On the neural level, we observed that, compared with a memory control group, compassion training elicited activity in a neural network including the medial orbitofrontal cortex, putamen, pallidum, and ventral tegmental area--brain regions previously associated with positive affect and affiliation. Taken together, these findings suggest that the deliberate cultivation of compassion offers a new coping strategy that fosters positive affect even when confronted with the distress of others.

Part 1: Executive summary

Jerry P. Nolan, Mary Fran Hazinski, John E. Billi +27 more · Resuscitation · 2010 · 697 citations

Tools and methods in participatory modeling: Selecting the right tool for the job

Alexey Voinov, Karen E. Jenni, Steven Gray +22 more · Environmental Modelling & Software · 2018 · 458 citations

Various tools and methods are used in participatory modelling, at different stages of the process and for different purposes. The diversity of tools and methods can create challenges for stakeholders and modelers when selecting the ones most appropriate for their projects. We offer a systematic overview, assessment, and categorization of methods to assist modelers and stakeholders with their choices and decisions. Most available literature provides little justification or information on the reasons for the use of particular methods or tools in a given study. In most of the cases, it seems that the prior experience and skills of the modelers had a dominant effect on the selection of the methods used. While we have not found any real evidence of this approach being wrong, we do think that putting more thought into the method selection process and choosing the most appropriate method for the project can produce better results. Based on expert opinion and a survey of modelers engaged in participatory processes, we offer practical guidelines to improve decisions about method selection at different stages of the participatory modeling process.

Adaptive representation of dynamics during learning of a motor task

Reza Shadmehr, FA Mussa-Ivaldi · Journal of Neuroscience · 1994 · 2,666 citations

We investigated how the CNS learns to control movements in different dynamical conditions, and how this learned behavior is represented. In particular, we considered the task of making reaching movements in the presence of externally imposed forces from a mechanical environment. This environment was a force field produced by a robot manipulandum, and the subjects made reaching movements while holding the end-effector of this manipulandum. Since the force field significantly changed the dynamics of the task, subjects' initial movements in the force field were grossly distorted compared to their movements in free space. However, with practice, hand trajectories in the force field converged to a path very similar to that observed in free space. This indicated that for reaching movements, there was a kinematic plan independent of dynamical conditions. The recovery of performance within the changed mechanical environment is motor adaptation. In order to investigate the mechanism underlying this adaptation, we considered the response to the sudden removal of the field after a training phase. The resulting trajectories, named aftereffects, were approximately mirror images of those that were observed when the subjects were initially exposed to the field. This suggested that the motor controller was gradually composing a model of the force field, a model that the nervous system used to predict and compensate for the forces imposed by the environment. In order to explore the structure of the model, we investigated whether adaptation to a force field, as presented in a small region, led to aftereffects in other regions of the workspace. We found that indeed there were aftereffects in workspace regions where no exposure to the field had taken place; that is, there was transfer beyond the boundary of the training data. This observation rules out the hypothesis that the subject's model of the force field was constructed as a narrow association between visited states and experienced forces; that is, adaptation was not via composition of a look-up table. In contrast, subjects modeled the force field by a combination of computational elements whose output was broadly tuned across the motor state space. These elements formed a model that extrapolated to outside the training region in a coordinate system similar to that of the joints and muscles rather than end-point forces. This geometric property suggests that the elements of the adaptive process represent dynamics of a motor task in terms of the intrinsic coordinate system of the sensors and actuators.

The myth of language universals: Language diversity and its importance for cognitive science

Nicholas Evans, Stephen C. Levinson · Behavioral and Brain Sciences · 2009 · 2,619 citations

Talk of linguistic universals has given cognitive scientists the impression that languages are all built to a common pattern. In fact, there are vanishingly few universals of language in the direct sense that all languages exhibit them. Instead, diversity can be found at almost every level of linguistic organization. This fundamentally changes the object of enquiry from a cognitive science perspective. This target article summarizes decades of cross-linguistic work by typologists and descriptive linguists, showing just how few and unprofound the universal characteristics of language are, once we honestly confront the diversity offered to us by the world's 6,000 to 8,000 languages. After surveying the various uses of "universal," we illustrate the ways languages vary radically in sound, meaning, and syntactic organization, and then we examine in more detail the core grammatical machinery of recursion, constituency, and grammatical relations. Although there are significant recurrent patterns in organization, these are better explained as stable engineering solutions satisfying multiple design constraints, reflecting both cultural-historical factors and the constraints of human cognition. Linguistic diversity then becomes the crucial datum for cognitive science: we are the only species with a communication system that is fundamentally variable at all levels. Recognizing the true extent of structural diversity in human language opens up exciting new research directions for cognitive scientists, offering thousands of different natural experiments given by different languages, with new opportunities for dialogue with biological paradigms concerned with change and diversity, and confronting us with the extraordinary plasticity of the highest human skills.

Replication and Analysis of Ebbinghaus’ Forgetting Curve

Jaap M. J. Murre, Joeri Dros · PLoS ONE · 2015 · 752 citations

We present a successful replication of Ebbinghaus' classic forgetting curve from 1880 based on the method of savings. One subject spent 70 hours learning lists and relearning them after 20 min, 1 hour, 9 hours, 1 day, 2 days, or 31 days. The results are similar to Ebbinghaus' original data. We analyze the effects of serial position on forgetting and investigate what mathematical equations present a good fit to the Ebbinghaus forgetting curve and its replications. We conclude that the Ebbinghaus forgetting curve has indeed been replicated and that it is not completely smooth but most probably shows a jump upwards starting at the 24 hour data point.

Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes

Kevin W. Eva, Heather Armson, Eric S. Holmboe +4 more · Advances in Health Sciences Education · 2011 · 403 citations

Self-appraisal has repeatedly been shown to be inadequate as a mechanism for performance improvement. This has placed greater emphasis on understanding the processes through which self-perception and external feedback interact to influence professional development. As feedback is inevitably interpreted through the lens of one's self-perceptions it is important to understand how learners interpret, accept, and use feedback (or not) and the factors that influence those interpretations. 134 participants from 8 health professional training/continuing competence programs were recruited to participate in focus groups. Analyses were designed to (a) elicit understandings of the processes used by learners and physicians to interpret, accept and use (or not) data to inform their perceptions of their clinical performance, and (b) further understand the factors (internal and external) believed to influence interpretation of feedback. Multiple influences appear to impact upon the interpretation and uptake of feedback. These include confidence, experience, and fear of not appearing knowledgeable. Importantly, however, each could have a paradoxical effect of both increasing and decreasing receptivity. Less prevalent but nonetheless important themes suggested mechanisms through which cognitive reasoning processes might impede growth from formative feedback. Many studies have examined the effectiveness of feedback through variable interventions focused on feedback delivery. This study suggests that it is equally important to consider feedback from the perspective of how it is received. The interplay observed between fear, confidence, and reasoning processes reinforces the notion that there is no simple recipe for the delivery of effective feedback. These factors should be taken into account when trying to understand (a) why self-appraisal can be flawed, (b) why appropriate external feedback is vital (yet can be ineffective), and (c) why we may need to disentangle the goals of performance improvement from the goals of improving self-assessment.

Assessment practices of clinical neuropsychologists in the United States and Canada: A survey of INS, NAN, and APA Division 40 members

L Rabin, William Barr, Leslie A. Burton · Archives of Clinical Neuropsychology · 2004 · 812 citations

The present study surveyed assessment practices and test usage patterns among clinical neuropsychologists. Respondents were 747 North American, doctorate-level psychologists (40% usable response rate) affiliated with Division 40 of the American Psychological Association (APA), the National Academy of Neuropsychology (NAN), or the International Neuropsychological Society (INS). Respondents first provided basic demographic and practice-related information and reported their most frequently utilized instruments. Overall, the Wechsler Adult Intelligence Scales and Wechsler Memory Scales were most frequently used, followed by the Trail Making Test, California Verbal Learning Test, and Wechsler Intelligence Scale for Children. Respondents also reviewed a vignette about a traumatic brain injury patient, and then reported the instruments they would use to assess this patient's specific cognitive symptomatology, general cognitive ability, and capacity to return to work. Particular attention was paid to the areas of memory, attention, and executive functioning. The current study represents the largest and most comprehensive test usage survey conducted to date within the field of clinical neuropsychology. Survey results update and greatly expand knowledge about neuropsychologists' assessment practices. Following a review of findings, results are compared to those obtained in prior surveys and implications for the field of neuropsychology are discussed.

Active‐Constructive‐Interactive: A Conceptual Framework for Differentiating Learning Activities

T. H. Michelene · Topics in Cognitive Science · 2009 · 1,498 citations

Active, constructive, and interactive are terms that are commonly used in the cognitive and learning sciences. They describe activities that can be undertaken by learners. However, the literature is actually not explicit about how these terms can be defined; whether they are distinct; and whether they refer to overt manifestations, learning processes, or learning outcomes. Thus, a framework is provided here that offers a way to differentiate active, constructive, and interactive in terms of observable overt activities and underlying learning processes. The framework generates a testable hypothesis for learning: that interactive activities are most likely to be better than constructive activities, which in turn might be better than active activities, which are better than being passive. Studies from the literature are cited to provide evidence in support of this hypothesis. Moreover, postulating underlying learning processes allows us to interpret evidence in the literature more accurately. Specifying distinct overt activities for active, constructive, and interactive also offers suggestions for how learning activities can be coded and how each kind of activity might be elicited.

Theories of developmental dyslexia: insights from a multiple case study of dyslexic adults

Franck Ramus · Brain · 2003 · 1,479 citations

A multiple case study was conducted in order to assess three leading theories of developmental dyslexia: (i) the phonological theory, (ii) the magnocellular (auditory and visual) theory and (iii) the cerebellar theory. Sixteen dyslexic and 16 control university students were administered a full battery of psychometric, phonological, auditory, visual and cerebellar tests. Individual data reveal that all 16 dyslexics suffer from a phonological deficit, 10 from an auditory deficit, four from a motor deficit and two from a visual magnocellular deficit. Results suggest that a phonological deficit can appear in the absence of any other sensory or motor disorder, and is sufficient to cause a literacy impairment, as demonstrated by five of the dyslexics. Auditory disorders, when present, aggravate the phonological deficit, hence the literacy impairment. However, auditory deficits cannot be characterized simply as rapid auditory processing problems, as would be predicted by the magnocellular theory. Nor are they restricted to speech. Contrary to the cerebellar theory, we find little support for the notion that motor impairments, when found, have a cerebellar origin or reflect an automaticity deficit. Overall, the present data support the phonological theory of dyslexia, while acknowledging the presence of additional sensory and motor disorders in certain individuals.

Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach.

Fred Paas, Jeroen J. G. van Merriënboer · Journal of Educational Psychology · 1994 · 1,119 citations

Four computer-based training strategies for geometrical problem solving in the domain of computer numerically controlled machinery programming were studied with regard to their effects on training performance, transfer performance, and cognitive load. A low- and a high-variability conventional condition, in which conventional practice problems had to be solved (followed by worked examples), were compared with a low- and a high-variability worked condition, in which worked examples had to be studied. Results showed that students who studied worked examples gained most from high-variability examples, invested less time and mental effort in practice, and attained better and less effort-demanding transfer performance than students who first attempted to solve conventional problems and then studied work examples. In complex cognitive domains such as mathematics, phys-ics, or computer programming, problem solutions can often be characterized by a hierarchical goal structure. The goal of these solutions can be attained only by successfully attaining all subgoals. Learning and performance of complex cogni-tive tasks are typically constrained by limited processing ca-

Beyond feedback: developing student capability in complex appraisal

D. Royce Sadler · Assessment & Evaluation in Higher Education · 2010 · 1,112 citations

A common practice is to give students detailed feedback about the strengths and weaknesses of their work, with suggestions for improvement. However, the impact of feedback often seems negligible, despite the investment of considerable time and effort put into its construction. With a view to increasing its effectiveness, extensive theoretical and empirical research has been carried out into structure, timing and other parameters. For students to be able to apply feedback, they need to understand the meaning of the feedback statements. They also need to identify, with near certainty, the particular aspects of their work that need attention. For this to occur, students must possess critical background knowledge. This article sets out the nature of this knowledge and how students can acquire it. They must appropriate for themselves three fundamental concepts, namely response genre, quality, and criteria, and in addition develop a cache of relevant tacit knowledge.