|Year : 2016 | Volume
| Issue : 1 | Page : 32-40
Establishing an instrumented training environment for simulation-based training of health care providers: An initial proof of concept
Scott M Pappada1, Thomas John Papadimos2, Jonathan A Lipps2, John J Feeney3, Kevin T Durkee3, Scott M Galster4, Scott R Winfield5, Sheryl A Pfeil6, Sujatha P Bhandary2, Karina Castellon-Larios2, Nicoleta Stoicea2, Susan D Moffatt-Bruce7
1 Department of Anesthesiology, The Ohio State University Wexner Medical Center, Columbus; Aptima, Inc., Dayton, OH, USA
2 Department of Anesthesiology, The Ohio State University Wexner Medical Center, Columbus, USA
3 Aptima, Inc., Dayton, OH, USA
4 United States Air Force Research Laboratory, Dayton, OH, USA
5 Clinical Skills Education and Assessment Center, The Ohio State University College of Medicine, Columbus, OH, USA
6 Clinical Skills Education and Assessment Center, The Ohio State University College of Medicine; Department of Medicine, Clinical Skills Education and Assessment Center, The Ohio State University College of Medicine, The Ohio State University Wexner Medical Center, Columbus, OH, USA
7 Department of Surgery, Division of Thoracic Surgery, The Ohio State University Wexner Medical Center, Columbus, USA
|Date of Submission||10-Nov-2015|
|Date of Acceptance||19-Jan-2016|
|Date of Web Publication||2-Jun-2016|
Thomas John Papadimos
Department of Anesthesiology, The Ohio State University Wexner Medical Center, 410 West 10th Avenue, Columbus, OH 43210
Source of Support: None, Conflict of Interest: None
Objective: Several decades of armed conflict at a time of incredible advances in medicine have led to an acknowledgment of the importance of cognitive workload and environmental stress in both war and the health care sector. Recent advances in portable neurophysiological monitoring technologies allow for the continuous real-time measurement and acquisition of key neurophysiological signals that can be leveraged to provide high-resolution temporal data indicative of rapid changes in functional state, (i.e., cognitive workload, stress, and fatigue). Here, we present recent coordinated proof of concept pilot project between private industry, the health sciences, and the USA government where a paper-based self-reporting of workload National Aeronautics and Space Administration Task Load Index Scale (NASA TLX) was successfully converted to a real-time objective measure through an automated cognitive load assessment for medical staff training and evaluation (ACLAMATE).
Methods: These real-time objective measures were derived exclusively through the processing and modeling of neurophysiological data. This endeavor involved health care education and training with real-time feedback during high fidelity simulations through the use of this artificial modeling and measurement approach supported by Aptima Corporation's FuSE2, SPOTLITE, and PM Engine technologies.
Results: Self-reported NASA TLX workload indicators were converted to measurable outputs through the development of a machine learning-based modeling approach. Workload measurements generated by this modeling approach were represented as a NASA TLX anchored scale of 0–100 and were displayed on a computer screen numerically and visually as individual outputs and as a consolidated team output.
Conclusions: Cognitive workloads for individuals and teams can be modeled through use of feed forward back-propagating neural networks thereby allowing healthcare systems to measure performance, stress, and cognitive workload in order to enhance patient safety, staff education, and overall quality of patient care.
The following core competencies are addressed in this article: Medical Knowledge, Interpersonal Skills, Patient Care, and Professionalism.
Keywords: Cognition, education, health care sector, neural network, psychological, stress, workload
|How to cite this article:|
Pappada SM, Papadimos TJ, Lipps JA, Feeney JJ, Durkee KT, Galster SM, Winfield SR, Pfeil SA, Bhandary SP, Castellon-Larios K, Stoicea N, Moffatt-Bruce SD. Establishing an instrumented training environment for simulation-based training of health care providers: An initial proof of concept. Int J Acad Med 2016;2:32-40
|How to cite this URL:|
Pappada SM, Papadimos TJ, Lipps JA, Feeney JJ, Durkee KT, Galster SM, Winfield SR, Pfeil SA, Bhandary SP, Castellon-Larios K, Stoicea N, Moffatt-Bruce SD. Establishing an instrumented training environment for simulation-based training of health care providers: An initial proof of concept. Int J Acad Med [serial online] 2016 [cited 2019 Jul 16];2:32-40. Available from: http://www.ijam-web.org/text.asp?2016/2/1/32/183324
| Introduction|| |
Factors affecting stress and workload in health care settings, such as the number of patients, patient acuity, and team communication, have been correlated with quality patient outcomes.,,, Cognitive overload is a challenge in today's health care environments., Health care providers (HCPs) frequently treat unstable patients, thereby requiring them to constantly revisit treatment plans. Increased multitasking and advanced technology utilization further exacerbate task demands/workload. Increased use of monitoring technologies and electronic medical records that enhance clinical processes also contribute to cognitive and information overload.
There is increased interest in using unobtrusive monitoring technologies to provide objective measures of functional state (e.g. cognitive workload) during training and day-to-day tasking. Prior research has shown key features derived from electrocardiogram (ECG) and electroencephalogram (EEG) digital signal processing to be correlated with the workload.,,,,,,, Recent advances in portable neurophysiological monitoring technologies allow for continuous real-time acquisition of neurophysiological signals that provide high-resolution temporal data indicative of rapid changes in functional state.,, Unobtrusive assessment technologies/solutions are beneficial because they provide a real-time continuous objective measure of an HCPs functional state (i.e., cognitive workload, stress, and fatigue).
While the use of simulation for training technical and teamwork skills of HCPs has accelerated, the assessment of learner skills, performance, and functional state during simulation is in its infancy. Traditional low-resolution (e.g., high/low or high/medium/low) ratings fail to capture accurate assessments of learner performance and functional state., To address these limitations, Aptima, Inc., (Dayton, OH), has developed comprehensive measurement and assessment technologies that provide real-time high-resolution measures of functional state and performance through advanced processing and modeling of neurophysiological, behavioral, observer-based, and environmental/system data (collected from training and real-world settings).
Here, we present the establishment of a novel instrumented training environment using an artificial neural network (ANN; advanced machine learning models) to provide an objective measure of functional state. The advanced technologies needed for this instrumented training environment were integrated into simulation-based training at OSUWMC's Clinical Skills Education and Assessment Center (CSEAC). Over time, data collected from these newly established environments will help to answer two key questions: (1) What is the optimal team composition/structure for specific clinical events, and (2) will use of instrumented training environments translate to substantial improvements of real-world health care operations and contribute to the enhancement of patient outcomes? This study was a qualitative proof of concept showing a successful demonstration of transitioning the paper-based National Aeronautics and Space Administration Task Load Index (NASA TLX) Scale of evaluation  to an objective ANN-derived evaluation of medical team's cognitive workload via an integrated networked suite of wireless sensors.
This study was approved by the OSUWMC Institutional Review Board as an initial proof-of-concept study. A group of three HCPs (the group consisted of a surgeon, a nurse, and a student acting as an anesthesia provider) completed a simulation-based training within an instrumented training environment. Each participant provided written informed consent. Neurophysiological and performance data were collected from three individuals. This endeavor was a part of a Phase I project entitled automated cognitive load assessment for medical staff training and evaluation (ACLAMATE) funded by a Small Business Innovation Research Grant (SBIR) awarded to Aptima, Inc., through the Defense Health Program. The Ohio State University Wexner Medical Center was contracted to provide the clinical platform for the project (Award Number: W81XWH-14-C-0021) (ACLAMATE).
The goals of this case study were as follows: (1) the initial establishment and testing of an instrumented training environment at OSUWMCs CSEAC, and (2) an initial evaluation of neurophysiological-derived functional state and algorithm-based performance measures derived by the ANN-based models and associated technologies.
The establishment of such a training environment was a collaborative effort between researchers at OSUWMC, government and private industry (Aptima, Inc.). This instrumented training environment tested three of Aptima's automated performance measurement and assessment technologies: (1) The Functional State Estimation Engine (FuSE 2), (2) Scenario-based Performance Observation Tool for Learning in Team Environments (SPOTLITE ®), and (3) Performance Measurement Engine (PMEngine™). Descriptions of each of these technologies follow.
FuSE 2 provided real-time second by second high-resolution individual and team-level functional state measurements (the development of FuSE 2 was funded by an SBIR Phase III award provided by a government research laboratory that is a part of the Human Universal Measurement and Assessment Network [HUMAN] Program). As a part of the HUMAN Program, FuSE 2 has been successfully used to measure remotely piloted aircraft (RPA) pilot workload during simulated operations. Through FuSE 2, at any instant in time, an individual's functional state can be determined. Input sources include neural, physiological, and behavioral data; changes in the training or operational environment (increased task demands or difficulty); antecedent lifestyle factors (e.g., poor sleep quality, caffeine consumption, etc.); and the operational domain (e.g., mission goals and/or requirements, etc.). These data/measurement sources provided instantaneous estimates of cognitive workload states. FuSE 2 provides real-time measures of cognitive workload changes. The FuSE 2 modeling engine consists of time delay neural network (TDNN) models that provide continuous real-time estimates of functional state reflecting and correlating training performance. TDNN models are a common type of temporal supervised network and have memory components that serve to store historical values passed through the network and contain processing capabilities that learn relationships in data over time. The embedded models can account for model uncertainty in sensor data and features derived by the feature extraction module. This increases functional state estimation accuracy over time. The FuSE 2 workload functional state model was validated in Dayton, OH, USA.
SPOTLITE is a customizable mobile application for capturing observer-based performance measures. The SPOTLITE tool was originally developed for the United States Air Force as a domain agnostic tool for capturing observer-based measures. SPOTLITE, populated with scenario specific measures, provides improved data collection/measurements through: (1) focusing observers on salient features of trainee performance throughout the scenarios, (2) integrating SPOTLITE measurements with system-based and self-report measures to provide an improved picture of the trainee's experience, and (3) increased inter-rater reliability thereby improving consistency across training sessions and trainee evaluations.
PMEngine™ technology provides real-time performance measurements by processing data collected from any task/training environments in real-time. PMEngine™ has been deployed and utilized across depart of defense to provide performance measurement and assessment within a variety of operational environments. It uses human performance markup language to ingest data from various sources, such as a distributed simulation network, log files, as well as FuSE 2, and SPOTLITE technologies.
The above technologies provide performance measurements that may identify the causes of cognitive workload imbalance in health care delivery such as environment, resources, inappropriate staffing, and/or the need for further training (of a person or aspect of care). The instrumented training/evaluation demonstrated here provides significant sources of objective measures that better quantify individual performance and support optimal team design.
Designing the right team for the right circumstances involves education (practicing real life scenarios) in order to alleviate the cognitive workload stress in real word situations and enhance safety (resulting in better patient outcomes through limiting morbidity and mortality) [Figure 1].
|Figure 1: Venn diagram showing the relationships between education, safety, and the stress of an increased cognitive workload|
Click here to view
| Methods|| |
Neurophysiological data acquisition
Each participant was monitored using a Bioradio 150 (Great Lakes Neurotechnologies, Cleveland, OH) [Figure 2]a. The Bioradio 150 is a wireless device enabling real-time acquisition of neurophysiological signals. The BioRadio was used to collect seven differential channels of EEG and a single channel of ECG. The EEG data were collected via an electrode cap [Figure 2]b. The reference electrode for the EEG cap and ground for the BioRadio were located behind the mastoids. A single channel of ECG data was acquired by placing two electrodes on the torso [Figure 2]c. A handheld impedance meter was utilized to measure electrode impedances to ensure appropriate electrode-skin contact. Appropriate impedance values were defined as < 5 kΩ for EEG electrodes, and <20 kΩ for ECG electrodes. The BioRadio data were analyzed by FuSE 2 and subjected to digital signal processing and feature extraction algorithms to derive model input features required for real-time participant workload monitoring as described below. Each participant was wearing an EEG cap (underneath their surgical cap) that had three major components: (1) A digital signal-processing module, (2) a feature extraction module, and (3) a functional state estimation module. FuSE 2 was integrated with a universal data bus (UDB) [Figure 3]. EEG data collection did not distract/disrupt a participant's normal operating conditions.
|Figure 2: Neurophysiological monitoring hardware and electrode setup used in pilot study. (a) Bioradio 150. (b) Electrode cap. (c) Electrodes on torso|
Click here to view
|Figure 3: Aptima's FuSE2 design and seamless integration with sensors and monitoring hardware via the Universal Data Bus|
Click here to view
The digital signal processing module of FuSE 2 has multiple algorithms that “clean” and “improve” raw data collected from sensors connected to the UDB, thus preparing raw data for FuSE 2 s feature extraction module (including correcting raw data for signal artifacts due such as eye blinks, respiration, body movement, or pressing on electrodes that can cause a phenomenon known as baseline drift) [Figure 4] and [Figure 5].
|Figure 4: Demonstration of FuSE2 wavelet denoising-based signal processing algorithm to filter out and correct for blinks present in raw EEG data (blink contaminated signal (top) corrected signal (bottom)|
Click here to view
|Figure 5: Demonstration of FuSE2 signal processing algorithms used to remove baseline drift present in raw ECG signal (original signal (top) corrected signal (bottom))|
Click here to view
FuSE 2 used workload models developed under prior efforts completed by Aptima on the HUMAN Program. These models provided second by second measures of participant workload on a NASA TLX anchored 0–100 scale. The NASA TLX is a subjective, multidimensional scale developed in 1982 to measure a subject's perceived workload. The NASA TLX can be utilized for assessment of a task, system, team effectiveness, or other aspects of performance and is regarded as the gold standard measure of perceived workload.
In the original validation study, the technology provided accurate classification of subject workload (within a certain range of TLX values) >83.0% of the time., In this study, the mean absolute difference between the average model output (meant to reflect self-reported workload for a trial) and self-reported NASA TLX was calculated as 10.4 across 112 of 150 trials used for FuSE 2 validation. In a separate validation study of the developed system, Pearson correlation coefficients calculated between computed average workload model estimates and self-reported NASA TLX values was 0.76 across over 320 five minute trials collected over ten different study participants (unpublished research study).
Measures of workload
In addition to FuSE 2, PMEngine™ was used to derive measures related to team workload. This included measures of relative workload (RWL) and workload balance (WLB). PMEngine™ derives its performance measures over a configurable N-second time window. As an example, derivation of RWL and WLB measures requires that PMEngine™ filters and averages second-by-second workload measures derived from FuSE 2 as shown in Equation 1.
Measures of average team workload at time t (TWL [t]) were calculated using Equation 2, where M is the number of team members and WLi (t) represents the filtered workload of team member i at time t.
A measure of RWL indicates the relative contribution of workload from each team member with respect to an averaged team workload value. This value can range from ± 100, where an increasing negative value indicates that the team member is underloaded, and increasing positive value indicates the team member is overloaded. Values of RWL are calculated by determining the mathematical distance between CL (t) and TWL (t) relative to an allowable range and bounds based on dmax defined in Equation 3. In Equation 3, T is defined as a threshold value to define an allowable range of TWL (t) based on T standard deviations from the mean of TWL (t) over the time window N.
Using dmax, RWLi (t) can be calculated separately for each team member i using Equation 4.
On the contrary, WLB provides an indication of team effectiveness based on the distribution of workload across all team members. WLB ranges from 0 to 100, where a higher value indicates that workload and tasking are distributed equally across the team and that no one team member is significantly over- or under-loaded for an extended period of time. Measures of CLB at time t are calculated using measures of RWLi (t) calculated for each team member (of team size M) over the time window N, as represented in Equation 5.
With real-time neurophysiological data acquisition, feature extraction, and the computation of workload measures on a second-by-second basis, the state of not only the team but also the individuals comprising the team is possible. This real-time data can then be compared to key events within the training simulation timeline to examine how events affected workload at either the team and/or individual level. With the time of scenario key events known, the team's and individuals' time series can be analyzed to look for peaks (e.g., maximum workload for a given key event) and trends (e.g., how long before workload peaks after a key event, or how long before workload returns steady state after a key event, etc.). This analysis is then available as feedback to the trainees to better understand team and individual performance and to the trainers to better understand how scenario events impact training.
We performed an observational study of three individuals (surgeon, nurse, and medical student in the role of an anesthesiologist). Neurophysiological and performance data from each participant were collected and derived in real-time by the three above-mentioned measurement technologies, including individual and team workload outputs. These data came from frequency band powers derived from the EEG power spectrum, heart rate, inter-beat intervals, and respiratory rate. For the 20 min simulated clinical scenario, well over 15,000 data points and measures per participant were generated (not specified in this report).
A simulation scenario was developed for training and programmed into patient manikins. The scenarios took place in a simulated operating room and utilized a high fidelity human simulator. Modifications were made to the human simulator to incorporate realistic clinical and surgical procedures. The scenario was one of abdominal hemorrhage. The scenario was developed with three intended levels of physical and cognitive difficulty such that the scenario would induce a variable workload and stress. During the scenario, the devices and modalities in the methods section were applied.
| Results|| |
[Figure 6] demonstrates a user interface (UI) developed for training environments. This UI contains a graph of workload measures generated by FuSE 2 across the scenario for the medical student playing the anesthesiologist. The postscenario debrief, review of the scenario video, and self-reported NASA TLX of the medical student playing the role of an anesthesiologist indicated that their workload was significantly lower than the participants playing the roles of the surgeon and nurse. This was reflected in FuSE 2-generated measures of workload derived during the simulation. The student's workload measures were slightly increased when interacting verbally with the surgical team or providing direct patient care [Figure 6]. Moreover, shown in this UI [Figure 6] are measures of average cognitive load (workload) and the RWL (see above) of the anesthesiologist as derived by PMEngine™. The measure of RWL indicates that the medical student acting as an anesthesiologist was underloaded with respect to the workload of the surgeon and nurse (this is likely to have occurred secondary to experience level). This was consistent with information and data collected during postscenario debriefing and the post hoc analysis of measures collected.
|Figure 6: User interface and visualization displaying FuSE2 output (for anesthesiologist) during pilot testing|
Click here to view
| Discussion|| |
This study demonstrated successful transitioning of a subjective paper-based NASA TLX workload scale to an objective physiologically sensor-based automated workload measurement in an academic environment. The abdominal hemorrhage scenario using a surgeon, a scrub nurse, and a medical student anesthesiologist quite effectively demonstrated the capabilities of the FuSE 2, PMEngine™, and SPOTLITE technologies to provide demonstrations of cognitive workload, overload, and underload during clinical training. This study has demonstrated HCPs can be outfitted with remote, mobile, neurophysiological, unobtrusive monitors during clinical practice and effectively measure stress/workload level and, potentially, the ability to learn or a documentation of a rate of learning.
This study provides an approach with which to optimize the current workload model present in FuSE 2 for use in remotely monitored mobile health care training and operational settings. This will allow additional algorithms to be developed for FuSE 2's digital signal processing module to correct for additional motion artifacts that occur within mobile simulation-based training environments.
In addition, measures of RWL and WLB will need to be adjusted and optimized during future data collection and system development efforts. Currently, WLB weights positive and negative values of RWL equally, future studies will provide necessary data that will allow better evaluation as to whether differential weighting is required. Measures of RWL and WLB will be modified to account for the varying demands of different team member roles (such as nurses, surgeons, and anesthesiologists). Current measures of WLB do not take into account different team roles and their effect on workload during processes of care; it thus establishes an equal weighting with respect to the average workload of the team.
Limitations of this pilot project include a limited number of subjects, inherent biases in the algorithms, motion artifacts affecting devices, and the theoretical nature of the work. However, the applications used in this pilot study are regularly used by the USA military in training and evaluation of their personnel. In addition, one of the interesting findings of this study was that workload measures generated by FuSE 2 were generally elevated in study participants. This is likely attributed to the fact that the workload model was initially developed using data collected during stationary and sedentary (RPA) operations. Therefore, more detailed studies are needed.
Broad range of applications for this technology
The application of this technology is extensive. Measuring cognitive workload will be valuable in measuring the extent of learning in regard to tasks thereby allowing measurements of the degree of difficulty in task completion among individuals, which then can lead to individualized learning plans. This technology will also allow a measurement of team cohesiveness and function. As described above, remote monitor readouts will indicate individual task and team task cognitive loading and performance, allowing not only improvement in performance, but also indicating which individuals should be paired up to make the most effective teams. These cognitive measurements can indicate who is underperforming in certain scenarios. Thus, leading to effective, nonpunitive interventions that may lead to improved personal and team performances. In above technological environment, different scenarios can be identified, whether theoretical or real, and these particular scenarios can be “gamed” to improved performance, and thus, patient safety. In the end, this endeavor will lead to improved patient safety and, consequently, improved patient outcomes. For instance, for a particular level of intensive care acuity, there may need to be a specific number of doctors and nurses and allied health providers available for optimum results. This technology could help leaders/administrators make personnel decisions regarding numbers of providers and the particular skill sets needed on a particular day. In once scenario, an Intensive Care Unit may need two very skilled physicians and ten nurses of a certain “training caliber” to provide a good result/outcome. However, the same scenario may also be accomplished with, hypothetically, through analysis of the scenario with the above-mentioned technology, with three less skilled, or junior physicians, but seven very skilled nurses. Such technology may allow a certain patient acuity level to have different combinations of individuals with different skill sets, or levels of experience and abilities, to achieve the same outstanding patient outcome. Moreover, once such a team composition is in place, cognitive workload measurement histories (analyses) will also help an organization decide which team/skill type needs to be thrown into action when there is a sudden increase in acuity that demands an immediate response (e.g., mass casualties, three organ transplants overnight, or multiple patient emergencies/physiological extremis in a short time period). In surgery and critical care, stressful decisions are made in stressful environments. This type of technology can lead to thoughtful, controlled, personnel, and technological interventions as described in a recent comprehensive textbook regarding the fundamentals of patient safety and outcome by Stawicki et al.
| Conclusion|| |
This study demonstrated the potential value an instrumented training environment consisting of state-of-the-art performance measurement and assessment capabilities. The capabilities demonstrated in this study may be the foundation for substantial advancements in training and learning in medical, educational environments. Further work will need to be done in order to enhance the measurement capabilities of the instrumented training environments in the medical education simulation and evaluation.
ACLAMATE study. Topic # DHP 13-002. Gov. Proposal #H132”002-0130. Pappada S, Principle Investigator, Aptima, Inc. Thomas J. Papadimos, Principal Site Investigator. Proposal to design, develop, and commercialize a fully functional data collection, assessment, and alerting tool that will allow instructors to dynamically modify simulation-based training scenarios. Awarded September 2013.
Dr. Pappada, Dr. Feeny, and Mr. Durkee are employed by the Aptima Corporation and work closely with university and government agencies.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Carayon P, Gürses AP. A human factors engineering conceptual framework of nursing workload and patient safety in intensive care units. Intensive Crit Care Nurs 2005;21:284-301.
Brookings JB, Wilson GF, Swain CR. Psychophysiological responses to changes in workload during simulated air traffic control. Biol Psychol 1996;42:361-77.
Gevins A, Smith ME. Neurophysiological measures of workload during human-computer interaction. Theor Issues Ergon Sci 2003;4:113-31.
Carayon P. The balance theory and the work system model. Twenty years later. Int J Hum Comput Interact 2009;25:313-27.
Patel VL, Kannampallil TG, Shortliffe EH. Role of cognition in generating and mitigating clinical errors. BMJ Qual Saf 2015;24:468-74.
Crane PW, Zhou Y, Sun Y, Lin L, Schneider SM. Entropy: A conceptual approach to measuring situation-level workload within emergency care and its relationship to emergency department crowding. J Emerg Med 2014;46:551-9.
Colligan L, Potts HW, Finn CT, Sinkin RA. Cognitive workload changes for nurses transitioning from a legacy system with paper documentation to a commercial electronic health record. Int J Med Inform 2015;84:469-76.
Carayon P, Alvarado C, Hundt AS. Reducing workload and increasing patient safety through work and workspace design. Washington, DC: Institute of Medicine; 2003.
Carayon P, Alvarado CJ; Systems Engineering Initiative for Patient Safety. Workload and patient safety among critical care nurses. Crit Care Nurs Clin North Am 2007;19:121-9.
Vogt J, Adolph L, Ayan T, Udovic A, Kastner M. Stress in modern air traffic control systems and potential influences on memory. Hum Factors Aerosp Saf 2002;2:355-78.
Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock PA, Meshkati N, editors. Human Mental Workload. Amsterdam, Netherlands: Elsevier Science; 1988. p. 139-83.
Paige JT, Garbee DD, Kozmenko V, Yu Q, Kozmenko L, Yang T, et al.
Getting a head start: High-fidelity, simulation-based operating room team training of interprofessional students. J Am Coll Surg 2014;218:140-9.
Kamzanova AT, Kustubayeva AM, Matthews G. Use of EEG workload indices for diagnostic monitoring of vigilance decrement. Hum Factors 2014;56:1136-49.
Estepp JR, Christensen JC. Electrode replacement does not affect classification accuracy in dual-session use of a passive brain-computer interface for assessing cognitive workload. Front Neurosci 2015;9:54.
Brouwer AM, Zander TO, van Erp JB, Korteling JE, Bronkhorst AW. Using neurophysiological signals that reflect cognitive or affective state: Six recommendations to avoid common pitfalls. Front Neurosci 2015;9:136.
Lin CJ, Hsieh TL, Yang CW, Huang RJ. The impact of computer-based procedures on team performance, communication, and situation awareness. Int J Ind Ergon 2016;52:21-29.
Zarea GV, Hazrati H, Ghojazadeh M. The efficacy of digital case scenario versus paper case scenario on clinical reasoning in problem-based learning: A systematic review and meta-analysis. Rev Dev Med Educ 2015;4:17-22.
Pappada SM, Durkee K, Geyer A, Ortiz A, Galster S, Cohn J. Developing Neurocognitive Workload Management Technologies for Multi-tasking, Presentation on US Navy Panel: Human Factors Challenges in UAS-Cognitive Workload, Situational Awareness, Decision-Making, and Trust in Automation, 85th
Annual Scientific Meeting of the Aerospace Medical Association, San Diego, CA; 2014. Available from: http://www.asma.org/annual-meetings/proceedings
. [Last accessed on 2015 Mar 30].
Yurko YY, Scerbo MW, Prabhu AS, Acker CE, Stefanidis D. Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc 2010;5:267-71.
Durkee K, Pappada SM, Ortiz A, Feeney J, Galster S. Using context to optimize a functional state estimation engine in unmanned aircraft system operations. In: Schmorrow DD, Fidopiastis CM, editors. Foundations of Augmented Cognition. Berlin, Germany: Springer; 2013. p. 24-35.
Stawicki SP, Galwankar SC, Papadimos TJ, Moffatt-Bruce SD, editors. The fundamentals of patient safety in medicine and surgery. New Delhi: Wolters Kluwer Health; 2015.
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6]