Advertisement

Flipping the classroom: Case-based learning, accountability, assessment, and feedback leads to a favorable change in culture

Open ArchivePublished:December 19, 2016DOI:https://doi.org/10.1016/j.jtcvs.2016.10.101

      Abstract

      Objective

      The 88-week Thoracic Surgery Curriculum is challenging to implement because of the large content in a traditional lecture format. This study investigates flipping the classroom by using a case-based format designed to stimulate resident preparation and engagement.

      Methods

      The didactic conference format was altered. Curricular reading assignments, case review, and conference participation prepared residents for novel formative assessment quizzes. Ten residents participated, and faculty served as controls. Scores were compared with the use of linear regression adjusted for clustering of responses for each person. A survey was administered to determine impressions of this educational technique.

      Results

      A majority of residents completed curricular readings (82%) and reviewed case presentations (79%). Resident performance initially lagged behind faculty but exceeded faculty performance by the conclusion (interaction P = .047). Junior resident overall performance was superior to senior residents over the entire analysis (P = .026); however, both groups improved over time similarly (P = .34) Increased reading from the curriculum (5% increase per level, P = .001) and case presentation review (6% increase per level, P < .0001) were associated with improved quiz performance. Residents presenting cases at their session performed no better than other quiz-takers for the same session (P = .38). The majority of residents viewed this method favorably.

      Conclusions

      This method stimulated increased resident participation and engagement in this pilot study. Assessment scores increased at both resident levels, and resident performance exceeded faculty performance with time. By using experiential learning principles, flipping the classroom in this manner may improve educational culture by enhancing accountability, assessment, and feedback.

      Key Words

      Abbreviations and Acronyms:

      ABTS (American Board of Thoracic Surgery), CI (confidence interval), LMS (learning management system), TSC (thoracic surgery curriculum)
      Figure thumbnail fx1
      Improving resident performance over time compared with faculty.
      By the use of experiential learning principles, flipping the classroom may improve educational culture by enhancing accountability, assessment, and feedback.
      In a pilot study, using a case-based format of curricular teaching, residents prepared and reviewed material while increasing their engagement and scores on quizzes and ultimately exceeded faculty performance. By the use of experiential learning principles, flipping the classroom may improve educational culture by enhancing accountability, assessment, and feedback.
      See Editorial Commentary page 997.
      The curricular requirements set forth by the American Board of Thoracic Surgery (ABTS) for the completion of suitable cardiothoracic training have been revised recently.

      American Board of Thoracic Surgery. Curriculum. Available at: https://www.abts.org/root/home/curriculum.aspx. Accessed April 16, 2016.

      This new paradigm was the basis for the redevelopment of the so-called 88-week curriculum by the Thoracic Surgery Directors Association and incorporation of this curriculum by the Joint Council for Thoracic Surgery Education (the Joint Council for Thoracic Surgery Education was incorporated into the Society of Thoracic Surgeons on July 1, 2016) into their new learning and content management systems and the subsequent thoracic surgery curriculum (TSC).

      Thoracic Surgery Directors Association. Thoracic surgical curriculum. Available at: http://www.tsda.org/education/thoracic-surgery-curricula. Accessed April 16, 2016.

      Joint Council on Thoracic Surgery Education. Thoracic surgical curriculum. Available at: http://www.jctse.org/education/thoracic-surgical-curriculum. Accessed April 16, 2016.

      This curriculum has oversight from an editorial board and consists of book chapters, peer-reviewed journal articles, lectures, and reviews.
      It has been recognized by educators and learners alike that the current curricular content has the potential to be overwhelming in volume and scope. As an example, the content for “Management of Benign Esophageal Disorders III,” includes 3 book chapters, 2 review articles, and 2 peer-reviewed publications.

      Joint Council on Thoracic Surgery Education. JCTSE online course. Available at: http://jctse.mrooms.net. Accessed April 16, 2016.

      Because many programs focus their teaching conferences based on progress in the curriculum, content covered during these sessions can devolve into dense lectures driven by a perceived need to cover all the assigned content.
      • Vaporciyan A.A.
      • Yang S.C.
      • Baker C.J.
      • Fann J.I.
      • Verrier E.D.
      Cardiothoracic surgery residency training: past, present and future.
      At our institution, we have used didactic (traditional lectures), case-based presentations, review of standardized examinations, and other educational modalities to maintain resident and faculty engagement. It had been noted by our faculty and residents that the case-based presentations often were the most dynamic and interactive. Partially driven by this observation, more than 100 case-based presentations were incorporated into the TSC to help guide teaching sessions.

      Joint Council on Thoracic Surgery Education. JCTSE online course. Available at: http://jctse.mrooms.net. Accessed April 16, 2016.

      Anecdotally, these cases were used sparingly, and there was little resident accountability by the TSC platform for the entire content matter.
      • Antonoff M.B.
      • Verrier E.D.
      • Allen M.S.
      • Aloia L.
      • Baker C.
      • Fann J.I.
      • et al.
      Impact of Moodle-based online curriculum on thoracic surgery in-training examination scores.
      Furthermore, the cases were criticized as lacking assessment and feedback capabilities. Although the original learning management system (LMS) for the TSC did have quiz functions, it lacked robust reporting tools thereby limiting its overall value. This year, a new LMS was launched with additional functionality that has yet to be explored fully.
      It also has been recognized in our specialty (and others) that traditional methods of teaching have faced significant challenges over the last decade.
      • Vaporciyan A.A.
      • Yang S.C.
      • Baker C.J.
      • Fann J.I.
      • Verrier E.D.
      Cardiothoracic surgery residency training: past, present and future.
      Flipping the classroom is characterized by learners engaged in active learning and assessment with feedback and increasingly has been used in many environments.
      • McLaughlin J.E.
      • Roth M.T.
      • Glatt D.M.
      • Gharkholonarehe N.
      • Davidson C.A.
      • Griffin L.M.
      • et al.
      The flipped classroom: a course redesign to foster learning and engagement in a health professions school.
      It requires the learners to acquire the majority of content during nonclass time, so that direct teacher−student interactions are optimized.
      This pilot study sought to examine a method by which to engage residents and faculty in educational conferences that required case-based learning via a new conference format, incorporated elements of assessment by including quizzes, promoted accountability for content by querying resident preparation and self-assessment, and provided meaningful feedback by providing live feedback during conferences and by furnishing quiz results. After a thorough review of the literature, we believed this teaching style would be well suited to our cardiothoracic educators and trainees, given the limited work hours and breadth and depth of content. We elected to pursue this strategy because of our existing experiences with case-based learning, its intuitive nature to our (local) teaching style, and the incorporation of cases embedded in the TSC, as this could provide the basis for other educators on their use.

      Methods

       Conference Conduct

      In a prospective fashion, over the course of 12 educational sessions (5 cardiac, 7 thoracic), the regularly scheduled didactic conference format was altered. A resident−faculty dyad was created for each conference. Together, they prepared an illustrative case covering the assigned topic, some of which were derived directly from the TSC. (Example presentation from “Management of Benign Esophageal Disorders III”: Video 1)

      Yates R, Verrier E. Achalasia. In: Thoracic Surgery Curriculum, TS 34: Management of Benign Esophageal Disorders III. Available at: http://learnctsurgery.sts.org/lms/libraries. Accessed August 25, 2016.

      Reading assignments, based on the curriculum, and focused on the case, as well as the case itself, were distributed to the residents (6 integrated, 4 traditional, or 4/3) and faculty for preview and preparation before the conference date. Via an interactive format, these topics were dissected by all members at the educational session as follows: After learning objectives are outlined, a case presentation is performed with pertinent findings. Both basic and advanced elements of the history, physical, and workup are included. The Socratic method is used by faculty to gauge level of understanding in learners of all levels. Diagnostic and therapeutic maneuvers are discussed in a group format, including risks and benefits of surgical options. Residents of all levels are given opportunities to ask questions and solicit faculty opinions in clinically vague areas. The case is then summarized and learning points once again highlighted.
      Figure thumbnail fx2
      Video 1A video of the slide set for from “Management of Benign Esophageal Disorders II” conference is included. After learning objectives are outlined, background reading is specifically identified from within the TSC. A case presentation is performed with pertinent findings. Both basic and advanced elements of the history, physical, and workup are included so as to engage learners of multiple levels. An in-depth discussion of manometry is conducted, which then leads to a discussion. The discussion includes therapeutic options and strategies, as well as surgical options. The case is summarized and learning points once again highlighted. Video available at: http://www.jtcvsonline.org/article/S0022-5223(16)31681-6/addons.

       Quiz Preparation

      A multiple-choice quiz was developed by the assigned faculty member to assess knowledge of the curriculum (Example quiz from “Management of Benign Esophageal Disorders III”: Video 2). The quiz items were newly written by the faculty to cover the assigned topic and were not obtained from commercially available sources. In all instances, the quiz content went beyond the content from the conference alone and was inclusive of the readings from the curriculum. Quizzes were distributed at the end of conference to all the residents (whether or not they were present at conference) and, as a control, to nonpresenting faculty members of the respective subspecialty—cardiac surgery faculty only took cardiac surgery quizzes, and thoracic surgery faculty only took thoracic surgery quizzes. A brief 4-question survey accompanied the quiz to quantify resident readings from the curriculum, case review before completing the quiz, conference attendance, and self-assessment of competence (using the modified Dreyfus scale) in the topic. Residents and faculty received a report of their individual performance on the quizzes.
      Figure thumbnail fx3
      Video 2Corresponding quiz items from “Management of Benign Esophageal Disorders II” conference. The quiz includes multiple choice questions and true/false questions. These items are designed to draw from the presentation and the reading materials for the session. Video available at: http://www.jtcvsonline.org/article/S0022-5223(16)31681-6/addons.

       Survey

      A survey consisting of 22 questions was administered anonymously to the residents, which explored demographics, career interests, conference preferences, preparation, and was specifically designed to evaluate the impact of the change in conference conduct (Appendix E1).

       Analysis

      Each resident's pre- and post-quiz engagement, self-assessment, and attendance were recorded for each quiz. Simple linear regression was used to evaluate engagement with preparation and review materials, self-assessment, and attendance over time. Analysis of accuracy was performed by the use of fixed-effects linear regression to adjust for clustering of responses at the level of residents or faculty. Interactions between successive sessions and seniority were performed based on visual inspection graphs of performance over time by seniority, which suggested differential accuracy by seniority and faculty/resident status. Predicted test scores were derived from the linear models to demonstrate the effect of engagement, attendance, and self-assessment. Stata (version 13; Stata Corp, College Station, Tex) was used for all analyses.

       Institutional Review Board

      The University of Washington institutional review board granted a waiver for these activities.

      Results

      Residents were well distributed across years of training, with 4 residents in their final 2 years of training (senior residents) and 6 residents with more than 2 years remaining (junior residents). Of the residents, 70% were male, and career interests were 30% adult cardiac, 20% general thoracic, 10% congenital, 10% cardiothoracic, and 30% undecided. One-half of the residents anticipated an academic career, whereas the remaining were undecided.
      Quiz performance improved with time as residents gained experience with the system (Figure 1). Resident performance was not statistically different from faculty over the course of the program (coefficient for faculty 0.057, 95% confidence interval [95% CI] −0.03, 0.15, P = .22); however, resident performance was lower than faculty initially but exceeded faculty performance by the end of the program (faculty*session interaction P = .047, Figure 2, A). Junior resident overall performance was superior to senior residents over the entire analysis (seniority coefficient −0.1; 95% CI, −0.19, −0.01; P = .026); however, both groups improved over time at a similar rate (seniority*session interaction P = .34, Figure 2, B).
      Figure thumbnail gr1
      Figure 1Comparison of scores between residents and faculty. Comp MI, Complications of myocardial infarction; Asc Ao, ascending aortic disease; Ao arch, aortic arch; MCS, mechanical circulatory support; TAAA, thoracoabdominal aortic aneurysm; Trachea, tracheal disease; TEF, tracheoesophageal fistula; DSL, diverticulae-stricture-lye; Diag, esophageal anatomy; diagnostic tools; GERD, gastroesophageal reflux disease; Eso CA, esophageal cancer.
      Figure thumbnail gr2
      Figure 2Comparison of resident and faculty quiz score trends over time. Lines are derived from simple linear regression, whereas the interaction P values (faculty*session and seniority*session) were estimated from a fixed-effects model. A, Comparison of residents to faculty over time. B, Comparison of junior with senior residents over time. Junior residents overall performed better than senior residents (P = .026).
      A majority of residents studied the readings from the curriculum before conference (82%) and reviewed the case presentations after conference (79%) before their assessment, whereas more than one-third of residents were absent from conference (35.7%). The majority of residents (66.9%) considered themselves novice or advanced beginners (Table 1). Increased reading from the curriculum (coefficient 0.05; 95% CI, 0.02-0.08, P = .001) and increased case presentation review (coefficient 0.06; 95% CI, 0.03-0.08, P < .0001) were associated with improved quiz performance. Residents presenting cases demonstrated a nonsignificant signal toward improvement on quizzes compared with other quiz-takers for the same session (vs absent coefficient 0.1, P = .38, vs present coefficient 0.05, P = .68). Power to detect differences was limited to analysis of only 12 sessions. Resident self-assessment of increasing proficiency with material was consistent with performance on quizzes (coefficient 0.05; 95% CI, 0.01-0.09, P = .018). Predicted test performance scores for each level of engagement, attendance, and self-assessment also are provided. For the entire group, average engagement with case readings (P = .84), case presentation reviews (P = .954), and self-assessment values (P = .129) did not change over the course of the program.
      Table 1Resident preparation, attendance, and self-assessment for quizzes and effect on quiz scores
      PercentRegression coefficientPredicted quiz scoreP value
      Readings from curriculum
       Read all9.80.0572.6%.001
       Most26.867.6%
       Some34.862.6%
       Little10.757.6%
       None17.952.6%
      Case review before quiz
       All reviewed330.0672.1%<.0001
       Most19.666.4%
       Some18.860.7%
       Little855.0%
       None20.549.3%
      Attendance at conference
       Presented2.7Reference71.9%
       Attended54.5−0.0567.2%.684
       Absent35.7−0.1061.6%.379
      Self-assessment of competence on topic
       Proficient10.70.0572.2%.018
       Competent22.367.2%
       Advanced beginner33.962.2%
       Novice3357.2%
      Regression coefficients correspond to improvement in performance (quiz percentage score) by each activity or versus all other groups in the case of attendance. Predicted test performance scores were based upon resident reported preparation, participation, and confidence.
      Seventy percent of the residents viewed this conference format as superior to formal resident or faculty didactic sessions, review of standardized questions, or guest lectures. The majority of the residents (60%) reported that the case-based format motivated them to prepare for conference, despite the fact that 60% also believed that the TSC reading assignments were “too much.” Multiple residents cited faculty engagement as one of the most beneficial aspects of this effort. Multiple residents also sought more elaborate feedback than was provided. Almost all the residents commented that they were motivated by this process to prepare more overall.

      Discussion

      The overarching aim of this study is to move the needle in changing the culture of didactic education in cardiothoracic surgery. Surgical culture has been studied qualitatively and quantitatively in other domains, generally related to operating room performance and changes in clinical outcomes, with encouraging, but mixed, results
      • Bosk C.
      Forgive and Remember: Managing Medical Failure.
      • Yule S.
      • Sacks G.D.
      • Maggard-Gibbons M.
      Innovative approaches for modifying surgical culture.
      ; however, much of the attention in our specialty has focused on culture at a national level—duty hours, ABTS pass rates, and innovative ways for acquisition of surgical skills.
      • Connors R.C.
      • Doty J.R.
      • Bull D.A.
      • May H.T.
      • Fullerton D.A.
      • Robbins R.C.
      Effect of work-hour restriction on operative experience in cardiothoracic surgical residency training.
      • Moffatt-Bruce S.D.
      • Ross P.
      • Williams T.E.
      American Board of Thoracic Surgery examination: fewer graduates, more failures.
      • Mokadam N.A.
      • Lee R.
      • Vaporciyan A.A.
      • Walker J.D.
      • Cerfolio R.J.
      • Hermsen J.L.
      • et al.
      Gamification in thoracic surgical education: using competition to fuel performance.
      The TSC provides a framework for national culture change, but this does not equate to direct transmission of that change to implementing programs. For a curriculum to be complete, it requires a needs assessment, objectives, assessment of the learner, and assessment of the curriculum itself.
      • Vaporciyan A.A.
      Teaching and learning surgical skill.
      This pilot study allowed our institution the opportunity to evaluate the TSC (and more specifically, case-based education) in a manner not previously done.
      Substantial attention decline occurs in as little as 10 minutes, and even in high-functioning medical students, only lasts an average of 20 minutes.
      • Stuart J.
      • Rutherford R.J.
      Medical student concentration during lectures.
      • Hartley J.
      • Cameron A.
      Some observations on the efficiency of learning.
      Furthermore, lecture-based information has a low overall retention rate of about 20%, making the traditional hour-long lecture an inefficient mechanism by which to teach complex topics such as cardiothoracic surgery.
      • Hartley J.
      • Davies J.K.
      Note-taking: a critical review.
      Experiential learning has emerged as one important element of education, which draws on understanding and rationale, rather than traditional dictum. This is in contrast to the concept of knowledge transmission during didactic learning, which is the basis for our traditional education, and referred to by our faculty as the “dump truck of knowledge.” Experiential learning theory is based on the following concepts: learning is a process that requires engagement and feedback; all learning is relearning; learning requires resolution of conceptual conflict; learning involves integrative functioning; learning results from synergy between the learner and the environment; learning is the process of creating knowledge.
      • Kolb A.Y.
      • Kolb D.A.
      Learning styles and learning spaces: enhancing experiential learning in higher education.
      The concepts of experiential learning theory are integral in adult learning theory—active involvement in goals, building on previous knowledge, intrinsic motivation, and responsibility.
      • Goldman S.
      The educational Kanban: promoting effective self-directed adult learning in medical education.
      The benefit of learner engagement by altering the traditional lecture-based teaching style has been studied in other settings. In a physics class, the use of preclass reading, questions, discussions, and feedback increased student class attendance and increased scores compared with control subjects.
      • Deslauriers L.
      • Schelew E.
      • Wieman C.
      Improved learning in a large-enrollment physics class.
      This also has been observed in meta-analysis examining the effectiveness of Continuing Medical Education.
      • Davis D.
      • O’Brien M.A.T.
      • Freemantle N.
      • Wolf F.M.
      • Mazmanian P.
      • Taylor-Vaisey A.
      Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?.
      In that study, the use of didactic sessions did not translate into changes in performance or differences in healthcare outcomes. This finding is in contrast to the observation that case discussions and hands-on sessions were more effective and led to a change in behavior and practice. It is also notable that experiential learning is one type of learning that may be complementary other learning types, or may be more or less effective in certain individuals depending on their individual learning styles.
      Interestingly, junior residents performed better overall than senior residents in this study. This may be attributed to time constraints in the more clinically demanding senior years, junior resident enthusiasm and thirst for cardiothoracic content, or other factors. It is encouraging that naïve learners in cardiothoracic surgery can be educated with regard to complex concepts when using novel educational methods. Equally interesting is the observation that resident performance, through preparation, exceeded faculty performance on some of these assessments. From the perspective of educators, this may be the most encouraging finding in the study—if the trainee can achieve the competence of the educator, then for this point in time, the trainee has learned and advanced towards competence. This also could indicate acquisition of expertise in this process (test-taking skills) without necessarily reflecting expertise in content over the short or long term. Indeed, it may be surprising that 3 groups (faculty, senior residents, presenting residents) who would have a priori been predicted to perform better on exams in fact did not. This may reflect limitations in the assessment, or may also reflect a higher level of understanding that exceeded the fidelity of a series of multiple-choice questions.
      This study incorporated these experiential learning principles into the cardiothoracic training arena to enact culture change. This process stimulated increased resident and faculty engagement. This increased engagement may be responsible for the overall satisfaction results seen in the survey. Nonetheless, over time, both junior and senior residents improved performance at an equivalent rate, suggesting synergy between the learner and the environment. This synergy also may be a product of a relatively large resident cohort incorporating learners of many levels. Extrapolation of these data to other programs bears further study.
      Not surprisingly, but statistically meaningful, reading from the curriculum, reviewing the cases (relearning), attending conference (resolution of conceptual conflict), and overall confidence (creating knowledge) predicted greater scores (feedback) in residents of all levels. The majority of residents had a favorable impression of this process and reported that it motivated them both to prepare before and review after the weekly teaching conferences. The new TSC LMS may enable this type of assessment and feedback with its ability to have benchmark quizzes and normalize the results across the national sample. These data also may provide evidence to encourage educators of any level (including those who have “always done it this way”) to consider other teaching styles in their curricula. Furthermore, use of the case-based presentations already embedded in the TSC can provide an ample starting point from which to begin. All culture is of course local culture, and the experience described in this paper may best serve as a blueprint for recapitulation in the broader surgical educational arena. It may be reasonable to examine the goals, objectives, and motivators of the educators to better define teaching practices in future studies. There also may be an opportunity to further explore this methodology in a multicenter fashion.

       Limitations

      Although this study was prospective, it was not possible to provide resident-level control observations. Furthermore, the assessment quizzes used were not subject to psychometric evaluation and therefore cannot be extrapolated to performance on standardized examinations such as the Thoracic Surgery Directors Association in-training examination or the ABTS qualifying examination. There was no analysis of long-term knowledge retention that may benefit from independent investigation. Finally, this is a preliminary pilot evaluation of this methodology for our specialty and warrants further study.

      Conclusions

      Using a case-based format of curricular teaching, residents prepared and reviewed material while increasing their engagement and scores on quizzes and ultimately exceeded faculty performance. By using experiential learning principles, flipping the classroom may improve educational culture by enhancing accountability, assessment, and feedback.

       Conflicts of Interest Statement

      Dr Mokadam is an investigator for St Jude Medical, HeartWare, and Syncardia and a consultant for St Jude Medical and HeartWare. Dr Dardas has research support from ISHLT/HeartWare. Dr Pal is a consultant for HeartWare. All other authors have nothing to disclose with regard to commercial support.

      Appendix E1. Survey items

       Survey Items for Flipping the Classroom: Case-Based Learning, Accountability, Assessment, and Feedback Leads to a Favorable Change in Culture

      • 1.
        Year of training:
      • 2.
        Sex:
      • 3.
        Last (thoracic) American Board of Surgery In-Training Exam score:
      • 4.
        Career interest (specialty):
      • 5.
        Career interest (academic vs private):
      • 6.
        Residency format (I6, traditional, 4/3):
      • 7.
        Which conference format is most educationally valuable?
      • (Please rank 1-6, 1 being most valuable and 6 being least valuable.)
        • Resident didactic presentation (ppt)
        • Faculty didactic presentation (ppt)
        • Case-based format
        • Self-Education Self-Assessment in Thoracic Surgery
        • Guest lecturer from outside the division
        • Other
      • 8.
        Does the case-based format change your preparation for conference (yes, no)?
      • 9.
        Do you find the reading assignments for the weekly thoracic surgery curriculum 88 curriculum (too much, too little, just right)?
      • 10.
        Do you find the readings assigned for the weekly University of Washington case-based conferences (too much, too little, just right)?
      • 11.
        Do you preview the distributed cases prior to conference (no, less than 50%, more than 50%)?
      • 12.
        Have your self-study habits changed over the course of this pilot project (no, less preparation, more preparation, more self-study)?
      • 13.
        Have you reviewed the cases after conference with their content (no, less than 50%, more than 50%)?
      • 14.
        Do the quizzes adequately reflect the reading or case presentations (yes, no)?
      • 15.
        Do the quizzes encourage you to prepare for conference (yes, no)?
      • 16.
        The difficulty level of the quizzes is (too easy, just right, too hard)?
      • 17.
        What are barriers to completing the quizzes (clinical duties, sleep deprivation, not educational, performance anxiety, other)?
      • 18.
        When do you typically complete the quizzes (immediately after conference, immediately after review, whenever there is time)?
      • 19.
        What aspects of reporting would be most valuable to your learning (write in)?
      • 20.
        How should we improve this process (write in)?
      • 21.
        How do you plan to change your learning habits in response to this (write in)?
      • 22.
        How can the faculty improve their interaction with you to optimize your conference learning (write in)?

      Supplementary Data

      References

      1. American Board of Thoracic Surgery. Curriculum. Available at: https://www.abts.org/root/home/curriculum.aspx. Accessed April 16, 2016.

      2. Thoracic Surgery Directors Association. Thoracic surgical curriculum. Available at: http://www.tsda.org/education/thoracic-surgery-curricula. Accessed April 16, 2016.

      3. Joint Council on Thoracic Surgery Education. Thoracic surgical curriculum. Available at: http://www.jctse.org/education/thoracic-surgical-curriculum. Accessed April 16, 2016.

      4. Joint Council on Thoracic Surgery Education. JCTSE online course. Available at: http://jctse.mrooms.net. Accessed April 16, 2016.

        • Vaporciyan A.A.
        • Yang S.C.
        • Baker C.J.
        • Fann J.I.
        • Verrier E.D.
        Cardiothoracic surgery residency training: past, present and future.
        J Thor Cardiovasc Surg. 2013; 146: 759-767
        • Antonoff M.B.
        • Verrier E.D.
        • Allen M.S.
        • Aloia L.
        • Baker C.
        • Fann J.I.
        • et al.
        Impact of Moodle-based online curriculum on thoracic surgery in-training examination scores.
        Ann Thor Surg. 2016; 102: 1381-1386
        • McLaughlin J.E.
        • Roth M.T.
        • Glatt D.M.
        • Gharkholonarehe N.
        • Davidson C.A.
        • Griffin L.M.
        • et al.
        The flipped classroom: a course redesign to foster learning and engagement in a health professions school.
        Acad Med. 2014; 89: 236-243
      5. Yates R, Verrier E. Achalasia. In: Thoracic Surgery Curriculum, TS 34: Management of Benign Esophageal Disorders III. Available at: http://learnctsurgery.sts.org/lms/libraries. Accessed August 25, 2016.

        • Bosk C.
        Forgive and Remember: Managing Medical Failure.
        University of Chicago Press, Chicago1979: 2003
        • Yule S.
        • Sacks G.D.
        • Maggard-Gibbons M.
        Innovative approaches for modifying surgical culture.
        JAMA Surg. 2016; 151: 791-792
        • Connors R.C.
        • Doty J.R.
        • Bull D.A.
        • May H.T.
        • Fullerton D.A.
        • Robbins R.C.
        Effect of work-hour restriction on operative experience in cardiothoracic surgical residency training.
        J Thorac Cardiovasc Surg. 2009; 137: 710-713
        • Moffatt-Bruce S.D.
        • Ross P.
        • Williams T.E.
        American Board of Thoracic Surgery examination: fewer graduates, more failures.
        J Thorac Cardiovasc Surg. 2014; 147: 1464-1470
        • Mokadam N.A.
        • Lee R.
        • Vaporciyan A.A.
        • Walker J.D.
        • Cerfolio R.J.
        • Hermsen J.L.
        • et al.
        Gamification in thoracic surgical education: using competition to fuel performance.
        J Thorac Cardiovasc Surg. 2015; 150: 1052-1058
        • Vaporciyan A.A.
        Teaching and learning surgical skill.
        Ann Thor Surg. 2016; 101: 12-14
        • Stuart J.
        • Rutherford R.J.
        Medical student concentration during lectures.
        Lancet. 1978; 312: 514-516
        • Hartley J.
        • Cameron A.
        Some observations on the efficiency of learning.
        Educ Rev. 1967; 20: 30-37
        • Hartley J.
        • Davies J.K.
        Note-taking: a critical review.
        Innov Educ Train Int. 1978; 15: 207-224
        • Kolb A.Y.
        • Kolb D.A.
        Learning styles and learning spaces: enhancing experiential learning in higher education.
        Acad Manag Learn Educat. 2005; 4: 193-212
        • Goldman S.
        The educational Kanban: promoting effective self-directed adult learning in medical education.
        Acad Med. 2009; 84: 927-934
        • Deslauriers L.
        • Schelew E.
        • Wieman C.
        Improved learning in a large-enrollment physics class.
        Science. 2011; 332: 862-864
        • Davis D.
        • O’Brien M.A.T.
        • Freemantle N.
        • Wolf F.M.
        • Mazmanian P.
        • Taylor-Vaisey A.
        Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?.
        J Am Med Assoc. 1999; 282: 867-874