Validation of a performance assessment instrument in problem-based learning tutorials using two cohorts of medical students

详细信息    查看全文

• 作者：Ming Lee ; Paul F. Wimmers
• 关键词：PBL assessment ; Psychometric validation ; Generalizability theory ; Reliability and validity ; Standards for Educational and Psychological Testing ; Medical Education ; Undergraduate
• 刊名：Advances in Health Sciences Education
• 出版年：2016
• 出版时间：May 2016
• 年：2016
• 卷：21
• 期：2
• 页码：341-357
• 全文大小：550 KB
• 参考文献：Albanese, M. A., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68, 52–81.CrossRef
  American Educational Research Association, American Psychological Association & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
  Baeten, M., Kyndt, E., Struyven, K., & Dochy, F. (2010). Using student-centered learning environments to stimulate deep approaches to learning: Factors encouraging or discouraging their effectiveness. Educational Research Review, 5, 243–260.CrossRef
  Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481–486.CrossRef
  Berkson, I. (1993). Problem-based learning: Have the expectations been met? Academic Medicine, 68, S79–S88.CrossRef
  Biggs, J. (2003). Teaching for quality learning at university. Buckingham: Open University Press.
  Blumberg, P. (2005). Assessing students during the problem-based learning (PBL) process. Journal of the International Association of Medical Science Educators, 15, 1–9. http://​www.​iamse.​org/​artman/​publish/​article_​289.​shtml . Accessed 11 April 2014.
  Bowerman, B. L., & O’Connell, R. T. (1990). Linear statistical models: An applied approach (2nd ed.). Belmont, CA: Duxbury.
  Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
  Colliver, J. A. (2000). Effectiveness of problem-based learning curricula: Research and theory. Academic Medicine, 75, 259–266.CrossRef
  Crick, J. E., & Brennan, R. L. (1983). Manual for GENOVA: A generalized analysis of variance system. Iowa City, IA: The American College Testing Program.
  Distlehorst, L. H., Dawson, E., Robbs, R. S., & Barrows, H. S. (2005). Problem-based learning outcomes: The glass half-full. Academic Medicine, 80, 294–299.CrossRef
  Dolmans, D., & Gijbels, D. (2013). Research on problem-based learning: Future challenges. Medical Education, 47, 214–218.CrossRef
  Dolmans, D., Gijselaers, W. H., Moust, J. H. C., de Grave, W. S., Wolfhagen, H. A. P., & van der Vleuten, C. P. M. (2002). Trends in research on the tutor in PBL: Conclusions and implications for educational practice and research. Medical Education, 24, 173–180.
  Elizondo-Montemayor, L. L. (2004). Formative and summative assessment of the problem-based learning tutorial session using a criterion-referenced system. Journal of the International Association of Medical Science Educators, 14, 8–14.
  Gijbels, D., Dochy, F., van den Bossche, P., & Segers, M. (2005). Effects of problem-based learning: A meta-analysis from the angle of assessment. Review of Educational Research, 75, 27–61.CrossRef
  Gijbels, D., van den Bossche, P., & Loyens, S. (2012). Student achievement in problem-based learning. In J. A. C. Hattie & E. M. Anderman (Eds.), International guide to student achievement (pp. 382–384). New York, NY: Routledge.
  Hak, T., & Maguire, P. (2000). Group process: The black box of studies on problem-based learning. Academic Medicine, 75, 769–772.CrossRef
  Hébert, R., & Bravo, G. (1996). Development and validation of an evaluation instrument for medical students in tutorials. Academic Medicine, 71, 488–494.CrossRef
  Kilroy, D. A. (2004). Problem based learning. Emergency Medicine Journal, 21, 411–413.CrossRef
  Kinkade, S. (2005). A snapshot of the status of problem-based learning in US medical schools, 2003–2004. Academic Medicine, 80, 300–301.CrossRef
  Koenig, J. A., Sireci, S. G., & Wiley, A. (1998). Evaluating the predictive validity of MCAT scores across diverse applicant groups. Academic Medicine, 73, 1095–1106.CrossRef
  Marzano, R. J., Pickering, D., & McTighe, J. (1993). Assessing student outcomes. Alexandria, VA: Association for Supervision and Curriculum Development.
  Menard, S. (1995). Applied logistic regression analysis. Sage university paper series on quantitative applications in the social sciences, 07-106. Thousand Oaks, CA: Sage.
  Myers, R. (1990). Classical and modern regression with applications (2nd ed.). Boston, MA: Duxbury.
  Nijhuis, J., Segers, M., & Gijselaers, W. (2008). The extent of variability in learning strategies and students’ perceptions of the learning environment. Learning and Instruction, 18, 121–134.CrossRef
  Norman, G. R., & Schmidt, H. G. (1992). The psychological basis of problem-based learning: A review of the evidence. Academic Medicine, 73, 1068–1071.
  Norman, G. R., & Schmidt, H. G. (2000). Effectiveness of problem-based learning curricula: Theory, practice and paper darts. Medical Education, 34, 721–728.CrossRef
  Painvin, C., Neufeld, V., Norman, G., Walker, I., & Wheelan, G. (1979). The “triple jump” exercise—A structured measure of problem solving and self-directed learning. Annual Conference on Research in Medical Education. Conference Proceedings, 18, 73–77.
  Schmidt, H. G., Rotgans, J. I., & Yew, E. H. J. (2011). The process of [problem-based learning: What works and why. Medical Education, 45, 792–806.CrossRef
  Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215–232.CrossRef
  Shavelson, R. J., Ruiz-Primo, M. A., & Wiley, E. W. (1999). Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36, 61–71.CrossRef
  Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
  Sim, S., Azila, N. M. A., Lian, L., Tan, C. P. L., & Tan, N. (2006). A simple instrument for the assessment of student performance in problem-based learning tutorials. Annals Academy of Medicine Singapore, 35, 634–641.
  Strobel, J., & van Barneveld, A. (2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem Based Learning, 3, 44–58.CrossRef
  Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Science Education, 18, 291–303.CrossRef
  Ten Cate, T. J., Kusurkar, R. A., & Williams, G. C. (2011). How self-determination theory can assist our understanding of the teaching and learning processes in medical education. AMEE Guide No. 59. Medical Teacher, 33, 961–973.CrossRef
  Valle, R., Petra, I., Martinez-Gonzalez, A., Rojas-Ramirez, J., Morales-Lopez, S., & Pina-Garza, B. (1999). Assessment of student performance in problem-based learning tutorial sessions. Medical Education, 33, 818–822.CrossRef
  Vernon, D. T. A., & Blake, R. L. (1993). Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine, 68, 550–563.CrossRef
  Walker, A., & Leary, H. (2009). A problem-based learning meta-analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem Based Learning, 3, 12–43.CrossRef
  Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006). Reliability coefficients and generalizability theory. Handbook of Statistics, 26, 1–44.CrossRef
  Williams, R., Klamen, D., & McGaghie, W. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15, 270–292.CrossRef
  Wimmers, P. F., & Lee, M. (2015). Identifying longitudinal growth trajectories of learning domains in problem-based learning: A latent growth curve modeling approach using SEM. Advances in Health Sciences Education, 20, 467–478.CrossRef
  
• 作者单位：Ming Lee (1)
  Paul F. Wimmers (1)
  
  1. Center for Educational Development and Research, David Geffen School of Medicine at University of California, Los Angeles, PO Box 951722, 60-051, Los Angeles, CA, USA
  
• 刊物类别：Humanities, Social Sciences and Law
• 刊物主题：Education
  Medical Education
  
• 出版者：Springer Netherlands
• ISSN：1573-1677

文摘

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60 %), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep2 = .68, ϕ = .60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < .01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.