Poor inter-rater reliability on mock anesthesia oral examinations

详细信息    查看全文

• 作者：Eric Jacobsohn (1)
  P. Alan Klock (2)
  Michael Avidan (1)
  
• 刊名：Canadian Journal of Anesthesia/Journal canadien d'anesth篓娄sie
• 出版年：2006
• 出版时间：July 2006
• 年：2006
• 卷：53
• 期：7
• 页码：659-668
• 全文大小：215KB
• 参考文献：1. / Eagle CJ, Martineau R, Hamilton K. The oral examination in anaesthetic resident evaluation. Can J Anaesth 1993; 40: 947鈥?3.
  2. / Schubert A, Tetzlaff JE, Tan M, Ryckman JV, Mascha E. Consistency, inter-rater reliability, and validity of 441 consecutive mock oral examinations in anesthesiology: implications for use as a tool for assessment of residents. Anesthesiology 1999; 91: 288鈥?8. CrossRef
  3. / Kearney RA, Puchalski SA, Yang HY, Skakun EN. The inter-rater and intra-rater reliability of a new Canadian oral examination format in anesthesia is fair to good. Can J Anesth 2002; 49: 232鈥?.
  4. / Muzzin LJ, Hart L. Oral examinations. / In: Neufeld VR, Norman GR (Eds). Assessing Clinical Competence. New York: Springer Publishing Co.; 1985: 71鈥?3.
  5. / Burchard KW, Rowland-Morin PA, Coe NP, Garb JL. A surgery oral examination: interrater agreement and the influence of rater characteristics. Acad Med 1995; 70: 1044鈥?. CrossRef
  6. / McGuire CH. The oral examination as a measure of professional competence. J Med Educ 1966; 41: 267鈥?4. 7 / Eagle C. Anaesthesia and education. Can J Anaesth 1992; 39: 158-65. 8 / Linn BS, Zeppa R. Stress in junior medical students: relationship to personality and performance. J Med Educ 1984; 59: 7-12.
  7. / Schubert A, Tetzlaff JE, Licina M, Mascha E, Smith MP. Organization of a comprehensive anesthesiology oral practice examination program: planning, structure, startup, administration, growth, and evaluation. J Clin Anesth 1999; 11: 504鈥?8. CrossRef
  8. / Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. Med Educ 1988; 22: 325鈥?4. CrossRef
  9. / Colliver JA, Swartz MH, Robbs RS, Cohen DS. Relationship between clinical competence and interpersonal and communication skills in standardized-patient assessment. Acad Med 1999; 74: 271鈥?. CrossRef
  
• 作者单位：Eric Jacobsohn (1)
  P. Alan Klock (2)
  Michael Avidan (1)
  
  1. Departments of Anesthesiology and Surgery, Washington University School of Medicine, 660 S. Euclid MC 8054, 63110, St. Louis, MO, USA
  2. Anesthesia and Critical Care, University of Chicago, Chicago, Illinois, USA
  

文摘

Purpose Inter-rater reliability (IRR) and communication skills are both important factors that have been shown to affect oral examination scores. This study was designed to test: 1) IRR of a group of American Board of Anesthesiology (ABA) and the Royal College of Physicians and Surgeons of Canada (RCPSC) graders who graded in true isolation; 2) the effect of teaching residents examination techniques. Methods This was a randomized, pretest-posttest trial. Twenty-five residents did an initial oral examination (E1) resembling the ABA examination. They were then randomized into two groups, a routine education group, and an intervention group that was taught oral examination skills. Six weeks later they did another oral examination (E2). The videotaped examinations were subsequently scored by six experienced RCPSC and ABA graders. Results There was very poor IRR on E1 (weighted Kappa = 0.166, intraclass correlation coefficient 0.243), which improved only slightly on E2 (weighted Kappa = 0.275,P = NS; intraclass correlation coefficient = 0.405,P < 0.01). Pass rate for grader-pairs increased from E1 to E2 (15%vs 43%,P = 0.01). The improved pass rate on E2 occurred in both the routine education group and in the intervention group. There was no significant difference between RCPSC and ABA graders. Teaching examination skillsper se did not improve performance, but this conclusion may be limited by the poor IRR. Practice orals do appear to improve performance on future examinations. Conclusions Inter-rater reliability may be poor when graders score an oral examination in true isolation. Teaching candidates an oral examination communication and presentation technique did not appear to improve performance. Oral examination practice may be of value in training for future examinations.