Inter-rater reliability and generalizability of patient note scores using a scoring rubric based on the USMLE Step-2 CS format

详细信息    查看全文

• 作者：Yoon Soo Park ; Abbas Hyderi ; Georges Bordage…
• 关键词：Patient note ; USMLE Step ; 2 CS ; Validity ; Rater effects
• 刊名：Advances in Health Sciences Education
• 出版年：2016
• 出版时间：October 2016
• 年：2016
• 卷：21
• 期：4
• 页码：761-773
• 全文大小：428 KB
• 刊物类别：Humanities, Social Sciences and Law
• 刊物主题：Education
  Medical Education
  
• 出版者：Springer Netherlands
• ISSN：1573-1677
• 卷排序：21

文摘

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November–December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was .79 (Documentation = .63, Differential Diagnosis = .90, Justification = .48, and Workup = .54). The majority of score variance was due to case specificity (13 %) and case-task specificity (31 %), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5 %). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.