- 作者：Bj?rn Slaug ; Oliver Schilling ; Tina Helle ; Susanne Iwarsson ; Gunilla Carlsson ; ?se Br ; t
- 关键词：Interrater ; Reliability ; Agreement ; Kappa ; Methodology ; Recommendations
- 刊名：Journal of Clinical Epidemiology
- 出版年：2012
- 出版时间：September, 2012
- 年：2012
- 卷：65
- 期：9
- 页码：1016-1025
- 全文大小：216 K
Objective
The overall objective was to unfold the phenomenon of interrater agreement: to identify potential sources of variation in agreement data and to explore how they can be statistically accounted for. The ultimate aim was to propose recommendations for in-depth examination of agreement to improve the reliability of assessment instruments.Study Design and Setting
Using a sample where 10 rater pairs had assessed the presence/absence of 188 environmental barriers by a systematic rating form, a raters ¡Á items data set was generated (N = 1,880). In addition to common agreement indices, relative shares of agreement variation were calculated. Multilevel regression analysis was carried out, using rater and item characteristics as predictors of agreement variation.
Results
Following a conceptual decomposition, the agreement variation was statistically disentangled into relative shares. The raters accounted for 6-11 % , the items for 32-33 % , and the residual for 57-60 % of the variation. Multilevel regression analysis showed barrier prevalence and raters¡¯ familiarity with using standardized instruments to have the strongest impact on agreement.
Conclusion
Supported by a conceptual analysis, we propose an approach of in-depth examination of agreement variation, as a strategy for increasing the level of interrater agreement. By identifying and limiting the most important sources of disagreement, instrument reliability can be improved ultimately.