Judging Relevance Using Magnitude Estimation

详细信息    查看全文

• 作者：Eddy Maddalena (19)
  Stefano Mizzaro (19)
  Falk Scholer (20)
  Andrew Turpin (21)
  
  19. University of Udine ; Udine ; Italy
  20. RMIT University ; Melbourne ; Australia
  21. University of Melbourne ; Melbourne ; Australia
  
• 刊名：Lecture Notes in Computer Science
• 出版年：2015
• 出版时间：2015
• 年：2015
• 卷：9022
• 期：1
• 页码：215-220
• 全文大小：3,866 KB
• 参考文献：1. Eisenberg, M. (1988) Measuring relevance judgements. Information Processing and Management 24: pp. 373-389 CrossRef
  2. Gescheider, G.: Psychophysics: The Fundamentals. Lawrence Erlbaum Associates, 3rd edn. (1997)
  3. McGee, M. (2003) Usability magnitude estimation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 47: pp. 691-695 CrossRef
  4. Moskowitz, H.R. (1977) Magnitude estimation: notes on what, how, when, and why to use it. Journal of Food Quality 1: pp. 195-227 CrossRef
  5. Sormunen, E. (2002) Liberal relevance criteria of TREC: Counting on negligible documents?. 25th SIGIR. ACM, New York, pp. 324-330
  6. Spink, A., Greisdorf, H. (2001) Regions and levels: Measuring and mapping users鈥?relevance judgments. JASIST 52: pp. 161-173 CrossRef
  7. Stevens, S.S. (1966) A metric for the social consensus.. Science 151: pp. 530-541 CrossRef
  
• 作者单位：Advances in Information Retrieval
• 丛书名：978-3-319-16353-6
• 刊物类别：Computer Science
• 刊物主题：Artificial Intelligence and Robotics
  Computer Communication Networks
  Software Engineering
  Data Encryption
  Database Management
  Computation by Abstract Devices
  Algorithm Analysis and Problem Complexity
  
• 出版者：Springer Berlin / Heidelberg
• ISSN：1611-3349

文摘

Magnitude estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if magnitude estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) magnitude estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable magnitude estimation scores.