Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments

详细信息    查看全文

• 作者：Laura Germine (1) (3)
  Ken Nakayama (2)
  Bradley C. Duchaine (4)
  Christopher F. Chabris (5)
  Garga Chatterjee (6)
  Jeremy B. Wilmer (3)
  
• 关键词：Web ; based testing ; Cognition ; Visual perception ; Face recognition
• 刊名：Psychonomic Bulletin & Review
• 出版年：2012
• 出版时间：October 2012
• 年：2012
• 卷：19
• 期：5
• 页码：847-857
• 全文大小：397KB
• 参考文献：1. Australian Twin Registry. (n.d.). Australian Twin Registry, publications. Retrieved April 20, 2011, from www.twins.org.au/index.php?page=52
  2. Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “Reading the Mind in the Eyes-Test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. / Journal of Child Psychology and Psychiatry, 42, 241-51. CrossRef
  3. Birnbaum, M. H. (2000). Decision making in the lab and on the Web. In M. H. Birnbaum (Ed.), / Psychological experiments on the Internet (pp. 3-4). San Diego, CA: Academic Press. CrossRef
  4. Bowles, D. C., McKone, E., Dawel, A., Duchaine, B., Palermo, R., Schmalzl, L., et al. (2009). Diagnosing prosopagnosia: Effects of ageing, sex, and participant-stimulus ethnic match on the Cambridge Face Memory Test and Cambridge Face Perception Test. / Cognitive Neuropsychology, 26, 423-55. CrossRef
  5. Buchanan, T. (2007). Personality testing on the Internet: What we know, and what we do not. In A. N. Joinson, K. Y. McKenna, T. Postmes, & U.-D. Reips (Eds.), / The Oxford handbook of Internet psychology (pp. 447-60). New York, NY: Oxford University Press.
  6. Buchanan, T., Ali, T., Heffernan, T. M., Ling, J., Parrott, A. C., Rodgers, J., et al. (2005a). Nonequivalence of online and paper-and-pencil psychological tests: The case of the Prospective Memory Questionnaire. / Behavior Research Methods, 37, 148-54. CrossRef
  7. Buchanan, T., Johnson, J. A., & Goldberg, L. (2005b). Implementing a Five-Factor personality inventory for use on the Internet. / European Journal of Psychological Assessment, 21, 115-27. CrossRef
  8. Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. / British Journal of Psychology, 90, 125-44. CrossRef
  9. Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2010). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? / Perspectives on Psychological Science, 6, 3-. CrossRef
  10. Cohen, J. (1992). A power primer. / Psychological Bulletin, 112, 155-59. CrossRef
  11. College Board. (2012). SAT percentile ranks for males, females and total group. Retrieved January 20, 2012, from http://professionals.collegeboard.com/profdownload/sat_percentile_ranks_composite_cr_m_w.pdf
  12. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. / Psychometrika, 16, 297-34. CrossRef
  13. Duchaine, B., Germine, L., & Nakayama, K. (2007). Family resemblance: Ten family members with prosopagnosia and within-class object agnosia. / Cognitive Neuropsychology, 24, 419-30. CrossRef
  14. Duchaine, B., & Nakayama, K. (2006). The Cambridge Face Memory Test: Results for neurologically intact individuals and an investigation of its validity using inverted face stimuli and prosopagnosic participants. / Neuropsychologia, 44, 576-85. CrossRef
  15. Frey, M. C., & Detterman, D. K. (2004). Scholastic assessment or g? The relationship between the Scholastic Assessment Test and general cognitive ability. / Psychological Science, 15, 373-78. CrossRef
  16. Germine, L., Cashdollar, N., Düzel, E., & Duchaine, B. (2011a). A new selective developmental deficit: Impaired object recognition with normal face recognition. / Cortex, 47, 598-07. CrossRef
  17. Germine, L. T., Duchaine, B., & Nakayama, K. (2011b). Where cognitive development and aging meet: Face learning ability peaks after age 30. / Cognition, 118, 201-10. CrossRef
  18. Germine, L. T., & Hooker, C. I. (2011). Face emotion recognition is related to individual differences in psychosis-proneness. / Psychological Medicine, 41, 937-48. CrossRef
  19. Gosling, S., Vazire, S., Srivastava, S., & John, O. (2004). Should we trust Web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. / American Psychologist, 59, 93-04. CrossRef
  20. Haworth, C., Harlaar, N., Kovas, Y., Davis, O., Oliver, B., Hayiou-Thomas, M., et al. (2007). Internet cognitive testing of large samples needed in genetic research. / Twin Research and Human Genetics, 10, 554-63. CrossRef
  21. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Beyond WEIRD: Towards a broad-based behavioral science. / The Behavioral and Brain Sciences, 33, 111-35. CrossRef
  22. Herzmann, G., Danthiir, V., Schacht, A., Sommer, W., & Wilhelm, O. (2008). Toward a comprehensive test battery for face cognition: Assessment of the tasks. / Behavior Research Methods, 40, 840-57. doi:10.3758/BRM.40.3.840 CrossRef
  23. Kendler, K., Myers, J., Potter, J., & Opalesky, J. (2009). A Web-based study of personality, psychopathology and substance use in twin, other relative and relationship pairs. / Twin Research and Human Genetics, 12, 137-41. CrossRef
  24. Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness. / Behavior Research Methods, Instruments, & Computers, 29, 264-69. CrossRef
  25. Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. In M. H. Birnbaum (Ed.), / Psychological experiments on the Internet (pp. 35-0). San Diego, CA: Academic Press. CrossRef
  26. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs-Advisory Group on the Conduct of Research on the Internet. / American Psychologist, 59, 105-17. CrossRef
  27. McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). The integrity of Web-delivered experiments: Can you trust the data? / Psychological Science, 11, 502-06. CrossRef
  28. Nosek, B. A., Banaji, M., & Greenwald, A. G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration Web site. / Group Dynamics: Theory, Research, and Practice, 6, 101-15. CrossRef
  29. Pew Research Center. (2012a). Pew Internet, Internet adoption 1995-011. Retrieved January 20, 2012, from www.pewinternet.org/Trend-Data/Internet-Adoption.aspx
  30. Pew Research Center. (2012b). Pew Internet, Online activities, 2000-009. Retrieved January 20, 2012, from www.pewinternet.org/Trend-Data/Online-Activities-20002009.aspx
  31. Plomin, R., DeFries, J. C., McClearn, G. E., & McGuffin, P. (2008). / Behavioral genetics (5th ed.). New York, NY: W. H. Freeman.
  32. Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), / Psychological experiments on the Internet (pp. 89-17). San Diego, CA: Academic Press. CrossRef
  33. Reips, U.-D. (2002). Standards for Internet-based experimenting. / Experimental Psychology, 49, 243-56. CrossRef
  34. Reips, U.-D. (2007). The methodology of Internet-based experiments. In A. N. Joinson, K. Y. McKenna, T. Postmes, & U.-D. Reips (Eds.), / The Oxford handbook of Internet psychology (pp. 373-90). New York, NY: Oxford University Press.
  35. Ronalds, G. A., DeStavola, B. L., & Leon, D. A. (2005). The cognitive cost of being a twin: Evidence from comparisons within families in the Aberdeen children of the 1950s cohort study. / British Medical Journal, 331, 1306. CrossRef
  36. Senior, C., Phillips, M. L., Barnes, J., & David, A. S. (1999). An investigation into the perception of dominance from schematic faces: A study using the World-Wide Web. / Behavior Research Methods, 31, 341-46.
  37. Soto, C. J., John, O. P., Gosling, S. D., & Potter, J. (2011). Age differences in personality traits from 10 to 65: Big Five domains and facets in a large cross-sectional sample. / Journal of Personality and Social Psychology, 100, 330-48. CrossRef
  38. Uttl, B., Graf, P., & Richter, L. K. (2002). Verbal paired associates tests: Limits on validity and reliability. / Archives of Clinical Neuropsychology, 17, 569-83.
  39. Wechsler, D. (2008). / Wechsler Adult Intelligence Scale (4th ed.). San Antonio, TX: Pearson.
  40. Wellesley College. (n.d.). Wellesley College—Admission & financial aid—Testing. Retrieved January 20, 2012, from http://web.wellesley.edu/web/Admission/Apply/FAQ/testing.psml
  41. Wilmer, J. B., Germine, L., Chabris, C. F., Chatterjee, G., Williams, M., Loken, E., et al. (2010). Human face recognition ability is specific and highly heritable. / Proceedings of the National Academy of Sciences, 107, 5238-241. CrossRef
  
• 作者单位：Laura Germine (1) (3)
  Ken Nakayama (2)
  Bradley C. Duchaine (4)
  Christopher F. Chabris (5)
  Garga Chatterjee (6)
  Jeremy B. Wilmer (3)
  
  1. Department of Psychology, Harvard University, 768 William James Hall, 33 Kirkland Street, Cambridge, MA, 02138, USA
  3. Psychology Department, Wellesley College, 106 Central Street, Wellesley, MA, 02481, USA
  2. Department of Psychology, Harvard University, 770 William James Hall, 33 Kirkland Street, Cambridge, MA, 02138, USA
  4. Department of Psychological and Brain Sciences, Dartmouth College, Moore Hall 6207, Hanover, New Hampshire, 03755, USA
  5. Department of Psychology, Union College, 807 Union Street, Schenectady, NY, 12308, USA
  6. Department of Psychology, Harvard University, 33 Kirkland Street, Cambridge, MA, 02138, USA
  

文摘

With the increasing sophistication and ubiquity of the Internet, behavioral research is on the cusp of a revolution that will do for population sampling what the computer did for stimulus control and measurement. It remains a common assumption, however, that data from self-selected Web samples must involve a trade-off between participant numbers and data quality. Concerns about data quality are heightened for performance-based cognitive and perceptual measures, particularly those that are timed or that involve complex stimuli. In experiments run with uncompensated, anonymous participants whose motivation for participation is unknown, reduced conscientiousness or lack of focus could produce results that would be difficult to interpret due to decreased overall performance, increased variability of performance, or increased measurement noise. Here, we addressed the question of data quality across a range of cognitive and perceptual tests. For three key performance metrics—mean performance, performance variance, and internal reliability—the results from self-selected Web samples did not differ systematically from those obtained from traditionally recruited and/or lab-tested samples. These findings demonstrate that collecting data from uncompensated, anonymous, unsupervised, self-selected participants need not reduce data quality, even for demanding cognitive and perceptual experiments.