The impact of faking on Cronbach’s alpha for dichotomous and ordered rating scores

详细信息    查看全文

• 作者：Massimiliano Pastore (1)
  Luigi Lombardi (2)
  
• 关键词：Sample generation by replacement ; Cronbach’s alpha ; Faking good ; Monte Carlo
• 刊名：Quality and Quantity
• 出版年：2014
• 出版时间：May 2014
• 年：2014
• 卷：48
• 期：3
• 页码：1191-1211
• 全文大小：511 KB
• 参考文献：1. Allen, M.J., Yen, W.M.: Introduction to Measurement Theory. Brooks/Cole, Monterey (1979)
  2. Bentler, P.A., Woodward, J.A.: Inequalities among lower bounds to reliability: with applications to test construction and factor analysis. Psychometrika 45, 249-67 (1980) CrossRef
  3. Cohen, J.: Things I have learned (so far). Am. Psychol. 45, 1304-312 (1990) CrossRef
  4. Cortina, J.M.: What is coefficient alpha? An examination of theory and applications. J. Appl. Psychol. 78, 98-04 (1993) CrossRef
  5. Cronbach, L.J.: Coefficient alpha and the internal structure of tests. Psychometrika 16, 297-34 (1951) CrossRef
  6. Duhachek, A., Coughlan, A.T., Iacobucci, D.: Results on the standard error of the coefficient alpha index of reliability. Mark. Sci. 24, 294-01 (2005) CrossRef
  7. Fan, X., Felsovalyi, A., Sivo, S.A., Keenan, S.: SAS for Monte Carlo Studies: A Guide for Quantitative Researchers. SAS Institute, Inc., Cary (2002)
  8. Ferrando, P.J.: Factor analytic procedures for assessing social desiderability in binary items. Multivar. Behav. Res. 40, 331-49 (2005) CrossRef
  9. Ferrando, P.J., Anguiano-Carrasco, C.: Assessing the impact of faking on binary personality measures: an IRT-based multiple-group factor analytic procedure. Multivar. Behav. Res. 44, 497-24 (2009) CrossRef
  10. Ferrando, P.J., Anguiano-Carrasco, C.: A structural eqution model at the individual and group level for assessing faking-related change. Struct. Equ. Model. 18, 91-09 (2011) CrossRef
  11. Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley, New York (1981)
  12. Furnham, A.: Response bias, social desirability and dissimulation. Personal. Individ. Differ. 7, 385-00 (1986) CrossRef
  13. Gray, N.S., MacCulloch, M.J., Smith, J., Morris, M., Snowden, R.J.: Forensic psychology: violence viewed by psychopathic murderers. Nature 423, 497-98 (2003) CrossRef
  14. Green, S.B., Lissitz, R.W., Mulaik, S.A.: Limitations of coefficient alpha as an index of test unidimensionality. Educ. Psychol. Meas. 37, 827-38 (1977) CrossRef
  15. Guttman, L.: A basis for analyzing test–retest reliability. Psychometrika 10, 255-82 (1945) CrossRef
  16. Hall, R.C., Hall, R.C.: Detection of malingered PTSD: an overview of clinical, psychometric, and physiological assessment: where do we stand? J. Forensic Sci. 52, 717-25 (2007) CrossRef
  17. Hopwood, C.J., Talbert, C.A., Morey, L.C., Rogers, R.: Testing the incremental utility of the negative impression-positive impression differential in detecting simulated personality assessment inventory profiles. J. Clin. Psychol. 64, 338-43 (2008) CrossRef
  18. J?reskog, K., S?rbom, D.: LISREL 8: User’s Reference Guide. Scientific Software International, Inc., Lincolnwood (1996a)
  19. J?reskog, K., S?rbom, D.: PRELIS 2: User’s Reference Guide. Scientific Software International, Inc., Lincolnwood (1996b)
  20. Kaiser, H.F., Dickman, K.: Sample and population score matrices and sample correlation matrices from an arbitrary population correlation matrix. Psychometrika 27, 179-82 (1962) CrossRef
  21. Komorita, S.S., Graham, W.K.: Number of scale points and the reliability of scales. Educ. Psychol. Meas. 25, 987-95 (1965) CrossRef
  22. Leite, W.L., Cooper, L.A.: Detecting social desiderability bias using factor mixture models. Multivar. Behav. Res. 45, 271-93 (2010) CrossRef
  23. Li Gwet, K.: K.: Variance estimation of nominal-scale inter-rater reliability with random selection of raters. Psychometrika 73, 407-30 (2008) CrossRef
  24. Lissitz, R.W., Green, S.B.: Effect of the number of scale points on reliability: a Monte Carlo approach. J. Appl. Psychol. 60, 10-3 (1975) CrossRef
  25. Liu, Y., Wu, A.D., Zumbo, B.D.: The impact of outliers on Cronbachs coefficient alpha estimate of reliability: ordinal/rating scale item responses. Educ. Psychol. Meas. 70, 5-1 (2010) CrossRef
  26. Lombardi, L., Pastore, M.: Sensitivity of fit indices to fake perturbation of ordinal data: a sample by replacement approach. Multivar. Behav. Res. 47, 519-46 (2012) CrossRef
  27. Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Wesley, Reading (1968)
  28. Marshall, E.: How prevalent is fraud? That’s a million-dollar question. Science 290, 1662-663 (2000) CrossRef
  29. Maydeu-Olivares, A., Coffman, D.L., Hartmann, W.M.: Asymptotically distribution-free (ADF) interval estimation of coefficient alpha. Psychol. Methods 12, 157-76 (2007)
  30. McDonald, R.P.: Factor Analysis and Related Methods. Erlbaum, Hillsdale (1985)
  31. McFarland, L.A., Ryan, A.M.: Variance in faking across noncognitive measures. J. Appl. Psychol. 85, 812-21 (2000) CrossRef
  32. Mittenberg, W., Patton, C., Canyock, E.M., Condit, D.C.: Baserates of malingering and symptom exaggeration. J. Clin. Exp. Neuropsychol. 24, 1094-102 (2002) CrossRef
  33. Morgan, M.G., Henrion, M., Small, M.: Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, New York (1990) CrossRef
  34. Muthén, B.: A general structural equation model with dichotomous, ordered categorical and continuous latent variables indicators. Psychometrika 49, 115-32 (1984) CrossRef
  35. Novick, M.R.: The axioms and principal results of classical test theory. J. Math. Psychol. 3, 1-8 (1966) CrossRef
  36. Novick, M.R., Lewis, C.: Coefficient alpha and the reliability of composite measurements. Psychometrika 32, 1-3 (1967) CrossRef
  37. Nunnally, J., Bernstein, I.: Psychometric Theory. McGraw-Hill, New York (1994)
  38. Paulhus, D.L.: Two-component models of socially desirable responding. J. Personal. Soc. Psychol. 46, 598-09 (1984) CrossRef
  39. Paulhus, D.L.: Measurement and control of response bias. In: Robinson, J.P., Shaver, P.R., Wrightsman, L.S. (eds.) Measures of Personality and Social-Psychological Attitudes, pp. 17-9. Academic Press, New York (1991) CrossRef
  40. R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org (2012)
  41. Revelle, W.: Hierarchical cluster-analysis and the internal structure of tests. Multivar. Behav. Res. 14, 57-4 (1979) CrossRef
  42. Rice, J.A.: Mathematical Statistics and Data Analysis, 2nd edn. Duxbury Press, Belmont (1995)
  43. Robert, C.P., Casella, G.: Monte Carlo Statistical Methods, 2nd edn. Springer- Verlag, New York (2004) CrossRef
  44. Rosse, J.G., Stecher, M.D., Miller, J.L., Levin, R.A.: The impact of response distortion on preemployment personality testing and hiring decisions. J. Appl. Psychol. 83, 634-44 (1998) CrossRef
  45. Sijtsma, K.: On the use, the misuse, and the very limited uselfulness of Cronbach’s alpha. Psychometrika 74, 107-20 (2009) CrossRef
  46. Van der Geest, S., Sarkodie, S.: The fake patient: a research experiment in a Ghanaian hospital. Social Science & Medicine. Medicine 47, 1373-381 (1998)
  47. Vidotto, G., Marchesini, C.: La realizzazione professionale. Franco Angeli, Analisi delle risorse personali e dei processi decisionali per l’orientamento scolastico-professionale, Milano (2000)
  48. Weng, L.: Impact of the number of response categories and anchor labels on coefficient alpha and test–retest reliability. Educ. Psychol. Meas. 64, 956-72 (2004) CrossRef
  49. Whitby, O.: Estimation of Parameters in the Generalized Beta Distribution (Technical Report No. 29). Department of Statistics, Standford University, Stanford (1971).
  50. Woods, C.M.: Careless responding to reverse-worded items: implications for confirmatory factor analysis. J. Psychopathol. Behav. Assess. 28, 189-94 (2006)
  51. Yuan, K.-H., Bentler, P.M.: On normal theory based inference for multilevel models with distributional violations. Psychometrika 67, 539-61 (2002) CrossRef
  52. Zickar, M.J., Gibby, R.E., Robie, C.: Uncovering faking samples in applicant, incumbent, and experimental data sets: an application of mixed-model item response theory. Organ. Res. Methods 7, 168-90 (2004) CrossRef
  53. Zickar, M.J., Robie, C.: Modeling faking good on personality items: an item-level analysis. J. Appl. Psychol. 84, 551-63 (1999) CrossRef
  54. Zinbarg, R.E., Revelle, W., Yovel, I., Li, W.: Cronbach’s $\alpha $ , Revelle’s $\beta $ , and McDonald’s $\omega _H$ : their relations with each other and two alternative conceptualizations of reliability. Psychometrika 70, 123-33 (2005) CrossRef
  
• 作者单位：Massimiliano Pastore (1)
  Luigi Lombardi (2)
  
  1. Department of Developmental and Social Psychology, University of Padova, Via Venezia, 8, 35131, Padua, Italy
  2. Department of Psychology and Cognitive Science, University of Trento, Corso Bettini, 31, 38068, Rovereto, TN, Italy
  
• ISSN：1573-7845

文摘

In many psychological inventories (i.e., personnel selection surveys and diagnostic tests) the collected samples often include fraudulent records. This confronts the researcher with the crucial problem of biases yielded by the usage of standard statistical models. In this paper we applied a recent probabilistic perturbation procedure, called sample generation by replacement (SGR)-Lombardi and Pastore, Multivar. Behav. Res 47:519-46, 2012), to study the sensitivity of Cronbach’s alpha index to fake perturbations in dichotomous and ordered data, respectively. We used SGR to perform two distinct SGR simulation studies involving two sample size conditions, three item set sizes, and twenty levels of faking perturbations. Moreover, in the second SGR simulation study we also evaluated an additional factor, type of faking model, to study sample reliability under different modulations of graded faking (uniform faking, average faking, slight faking, and extreme faking). To simulate these more complex faking models we proposed a novel extension of the SGR perturbation procedure based on a discrete version of the generalized beta density distribution. We also applied the new procedure to real behavioral data on emotional instability.