基于混合模型(Mixed-CDMs)视角的CD-CAT及其应用研究
|
摘要
传统CD-CAT通常选择一个认知诊断模型(Cognitive Diagnosis Model, CDM)标定题库参数,但在实际应用中一个CDM很难完全拟合题库中所有的题目。本文提出了一种基于混合模型(Mixed-CDMs)建立CD-CAT的方法,该方法通过Wald检验为题库中每一题目选择一个恰当的CDM,并通过模拟研究和实际数据的应用分别比较了基于传统单一CDM(G-DINA, DINA, DINO,A-CDM, LLM, RRUM)和Mixed-CDMs建立CD-CAT的效果,选题策略包括SHE和MPWKL,终止规则采用了定长的方式。结果发现:基于Mixed-CDMs建立的CD-CAT在模式判准率和题库安全性的表现要全面优于传统方法,因此本文提出的基于Mixed-CDMs的CD-CAT具有较强的理论和实用价值。
Cognitive Diagnostic Computerized Adaptive Testing(CD-CAT) combines the advantages of both cognitive diagnosis and CAT, which can make adaptive diagnosis for different individuals, and provide more detailed diagnostic information on the knowledge competence of the examinees. Currently, CD-CAT has been a promising research area and gained more and more attention.The first step in the implementation of CD-CAT is to build a high-quality item bank. One difficulty that practitioners face is that of how to select the most appropriate cognitive diagnostic model(CDM) from such a large number of models. A wide array of CDMs has been developed based on different assumptions, for example, some reduced CDMs include the Deterministic Inputs, Noisy and Gate(DINA) model, the Deterministic Inputs, Noisy or Gate(DINO) model, the Additive Cognitive Diagnostic Model(A-CDM), the Linear Logistic Model(LLM) and the Reduced Reparametrized Unified Model(RRUM). Apart from these reduced CDMs, some generalized models have also been developed, including the Generalized DINA(G-DINA) model, the General Diagnostic Model(GDM), and the Log-linear CDM(LCDM). Compared with the reduced CDMs, generalized CDMs are more complex and require a larger sample size to yield accurate estimates. In addition, compared with the complex generalized model, using reduced models can improve the accuracy of diagnostic test and lead to more straightforward and meaningful interpretations. However, almost all the research and application of CD-CAT has been conducted using only one CDM to estimate the item parameters. Analyses of real test data indicate that no single reduced model can be expected to satisfactorily fit all the items. The Wald test is developed as an item-level statistical test to examine whether the G-DINA model can be replaced by a reduced CDM without losing model data fit significantly in order to select an appropriate CDM for each item. Based on this, the study developed a new selecting model method, namely, mixed CDMs(Mixed-CDMs) method, to the construction of item bank in CD-CAT. The mixed CDMs had the advantages of both generalized model and reduced model.To explore the effectiveness of the mixed CDMs method, a simulation experiment was carried out to investigate the feasibility of the proposed mixed CDMs and whether the efficiency of CD-CAT could be reduced when using a single CDM, such as G-DINA, DINA, DINO, ACDM, LLM and RRUM, to fit data generated from mixed model. The simulation study considered a variety of factors, namely, item selection strategy(SHE and MPWKL), and the test length(10, 15, 20 and 25). The number of attributes was fixed to K = 7. An item bank of 360 items was simulated with the highest and lowest probabilities of success, P(1) and P(0), generated from uniform distributions with U(.7,.95) and U(.05,.3), respectively. The second study was to apply the mixed CDMs method to an empirical data from the depression diagnostic test. Two experimental results showed that compared with traditional method using a single CDM, the mixed CDMs method used in the construction of CD-CAT could improve both correct attribute classification rates and the utilization rate of the item bank.
Cognitive Diagnostic Computerized Adaptive Testing(CD-CAT) combines the advantages of both cognitive diagnosis and CAT, which can make adaptive diagnosis for different individuals, and provide more detailed diagnostic information on the knowledge competence of the examinees. Currently, CD-CAT has been a promising research area and gained more and more attention.The first step in the implementation of CD-CAT is to build a high-quality item bank. One difficulty that practitioners face is that of how to select the most appropriate cognitive diagnostic model(CDM) from such a large number of models. A wide array of CDMs has been developed based on different assumptions, for example, some reduced CDMs include the Deterministic Inputs, Noisy and Gate(DINA) model, the Deterministic Inputs, Noisy or Gate(DINO) model, the Additive Cognitive Diagnostic Model(A-CDM), the Linear Logistic Model(LLM) and the Reduced Reparametrized Unified Model(RRUM). Apart from these reduced CDMs, some generalized models have also been developed, including the Generalized DINA(G-DINA) model, the General Diagnostic Model(GDM), and the Log-linear CDM(LCDM). Compared with the reduced CDMs, generalized CDMs are more complex and require a larger sample size to yield accurate estimates. In addition, compared with the complex generalized model, using reduced models can improve the accuracy of diagnostic test and lead to more straightforward and meaningful interpretations. However, almost all the research and application of CD-CAT has been conducted using only one CDM to estimate the item parameters. Analyses of real test data indicate that no single reduced model can be expected to satisfactorily fit all the items. The Wald test is developed as an item-level statistical test to examine whether the G-DINA model can be replaced by a reduced CDM without losing model data fit significantly in order to select an appropriate CDM for each item. Based on this, the study developed a new selecting model method, namely, mixed CDMs(Mixed-CDMs) method, to the construction of item bank in CD-CAT. The mixed CDMs had the advantages of both generalized model and reduced model.To explore the effectiveness of the mixed CDMs method, a simulation experiment was carried out to investigate the feasibility of the proposed mixed CDMs and whether the efficiency of CD-CAT could be reduced when using a single CDM, such as G-DINA, DINA, DINO, ACDM, LLM and RRUM, to fit data generated from mixed model. The simulation study considered a variety of factors, namely, item selection strategy(SHE and MPWKL), and the test length(10, 15, 20 and 25). The number of attributes was fixed to K = 7. An item bank of 360 items was simulated with the highest and lowest probabilities of success, P(1) and P(0), generated from uniform distributions with U(.7,.95) and U(.05,.3), respectively. The second study was to apply the mixed CDMs method to an empirical data from the depression diagnostic test. Two experimental results showed that compared with traditional method using a single CDM, the mixed CDMs method used in the construction of CD-CAT could improve both correct attribute classification rates and the utilization rate of the item bank.
引文
高旭亮,汪大勋,韩雨婷,蔡艳,涂冬波.(2017).可修改答案的认知诊断计算机化自适应测验研究.心理科学,40(3),721-727.
涂冬波,蔡艳,戴海崎.(2012).基于DINA模型的Q矩阵修正方法.心理学报,44(4),558-568.
汪文义,丁树良,游晓锋.(2011).计算机化自适应诊断测验中原始题的属性标定.心理学报,43(8),964-976.
Chen,J.S.,&de la Torre,J.(2013).A general cognitive diagnosis model for expert-defined polytomous attributes.Applied Psychological Measurement,37(6),419-437.
Cheng,Y.(2009).When cognitive diagnosis meets computerized adaptive testing:CD-CAT.Psychometrika,74(4),619-632.
de la Torre,J.(2011).The generalized DINA model framework.Psychometrika,76(2),179-199.
de la Torre,J.,van der Ark,L.A.,&Rossi,G.(2017).Analysis of clinical data from a cognitive diagnosis modeling framework.Measurement and Evaluation in Counseling and Development,1-16.
Haertel,E.H.(1989).Using restricted latent class models to map the skill structure of achievement items.Journal of Educational Measurement,26(4),301-321.
Hartz,S.M.(2002).A Bayesian framework for the unified model for assessing cognitive abilities:Blending theory with practicality.Doctoral dissertation of the University of Illinois at Urbana-Champaign.
Henson,R.A.,Templin,J.L.,&Willse,J.T.(2009).Defining a family of cognitive diagnosis models using log-linear models with latent variables.Psychometrika,74(2),191-210.
Jang,E.E.(2008).A framework for cognitive diagnostic assessment.In Towards adaptive CALL:Natural language processing for diagnostic language assessment(pp.117-131).Ames,IA:Iowa State University.
Kaplan,M.,de la Torre,J.,&Barrada,J.R.(2015).New item selection methods for cognitive diagnosis computerized adaptive testing.Applied Psychological Measurement,39(3),167-188.
Kunina-Habenicht,O.,Rupp,A.A.,&Wilhelm,O.(2012).The impact of model misspecification on parameter estimation and item-fit assessment in loglinear diagnostic classification models.Journal of Educational Measurement,49(1),59-81.
Lei,P.W.,&Li,H.L.(2016).Performance of fit indices in choosing correct cognitive diagnostic models and Q-matrices.Applied Psychological Measurement,40(6),405-417.
Liu,H.Y.,You,X.F.,Wang,W.Y.,Ding,S.L.,&Chang,H.H.(2013).The development of computerized adaptive testing with cognitive diagnosis for an English achievement test in China.Journal of Classification,30(2),152-172.
Ma,W.C.,&de la Torre,J.(2016).A sequential cognitive diagnosis model for polytomous responses.British Journal of Mathematical and Statistical Psychology,69(3),253-275.
Ma,W.C.,Iaconangelo,C.,&de la Torre,J.(2016).Model similarity,model selection,and attribute classification.Applied Psychological Measurement,40(3),200-217.
Mao,X.Z.,&Xin,T.(2013).The application of the Monte Carlo approach to cognitive diagnostic computerized adaptive testing with content constraints.Applied Psychological Measurement,37(6),482-496.
Rojas,G.,de la Torre,J.,&Olea,J.(2012).Choosing between general and specific cognitive diagnosis models when the sample size is small.Paper presented at the meeting of the national council on measurement in education,Vancouver,Canada.
Templin,J.L.,&Henson,R.A.(2006).Measurement of psychological disorders using cognitive diagnosis models.Psychological Methods,11(3),287-305.
Tu,D.B.,Gao,X.L.,Wang,D.X.,&Cai,Y.(2017).A new measurement of internet addiction using diagnostic classification models.Frontiers in Psychology,8,1768.
von Davier,M.(2008).A general diagnostic model applied to language testing data.British Journal of Mathematical and Statistical Psychology,61(2),287-307.
Wang,C.(2013).Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length.Educational and Psychological Measurement,73(6),1017-1035.
Wang,C.,Chang,H.H.,&Huebner,A.(2011).Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing.Journal of Educational Measurement,48(3),255-273.
涂冬波,蔡艳,戴海崎.(2012).基于DINA模型的Q矩阵修正方法.心理学报,44(4),558-568.
汪文义,丁树良,游晓锋.(2011).计算机化自适应诊断测验中原始题的属性标定.心理学报,43(8),964-976.
Chen,J.S.,&de la Torre,J.(2013).A general cognitive diagnosis model for expert-defined polytomous attributes.Applied Psychological Measurement,37(6),419-437.
Cheng,Y.(2009).When cognitive diagnosis meets computerized adaptive testing:CD-CAT.Psychometrika,74(4),619-632.
de la Torre,J.(2011).The generalized DINA model framework.Psychometrika,76(2),179-199.
de la Torre,J.,van der Ark,L.A.,&Rossi,G.(2017).Analysis of clinical data from a cognitive diagnosis modeling framework.Measurement and Evaluation in Counseling and Development,1-16.
Haertel,E.H.(1989).Using restricted latent class models to map the skill structure of achievement items.Journal of Educational Measurement,26(4),301-321.
Hartz,S.M.(2002).A Bayesian framework for the unified model for assessing cognitive abilities:Blending theory with practicality.Doctoral dissertation of the University of Illinois at Urbana-Champaign.
Henson,R.A.,Templin,J.L.,&Willse,J.T.(2009).Defining a family of cognitive diagnosis models using log-linear models with latent variables.Psychometrika,74(2),191-210.
Jang,E.E.(2008).A framework for cognitive diagnostic assessment.In Towards adaptive CALL:Natural language processing for diagnostic language assessment(pp.117-131).Ames,IA:Iowa State University.
Kaplan,M.,de la Torre,J.,&Barrada,J.R.(2015).New item selection methods for cognitive diagnosis computerized adaptive testing.Applied Psychological Measurement,39(3),167-188.
Kunina-Habenicht,O.,Rupp,A.A.,&Wilhelm,O.(2012).The impact of model misspecification on parameter estimation and item-fit assessment in loglinear diagnostic classification models.Journal of Educational Measurement,49(1),59-81.
Lei,P.W.,&Li,H.L.(2016).Performance of fit indices in choosing correct cognitive diagnostic models and Q-matrices.Applied Psychological Measurement,40(6),405-417.
Liu,H.Y.,You,X.F.,Wang,W.Y.,Ding,S.L.,&Chang,H.H.(2013).The development of computerized adaptive testing with cognitive diagnosis for an English achievement test in China.Journal of Classification,30(2),152-172.
Ma,W.C.,&de la Torre,J.(2016).A sequential cognitive diagnosis model for polytomous responses.British Journal of Mathematical and Statistical Psychology,69(3),253-275.
Ma,W.C.,Iaconangelo,C.,&de la Torre,J.(2016).Model similarity,model selection,and attribute classification.Applied Psychological Measurement,40(3),200-217.
Mao,X.Z.,&Xin,T.(2013).The application of the Monte Carlo approach to cognitive diagnostic computerized adaptive testing with content constraints.Applied Psychological Measurement,37(6),482-496.
Rojas,G.,de la Torre,J.,&Olea,J.(2012).Choosing between general and specific cognitive diagnosis models when the sample size is small.Paper presented at the meeting of the national council on measurement in education,Vancouver,Canada.
Templin,J.L.,&Henson,R.A.(2006).Measurement of psychological disorders using cognitive diagnosis models.Psychological Methods,11(3),287-305.
Tu,D.B.,Gao,X.L.,Wang,D.X.,&Cai,Y.(2017).A new measurement of internet addiction using diagnostic classification models.Frontiers in Psychology,8,1768.
von Davier,M.(2008).A general diagnostic model applied to language testing data.British Journal of Mathematical and Statistical Psychology,61(2),287-307.
Wang,C.(2013).Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length.Educational and Psychological Measurement,73(6),1017-1035.
Wang,C.,Chang,H.H.,&Huebner,A.(2011).Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing.Journal of Educational Measurement,48(3),255-273.