一种基于多阶认知诊断模型测评科学素养的方法
|
摘要
科学素养是指作为一名有反思意识的公民所具有的解决科学问题和运用科学理念的能力。为实现在认知诊断中对科学素养的测评,本文基于PISA2015科学素养测评框架首次提出科学素养包含的三阶潜在结构,使用新提出的多阶认知诊断模型对PISA 2015科学测评数据进行分析,并通过模拟研究探究新模型的心理测量学性能。结果表明:(1)新模型能够较好地分析包含三阶潜在结构的科学素养;(2)科学知识对科学素养的影响最大,科学背景次之,科学能力的影响最小;(3)全贝叶斯MCMC算法能够为新模型提供较精准的参数估计。
In PISA 2015, scientific literacy is defined as "the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen". There are four interdependent dimensions are specified in the scientific literacy assessment framework for PISA 2015: Competencies, Knowledge, Contexts, and Attitudes.Given that knowledge of scientific literacy contributes significantly to individuals' personal, social, and professional lives, it is of vital importance to find an objectively and accurately assessment method for scientific literacy. However, only unidimensional IRT models were used in the analysis in PISA 2015. Which means that the analysis model does not match with such a multidimensional assessment framework. It is desired to develop a new analysis model. This study attempts to measure scientific literacy in cognitive diagnostic assessment for the first time.According to the scientific literacy assessment framework for PISA 2015, a third-order latent structure for scientific literacy is first pointed out. Specifically, the scientific literacy is treated as the third-order latent trait;Competencies, Knowledge, Contexts, and Attitudes are all treated as second-order latent traits; And nine subdomains, e.g., explain phenomena scientifically and content knowledge, were treated as first-order traits(or attributes). Unfortunately, however, there is still a lack of cognitive diagnosis models that can deal with such a third-order latent structure. To this end, a multi-order DINA(MO-DINA) model was developed in this study.The new model is an extension of the higher-order(HO-DINA) model, which is similar to the third-order IRT models. To illustrate the application and advantages of the MO-DINA model, a sub-data of PISA 2015 science assessment data were analyzed. Items were chosen from the S01 cluster, and participants were chosen from China. After data cleaning, 1076 participants with 18 items were retained. Three models were fitted to this sub-data and compared, the MO-DINA model, in which the third-order latent structure of scientific literacy was considered; the HO-DINA model, in which the scientific literacy was treated as a second-order latent trait and contacted with attributes directly; and the DINA model.All three models appear to provide a reasonably good fit to data according to the posterior predictive model checking. According to the –2 LL, AIC, BIC, and DIC, the DINA model fits the data worst, and the MO-DINA model fits the data best, the results of MO-DINA model are used to make further interpretations. The results indicated that(1) the quality of 18 items are not good enough;(2) The correlations among second-order latent traits are high(0.8, approximately);(3) Knowledge has the greatest influence on scientific literacy, Contexts second, and Competencies least;(4) Explain phenomena scientifically, procedural knowledge, and local/national has the greatest influence on Competencies, Knowledge, and Contexts, respectively. In addition, a simulation study was conducted to evaluate the psychometric properties of the proposed model. The results showed that the proposed Bayesian MCMC estimation algorithm can provide accurate model parameter estimation.Overall, the proposed MO-DINA model works well in real data analysis and simulation study and meets the needs of assessment for PISA 2015 scientific literacy which included a third-order latent structure.
In PISA 2015, scientific literacy is defined as "the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen". There are four interdependent dimensions are specified in the scientific literacy assessment framework for PISA 2015: Competencies, Knowledge, Contexts, and Attitudes.Given that knowledge of scientific literacy contributes significantly to individuals' personal, social, and professional lives, it is of vital importance to find an objectively and accurately assessment method for scientific literacy. However, only unidimensional IRT models were used in the analysis in PISA 2015. Which means that the analysis model does not match with such a multidimensional assessment framework. It is desired to develop a new analysis model. This study attempts to measure scientific literacy in cognitive diagnostic assessment for the first time.According to the scientific literacy assessment framework for PISA 2015, a third-order latent structure for scientific literacy is first pointed out. Specifically, the scientific literacy is treated as the third-order latent trait;Competencies, Knowledge, Contexts, and Attitudes are all treated as second-order latent traits; And nine subdomains, e.g., explain phenomena scientifically and content knowledge, were treated as first-order traits(or attributes). Unfortunately, however, there is still a lack of cognitive diagnosis models that can deal with such a third-order latent structure. To this end, a multi-order DINA(MO-DINA) model was developed in this study.The new model is an extension of the higher-order(HO-DINA) model, which is similar to the third-order IRT models. To illustrate the application and advantages of the MO-DINA model, a sub-data of PISA 2015 science assessment data were analyzed. Items were chosen from the S01 cluster, and participants were chosen from China. After data cleaning, 1076 participants with 18 items were retained. Three models were fitted to this sub-data and compared, the MO-DINA model, in which the third-order latent structure of scientific literacy was considered; the HO-DINA model, in which the scientific literacy was treated as a second-order latent trait and contacted with attributes directly; and the DINA model.All three models appear to provide a reasonably good fit to data according to the posterior predictive model checking. According to the –2 LL, AIC, BIC, and DIC, the DINA model fits the data worst, and the MO-DINA model fits the data best, the results of MO-DINA model are used to make further interpretations. The results indicated that(1) the quality of 18 items are not good enough;(2) The correlations among second-order latent traits are high(0.8, approximately);(3) Knowledge has the greatest influence on scientific literacy, Contexts second, and Competencies least;(4) Explain phenomena scientifically, procedural knowledge, and local/national has the greatest influence on Competencies, Knowledge, and Contexts, respectively. In addition, a simulation study was conducted to evaluate the psychometric properties of the proposed model. The results showed that the proposed Bayesian MCMC estimation algorithm can provide accurate model parameter estimation.Overall, the proposed MO-DINA model works well in real data analysis and simulation study and meets the needs of assessment for PISA 2015 scientific literacy which included a third-order latent structure.
引文
Birnbaum,A.(1968).Some latent trait models and their use in inferring a student’s ability.In F.M.Lord&M.R.Novick(Eds.).Statistical theories of mental test scores.AddisonWesley,Reading,MA.
Brooks,S.P.,&Gelman,A.(1998).General methods for monitoring convergence of iterative simulations.Journal of Computational and Graphical Statistics.7(4),434-455.
Chen,F.,Zhan,P.,Wang,L.,Chen,C.,&Cai,M.(2015).The development and application of higher-order item response models.Advances in Psychological Science,23,150-157.[陈飞鹏,詹沛达,王立君,陈春晓,蔡毛.(2015).高阶项目反应模型的发展与应用.心理科学进展,23,150-157.]
de la Torre,J.(2008).An empirically-based method of Q-matrix validation for the DINA model:Development and applications.Journal of Educational Measurement,45(4),343-362.
de la Torre,J.(2009).DINA model and parameter estimation:A didactic.Journal of Educational and Behavioral Statistics,34(1),115-130.
de la Torre,J.,&Douglas,J.A.(2004).Higher-order latent trait models for cognitive diagnosis.Psychometrika,69(3),333-353.
de la Torre,J.,&Song,H.(2009).Simultaneously estimation of overall and domain abilities:A higher-order IRT model approach.Applied Psychological Measurement,33(8),620-639.
Gao,H.B.(2011).Results of the eighth survey on Chinese citizens'scientific literacy were released.Bulletin of National Natural Science Foundation of China,25,63-64.[高宏斌.(2011).第八次中国公民科学素养调查结果发布.中国科学基金,25,63-64.]
Hu,Y.,Yang,S.,&Lu,K.(2012).The research of assessment tools of adolescents’scientific literacy and its quality analysis.Education Research Monthly,3,16-21.[胡咏梅,杨素红,卢珂.(2012).青少年科学素养测评工具研发及质量分析.教育学术月刊,3,16-21.]
Huang,H.-Y.,Wang,W.-C.,Chen,P.-H.,&Su,C.-M.(2013).Higher-order item response models for hierarchical latent traits.Applied Psychological Measurement,37(8),619-637.
Junker,B.W.,&Sijtsma,K.(2001).Cognitive assessment models with few assumptions,and connections with nonparametric item response theory.Applied Psychological Measurement,25,258-272.
Karelitz,T.M.(2004).Ordered category attribute coding framework for cognitive assessments(Unpublished doctoral dissertation).University of Illinois at Urbana-Champaign
Kang,C.,&Xin,T.(2010).New development in test theory:multidimensional item response theory.Advances in Psychological Science,18(3),530-536[康春花,辛涛.(2010).测验理论的新发展:多维项目反应理论.心理科学进展,18(3),530-536.]
K?hn,H.-F.,&Chiu,C.-Y.(2017).A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests.Psychometrika,82(1),112-132
Leighton,J.,&Gierl,M.(Eds.).(2007).Cognitive diagnostic assessment for education:Theory and applications.Cambridge University Press.
Leighton,J.P.,Gierl,M.J.,&Hunka,S.M.(2004).The attribute hierarchy method for cognitive assessment:Avariation on Tatsuoka’s rule-space approach.Journal of Educational Measurement,41(1),205-237.
Li,F.,Cohen,A.,Bottge,B,&Templin,J.(2016).A latent transition analysis model for assessing change in cognitive skills.Educational and Psychological Measurement,76(2),181-204.
Liu,H.,Liu,Y.,&Li,M.(2018).Analysis of process data of PISA 2012 computer-based problem solving:Application of the modified multilevel mixture IRT model.Frontiers in Psychology,9,1372.
Liu,K.,Li,C.(2015).The content and characteristic of PISA2015 draft science framework.Comparative Education Review,37(7),98-105.[刘克文,李川.(2015).PISA 2015科学素养测试内容及特点.比较教育研究,37(7),98-105.]
Macready,G.B.,&Dayton,C.M.(1977).The use of probabilistic models in the assessment of mastery.Journal of Educational and Behavioral Statistics,2(2),99-120.
Miller,J.D.(1983).Scientific literacy:A conceptual and empirical review.Daedalus,112(2),29-48.
OECD.(2006).Assessing Scientific,Reading and Mathematical Literacy:A Framework for PISA 2006.Paris:PISA,OECDPublishing
OECD.(2016).PISA 2015 Assessment and Analytical Framework:Science,Reading,Mathematic and Financial Literacy.Paris:PISA,OECD Publishing
OECD.(2017).PISA 2015 Technical Report.Paris:PISA,OECD Publishing
Qin,H.,&Qian,Y.(2008).A survey report on Shanghai adolescents’scientific literacy.Research in Educational Development,(24),31-35.[秦浩正,钱源伟.(2008).上海青少年科学素养调查报告.教育发展研究,(24),31-35.]
Reckase,M.D.(2009).Multidimensional item response theory.New York:Springer.
Ren,L.,Zhang,C.,&He,W.(2013).Constructing and anallysis of the model of how the factors affect the scientific literacy of Chinese citizens and a comparative investigation.Studies in Science of Science,31,983-990.[任磊,张超,何薇.(2013).中国公民科学素养及其影响因素模型的建构与分析.科学学研究,31(7),983-990.]
Rijmen,F.,Jeon,M.,von Davier,M.,&Rabe-Hesketh,S.(2014).A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys.Journal of Educational and Behavioral Statistics,39(4),235-256.
Roos,J.M.(2014).Measuring science or religion?Ameasurement analysis of the National Science Foundation sponsored science literacy scale 2006-2010.Public Understanding of Science,23(7),797-813.
Rupp,A.A.,Templin,J.,&Henson,R.A.(2010).Diagnostic Measurement:Theory,Methods,and Applications.New York,NY:Guilford Press
Ryan,J.J.,&Schnakenberg-Ott,S.D.(2003).Scoring reliability on the Wechsler Adult Intelligence Scale-Third Edition(WAIS-III).Assessment,10(2),151-159.
Templin,J.L.,Henson,R.A.,Templin,S.E.,&Roussos,L.(2008).Robustness of hierarchical modeling of skill association in cognitive diagnosis models.Applied Psychological Measurement,32(7),559-574.
The Ministry of Education of the People's Republic of China.(2017).Compulsory education primary school curriculum standards.Retrieved June 2,2017,from http://www.moe.edu.cn/srcsite/A26/s8001/201702/t20170215_296305.html[中华人民共和国教育部.(2017).义务教育小学科学课程标准.2017-06-02取自http://www.moe.edu.cn/srcsite/A26/s8001/201702/t20170215_296305.html]
Tu,D.,Cai,Y.,&Ding,S.(2012).Cognitive diagnosis:Theory,Methods,and Applications.Beijing:Beijing Normal University Publishing Group.[涂冬波,蔡艳,丁树良.(2012).认知诊断理论、方法与应用.北京:北京师范大学出版社。]
Wang,W.-C.,&Chen,P.-H.(2004).Implementation and measurement efficiency of multidimensional computerized adaptive testing.Applied Psychological Measurement,28(5),295-316.
Zhan,P.,Chen,P.,&Bian,Y.(2016).Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis.Acta Psychologica Sinica,48(10),1347-1356.[詹沛达,陈平,边玉芳.(2016).使用验证性补偿多维IRT模型进行认知诊断评估.心理学报,48(10),1347-1356.]
Zhan,P.,Jiao,H.,&Liao,D.(2018).Cognitive diagnosis modelling incorporating item response times.British Journal of Mathematical and Statistical Psychology,71(2),262-286.
Zhan,P.,Jiao,H.,Liao,D.,&Li,F.(in press).A longitudinal higher-order diagnostic classification model.Journal of Educational and Behavioral Statistics.
Zhan,P.,Jiao,H.,Liao,M.,&Bian,Y.(2018).Bayesian DINAmodeling incorporating within-item characteristic dependency.Applied Psychological Measurement.Advanced online publication.URL https://doi.org/10.1177/0146621618781594
Zhan,P.,Jiao,H.,Man,K.,&Wang,L.(in press).Using JAGS for Bayesian cognitive diagnosis modeling:Atutorial.Journal of Educational and Behavioral Statistics.
Zhan,P.,Ma,W.,Jiao,H.,&Ding,S.(in press).A sequential higher-order latent structural model for hierarchical attributes in cognitive diagnostic assessments.Applied Psychological Measurement.
Zhan,P.,Wang,W.-C.,&Li,X.(in press).A partial mastery,higher-order latent structural model for polytomous attributes in cognitive diagnostic assessments.Journal of Classification.
Zhan,P.,Wang,W.-C.,&Wang,L.(2013).Testlet response theory:an introduction and new developments.Advances in Psychological Science,21(12),2265-2280.[詹沛达,王文中,王立君.(2013).项目反应理论新进展之题组反应理论.心理科学进展,21(12),2265-2280.]
Brooks,S.P.,&Gelman,A.(1998).General methods for monitoring convergence of iterative simulations.Journal of Computational and Graphical Statistics.7(4),434-455.
Chen,F.,Zhan,P.,Wang,L.,Chen,C.,&Cai,M.(2015).The development and application of higher-order item response models.Advances in Psychological Science,23,150-157.[陈飞鹏,詹沛达,王立君,陈春晓,蔡毛.(2015).高阶项目反应模型的发展与应用.心理科学进展,23,150-157.]
de la Torre,J.(2008).An empirically-based method of Q-matrix validation for the DINA model:Development and applications.Journal of Educational Measurement,45(4),343-362.
de la Torre,J.(2009).DINA model and parameter estimation:A didactic.Journal of Educational and Behavioral Statistics,34(1),115-130.
de la Torre,J.,&Douglas,J.A.(2004).Higher-order latent trait models for cognitive diagnosis.Psychometrika,69(3),333-353.
de la Torre,J.,&Song,H.(2009).Simultaneously estimation of overall and domain abilities:A higher-order IRT model approach.Applied Psychological Measurement,33(8),620-639.
Gao,H.B.(2011).Results of the eighth survey on Chinese citizens'scientific literacy were released.Bulletin of National Natural Science Foundation of China,25,63-64.[高宏斌.(2011).第八次中国公民科学素养调查结果发布.中国科学基金,25,63-64.]
Hu,Y.,Yang,S.,&Lu,K.(2012).The research of assessment tools of adolescents’scientific literacy and its quality analysis.Education Research Monthly,3,16-21.[胡咏梅,杨素红,卢珂.(2012).青少年科学素养测评工具研发及质量分析.教育学术月刊,3,16-21.]
Huang,H.-Y.,Wang,W.-C.,Chen,P.-H.,&Su,C.-M.(2013).Higher-order item response models for hierarchical latent traits.Applied Psychological Measurement,37(8),619-637.
Junker,B.W.,&Sijtsma,K.(2001).Cognitive assessment models with few assumptions,and connections with nonparametric item response theory.Applied Psychological Measurement,25,258-272.
Karelitz,T.M.(2004).Ordered category attribute coding framework for cognitive assessments(Unpublished doctoral dissertation).University of Illinois at Urbana-Champaign
Kang,C.,&Xin,T.(2010).New development in test theory:multidimensional item response theory.Advances in Psychological Science,18(3),530-536[康春花,辛涛.(2010).测验理论的新发展:多维项目反应理论.心理科学进展,18(3),530-536.]
K?hn,H.-F.,&Chiu,C.-Y.(2017).A procedure for assessing the completeness of the Q-matrices of cognitively diagnostic tests.Psychometrika,82(1),112-132
Leighton,J.,&Gierl,M.(Eds.).(2007).Cognitive diagnostic assessment for education:Theory and applications.Cambridge University Press.
Leighton,J.P.,Gierl,M.J.,&Hunka,S.M.(2004).The attribute hierarchy method for cognitive assessment:Avariation on Tatsuoka’s rule-space approach.Journal of Educational Measurement,41(1),205-237.
Li,F.,Cohen,A.,Bottge,B,&Templin,J.(2016).A latent transition analysis model for assessing change in cognitive skills.Educational and Psychological Measurement,76(2),181-204.
Liu,H.,Liu,Y.,&Li,M.(2018).Analysis of process data of PISA 2012 computer-based problem solving:Application of the modified multilevel mixture IRT model.Frontiers in Psychology,9,1372.
Liu,K.,Li,C.(2015).The content and characteristic of PISA2015 draft science framework.Comparative Education Review,37(7),98-105.[刘克文,李川.(2015).PISA 2015科学素养测试内容及特点.比较教育研究,37(7),98-105.]
Macready,G.B.,&Dayton,C.M.(1977).The use of probabilistic models in the assessment of mastery.Journal of Educational and Behavioral Statistics,2(2),99-120.
Miller,J.D.(1983).Scientific literacy:A conceptual and empirical review.Daedalus,112(2),29-48.
OECD.(2006).Assessing Scientific,Reading and Mathematical Literacy:A Framework for PISA 2006.Paris:PISA,OECDPublishing
OECD.(2016).PISA 2015 Assessment and Analytical Framework:Science,Reading,Mathematic and Financial Literacy.Paris:PISA,OECD Publishing
OECD.(2017).PISA 2015 Technical Report.Paris:PISA,OECD Publishing
Qin,H.,&Qian,Y.(2008).A survey report on Shanghai adolescents’scientific literacy.Research in Educational Development,(24),31-35.[秦浩正,钱源伟.(2008).上海青少年科学素养调查报告.教育发展研究,(24),31-35.]
Reckase,M.D.(2009).Multidimensional item response theory.New York:Springer.
Ren,L.,Zhang,C.,&He,W.(2013).Constructing and anallysis of the model of how the factors affect the scientific literacy of Chinese citizens and a comparative investigation.Studies in Science of Science,31,983-990.[任磊,张超,何薇.(2013).中国公民科学素养及其影响因素模型的建构与分析.科学学研究,31(7),983-990.]
Rijmen,F.,Jeon,M.,von Davier,M.,&Rabe-Hesketh,S.(2014).A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys.Journal of Educational and Behavioral Statistics,39(4),235-256.
Roos,J.M.(2014).Measuring science or religion?Ameasurement analysis of the National Science Foundation sponsored science literacy scale 2006-2010.Public Understanding of Science,23(7),797-813.
Rupp,A.A.,Templin,J.,&Henson,R.A.(2010).Diagnostic Measurement:Theory,Methods,and Applications.New York,NY:Guilford Press
Ryan,J.J.,&Schnakenberg-Ott,S.D.(2003).Scoring reliability on the Wechsler Adult Intelligence Scale-Third Edition(WAIS-III).Assessment,10(2),151-159.
Templin,J.L.,Henson,R.A.,Templin,S.E.,&Roussos,L.(2008).Robustness of hierarchical modeling of skill association in cognitive diagnosis models.Applied Psychological Measurement,32(7),559-574.
The Ministry of Education of the People's Republic of China.(2017).Compulsory education primary school curriculum standards.Retrieved June 2,2017,from http://www.moe.edu.cn/srcsite/A26/s8001/201702/t20170215_296305.html[中华人民共和国教育部.(2017).义务教育小学科学课程标准.2017-06-02取自http://www.moe.edu.cn/srcsite/A26/s8001/201702/t20170215_296305.html]
Tu,D.,Cai,Y.,&Ding,S.(2012).Cognitive diagnosis:Theory,Methods,and Applications.Beijing:Beijing Normal University Publishing Group.[涂冬波,蔡艳,丁树良.(2012).认知诊断理论、方法与应用.北京:北京师范大学出版社。]
Wang,W.-C.,&Chen,P.-H.(2004).Implementation and measurement efficiency of multidimensional computerized adaptive testing.Applied Psychological Measurement,28(5),295-316.
Zhan,P.,Chen,P.,&Bian,Y.(2016).Using confirmatory compensatory multidimensional IRT models to do cognitive diagnosis.Acta Psychologica Sinica,48(10),1347-1356.[詹沛达,陈平,边玉芳.(2016).使用验证性补偿多维IRT模型进行认知诊断评估.心理学报,48(10),1347-1356.]
Zhan,P.,Jiao,H.,&Liao,D.(2018).Cognitive diagnosis modelling incorporating item response times.British Journal of Mathematical and Statistical Psychology,71(2),262-286.
Zhan,P.,Jiao,H.,Liao,D.,&Li,F.(in press).A longitudinal higher-order diagnostic classification model.Journal of Educational and Behavioral Statistics.
Zhan,P.,Jiao,H.,Liao,M.,&Bian,Y.(2018).Bayesian DINAmodeling incorporating within-item characteristic dependency.Applied Psychological Measurement.Advanced online publication.URL https://doi.org/10.1177/0146621618781594
Zhan,P.,Jiao,H.,Man,K.,&Wang,L.(in press).Using JAGS for Bayesian cognitive diagnosis modeling:Atutorial.Journal of Educational and Behavioral Statistics.
Zhan,P.,Ma,W.,Jiao,H.,&Ding,S.(in press).A sequential higher-order latent structural model for hierarchical attributes in cognitive diagnostic assessments.Applied Psychological Measurement.
Zhan,P.,Wang,W.-C.,&Li,X.(in press).A partial mastery,higher-order latent structural model for polytomous attributes in cognitive diagnostic assessments.Journal of Classification.
Zhan,P.,Wang,W.-C.,&Wang,L.(2013).Testlet response theory:an introduction and new developments.Advances in Psychological Science,21(12),2265-2280.[詹沛达,王文中,王立君.(2013).项目反应理论新进展之题组反应理论.心理科学进展,21(12),2265-2280.]