Process Mining of Interactions During Computer-Based Testing for Detecting and Modelling Guessing Behavior

详细信息    查看全文

• 关键词：Assessment analytics ; Educational data mining ; Guessing behavior ; Pattern recognition ; Process mining ; Student interaction analysis
• 刊名：Lecture Notes in Computer Science
• 出版年：2016
• 出版时间：2016
• 年：2016
• 卷：9753
• 期：1
• 页码：437-449
• 全文大小：800 KB
• 参考文献：1.Papamitsiou, Z., Economides, A.A.: Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Educ. Technol. Soc. 17(4), 49–64 (2014)
  2.Jordan, B., Henderson, A.: Interaction analysis: foundations and practice. J. Learn. Sci. 4(1), 39–103 (1995)CrossRef
  3.Donnelly, R.: Interaction analysis in a “Learning by Doing” problem-based professional development context. Comput. Educ. 55, 1357–1366 (2010)CrossRef
  4.Huang, S., Fang, N.: Predicting student academic performance in an engineering dynamics course: a comparison of four types of predictive mathematical models. Comput. Educ. 61, 133–145 (2013)CrossRef
  5.Macfadyen, L.P., Dawson, S.: Mining LMS data to develop an “early warning system” for educators: a proof of concept. Comput. Educ. 54(2), 588–599 (2010)CrossRef
  6.Papamitsiou, Z., Terzis, V., Economides, A.A.: Temporal learning analytics during computer based testing. In: 4th International Conference on Learning Analytics and Knowledge, pp. 31−35. ACM, New York (2014)
  7.Lee, Y.-H., Jia, Y.: Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments Educ. 2(1), 1–24 (2014)MathSciNet CrossRef
  8.Kong, X.J., Wise, S.L., Bhola, D.S.: Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behaviour. Educ. Psychol. Meas. 67(4), 606–619 (2007)MathSciNet CrossRef
  9.Baker, R.S., Corbett, A.T., Gowda, S.M., Wagner, A.Z., MacLaren, B.A., Kauffman, L.R., Mitchell, A.P., Giguere, S.: Contextual slip and prediction of student performance after use of an intelligent tutor. In: De Bra, P., Kobsa, A., Chin, D. (eds.) UMAP 2010. LNCS, vol. 6075, pp. 52–63. Springer, Heidelberg (2010)CrossRef
  10.Schnipke, D.L., Scrams, D.J.: Modeling item response times with a two-state mixture model: a new method of measuring speededness. J. Educ. Meas. 34, 213–232 (1997)CrossRef
  11.Wise, S.L., Kong, X.: Response time effort: a new measure of examinee motivation in computer-based tests. Appl. Meas. Educ. 16, 163–183 (2005)CrossRef
  12.Cao, J., Stokes, S.L.: Bayesian IRT guessing models for partial guessing behaviors. Psychometrika 73(2), 209–230 (2008)MathSciNet CrossRef MATH
  13.Baker, R.S.J.d., Corbett, A.T., Aleven, V.: Improving contextual models of guessing and slipping with a truncated training set. In: ‎Baker, R.S.J.d., Barnes, T., Beck, J.E. (eds.) 1st International Conference on Educational Data Mining, pp. 67−76 (2008)
  14.Gowda, S.M., Rowe, J.P., Baker, R.S.J.d., Chi, M., Koedinger, K.R.: Improving models of slipping, guessing and moment-by-moment learning with estimates of skill difficulty. In: Pechenizkiy, M., Calders, T., Conati, C., Ventura, S., Romero, C., and Stamper, J. (eds.) 4th International Conference on Educational Data Mining, pp. 199−208 (2011)
  15.Trčka, N., Pechenizkiy, M., van der Aalst, W.P.M.: Process mining from educational data (Chapter 9). In: Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (eds.) Handbook of Educational Data Mining. pp. 123−142. CRC Press, London (2010)
  16.van der Aalst, W.P.M.: Process Mining: Discovery, Conformance and Enhancement of Business Processes. Springer, Berlin (2011)CrossRef MATH
  17.van der Aalst, W.P.M.: Process mining: making knowledge discovery process centric. SIGKDD Explor. Newsl. 13(2), 45–49 (2012)CrossRef
  18.van Dongen, B.F., de Medeiros, A.K.A., Verbeek, H., Weijters, A., van der Aalst, W.M.: The ProM framework: a new era in process mining tool support. In: Ciardo, G., Darondeau, P. (eds.) ICATPN 2005. LNCS, vol. 3536, pp. 444–454. Springer, Heidelberg (2005)CrossRef
  19.van der Aalst, W.M.: Process mining in the large: a tutorial. In: Zimányi, E. (ed.) eBISS 2013. LNBIP, vol. 172, pp. 33–76. Springer, Heidelberg (2014)CrossRef
  20.Bannert, M., Reimann, P., Sonnenberg, C.: Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacogn. Learn. 9(2), 161–185 (2014)CrossRef
  21.Schoor, C., Bannert, M.: Exploring regulatory processes during a CSCL task using process mining. Comput. Hum. Behav. 28(4), 1321–1331 (2012)CrossRef
  22.Bergenthum, R., Desel, J., Harrer, A., Mauser, S.: Modeling and mining of learnflows. In: Jensen, K., Donatelli, S., Kleijn, J. (eds.) ToPNoC V. LNCS, vol. 6900, pp. 22–50. Springer, Heidelberg (2012)CrossRef
  23.Southavilay, V., Yacef, K., Calvo, R.A.: Process mining to support students’ collaborative writing. In: Baker, R.S.J.d., Merceron, A., Pavlik, P.I.Jr. (eds.) 3rd International Conference on Educatinal Data Mining, pp. 257−266 (2010)
  24.Calvo, R.A., O’Rourke, S.T., Jones, J., Yacef, K., Reimann, P.: Collaborative writing support tools on the cloud. IEEE Trans. Learn. Technol. 4(1), 88–97 (2011)CrossRef
  25.Pechenizkiy, M., Trčka, N., Vasilyeva, E., van der Aalst, W.M.P., de Bra, P.: Process mining online assessment data. In: Barnes, T., Desmarais, M.C., Romero, C. and Ventura, S. (eds.) 2nd International Conference on Educational Data Mining, pp. 279–288 (2009)
  26.Mukala, P., Buijs, J.C.A.M., Leemans, M., van der Aalst, W.M.P.: Learning analytics on Coursera event data: a process mining approach. In: Ceravolo, P., Rinderle-Ma, S. (eds.) 5th SIMPDA, pp. 18−32 (2015)
  27.Sedrakyan, G., De Weerdt, J., Snoeck, M.: Process-mining enabled feedback: “tell me what I did wrong” vs. “tell me how to do it right”. Comput. Hum. Behav. 57, 352–376 (2016)CrossRef
  28.Papamitsiou, Z., Economides, A.A.: A temporal estimation of students’ on-task mental effort and its effect on students’ performance during computer based testing. In: 18th International Conference on Interactive Collaborative Learning, pp. 1136–1144. IEEE (2015)
  29.Terzis, V., Economides, A.A.: The acceptance and use of computer based assessment. Comput. Educ. 56(4), 1032–1044 (2011)CrossRef
  30.van der Aalst, W.M.P.: The application of petri nets to workflow management. J. Circuit. Syst. Comput. 8(1), 21–66 (1998)CrossRef
  31.Sugar, C.A., James, G.M.: Finding the number of clusters in a dataset: an information-theoretic approach. J. Am. Stat. Assoc. 98, 750–763 (2003)MathSciNet CrossRef MATH
  32.van der Werf, J.M.E.M., Dongen, B.F., van Hurkens, C.A.J., Serebrenik, A.: Process discovery using integer linear programming. Fundam. Inform. 94(3–4), 387–412 (2009)MathSciNet MATH
  33.Bereby-Meyer, Y., Meyer, J., Flascher, O.M.: Prospect theory analysis of guessing in multiple choice tests. J. Behav. Decis. Making 15(4), 313–327 (2002)CrossRef
  34.Lamboudis, D., Economides, A.A.: Adaptive exploration of user knowledge in computer based testing. WSEAS Trans. Commun. 3(1), 322–327 (2004)
  
• 作者单位：Zacharoula Papamitsiou (15)
  Anastasios A. Economides (15)
  
  15. IPPS in Information Systems, University of Macedonia, Thessaloniki, Greece
  
• 丛书名：Learning and Collaboration Technologies
• ISBN：978-3-319-39483-1
• 刊物类别：Computer Science
• 刊物主题：Artificial Intelligence and Robotics
  Computer Communication Networks
  Software Engineering
  Data Encryption
  Database Management
  Computation by Abstract Devices
  Algorithm Analysis and Problem Complexity
  
• 出版者：Springer Berlin / Heidelberg
• ISSN：1611-3349
• 卷排序：9753

文摘

Detecting, recognizing and modelling patterns of observed examinee behaviors during assessment is a topic of great interest for the educational research community. In this paper we investigate the perspectives of process-centric inference of guessing behavior patterns. The underlying idea is to extract knowledge from real processes (i.e., not assumed nor truncated), logged automatically by the assessment environment. We applied a three-step process mining methodology on logged interaction traces from a case study with 259 undergraduate university students. The analysis revealed sequences of interactions in which low goal-orientation students answered quickly and correctly on difficult items, without reviewing them, while they submitted wrong answers on easier items. We assumed that this implies guessing behavior. From the conformance checking and performance analysis we found that the fitness of our process model is almost 85 %. Hence, initial results are encouraging towards modelling guessing behavior. Potential implications and future work plans are also discussed.