多核并行架构:多模态研究的范式、路径及领域问题之辨
|
摘要
根据库恩关于科学范式的定义,多模态研究可以视为一个类似于计算机科学中"多核并行架构"、综合多种路径的整合性研究范式。多模态概念可归结为三个:1)将其视为感官及相应的神经系统; 2)将其视为在社会文化中形成的创造意义的符号资源; 3)将其定义为人机交互中的信息呈现方式。多模态研究的路径及领域包括:1)符号学属性的研究; 2)语料库语言学研究; 3)神经科学、人机交互与学习科学中的研究。目前,多模态研究范式下的不同路径被应用于多个领域,正向探索人类多模态互动的脑机制、行为规律等基础性研究以及多模态应用性研发两个方向不断拓展。
Inspired by the definition of scientific paradigm by Thomas Kuhn,we regard multimodal study as a research paradigm integrating different approaches,which can be compared to the"Multi-core Parallel System"in Computer Science. The concept of"modality"mainly has three definitions: 1) the sense organs and the related nervous systems,2) the semiotic resources for meaning construction,and 3) the way of representing information through some physical media. As an integrated paradigm,multimodal study is consisted of several different approaches and fields,including: 1) multimodal discourse analysis rooted in semiotics,2) multimodal corpus-based study,and 3) multimodal study in neuroscience,HCI and learning science. These approaches have been applied to many different fields,focusing on both fundamental research and technological development.
Inspired by the definition of scientific paradigm by Thomas Kuhn,we regard multimodal study as a research paradigm integrating different approaches,which can be compared to the"Multi-core Parallel System"in Computer Science. The concept of"modality"mainly has three definitions: 1) the sense organs and the related nervous systems,2) the semiotic resources for meaning construction,and 3) the way of representing information through some physical media. As an integrated paradigm,multimodal study is consisted of several different approaches and fields,including: 1) multimodal discourse analysis rooted in semiotics,2) multimodal corpus-based study,and 3) multimodal study in neuroscience,HCI and learning science. These approaches have been applied to many different fields,focusing on both fundamental research and technological development.
引文
[1]Adolphs,S.&R. Carter. Spoken Corpus Linguistics:From Monomodal to Multimodal[M]. New York:Routledge,2013.
[2]Baldry,A. Multimodality and Multimediality in the Distance Learning Age[C]. Campobasso,Italy:Palladino Editore,2000.
[3]Bernsen,N. O. Multimodality theory[A]. In D. Tzovaras(ed.). Multimodal User Interfaces:From Signals to Interaction[C]. Springer Berlin Heidelberg,2008:5-30.
[4]Forceville,C.&E. Urios-Aparisi(eds.). Multimodal Metaphr[C]. Berlin:Mouton de Gruyter,2009.
[5]Gibbons,A. Multimodality,Cognition,and Experimental Literature[M].London:Routledge,2012.
[6]Gu,Y. G. Multimodal text analysis:A corpus linguistic approach to situated discourse[J]. Text&Talk,2006(26):127-167.
[7]Gu,Y. G. From real-life situated discourse to video-stream data-mining:An argument for agent-oriented modeling for multimodal corpus compilation[J]. International Journal of Corpus Linguistics,2009(14):433-466.
[8]Huang,L. H. Issues on multimodal corpus of Chinese speech acts:A case in multimodal pragmatics[J]. Digital Scholarship in the Humanities,2018(2):316-326.
[9] Jewitt,C. The Routledge Handbook of Multimodal Analysis[C]. Rougledge:London,2009.
[10]Jewitt,C.,Bezemer,J.&K. O’Halloran. Introducing Multimodality[M]. New York:Routledge,2016.
[11]Knight,D. Multimodality and Active Listenership:A Corpus Approach[M]. London and New York:Continuum,2011.
[12]Kolb,B.&I. Q. Whishaw. Fundamentals of Human Neuropsychology[M]. New York:Worth Publishers,2005.
[13]Kress,G.&T. van Leeuwen. Multimodal Discourse:The Modes and Media of Contemporary Communication[M]. London:Arnold,2001.
[14]Kress,G. A Social Semiotic Approach to Contemporary Communication[M]. Routledge:London,2010.
[15]Kuhn,T. S. The Structure of Scientific Revolutions(3rd edn.)[M]. Chicago:Te University of Chicago Press,1996.
[16]Maragos,P.,Potamianos,A.&P. Gros. Multimodal Processing and Interaction:Audio,Video,Text[C]. Springer,2008.
[17]Norris,S. Analyzing Multimodal Interaction:A Methodological Framework[M]. London:Routledge,2004.
[18]Norris,S.&R. H. Jones(eds.). Discourse in Action:Introducing Mediated Discourse Analysis[C]. London:Routledge,2005.
[19]O’Halloran,K. L. Mathematical Discourse:Language,Symbolism and Visual Image[M]. London:Continuum,2005.
[20]Scollon,R.&R. Scollon. Nexus Analysis:Discourse and the Emerging Internet[M]. London:Routledge,2004.
[21]van Kuppevelt,Jan,C. J.,Dybkjr,Laila&N. O. Bernsen. Advances in Natural Multimodal Dialogue Systems[C]. Dordrecht:Springer,2005.
[22]van Leeuwen,T. Introducing Social Semiotics[M]. London:Routledge,2005.
[23]Wahlster,W. Smart Kom:Foundations of Multimodal Dialogue Systems[M]. NJ,USA:Springer,2006.
[24]丁肇芬,张德禄.儿童话语分析的多模态意义模块建构探索——模态系统框架[J].西安外国语大学学报,2018(1):19-24.
[25]顾曰国.多媒体、多模态学习剖析[J].外语电化教学,2007(4):3-12.
[26]顾曰国.论言思情貌整一原则与鲜活话语研究[J].当代修辞学,2013(6):1-19.
[27]顾曰国.多模态感官系统与语言研究[J].当代语言学,2015(4):1-22.
[28]胡壮麟.社会符号学研究中的多模态化[J].语言教学与研究,2007(1):1-10.
[29]黄立鹤.基于多模态理论的大学英语拓展学习体系研究[J].语言教育,2014(3):11-16,21.
[30]黄立鹤.语料库4. 0:多模态语料库建设及其应用[J].解放军外国语学院学报,2015a(3):1-7,48.
[31]黄立鹤.近十年老年人语言衰老现象研究:回顾与前瞻[J].北京第二外国语学院学报,2015b(10):17-24.
[32]黄立鹤.通感、隐喻与多模态隐喻[J].语文与国际研究(中国台湾),2015c(13):1-13.
[33]黄立鹤.现场即席话语与多模态研究:意义、理论与方法[J].北京科技大学学报(社会科学版),2016(5):29-36.
[34]黄立鹤.言语行为理论与多模态研究:兼论多模态(语料库)语用学的逻辑[J].北京第二外国语学院学报,2017(2):12-30,133.
[35]黄立鹤.基于多模态语料库的语力研究:多模态语用学新探索[M].上海:上海外语教育出版社,2018a.
[36]黄立鹤.多模态修辞学的构建与研究——兼论修辞学与语用学的连接[J].当代语言学,2018b(1):117-132.
[37]黄立鹤,何继红.从《Routledge多模态分析手册》看多模态语言研究[J].语言教育,2013(1):92-95,封三.
[38]陶建华,等.多模态融合的人机对话系统[J].中国计算机学会通讯,2011(11):30-39.
[39]王正,张德禄.基于语料库的多模态语类研究——以期刊封面语类为类[J].外语教学,2016(5):15-20.
[40]张德禄.多模态学习能力培养模式探索[J].外语研究,2012(2):9-14.
[41]张德禄.多模态论辩修辞框架探索[J].当代修辞学,2017(1):1-8.
[42]张德禄,王璐.多模态话语模态的协同及在外语教学中的体现[J].外语学刊,2010(2):97-102.
[43]赵秀凤.多模态认知诗学研究——认知诗学研究的新进展[J].外国语文,2013(6):43-51.
(1)参考《多核多线程技术》,英特尔软件学院教材编写组.上海:上海交通大学出版社,2011:10-20.
(2)关于现场即席话语的概念及其与多模态研究的关系,可参考黄立鹤(2016)。
[2]Baldry,A. Multimodality and Multimediality in the Distance Learning Age[C]. Campobasso,Italy:Palladino Editore,2000.
[3]Bernsen,N. O. Multimodality theory[A]. In D. Tzovaras(ed.). Multimodal User Interfaces:From Signals to Interaction[C]. Springer Berlin Heidelberg,2008:5-30.
[4]Forceville,C.&E. Urios-Aparisi(eds.). Multimodal Metaphr[C]. Berlin:Mouton de Gruyter,2009.
[5]Gibbons,A. Multimodality,Cognition,and Experimental Literature[M].London:Routledge,2012.
[6]Gu,Y. G. Multimodal text analysis:A corpus linguistic approach to situated discourse[J]. Text&Talk,2006(26):127-167.
[7]Gu,Y. G. From real-life situated discourse to video-stream data-mining:An argument for agent-oriented modeling for multimodal corpus compilation[J]. International Journal of Corpus Linguistics,2009(14):433-466.
[8]Huang,L. H. Issues on multimodal corpus of Chinese speech acts:A case in multimodal pragmatics[J]. Digital Scholarship in the Humanities,2018(2):316-326.
[9] Jewitt,C. The Routledge Handbook of Multimodal Analysis[C]. Rougledge:London,2009.
[10]Jewitt,C.,Bezemer,J.&K. O’Halloran. Introducing Multimodality[M]. New York:Routledge,2016.
[11]Knight,D. Multimodality and Active Listenership:A Corpus Approach[M]. London and New York:Continuum,2011.
[12]Kolb,B.&I. Q. Whishaw. Fundamentals of Human Neuropsychology[M]. New York:Worth Publishers,2005.
[13]Kress,G.&T. van Leeuwen. Multimodal Discourse:The Modes and Media of Contemporary Communication[M]. London:Arnold,2001.
[14]Kress,G. A Social Semiotic Approach to Contemporary Communication[M]. Routledge:London,2010.
[15]Kuhn,T. S. The Structure of Scientific Revolutions(3rd edn.)[M]. Chicago:Te University of Chicago Press,1996.
[16]Maragos,P.,Potamianos,A.&P. Gros. Multimodal Processing and Interaction:Audio,Video,Text[C]. Springer,2008.
[17]Norris,S. Analyzing Multimodal Interaction:A Methodological Framework[M]. London:Routledge,2004.
[18]Norris,S.&R. H. Jones(eds.). Discourse in Action:Introducing Mediated Discourse Analysis[C]. London:Routledge,2005.
[19]O’Halloran,K. L. Mathematical Discourse:Language,Symbolism and Visual Image[M]. London:Continuum,2005.
[20]Scollon,R.&R. Scollon. Nexus Analysis:Discourse and the Emerging Internet[M]. London:Routledge,2004.
[21]van Kuppevelt,Jan,C. J.,Dybkjr,Laila&N. O. Bernsen. Advances in Natural Multimodal Dialogue Systems[C]. Dordrecht:Springer,2005.
[22]van Leeuwen,T. Introducing Social Semiotics[M]. London:Routledge,2005.
[23]Wahlster,W. Smart Kom:Foundations of Multimodal Dialogue Systems[M]. NJ,USA:Springer,2006.
[24]丁肇芬,张德禄.儿童话语分析的多模态意义模块建构探索——模态系统框架[J].西安外国语大学学报,2018(1):19-24.
[25]顾曰国.多媒体、多模态学习剖析[J].外语电化教学,2007(4):3-12.
[26]顾曰国.论言思情貌整一原则与鲜活话语研究[J].当代修辞学,2013(6):1-19.
[27]顾曰国.多模态感官系统与语言研究[J].当代语言学,2015(4):1-22.
[28]胡壮麟.社会符号学研究中的多模态化[J].语言教学与研究,2007(1):1-10.
[29]黄立鹤.基于多模态理论的大学英语拓展学习体系研究[J].语言教育,2014(3):11-16,21.
[30]黄立鹤.语料库4. 0:多模态语料库建设及其应用[J].解放军外国语学院学报,2015a(3):1-7,48.
[31]黄立鹤.近十年老年人语言衰老现象研究:回顾与前瞻[J].北京第二外国语学院学报,2015b(10):17-24.
[32]黄立鹤.通感、隐喻与多模态隐喻[J].语文与国际研究(中国台湾),2015c(13):1-13.
[33]黄立鹤.现场即席话语与多模态研究:意义、理论与方法[J].北京科技大学学报(社会科学版),2016(5):29-36.
[34]黄立鹤.言语行为理论与多模态研究:兼论多模态(语料库)语用学的逻辑[J].北京第二外国语学院学报,2017(2):12-30,133.
[35]黄立鹤.基于多模态语料库的语力研究:多模态语用学新探索[M].上海:上海外语教育出版社,2018a.
[36]黄立鹤.多模态修辞学的构建与研究——兼论修辞学与语用学的连接[J].当代语言学,2018b(1):117-132.
[37]黄立鹤,何继红.从《Routledge多模态分析手册》看多模态语言研究[J].语言教育,2013(1):92-95,封三.
[38]陶建华,等.多模态融合的人机对话系统[J].中国计算机学会通讯,2011(11):30-39.
[39]王正,张德禄.基于语料库的多模态语类研究——以期刊封面语类为类[J].外语教学,2016(5):15-20.
[40]张德禄.多模态学习能力培养模式探索[J].外语研究,2012(2):9-14.
[41]张德禄.多模态论辩修辞框架探索[J].当代修辞学,2017(1):1-8.
[42]张德禄,王璐.多模态话语模态的协同及在外语教学中的体现[J].外语学刊,2010(2):97-102.
[43]赵秀凤.多模态认知诗学研究——认知诗学研究的新进展[J].外国语文,2013(6):43-51.
(1)参考《多核多线程技术》,英特尔软件学院教材编写组.上海:上海交通大学出版社,2011:10-20.
(2)关于现场即席话语的概念及其与多模态研究的关系,可参考黄立鹤(2016)。
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |