基于表层语义分析的自然手语动作序列生成的研究
摘要
近些年来,随着社会的发展,信息无障碍的研究与发展越来越受到重视,但聋人在获取信息,尤其是实时性极强的新闻类信息时,依然存在着很大障碍。如何使用计算机技术帮助聋人提高他们对信息的接收能力成为一个既有极高科研价值,又有极强实际意义的课题。
利用计算机多媒体技术并结合手语合成技术,使用卡通人通过手语来与聋人沟通,例如的手语新闻播报,已经在某些电视台使用了。这种信息传媒方式为聋人和正常人之间的信息交流提供了一个很好的沟通渠道。但是,目前手语新闻播报收视率相对较低,究其原因是聋人普遍反映看不懂。造成这种局面的原因,不仅有中国聋人手语手势标准化普及、播报中由于播音语速导致哑语手势过快、以及虚拟人面部表情单一等多种原因,还有一个很重要的原因就是目前卡通人的手语是按照正常人的语法而非聋人所使用的自然手语的语法来打手语的。这样最终造成了手语新闻让聋人,特别是文化基础差的聋人,很难看懂。
因此让卡通人按照自然手语的语法规则而非我们的现代汉语语法规则来打手语就成为了一个亟待解决的重要问题。本文的研究工作就是针对这一问题而进行的。本文研究的内容是首先整理聋人自然手语的规则并将其形式化,然后在建立自然手语形式规则库的基础上,将现代汉语的文本自动生成相应的依照自然手语规则的的手语动作序列。我们的工作取得了以下研究成果:
首先我们在对自然手语的语法研究相对匮乏的情况下建立了自然手语形式规则库。其包括:
1、从句子、短语和修饰词三个层次建立了语序系统的倒装规则体系;
2、在副词、量词等词典的基础上建立了自然手语的省略规则体系;
其次我们使用表层语义结构的分析方法获得汉语文本的表层语义结构,然后再依据自然手语形式规则库中的规则将其转换为自然手语的动作序列。
最后我们通过手语合成技术和卡通动画,搭建了一个手语的播报平台,用卡通人的动画来输出符合聋人习惯的自然手语播报动作序列。
In recent years, with the development of society, the research and development of information accessibility is attracting more and more attention, but there still exist some difficulties for the deaf in the access to information, in particular, strong real-time news information. How to use computer technology to assist the deaf to improve their ability to receive the information is a subject which has both high research value and great practical significance.
Through adopting the computer multimedia technology and sign language synthesis techniques, cartoon people use sign language to communicate with the deaf, such as sign language news broadcast has been used by some television stations. This media provides a good communication channel for the exchange of information between the deaf and the normal. At present, however, sign language news audience ratings are relatively low for the feedback from the deaf is that they can’t understand them. The reasons for causing such a situation lies not only there is no widely popularity of standard deaf sign language in China but also excessive speed of sign language in radio broadcast, as well as the single facial expression of virtual humans and the sign language of the current the cartoon are in accordance with the normal grammar rules used the normal but not by the deaf. Thus eventually lead to the deaf, especially the deaf with little foundation knowledge, difficult to understand news sign.
Therefore, it has become an urgent but essential issue for us to let the cartoon follow the grammar rules of natural sign language rather than grammar rules of modern Chinese language. This thesis mainly discusses this issue. The contents of this paper include: first of all, collect the rules of the natural deaf sign languages and formalize them; and then base on set- up natural sign language database, the version of modern Chinese can automaticly generate the corresponding sign language motion sequence in accordance with the natural sign language rules.
Our research has got the following achievements:
First of all, we have set up our natural sign language database with little referent information, including:
1. We have set up the inversion system from the three levels: sentences, phrases and modifiers.
2. We have set up natural sign language omission system based on adverbs, quantifiers, emotional words and so on.
Second, we explore the surface semantic structure analysis method to obtain the surface semantic structure of Chinese text, and then on the basis of rules of natural sign language database, we convert it to motion sequence of natural sign language.
Finally, we set up a broadcast platform with sign language to use the cartoon animation to output natural sign language broadcast motion sequence in line with the habits of the deaf through sign language technology synthesis and cartoon animation.
利用计算机多媒体技术并结合手语合成技术,使用卡通人通过手语来与聋人沟通,例如的手语新闻播报,已经在某些电视台使用了。这种信息传媒方式为聋人和正常人之间的信息交流提供了一个很好的沟通渠道。但是,目前手语新闻播报收视率相对较低,究其原因是聋人普遍反映看不懂。造成这种局面的原因,不仅有中国聋人手语手势标准化普及、播报中由于播音语速导致哑语手势过快、以及虚拟人面部表情单一等多种原因,还有一个很重要的原因就是目前卡通人的手语是按照正常人的语法而非聋人所使用的自然手语的语法来打手语的。这样最终造成了手语新闻让聋人,特别是文化基础差的聋人,很难看懂。
因此让卡通人按照自然手语的语法规则而非我们的现代汉语语法规则来打手语就成为了一个亟待解决的重要问题。本文的研究工作就是针对这一问题而进行的。本文研究的内容是首先整理聋人自然手语的规则并将其形式化,然后在建立自然手语形式规则库的基础上,将现代汉语的文本自动生成相应的依照自然手语规则的的手语动作序列。我们的工作取得了以下研究成果:
首先我们在对自然手语的语法研究相对匮乏的情况下建立了自然手语形式规则库。其包括:
1、从句子、短语和修饰词三个层次建立了语序系统的倒装规则体系;
2、在副词、量词等词典的基础上建立了自然手语的省略规则体系;
其次我们使用表层语义结构的分析方法获得汉语文本的表层语义结构,然后再依据自然手语形式规则库中的规则将其转换为自然手语的动作序列。
最后我们通过手语合成技术和卡通动画,搭建了一个手语的播报平台,用卡通人的动画来输出符合聋人习惯的自然手语播报动作序列。
In recent years, with the development of society, the research and development of information accessibility is attracting more and more attention, but there still exist some difficulties for the deaf in the access to information, in particular, strong real-time news information. How to use computer technology to assist the deaf to improve their ability to receive the information is a subject which has both high research value and great practical significance.
Through adopting the computer multimedia technology and sign language synthesis techniques, cartoon people use sign language to communicate with the deaf, such as sign language news broadcast has been used by some television stations. This media provides a good communication channel for the exchange of information between the deaf and the normal. At present, however, sign language news audience ratings are relatively low for the feedback from the deaf is that they can’t understand them. The reasons for causing such a situation lies not only there is no widely popularity of standard deaf sign language in China but also excessive speed of sign language in radio broadcast, as well as the single facial expression of virtual humans and the sign language of the current the cartoon are in accordance with the normal grammar rules used the normal but not by the deaf. Thus eventually lead to the deaf, especially the deaf with little foundation knowledge, difficult to understand news sign.
Therefore, it has become an urgent but essential issue for us to let the cartoon follow the grammar rules of natural sign language rather than grammar rules of modern Chinese language. This thesis mainly discusses this issue. The contents of this paper include: first of all, collect the rules of the natural deaf sign languages and formalize them; and then base on set- up natural sign language database, the version of modern Chinese can automaticly generate the corresponding sign language motion sequence in accordance with the natural sign language rules.
Our research has got the following achievements:
First of all, we have set up our natural sign language database with little referent information, including:
1. We have set up the inversion system from the three levels: sentences, phrases and modifiers.
2. We have set up natural sign language omission system based on adverbs, quantifiers, emotional words and so on.
Second, we explore the surface semantic structure analysis method to obtain the surface semantic structure of Chinese text, and then on the basis of rules of natural sign language database, we convert it to motion sequence of natural sign language.
Finally, we set up a broadcast platform with sign language to use the cartoon animation to output natural sign language broadcast motion sequence in line with the habits of the deaf through sign language technology synthesis and cartoon animation.
引文
[1] Lou Fant. The American Sign Language Phrase book. Chicago:the Contemporary Book, Inc. 1994, P17-73.
[2] William C. Stokoe,Jr. Sign Language Structure: An Outline of The Visual Communication Systems of the American Deaf. Journal of Deaf Studies and Deaf Education, 2005 10(1): 3-37.
[3] Ong S C W, Ranganath S. Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Transaction on Pattern Analysis and Machine Intelligence, 2005, 27(6): 873-891.
[4] Benjamin Bahan. The Licensing of Null Arguments in American Sign Language. Linguistic Inquiry, 2000, 3(1): 1-27.
[5] Lane.H, Bahan.B. Ethics of cochlear implantation in young children: A review and reply from a DEAF-WORLD Perspective. Otolaryngology—Head and Neck Surgery, 1998, 297-313.
[6] Yiqiang Chen, Wen Gao, etc. CSLDS: Chinese Sign Language Dialog System. Analysis and Modeling of Faces and Gestures, 2003.
[7] Yiqiang Chen, Wen Gao,etc.Text to Avatar in Multimodal Human Computer Interface. Asia-Pacific Human Computer Interface(APCHI2002). 2002, 1(2): 636-643
[8] Wen Gao, Yiqiang Chen, Gaolin Fang, etc. HandTalkerⅡ: a Chinese Sign Language Recognition and Synthesis System, Multimedia computing and Systems, Control, Automation, Robotics and Vision Conference, 2004.
[9] Wen Gao, Gaolin Fang, Debin Zhao,Yiqiang Chen. A Chinese sign language recognition system based on SOFM/SRN/HMM. Pattern Recgonition, 2004, 37:2389-2402
[10]姜峰,高文,姚鸿勋,陈熙霖.手势手语力效分析[J].计算机学报, 2007, 30(5): 851-860.
[11]王兆其,高文.基于虚拟人合成技术的中国手语合成方法[J].软件学报, 2002, 13: 2051-2056.
[12]陈益强,高文,刘军发,杨长水.手语合成中的多模式行为协同韵律模型[J].计算机学报, 2006, 29: 822-827.
[13]王兆其,杨长水,高文.基于VRML的中国手语三维显示[J].计算机研究与发展, 2003, 40: 459-463.
[14]吴婷婷.基于OpenGL与运动学模型的手势合成[D].上海海事大学, 2004.
[15]俞益洲.规则库块状结构的自动生成[J].计算机工程, 1991.
[16]徐琳,高文.基于规则的汉语至汉语手语的变换[J].哈尔滨工业大学学报, 1997.
[17]陈益强,周经野.基于神经网络的汉语语义网络生成[J].湘潭大学自然科学学报, 1999.
[18]高伟,郭瑾,曾碚凯.计算机手语的研究与应用[J].微计算机信息, 2002, 18.
[19]胡金柱,俞小娟,李琼,周毕吉.基于规则库和聚类分析的复句短语字段的自动识别研究[J].华中师范大学学报, 2008, 42.
[20]李向东,周清波.智能型俄汉机器翻译系统的句法规则库的设计原理[J].中文信息学报,1998, 13.
[21]国华.威廉姆·斯多基和他的手语语言学研究评介[J].中国特殊教育, 2006, 2: 35-40.
[22]兰继军,陈少毅.美国手语的语法特点及使用简介[J].中国特殊教育, 2001, 2:57-61.
[23]国华.美国手语语言学研究概述[J].中国特殊教育, 2006.
[24]李汝亚.《美国手语句法:功能语类和等级结构》述评[J].当代语言学, 2005, 7(3): 274-279.
[25]于松梅,张宁生.聋人手语的语言学研究[J].中国特殊教育, 2004, 9.
[26]梅次开.《中国手语浅谈》研读笔记[J].中国听力语言康复科学杂志, 2004, 5.
[27]冯建新.聋人与正常人汉语加工比较研究[J].中国特殊教育, 2000, 1. [28 ]傅逸亭,梅次开.聋人手语概论[M].上海:学林出版社, 1986.
[29]张茂聪,吴永玲等.中国手语与语言基础[M].山东:山东出版社,1998.
[30]冉再华.手语新闻收视的调查与思考[J].观察与思考.
[31]刘爱琴.聋人语言文字表述中的倒序现象.
[32]吴铃.汉语手语语法研究[J].中国特殊教育, 2005.
[33]吴铃.手语语法和汉语语法的比较研究[J].中国特殊教育, 2006.
[34]梅芙生.对我国聋人语言教学法的沉思[J].教育科学, 1999.
[35]杨军辉.中国手语和汉语双语教育初探[J].中国特殊教育, 2002.
[36]孙明欣,尹存燕,戴新宇,陈家俊.一种基于元规则的自然语言生成规则解释技术[J].南京大学学报, 2006, 42(1).
[37]戴目.论双语教学[J].中国听力语言康复科学杂志, 2004.
[38]邓慧兰.手语语言学与语言习得研究的关系[J].当代语言学, 2002, 4(3): 201-224.
[39]梅芙生.谈聋人手语与聋人教师的作用[J].教育科学, 2001, 3: 26-27.
[40]刘润楠.中国大陆手语语言学研究现状[J].中国特殊教育, 2005, 5: 26-29.
[41]杨军辉.中国手语和汉语双语教育初探[J].中国特殊教育, 2002, 1: 23-27.
[42]徐咏娟.重视发展聋人的母语—聋人手语[J].中国特殊教育, 2000, 2: 61-63.
[43]陈亚秋.北京地区自然手势语序特点的调查研究[J].中国特殊教育, 2006, 4: 26-30.
[44]哈平安.两种不同的手语[J].中国特殊教育, 2002, 3: 42-46.
[45]吕雪晶,王爱英.聋生自然手语的研究与思考[J].中国特殊教育, 2007, 6: 40-48.
[46]邱丽君.浅说“自然手语”[J].中国残疾人, 2007.
[47]吴铃.试论自然手语和文法手语的几个问题[J].中国特殊教育, 2005, 9: 45-49.
[48]国华.用手表达的语言[J].中国特殊教育, 2005, 9: 50-54.
[49]王静.中国聋人自然手语和古代汉语之比较[J].中国特殊教育, 2008, 2: 35-38.
[50]徐铁卫.中国手语的概念及内涵[J].中国特殊教育, 2004, 2: 9-12.
[51]顾定倩,宋晓华,于缘缘.中国手语基本词(基本动作)类型分析[J].中国特殊教育, 2005, 2: 65-72.
[52]王东.自然手语与规约手语之研究[J].中国特殊教育, 2003, 3: 33-39.
[53]崔淑云.浅谈中国手语的规范化[J].河南教育学院学报, 2007, 2: 84-86.
[54]王姣艳.从聋校学生的书面语谈其语言能力与教育对策[J].中国特殊教育, 2004, 7: 17-20.
[55]李尚生.从文化的视角探讨聋人语言教育[J].中国特殊教育, 2004, 1: 48-51.
[56]郭彩龙,张立材,孙丽娟.基于视觉的手语识别及其分类[J].科技资讯, 2008.
[57]高伟,郭瑾,曾碚凯.手语研究的方向与现状[J].电子技术应用, 2002, 11.
[58]沈玉林.手语多样性、标准化及手语语言建设的问题与思考[J].中国特殊教育, 2008, 6: 34-40.
[59]刘润楠,杨松.试论手语词汇的任意性和理据性[J].中国特殊教育, 2007, 5: 38-42.
[60]国华.自然手语习得与有声语言习得之比较及其启示[J].中国特殊教育, 2007, 3: 21-25.
[61]赖国芳.浅析“手语”的语言特性和运用[J].科技信息.
[62]陈凤芸.试论聋童汉语述宾结构的习得特征[J].中国特殊教育, 2008, 1: 50-55.
[63]孙锋.论聋人文化研究[J].中国听力语言康复科学杂志, 2007, 21.
[64]刘志军,刘晶,林萍.运用自然手语培养智障儿童、孤独症儿童沟通能力的研究[J].现代特殊教育, 2005.
[65]方俊明.聋儿的认知与综合语言教育[J].中国听力语言康复科学杂志, 2004, 6: 44-49.
[66]李英哲.“小句中枢说”在句法研究上的重要意义[J].汉语学报, 2005, 1: 52-55.
[67]国华.谈聋儿早期的手语语言习得[J].现代特殊教育, 2006.
[68]陈晓英.有关特师手语教材编写的两点思考[J].中国特殊教育, 2000, 4: 51-53.
[69]赵铁军等.机器翻译原理第1版[M].哈尔滨:哈尔滨工业大学出版社, 2000.
[2] William C. Stokoe,Jr. Sign Language Structure: An Outline of The Visual Communication Systems of the American Deaf. Journal of Deaf Studies and Deaf Education, 2005 10(1): 3-37.
[3] Ong S C W, Ranganath S. Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Transaction on Pattern Analysis and Machine Intelligence, 2005, 27(6): 873-891.
[4] Benjamin Bahan. The Licensing of Null Arguments in American Sign Language. Linguistic Inquiry, 2000, 3(1): 1-27.
[5] Lane.H, Bahan.B. Ethics of cochlear implantation in young children: A review and reply from a DEAF-WORLD Perspective. Otolaryngology—Head and Neck Surgery, 1998, 297-313.
[6] Yiqiang Chen, Wen Gao, etc. CSLDS: Chinese Sign Language Dialog System. Analysis and Modeling of Faces and Gestures, 2003.
[7] Yiqiang Chen, Wen Gao,etc.Text to Avatar in Multimodal Human Computer Interface. Asia-Pacific Human Computer Interface(APCHI2002). 2002, 1(2): 636-643
[8] Wen Gao, Yiqiang Chen, Gaolin Fang, etc. HandTalkerⅡ: a Chinese Sign Language Recognition and Synthesis System, Multimedia computing and Systems, Control, Automation, Robotics and Vision Conference, 2004.
[9] Wen Gao, Gaolin Fang, Debin Zhao,Yiqiang Chen. A Chinese sign language recognition system based on SOFM/SRN/HMM. Pattern Recgonition, 2004, 37:2389-2402
[10]姜峰,高文,姚鸿勋,陈熙霖.手势手语力效分析[J].计算机学报, 2007, 30(5): 851-860.
[11]王兆其,高文.基于虚拟人合成技术的中国手语合成方法[J].软件学报, 2002, 13: 2051-2056.
[12]陈益强,高文,刘军发,杨长水.手语合成中的多模式行为协同韵律模型[J].计算机学报, 2006, 29: 822-827.
[13]王兆其,杨长水,高文.基于VRML的中国手语三维显示[J].计算机研究与发展, 2003, 40: 459-463.
[14]吴婷婷.基于OpenGL与运动学模型的手势合成[D].上海海事大学, 2004.
[15]俞益洲.规则库块状结构的自动生成[J].计算机工程, 1991.
[16]徐琳,高文.基于规则的汉语至汉语手语的变换[J].哈尔滨工业大学学报, 1997.
[17]陈益强,周经野.基于神经网络的汉语语义网络生成[J].湘潭大学自然科学学报, 1999.
[18]高伟,郭瑾,曾碚凯.计算机手语的研究与应用[J].微计算机信息, 2002, 18.
[19]胡金柱,俞小娟,李琼,周毕吉.基于规则库和聚类分析的复句短语字段的自动识别研究[J].华中师范大学学报, 2008, 42.
[20]李向东,周清波.智能型俄汉机器翻译系统的句法规则库的设计原理[J].中文信息学报,1998, 13.
[21]国华.威廉姆·斯多基和他的手语语言学研究评介[J].中国特殊教育, 2006, 2: 35-40.
[22]兰继军,陈少毅.美国手语的语法特点及使用简介[J].中国特殊教育, 2001, 2:57-61.
[23]国华.美国手语语言学研究概述[J].中国特殊教育, 2006.
[24]李汝亚.《美国手语句法:功能语类和等级结构》述评[J].当代语言学, 2005, 7(3): 274-279.
[25]于松梅,张宁生.聋人手语的语言学研究[J].中国特殊教育, 2004, 9.
[26]梅次开.《中国手语浅谈》研读笔记[J].中国听力语言康复科学杂志, 2004, 5.
[27]冯建新.聋人与正常人汉语加工比较研究[J].中国特殊教育, 2000, 1. [28 ]傅逸亭,梅次开.聋人手语概论[M].上海:学林出版社, 1986.
[29]张茂聪,吴永玲等.中国手语与语言基础[M].山东:山东出版社,1998.
[30]冉再华.手语新闻收视的调查与思考[J].观察与思考.
[31]刘爱琴.聋人语言文字表述中的倒序现象.
[32]吴铃.汉语手语语法研究[J].中国特殊教育, 2005.
[33]吴铃.手语语法和汉语语法的比较研究[J].中国特殊教育, 2006.
[34]梅芙生.对我国聋人语言教学法的沉思[J].教育科学, 1999.
[35]杨军辉.中国手语和汉语双语教育初探[J].中国特殊教育, 2002.
[36]孙明欣,尹存燕,戴新宇,陈家俊.一种基于元规则的自然语言生成规则解释技术[J].南京大学学报, 2006, 42(1).
[37]戴目.论双语教学[J].中国听力语言康复科学杂志, 2004.
[38]邓慧兰.手语语言学与语言习得研究的关系[J].当代语言学, 2002, 4(3): 201-224.
[39]梅芙生.谈聋人手语与聋人教师的作用[J].教育科学, 2001, 3: 26-27.
[40]刘润楠.中国大陆手语语言学研究现状[J].中国特殊教育, 2005, 5: 26-29.
[41]杨军辉.中国手语和汉语双语教育初探[J].中国特殊教育, 2002, 1: 23-27.
[42]徐咏娟.重视发展聋人的母语—聋人手语[J].中国特殊教育, 2000, 2: 61-63.
[43]陈亚秋.北京地区自然手势语序特点的调查研究[J].中国特殊教育, 2006, 4: 26-30.
[44]哈平安.两种不同的手语[J].中国特殊教育, 2002, 3: 42-46.
[45]吕雪晶,王爱英.聋生自然手语的研究与思考[J].中国特殊教育, 2007, 6: 40-48.
[46]邱丽君.浅说“自然手语”[J].中国残疾人, 2007.
[47]吴铃.试论自然手语和文法手语的几个问题[J].中国特殊教育, 2005, 9: 45-49.
[48]国华.用手表达的语言[J].中国特殊教育, 2005, 9: 50-54.
[49]王静.中国聋人自然手语和古代汉语之比较[J].中国特殊教育, 2008, 2: 35-38.
[50]徐铁卫.中国手语的概念及内涵[J].中国特殊教育, 2004, 2: 9-12.
[51]顾定倩,宋晓华,于缘缘.中国手语基本词(基本动作)类型分析[J].中国特殊教育, 2005, 2: 65-72.
[52]王东.自然手语与规约手语之研究[J].中国特殊教育, 2003, 3: 33-39.
[53]崔淑云.浅谈中国手语的规范化[J].河南教育学院学报, 2007, 2: 84-86.
[54]王姣艳.从聋校学生的书面语谈其语言能力与教育对策[J].中国特殊教育, 2004, 7: 17-20.
[55]李尚生.从文化的视角探讨聋人语言教育[J].中国特殊教育, 2004, 1: 48-51.
[56]郭彩龙,张立材,孙丽娟.基于视觉的手语识别及其分类[J].科技资讯, 2008.
[57]高伟,郭瑾,曾碚凯.手语研究的方向与现状[J].电子技术应用, 2002, 11.
[58]沈玉林.手语多样性、标准化及手语语言建设的问题与思考[J].中国特殊教育, 2008, 6: 34-40.
[59]刘润楠,杨松.试论手语词汇的任意性和理据性[J].中国特殊教育, 2007, 5: 38-42.
[60]国华.自然手语习得与有声语言习得之比较及其启示[J].中国特殊教育, 2007, 3: 21-25.
[61]赖国芳.浅析“手语”的语言特性和运用[J].科技信息.
[62]陈凤芸.试论聋童汉语述宾结构的习得特征[J].中国特殊教育, 2008, 1: 50-55.
[63]孙锋.论聋人文化研究[J].中国听力语言康复科学杂志, 2007, 21.
[64]刘志军,刘晶,林萍.运用自然手语培养智障儿童、孤独症儿童沟通能力的研究[J].现代特殊教育, 2005.
[65]方俊明.聋儿的认知与综合语言教育[J].中国听力语言康复科学杂志, 2004, 6: 44-49.
[66]李英哲.“小句中枢说”在句法研究上的重要意义[J].汉语学报, 2005, 1: 52-55.
[67]国华.谈聋儿早期的手语语言习得[J].现代特殊教育, 2006.
[68]陈晓英.有关特师手语教材编写的两点思考[J].中国特殊教育, 2000, 4: 51-53.
[69]赵铁军等.机器翻译原理第1版[M].哈尔滨:哈尔滨工业大学出版社, 2000.