基于现代汉语动态流通语料库的通用词汇自动提取方法研究
|
摘要
人类社会正在从工业社会迈向信息社会,信息的主要载体是自然语言,即人类彼此交流所使用的语言。自然语言处理研究如何让计算机理解人类语言并开发有关的适用系统,然而自然语言中的通用词汇是一个民族的语言系统中最常见,使用频率较高的那些词汇,无论在汉语言教学上,还是在字典的编写上,甚至在计算机信息处理上,汉语言的通用词汇范围的准确界定都有着深远的意义。在某一时段内,通用词汇既是一个相对稳定又是一个开放的集合,既是一个相对动态又是一个静态的集合;传统的统计方法以及语言学家的经验法则等等都根本无法给与通用词汇准确界定。因此把计算机应用到具体语言学的通用词汇提取领域,就更体现其应用价值和意义。
运用“语料库”的科学数据方法来研究语言已经成为必然的趋势和必要的手段。本论文选择的是基于中国主流报纸的动态流通语料库,动态性和流通性是其本质特征。动态流通语料库的“动态性”贯穿着“历时中包含着共时”和“共时中包含有历时”的语言知识变化原则。也就是说,这种语料库既可以提供语言的共时描写,也可以提供语言的历时描写。流通性体现在栏目信息尽可能多样的报纸,发行地区应尽可能多样,语料的覆盖量要足够大。
本论文主要做了以下工作:
1.原始语料领域分类(自编程序)用程序实现按照报纸语料的栏目信息,将原始语料分为10类,分类结果见表4-3。
2.原始语料格式转换(自编程序)
原始下载语料为HTML\HML网页格式,需要按/领域分类/媒体/年月分别将原始语料转化为纯文本格式语料,同时应该滤除网页格式中的垃圾信息,只保留有效的文本信息内容。转换后文件格式为xml文件。
3.文本文件切分词(引用程序)、入库(自编程序)
按领域分类/媒体/年月分别将文本文件切分成词,将切分后的文件以词语为单位导入到数据库中等待进一步处理,实验时数据库软件使用的是SQL Server7.0。
4.对其进行校对
使用自行开发的人工校对系统(java语言编写)进行检查式校对,对其上面分词中不可避免的错误进行纠正,使结果更科学更准确。
5.词汇统计
计算每个词按月“词频度”、“领域通用度”、“时间通用度”。实验时使用的软件是微软的excel 2003。
6.通用词汇提取
按照词语的年“词汇通用度Ok”降序排序,提取通用词汇表,使通用词汇表中词语的总词次能覆盖全部语料词语总词次的85-95%。
Human society is moving from the industrial society into an information society, and information is the main carrier of natural language, which is used for communicating by human being. Natural language researches how to make computers understand human language and develop the suitable system. The common vocabulary of natural language is used frequently in a national language system, whatever in Chinese language teaching, or in making a dictionary, even in the computer information processing, so the clear conception of Chinese common vocabulary has a far-reaching significance. In a certain period of time, the common vocabulary is not only a relatively closed and open set, but also a dynamic and relatively stable set. Traditional statistical methods, as well as the experience of linguists can’t give a correct conception of common vocabulary. The computer technology is applied to the extraction for common vocabulary, that is an automatic extraction for common vocabulary based on DCC, which has its value and significance.
That by using the scientific data of "Corpus" to study languages has become an inevitable trend and necessary means in the language study field. This paper is based on DCC of the mainstream newspapers in china. the dynamic and the circulation are the essential character of DCC. "The dynamic" of DCC permeates a language change rule, which is“last contains simultaneity”, and "simultaneity contains last”. In other words, it not only can provide the language description at the same time, but also can provide the language description in different time.“The circulation”of DCC is reflected in the newspaper, which has more columns, more diverse areas, and more coverage of the corpus.
Main contents in this paper:
1.The classification of the original corpus
Designing a process the author divides the corpus into 10 categories according to the different columns in the newspaper, the classification results appears in table 4-3.
2.The format conversion of the original corpus
The format of original corpus is HTML \ HML, and it should be transformed into a XML file which has its own field classification, its own media, year and month. Meanwhile clean the useless information in the format of the Web and only retain the effective information content. After the conversion, the format of document is XML.
3.The segmentation, depositing of the text file into the database
The author cuts the word text file into the segmentation by the field classified / media / year and month and puts the segmentation whose unit is word into the database for further processing, the database software used in the experiments is SQL Server7.0
4.Check
Using self-developed artificial proofing system (developed by java language), the author checks and corrects the inevitable mistakes in above procedures, lets results much more scientific and more accurate.
5.The statistics of vocabulary
Calculate the "the frequency" "the usage" and " the circulation" of each word in a month. The software used in the experience is Microsoft excel 2003.
6.Extraction of the common vocabulary
Putting the vocabulary in descending order according to "the common vocabulary usage Ok" in a year the author extracts the common vocabulary; the words can cover 85-95% of all the words in the corpus terms.
运用“语料库”的科学数据方法来研究语言已经成为必然的趋势和必要的手段。本论文选择的是基于中国主流报纸的动态流通语料库,动态性和流通性是其本质特征。动态流通语料库的“动态性”贯穿着“历时中包含着共时”和“共时中包含有历时”的语言知识变化原则。也就是说,这种语料库既可以提供语言的共时描写,也可以提供语言的历时描写。流通性体现在栏目信息尽可能多样的报纸,发行地区应尽可能多样,语料的覆盖量要足够大。
本论文主要做了以下工作:
1.原始语料领域分类(自编程序)用程序实现按照报纸语料的栏目信息,将原始语料分为10类,分类结果见表4-3。
2.原始语料格式转换(自编程序)
原始下载语料为HTML\HML网页格式,需要按/领域分类/媒体/年月分别将原始语料转化为纯文本格式语料,同时应该滤除网页格式中的垃圾信息,只保留有效的文本信息内容。转换后文件格式为xml文件。
3.文本文件切分词(引用程序)、入库(自编程序)
按领域分类/媒体/年月分别将文本文件切分成词,将切分后的文件以词语为单位导入到数据库中等待进一步处理,实验时数据库软件使用的是SQL Server7.0。
4.对其进行校对
使用自行开发的人工校对系统(java语言编写)进行检查式校对,对其上面分词中不可避免的错误进行纠正,使结果更科学更准确。
5.词汇统计
计算每个词按月“词频度”、“领域通用度”、“时间通用度”。实验时使用的软件是微软的excel 2003。
6.通用词汇提取
按照词语的年“词汇通用度Ok”降序排序,提取通用词汇表,使通用词汇表中词语的总词次能覆盖全部语料词语总词次的85-95%。
Human society is moving from the industrial society into an information society, and information is the main carrier of natural language, which is used for communicating by human being. Natural language researches how to make computers understand human language and develop the suitable system. The common vocabulary of natural language is used frequently in a national language system, whatever in Chinese language teaching, or in making a dictionary, even in the computer information processing, so the clear conception of Chinese common vocabulary has a far-reaching significance. In a certain period of time, the common vocabulary is not only a relatively closed and open set, but also a dynamic and relatively stable set. Traditional statistical methods, as well as the experience of linguists can’t give a correct conception of common vocabulary. The computer technology is applied to the extraction for common vocabulary, that is an automatic extraction for common vocabulary based on DCC, which has its value and significance.
That by using the scientific data of "Corpus" to study languages has become an inevitable trend and necessary means in the language study field. This paper is based on DCC of the mainstream newspapers in china. the dynamic and the circulation are the essential character of DCC. "The dynamic" of DCC permeates a language change rule, which is“last contains simultaneity”, and "simultaneity contains last”. In other words, it not only can provide the language description at the same time, but also can provide the language description in different time.“The circulation”of DCC is reflected in the newspaper, which has more columns, more diverse areas, and more coverage of the corpus.
Main contents in this paper:
1.The classification of the original corpus
Designing a process the author divides the corpus into 10 categories according to the different columns in the newspaper, the classification results appears in table 4-3.
2.The format conversion of the original corpus
The format of original corpus is HTML \ HML, and it should be transformed into a XML file which has its own field classification, its own media, year and month. Meanwhile clean the useless information in the format of the Web and only retain the effective information content. After the conversion, the format of document is XML.
3.The segmentation, depositing of the text file into the database
The author cuts the word text file into the segmentation by the field classified / media / year and month and puts the segmentation whose unit is word into the database for further processing, the database software used in the experiments is SQL Server7.0
4.Check
Using self-developed artificial proofing system (developed by java language), the author checks and corrects the inevitable mistakes in above procedures, lets results much more scientific and more accurate.
5.The statistics of vocabulary
Calculate the "the frequency" "the usage" and " the circulation" of each word in a month. The software used in the experience is Microsoft excel 2003.
6.Extraction of the common vocabulary
Putting the vocabulary in descending order according to "the common vocabulary usage Ok" in a year the author extracts the common vocabulary; the words can cover 85-95% of all the words in the corpus terms.
引文
[1]张普.《关于动态语言知识更新与流通度研究》[J].《语言文字应用》,2000(4).
[2]于根元等.《动态:语言的本质(下)》[J].语文建设,1997(9).
[3]张普.《信息处理用动态语言知识更新的总体思考》[J].语言文字应用,2000(2).
[4]黎锦熙 国文学会丛刊 [M] 1992(1)
[5]王畛.语言统计学的重要性[J].外语教学与研究,1962(1)
[6]郑林曦.普通话三千常用词表(增订本)[M].语文出版社,1987.
[7]这七个动态性频度统计词典、词表和字表是:《现代汉语频率词典》(北京语言学院)、《现代汉语常用词词频典》(北京航空航天大学)、《中小学汉语教学用词表》(北京师范大学)、《现代汉语常用词库》(山东大学)、《信息处理现代汉语常用词表》(国家标准,北京航空航天大学)、《北京口语调查》(草稿,北京语言大学)、《现代汉语常用字表》(国家语委与国家教委)。
[8]杨奔.20 世纪的现代汉语词汇统计研究[J].玉林师范学院学报(哲学社会科学),2002(1):2-3.
[9]苏新春.国家语委“通用语料库·核心库”的词表提取及词汇构成分析[J].江苏大学学报(社科),2007(1).
[10]周祖谟.汉语词汇讲话[M].北京:人民教育出版社,1959.
[11]王铁琨. 中国语言生活状况报告( 2005) [ R ]. 北京:商务印书馆, 2006.
[12]徐婷. 现代汉语词汇系统中的行业词研究———基于“通用语料库”的定量研究[D ]. 厦门:厦门大学中文系, 2006.
[13]Jenny Thomas Mick Short. 《用语料库研究语言》[M].北京:商务印书馆,2001.
[14]教育部语言文字应用研究所. 国家语委现代汉语语料库介绍[ EB /OL ]. [ 2007 - 01 - 01 ]. http: ∥219. 238. 40. 213: 8080.
[15]张普《关于控制论与动态语言知识更新的思考》,《语言文字应用》2001 年 11 月第 4 期:79-80.
[16]张普《关于语感与流通度的思考》[J].《语言教学与研究》,1999(2).
[17]韩秀娟.基于动态流通语料库的通用词语用字研究及字词语关系考察[D].北京:北京语言大学,2007:26-28.
[18]冯志伟 《计算语言学基础》[M].北京:商务印书馆, 2001.
[19]Christopher D.Manning,Hinrich Schutze.《统计自然语言处理基础》[M]碗春法,李庆中等译.北京:电子工业出版社.2005.
[20]赵小兵.《基于 DCC 基本词汇自动识别与提取方法的研究》[D]2007 北京:北京语言大学,2007:51-52.
[21]赵小兵.《基于 DCC 基本词汇自动识别与提取方法的研究》[D]2007 北京:北京语言大学,2007:54-55.
[22]赵小兵.《基于 DCC 基本词汇自动识别与提取方法的研究》[D]2007 北京:北京语言大学,2007:58-60.
[23]张普.《关于语感与流通度的思考》[J].语言教学与研究,1999(2).
[24]黄昌宁.《关于处理大规模真实文本的谈话》[J].语言文字应用,1993(2).
[25]张普.《关于语感与流通度的思考》[J].语言教学与研究,1999(2).
[26]R.Khare,A.Rifkin.XML:A Door to Automated Web Applications[J].IEEE Internet Computing,July-August 1997;1(4):78-87.
[27]计磊,李里,周伟编著.精通 J2EE Eclipse、Struts、Hibernate、Spring 整合应用案例[M].北京:人民邮电出版社.2006.8.:47-48.
[28]HTML4.01SpecificationW3CRecommendation24December1999[R].http://www.w3.org/TR/1999/REC-html401-19991224.
[29]郑阿奇.Java 实用教程[M].北京:电子工业出版社.2005.4.:42-43.
[30]王小铭 软件工程辅导与提高[M] 北京:清华大学出版社 2004.2.第一版:29-32
[31]郎波 java 语言程序设计[M] 北京:清华大学出版社,2005:24-32.
[2]于根元等.《动态:语言的本质(下)》[J].语文建设,1997(9).
[3]张普.《信息处理用动态语言知识更新的总体思考》[J].语言文字应用,2000(2).
[4]黎锦熙 国文学会丛刊 [M] 1992(1)
[5]王畛.语言统计学的重要性[J].外语教学与研究,1962(1)
[6]郑林曦.普通话三千常用词表(增订本)[M].语文出版社,1987.
[7]这七个动态性频度统计词典、词表和字表是:《现代汉语频率词典》(北京语言学院)、《现代汉语常用词词频典》(北京航空航天大学)、《中小学汉语教学用词表》(北京师范大学)、《现代汉语常用词库》(山东大学)、《信息处理现代汉语常用词表》(国家标准,北京航空航天大学)、《北京口语调查》(草稿,北京语言大学)、《现代汉语常用字表》(国家语委与国家教委)。
[8]杨奔.20 世纪的现代汉语词汇统计研究[J].玉林师范学院学报(哲学社会科学),2002(1):2-3.
[9]苏新春.国家语委“通用语料库·核心库”的词表提取及词汇构成分析[J].江苏大学学报(社科),2007(1).
[10]周祖谟.汉语词汇讲话[M].北京:人民教育出版社,1959.
[11]王铁琨. 中国语言生活状况报告( 2005) [ R ]. 北京:商务印书馆, 2006.
[12]徐婷. 现代汉语词汇系统中的行业词研究———基于“通用语料库”的定量研究[D ]. 厦门:厦门大学中文系, 2006.
[13]Jenny Thomas Mick Short. 《用语料库研究语言》[M].北京:商务印书馆,2001.
[14]教育部语言文字应用研究所. 国家语委现代汉语语料库介绍[ EB /OL ]. [ 2007 - 01 - 01 ]. http: ∥219. 238. 40. 213: 8080.
[15]张普《关于控制论与动态语言知识更新的思考》,《语言文字应用》2001 年 11 月第 4 期:79-80.
[16]张普《关于语感与流通度的思考》[J].《语言教学与研究》,1999(2).
[17]韩秀娟.基于动态流通语料库的通用词语用字研究及字词语关系考察[D].北京:北京语言大学,2007:26-28.
[18]冯志伟 《计算语言学基础》[M].北京:商务印书馆, 2001.
[19]Christopher D.Manning,Hinrich Schutze.《统计自然语言处理基础》[M]碗春法,李庆中等译.北京:电子工业出版社.2005.
[20]赵小兵.《基于 DCC 基本词汇自动识别与提取方法的研究》[D]2007 北京:北京语言大学,2007:51-52.
[21]赵小兵.《基于 DCC 基本词汇自动识别与提取方法的研究》[D]2007 北京:北京语言大学,2007:54-55.
[22]赵小兵.《基于 DCC 基本词汇自动识别与提取方法的研究》[D]2007 北京:北京语言大学,2007:58-60.
[23]张普.《关于语感与流通度的思考》[J].语言教学与研究,1999(2).
[24]黄昌宁.《关于处理大规模真实文本的谈话》[J].语言文字应用,1993(2).
[25]张普.《关于语感与流通度的思考》[J].语言教学与研究,1999(2).
[26]R.Khare,A.Rifkin.XML:A Door to Automated Web Applications[J].IEEE Internet Computing,July-August 1997;1(4):78-87.
[27]计磊,李里,周伟编著.精通 J2EE Eclipse、Struts、Hibernate、Spring 整合应用案例[M].北京:人民邮电出版社.2006.8.:47-48.
[28]HTML4.01SpecificationW3CRecommendation24December1999[R].http://www.w3.org/TR/1999/REC-html401-19991224.
[29]郑阿奇.Java 实用教程[M].北京:电子工业出版社.2005.4.:42-43.
[30]王小铭 软件工程辅导与提高[M] 北京:清华大学出版社 2004.2.第一版:29-32
[31]郎波 java 语言程序设计[M] 北京:清华大学出版社,2005:24-32.