藏语音节标注研究
|
摘要
藏语的"音节"在词汇语法研究和文本信息处理研究中都十分重要,尤其在解决未登录词切分问题和标注中能够发挥积极的作用。然而在现有的研究中,对音节的重视还不够。该文提出在文本标注时,可以先进行音节的性质标注,然后通过音节构词的规律预测复合词的词性,尤其是未登录词的词性。该文作者对藏语音节的定义进行了界定,提出音节的性质分类及标注原则,利用统计模型,在约24万音节的中小学语文教材语料库上进行实验,音节性质标注的正确率为93.520 8%。在此基础上,把音节性质标注信息用到词性标注中。实验结果表明:即使在音节性质标注存在一定错误的情况下,词性标注的正确率也提高到94.196 7%;如果在保证音节性质标注完全正确的情况下,词性标注的正确率可以提高到97.775 4%,这说明音节性质标注信息对词性标注有帮助。
"Syllables"of Tibetan language are very important in vocabulary construction and text information processing,especially for solving the segmentation and annotation of OOVs.This paper proposes to tag the syllables,which can be applied to predict POS of compound words(especially OOVs)according to the rules of words-construction.This paper presents the definition of the Tibetan syllable,outlines and the principles of classification and labeling.The train and test texts are selected from teaching material of Tibetan language of primary and secondary schools,total 240 Ksyllables.Experiments reveals a precision of 93.5208%for syllable tagging,upon which an improved 94.1967% accuracy for POS tagging can be reached.And given the gold-standard of syllable tagging,the accuracy of POS tagging will be improved to 97.775 4%.
"Syllables"of Tibetan language are very important in vocabulary construction and text information processing,especially for solving the segmentation and annotation of OOVs.This paper proposes to tag the syllables,which can be applied to predict POS of compound words(especially OOVs)according to the rules of words-construction.This paper presents the definition of the Tibetan syllable,outlines and the principles of classification and labeling.The train and test texts are selected from teaching material of Tibetan language of primary and secondary schools,total 240 Ksyllables.Experiments reveals a precision of 93.5208%for syllable tagging,upon which an improved 94.1967% accuracy for POS tagging can be reached.And given the gold-standard of syllable tagging,the accuracy of POS tagging will be improved to 97.775 4%.
引文
[1]黄昌宁,赵海.中文分词十年回顾[J].中文信息学报,2007,21(3):8-19.
[2]康才畯,龙从军,江荻.基于词位的藏文黏写形式的切分[J].计算机工程与应用,2014(11):218-222.
[3]史晓东,卢亚军.央金藏文分词系统[J].中文信息学报,2011,25(4):54-56.
[4]于洪志,李亚超,汪昆,等.融合音节特征的最大熵藏文词性标注研究[J].中文信息学报,2013,27(5):160-165.
[5]康才畯.藏语分词与词性标注研究[D].上海师范大学博士学位论文,2014:53.
[6]于洪志,李亚超,汪昆,等.融合音节特征的最大熵藏文词性标注研究[J].中文信息学报,2013,27(5):160-165.
[7]华却才让,刘群,赵海兴,等.判别式藏语文本词性标注研究[J].中文信息学报,2014,28(2):56-60.
[8]Tomá2Mikolov,IlyaSutskever,Hai-Son Leetc.Subword language modeling with neural networks[EB/OL].www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf.
[9]赵小兵,孙媛,龙从军,等.藏文拉丁转写、分词和词性分类规范:信息处理用现代藏语分词规范(草案)[M].北京:商务印书馆,2015:1-10.
[10]张济川.藏语词族研究:古代藏族如何丰富发展他们的词汇[M].北京:社会科学文献出版社,2009:207.
[11]才智杰.藏文自动分词系统中紧缩词的识别[J].中文信息学报,2009,23(1):35-37,43.
[12]刘汇丹,藏文分词及文本资源挖掘研究[D],中国科学院博士学位论文,2012:46.
[13]Congjun Long,Caijun Kang,Di Jiang.The comparative research on the segmentation strategies of Tibetan bounded variant forms[C]//Proceedings of the A-sian Language Processing(IALP),2013International Conference on DOI:10.1109/IALP,2013:243-246.
[2]康才畯,龙从军,江荻.基于词位的藏文黏写形式的切分[J].计算机工程与应用,2014(11):218-222.
[3]史晓东,卢亚军.央金藏文分词系统[J].中文信息学报,2011,25(4):54-56.
[4]于洪志,李亚超,汪昆,等.融合音节特征的最大熵藏文词性标注研究[J].中文信息学报,2013,27(5):160-165.
[5]康才畯.藏语分词与词性标注研究[D].上海师范大学博士学位论文,2014:53.
[6]于洪志,李亚超,汪昆,等.融合音节特征的最大熵藏文词性标注研究[J].中文信息学报,2013,27(5):160-165.
[7]华却才让,刘群,赵海兴,等.判别式藏语文本词性标注研究[J].中文信息学报,2014,28(2):56-60.
[8]Tomá2Mikolov,IlyaSutskever,Hai-Son Leetc.Subword language modeling with neural networks[EB/OL].www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf.
[9]赵小兵,孙媛,龙从军,等.藏文拉丁转写、分词和词性分类规范:信息处理用现代藏语分词规范(草案)[M].北京:商务印书馆,2015:1-10.
[10]张济川.藏语词族研究:古代藏族如何丰富发展他们的词汇[M].北京:社会科学文献出版社,2009:207.
[11]才智杰.藏文自动分词系统中紧缩词的识别[J].中文信息学报,2009,23(1):35-37,43.
[12]刘汇丹,藏文分词及文本资源挖掘研究[D],中国科学院博士学位论文,2012:46.
[13]Congjun Long,Caijun Kang,Di Jiang.The comparative research on the segmentation strategies of Tibetan bounded variant forms[C]//Proceedings of the A-sian Language Processing(IALP),2013International Conference on DOI:10.1109/IALP,2013:243-246.