基于Lucene教学资源搜索引擎的设计与实现
摘要
20世纪最后20年至今,互联网的发展和普及为人们提供了一个全新的信息存储、加工、传递和使用的载体,网络信息也迅速成为社会成员获得知识和信息的主要渠道之一。传统的信息检索技术与网络应用实际相结合的搜索引擎就应运而生。
该系统源于北京某公司针对特定用户群体设计的一个主题搜索引擎系统。主要解决目前通用搜索引擎信息量广、搜索专业知识时结果不集中,用户很难从浩瀚的网络信息中检索到自己领域的知识等问题,基于此目的,为满足公司特定客户群体对于教学资源主题搜索引擎的需要,开发和设计此系统,并投入测试和应用。
论文以用户对教学资源信息需求为研究背景,采用主题搜索理念与技术,通过分析将国内互联网上与教学资源相关的信息作为采集对象,基于开源软件包Lucene,研究和实现了对教学资源信息搜索引擎的搜索引擎。
首先,介绍搜索引擎的基本概念以及搜索引擎的基本运行原理,并对搜索引擎的排序技术和评价指标进行了探讨。同时对开源项目Lucene进行概述,包括Lucene的系统结构、包结构、索引机制、搜索机制进行了论述。
其次,在对公司业务分析的基础上,运用UML用例分析并确定了系统的开发目标和基本功能需求,对系统的关键功能进行了详细的设计,同时对系统的运行架构、组织结构以及系统部署作了设计说明。
最后,对搜索引擎的相关开发技术进行介绍,基于Lucene开源框架,运用Spring、Struts 2等技术开发并实现了教学资源搜索引擎系统。
Since the 20th century for the last 20 years, the development and popularity of the Internet provides a new carrier for people who take as information storage, processing, transmission and making use of, Network information also quickly became one of the main channel which takes as an access to knowledge and information for the member of society. Thus the traditional information retrieval techniques combined with the practical application of network made the search engine.
The system which originated from a Beijing company for a project-specific user groups, is a subject search engine. It is mainly to solve the problems that information more widely for general search engine, search results not focusing on professional knowledge and the user is very difficult to retrieve their field of knowledge from the vast network of information, On this basis, the groups develop a search engine on teaching resources in order to meet company-specific customer, and put into testing and application.
The thesis takes the information needs of users for teaching resources as the study background, using the concepts and techniques of the theme search.
By analysis, the thesis takes the teaching resources information on the domestic Internet as a collection object based on the open source software package-Lucene, studies and achieves the search engines for the teaching resources information.
Firstly, this thesis introduces the basic concepts and basic operating principles of search engine, and discusses ranking techniques and evaluation indexes of the Search engine, then an overview of the open source Lucene project, including the Lucene system structure, including structure, indexing mechanism, the search mechanism, was discussed.
Secondly, the thesis determine the system's development objectives and basic functional requirements based on the use case analysis-UML, and carries out a detailed design for the critical functions of the system, and the operation of the system architecture, the organizational structure and the system deployment are made a design specification.
Finally, introducing the key technology of the search engine, and developing and achieving the search engine system for teaching resources by using Spring, Struts 2 and other technology based on the Lucene open-source framework.
该系统源于北京某公司针对特定用户群体设计的一个主题搜索引擎系统。主要解决目前通用搜索引擎信息量广、搜索专业知识时结果不集中,用户很难从浩瀚的网络信息中检索到自己领域的知识等问题,基于此目的,为满足公司特定客户群体对于教学资源主题搜索引擎的需要,开发和设计此系统,并投入测试和应用。
论文以用户对教学资源信息需求为研究背景,采用主题搜索理念与技术,通过分析将国内互联网上与教学资源相关的信息作为采集对象,基于开源软件包Lucene,研究和实现了对教学资源信息搜索引擎的搜索引擎。
首先,介绍搜索引擎的基本概念以及搜索引擎的基本运行原理,并对搜索引擎的排序技术和评价指标进行了探讨。同时对开源项目Lucene进行概述,包括Lucene的系统结构、包结构、索引机制、搜索机制进行了论述。
其次,在对公司业务分析的基础上,运用UML用例分析并确定了系统的开发目标和基本功能需求,对系统的关键功能进行了详细的设计,同时对系统的运行架构、组织结构以及系统部署作了设计说明。
最后,对搜索引擎的相关开发技术进行介绍,基于Lucene开源框架,运用Spring、Struts 2等技术开发并实现了教学资源搜索引擎系统。
Since the 20th century for the last 20 years, the development and popularity of the Internet provides a new carrier for people who take as information storage, processing, transmission and making use of, Network information also quickly became one of the main channel which takes as an access to knowledge and information for the member of society. Thus the traditional information retrieval techniques combined with the practical application of network made the search engine.
The system which originated from a Beijing company for a project-specific user groups, is a subject search engine. It is mainly to solve the problems that information more widely for general search engine, search results not focusing on professional knowledge and the user is very difficult to retrieve their field of knowledge from the vast network of information, On this basis, the groups develop a search engine on teaching resources in order to meet company-specific customer, and put into testing and application.
The thesis takes the information needs of users for teaching resources as the study background, using the concepts and techniques of the theme search.
By analysis, the thesis takes the teaching resources information on the domestic Internet as a collection object based on the open source software package-Lucene, studies and achieves the search engines for the teaching resources information.
Firstly, this thesis introduces the basic concepts and basic operating principles of search engine, and discusses ranking techniques and evaluation indexes of the Search engine, then an overview of the open source Lucene project, including the Lucene system structure, including structure, indexing mechanism, the search mechanism, was discussed.
Secondly, the thesis determine the system's development objectives and basic functional requirements based on the use case analysis-UML, and carries out a detailed design for the critical functions of the system, and the operation of the system architecture, the organizational structure and the system deployment are made a design specification.
Finally, introducing the key technology of the search engine, and developing and achieving the search engine system for teaching resources by using Spring, Struts 2 and other technology based on the Lucene open-source framework.
引文
[1]姚林涛.基于Lucene的Web搜索引擎实现[D].西安电子科技大学,2008.
[2]Rose,D.E.and Levinson,D.Understanding user goals in web search. In:Proceedings of the 13th international Conference on World Wide Web. New York:ACMPress,2004.13-19.
[3]中国互联网络信息中心.第27次中国互联网络发展状况统计报告.2011.1.
[4]张伟.垂直搜索引擎设计与实现.西安电子科技大学,2008.
[5]王学松Lucene与Nutch搜索引擎开发.人民邮电出版社,2008.
[6]李艳红.浅谈搜索引擎的研究现状.西安外事学院计算机中心,2009.
[7]苏新宁.信息检索理论与技术[M].北京：科学技术文献出版社,2004：3-29.
[8]李建林.基于Lucene的Web搜索引擎的研究.2011,4.
[9]Cohen,W.,R.Shapire,et al.. learning to order things[J].Journal of Artificial Intelligence Research,1999,10:243-270.
[10]孙鑫Struts2深入详解[M].北京：电子工业出版社,2007.
[11]刘奕群马少平等.搜索引擎技术基础.清华大学出版社.2010.
[12]孔伯煊等.基于Lucene\XML技术的Web搜索引擎设计与实现.航空计算技术.2006,4.
[13]JQuery官网[EB/OL]:http://docs.jquery.com/Plugins/Autocomplete.
[14]史鹏辉.专业服务网站搜索引擎的设计与实现.大连理工大学.2004.
[15]李毅,杨善林,刘业政,顾铁军.基于XML的网络课件环境中搜索引擎的实现[J].武汉理工大学学报,2003,7.
[16]硕网资讯.洞悉XML[M]北京：北京大学出版社,2001.
[17]《程序天下—Strusts2技术详解》ISBN 978-7-121-06221-6电子工业出版社,2008年6月.
[18]贾素玲,王强,姚琪琳,舒毕磊,柴庆慧.XM技术应用.清华大学出版社.2007,7.
[19]Otis Gospodnetic,Erik Hatcher.Lucene IN ATION电子工业出版社,2007,1.
[20]IKAnalyzer中文分词器v3.0使用手册.
[21]冯曼菲,精通Ajax—基础概念、核心技术与典型案例ISBN 978-7-121-17315-7人民邮电出版社,2008年2月.
[22]李刚Struts 2权威指南.电子工业出版社，2007年9月
[23]马晓星.分布式Web服务器技术综述.计算机科学.2002,1.
[24]于天恩,搜索引擎开发权威经典.中国铁道出版社,2008,10.
[25]高琰,谷士文,谭立球,费耀平.基于Lucene的搜索引擎设计与实现[J].微机发展,2004,(10).
[26]百度百科http://baike.baidu.com/view/7491.htm.
[27]Lucene[EB/OL].http://lucene.apache.org/java/docs/index.html.
[28]Hearst,M,Divoli,A,Guturu,H.et al. BioText Search Engine:beyond abstract search.Bioinformatics, 2007,23 (16):2196-2197.
[29]钟凯磊Lucene论述——构建自己的搜索引擎[J].大众商务,2009,(06).
[30]刘云吉,施晓秋.基于lucene的搜索引擎在Ajax中的应用[J].硅谷,2009,(19)
[31]任惠静.基于Lucene的面向主题搜索引擎的索引技术的研究[J].电脑知识与技术,2010,(04)
[32]朱学昊,王儒敬,余锋林,唐昱.基于Lucene的站内搜索设计与实现[J].计算机应用与软件,2008,(10).
[33]李颖,李志蜀,邓欢.基于Lucene的中文分词方法设计与实现[J].四川大学学报(自然科学版),2008,(05).
[34]周杨.基于Lucene的小型搜索引擎系统的架构与实现[J].电脑编程技巧与维护,2010,(11).
[35]王晓伟.垂直搜索引擎若干关键技术的研究[D].浙江：浙江大学,2007.
[36]Henry Song,Doreen Cheng,Alan Messer,Swaroop Kalasa-pur. Web Service Discovery Using General-Purpose Search En-gines .Web Services,ICWS2007.IEEE International Conference on. 2007,:pp.265-271.
[37]Web Service Description Language(WSDL).http://www.w3.org/TR/wsdl.
[38]J.Cho.H.Garcia-Molina and L. Page. Efficient crawling through URL ordering. In:Proceedings of the 7th International World Wide Web Conference. Brisbane:Elsevier Science,1998.161-172
[39]钟凯磊Lucene论述——构建自己的搜索引擎[J].大众商务,2009,(06).
[40]晁岳峰,曹作良,郭英玲.基于Lucene的搜索引擎在远程教育平台中的实现[J].天津理工大学学报,2005，,(06)
[41]吉亮.个性化搜索引擎的技术发展与应用[J].机械管理开发,2010,(05).
[42]张东振,张明.基于XML的站内检索系统[J].电脑知识与技术,2010,(02).
[43]黄波.主题搜索引擎的研究与应用[D].成都理工大学,2007
[44]李晓光.XML非完全结构查询处理中若干关键技术的研究[D].东北大学,2006.
[45]马楠.基于Spring的J2EE安全框架的研究与应用[D].北京交通大学,2009
[46]R.Baeza-Yates,C. Hurtado, and M. Mendoza. Query recommendation using query logs in search engines. EDBT 2004 Workshop on Current Trends in Database Tech.2004
[47]陈航哲.基于Lucene的垂直搜索引擎的研究与应用[D].暨南大学,2010.
[48]曹卫峰.中文分词关键技术研究[D].南京理工大学,2009.
[49]A. Singhal,M. Kaszkiel. A case study in web search using TREC algorithms.Proceedings of the tenth international conference on World Wide Web. Hong Kong.2001,.
[50]S H Lin,and J.M Ho. Discovering informative content blocks from Web documents.KDD - 2002 Proceedings of the Eight ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.2002,.
[51]刘敏娜,葛萌,陈娟.基于Lucene的全文搜索引擎设计与实现[J].福建电脑.2009,(05).
[52]张晓卫，朱巧明.一种基于Lucene的Web全文信息检索系统的设计与实现[J].计算机与现代化.2006,(12).
[53]潘亭沥.基于Lucene的面向商业应用的搜索引擎研究与实现[D].电子科技大学,2007.
[54]冯斌.基于Lucene小型搜索引擎的研究与实现[D].武汉理工大学,2008.
[2]Rose,D.E.and Levinson,D.Understanding user goals in web search. In:Proceedings of the 13th international Conference on World Wide Web. New York:ACMPress,2004.13-19.
[3]中国互联网络信息中心.第27次中国互联网络发展状况统计报告.2011.1.
[4]张伟.垂直搜索引擎设计与实现.西安电子科技大学,2008.
[5]王学松Lucene与Nutch搜索引擎开发.人民邮电出版社,2008.
[6]李艳红.浅谈搜索引擎的研究现状.西安外事学院计算机中心,2009.
[7]苏新宁.信息检索理论与技术[M].北京：科学技术文献出版社,2004：3-29.
[8]李建林.基于Lucene的Web搜索引擎的研究.2011,4.
[9]Cohen,W.,R.Shapire,et al.. learning to order things[J].Journal of Artificial Intelligence Research,1999,10:243-270.
[10]孙鑫Struts2深入详解[M].北京：电子工业出版社,2007.
[11]刘奕群马少平等.搜索引擎技术基础.清华大学出版社.2010.
[12]孔伯煊等.基于Lucene\XML技术的Web搜索引擎设计与实现.航空计算技术.2006,4.
[13]JQuery官网[EB/OL]:http://docs.jquery.com/Plugins/Autocomplete.
[14]史鹏辉.专业服务网站搜索引擎的设计与实现.大连理工大学.2004.
[15]李毅,杨善林,刘业政,顾铁军.基于XML的网络课件环境中搜索引擎的实现[J].武汉理工大学学报,2003,7.
[16]硕网资讯.洞悉XML[M]北京：北京大学出版社,2001.
[17]《程序天下—Strusts2技术详解》ISBN 978-7-121-06221-6电子工业出版社,2008年6月.
[18]贾素玲,王强,姚琪琳,舒毕磊,柴庆慧.XM技术应用.清华大学出版社.2007,7.
[19]Otis Gospodnetic,Erik Hatcher.Lucene IN ATION电子工业出版社,2007,1.
[20]IKAnalyzer中文分词器v3.0使用手册.
[21]冯曼菲,精通Ajax—基础概念、核心技术与典型案例ISBN 978-7-121-17315-7人民邮电出版社,2008年2月.
[22]李刚Struts 2权威指南.电子工业出版社，2007年9月
[23]马晓星.分布式Web服务器技术综述.计算机科学.2002,1.
[24]于天恩,搜索引擎开发权威经典.中国铁道出版社,2008,10.
[25]高琰,谷士文,谭立球,费耀平.基于Lucene的搜索引擎设计与实现[J].微机发展,2004,(10).
[26]百度百科http://baike.baidu.com/view/7491.htm.
[27]Lucene[EB/OL].http://lucene.apache.org/java/docs/index.html.
[28]Hearst,M,Divoli,A,Guturu,H.et al. BioText Search Engine:beyond abstract search.Bioinformatics, 2007,23 (16):2196-2197.
[29]钟凯磊Lucene论述——构建自己的搜索引擎[J].大众商务,2009,(06).
[30]刘云吉,施晓秋.基于lucene的搜索引擎在Ajax中的应用[J].硅谷,2009,(19)
[31]任惠静.基于Lucene的面向主题搜索引擎的索引技术的研究[J].电脑知识与技术,2010,(04)
[32]朱学昊,王儒敬,余锋林,唐昱.基于Lucene的站内搜索设计与实现[J].计算机应用与软件,2008,(10).
[33]李颖,李志蜀,邓欢.基于Lucene的中文分词方法设计与实现[J].四川大学学报(自然科学版),2008,(05).
[34]周杨.基于Lucene的小型搜索引擎系统的架构与实现[J].电脑编程技巧与维护,2010,(11).
[35]王晓伟.垂直搜索引擎若干关键技术的研究[D].浙江：浙江大学,2007.
[36]Henry Song,Doreen Cheng,Alan Messer,Swaroop Kalasa-pur. Web Service Discovery Using General-Purpose Search En-gines .Web Services,ICWS2007.IEEE International Conference on. 2007,:pp.265-271.
[37]Web Service Description Language(WSDL).http://www.w3.org/TR/wsdl.
[38]J.Cho.H.Garcia-Molina and L. Page. Efficient crawling through URL ordering. In:Proceedings of the 7th International World Wide Web Conference. Brisbane:Elsevier Science,1998.161-172
[39]钟凯磊Lucene论述——构建自己的搜索引擎[J].大众商务,2009,(06).
[40]晁岳峰,曹作良,郭英玲.基于Lucene的搜索引擎在远程教育平台中的实现[J].天津理工大学学报,2005，,(06)
[41]吉亮.个性化搜索引擎的技术发展与应用[J].机械管理开发,2010,(05).
[42]张东振,张明.基于XML的站内检索系统[J].电脑知识与技术,2010,(02).
[43]黄波.主题搜索引擎的研究与应用[D].成都理工大学,2007
[44]李晓光.XML非完全结构查询处理中若干关键技术的研究[D].东北大学,2006.
[45]马楠.基于Spring的J2EE安全框架的研究与应用[D].北京交通大学,2009
[46]R.Baeza-Yates,C. Hurtado, and M. Mendoza. Query recommendation using query logs in search engines. EDBT 2004 Workshop on Current Trends in Database Tech.2004
[47]陈航哲.基于Lucene的垂直搜索引擎的研究与应用[D].暨南大学,2010.
[48]曹卫峰.中文分词关键技术研究[D].南京理工大学,2009.
[49]A. Singhal,M. Kaszkiel. A case study in web search using TREC algorithms.Proceedings of the tenth international conference on World Wide Web. Hong Kong.2001,.
[50]S H Lin,and J.M Ho. Discovering informative content blocks from Web documents.KDD - 2002 Proceedings of the Eight ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.2002,.
[51]刘敏娜,葛萌,陈娟.基于Lucene的全文搜索引擎设计与实现[J].福建电脑.2009,(05).
[52]张晓卫，朱巧明.一种基于Lucene的Web全文信息检索系统的设计与实现[J].计算机与现代化.2006,(12).
[53]潘亭沥.基于Lucene的面向商业应用的搜索引擎研究与实现[D].电子科技大学,2007.
[54]冯斌.基于Lucene小型搜索引擎的研究与实现[D].武汉理工大学,2008.