企业级海量代码的检索与管理技术
|
摘要
在大型IT企业中,尤其像Google或者百度,代码搜索已是软件开发过程中不可或缺且频繁的活动,其通过借鉴或复用已有代码,加速开发过程的速度.多年以来,已有大量的研究人员关注代码搜索,且设计出很多优秀的工具.但是已有的研究和工具主要是在小规模或者编程语言单一的代码数据集上,没有从企业实际搜索需求出发,且对用户的查询输入也有所限制,尚缺少一套针对企业级海量代码的检索与管理技术方案.提出了一套企业级海量数据代码搜索引擎的方案和系统实现,面向开发过程中用户最直接的需求,通过离线分析与在线分析,完成对海量代码库的索引构建与检索.其中,离线分析负责代码相关数据的获取与分析、构建索引集群.在线过程负责变换用户的query、对搜索的结果进行高级排序、生成摘要.本系统部署在百度代码库上,为数十TB级的Git代码库构建了索引,平均一次检索时间在1s之内.在百度推出应用以来,访问量逐步增加,现每周平均用户有数千人,每周查询平均有数万次,广受百度工程师好评.
In large IT companies, especially like Google or Baidu, code search is an indispensable and frequent activity in the software development process, which speeds up the development process by learning or reusing existing code. Over the years, a large number of researchers have focused on code search and designed many excellent tools. However, the existing research and tools are mainly on a small-scale or single programming language code data set, not from the actual requirement of industries, and the user's query input is also limited; there is still a lack of a set of industrial-scale massive code retrieval and management technology solutions. This study proposes a code search engine solution and system implementation based on industrial-scale massive data, oriented to the most direct needs of users in the development process, through offline analysis and online analysis, complete the index construction and retrieval of massive code base. Among them, offline analysis is responsible for the acquisition and analysis of code-related data and building an index cluster. The online process is responsible for transforming the user's query, sorting the results of the search, and generating a summary. The system is deployed on the Baidu code base, and the index is built for dozens of TB-level Git code bases. The average retrieval time is within 1 s.Since the launch of Baidu's application, the number of visits has gradually increased. There are thousands of users per week and tens of thousands of times searching. The system is widely praised by Baidu engineers.
In large IT companies, especially like Google or Baidu, code search is an indispensable and frequent activity in the software development process, which speeds up the development process by learning or reusing existing code. Over the years, a large number of researchers have focused on code search and designed many excellent tools. However, the existing research and tools are mainly on a small-scale or single programming language code data set, not from the actual requirement of industries, and the user's query input is also limited; there is still a lack of a set of industrial-scale massive code retrieval and management technology solutions. This study proposes a code search engine solution and system implementation based on industrial-scale massive data, oriented to the most direct needs of users in the development process, through offline analysis and online analysis, complete the index construction and retrieval of massive code base. Among them, offline analysis is responsible for the acquisition and analysis of code-related data and building an index cluster. The online process is responsible for transforming the user's query, sorting the results of the search, and generating a summary. The system is deployed on the Baidu code base, and the index is built for dozens of TB-level Git code bases. The average retrieval time is within 1 s.Since the launch of Baidu's application, the number of visits has gradually increased. There are thousands of users per week and tens of thousands of times searching. The system is widely praised by Baidu engineers.
引文
[1]Singer J,Lethbridge T,Vinson N,Anquetil N.An examination of software engineering work practices.In:Proc.of the’97 Conf.of the Centre for Advanced Studies on Collaborative Research(CASCON’97).IBM Press,1997.174-188.
[2]Sadowski C,Stolee KT,Elbaum S.How developers search for code:A case study.In:Proc.of the 2015 10th Joint Meeting on Foundations of Software Engineering.Bergamo,2015.191-201.
[3]Xia X,Bao LF,Lo D,Kochhar PS,Hassan AE,Xing ZC.What do developers search for on the Web?Empirical Software Engineering,2017,22(6):3149-3185.
[4]Potvin R,Levenberg J.Why Google stores billions of lines of code in a single repository.Communications of the ACM,2016,59(7):78-87.[doi:10.1145/2854146]
[5]Gu X,Zhang H,Kim S.Deep code search.In:Proc.of the 40th Int’l Conf.New York:ACM Press,2018.933-944.http://doi.org/10.1145/3180155.3180167
[6]Kashyap V,Brown DB,Liblit B,Melski D,Reps TW.Source forager-A search engine for similar source code.arXiv preprint arXiv:1706.02769,2017.
[7]Linstead E,Bajracharya SK,Ngo TC,Rigor P,Lopes CV,Baldi P.Sourcerer:Mining and searching Internet-scale software repositories.Data Mining and Knowledge Discovery,2009,18(2):300-336.
[8]Reiss SP.Semantics-based code search.In:Proc.of the 31st Int’l Conf.on Software Engineering.2009.243-253.
[9]Stollee KT,Elbaum SG,Dobos D.Solving the search for source code.ACM Trans.on Software Engineering and Methodology,2014,23(3):1-45.
[10]Sahavechaphan N,Claypool KT.XSnippet:Mining for sample code.In:Proc.of the 21st Annual ACM SIGPLAN Conf.on Object-oriented Programming Systems,Languages,and Applications.2006.413-430.
[11]Begel A.Codifier:A programmer-centric search user interface.In:Proc.of the Workshop on Human-computer Interaction and Information Retrieval.2007.23-24.
[12]Heydon A,Najork M.Mercator:A scalable,extensible Web crawler.In:Proc.of the World Wide Web 2.4.1999.219-229.
[13]Ahmadi-Abkenari F,Selamat A.An architecture for a focused trend parallel Web crawler with the application of click stream analysis.Information Sciences,2012,184(1):266-281.
[14]Hsieh JM,Gribble SD,Levy HM.The architecture and implementation of an extensible Web crawler.In:Proc.of the InNSDI 2010.2010.329-344.
[15]Furnas GW,Landauer TK,Gomez LM,Dumais ST.The vocabulary problem in human-system communication.Communications of the ACM,1987,30(11):964-971.
[16]Wang S,Lo D,Jiang L.Active code search:Incorporating user feedback to improve code search relevance.In:Proc.of the 29th ACM/IEEE Int’l Conf.on Automated Software Engineering.Vasteras,2014.677-682.
[17]Roldan-Vega M,Mallet G,Hill E,Fails JA.CONQUER:A tool for NL-based query refinement and contextualizing code search results.In:Proc.of the 2013 29th IEEE Int’l Conf.on Software Maintenance(ICSM).2013.512-515.
[18]Lu M,Sun X,Wang S,Lo D,Duan Y.Query expansion via WordNet for effective code search.In:Proc.of the 2015 IEEE 22nd Int’l Conf.on Software Analysis,Evolution and Reengineering(SANER).2015.545-549.
[19]Miller GA.WordNet:A lexical database for English.Communications of the ACM,1995,38(11):39-41.
[20]Lemos OA,de Paula AC,Zanichelli FC,Lopes CV.Thesaurus-based automatic query expansion for interface-driven code search.In:Proc.of the 11th Working Conf.on Mining Software Repositories.Hyderabad,2014.212-221.
[21]Ko AJ,Myers BA,Coblenz MJ,Aung HH.An exploratory study of how developers seek,relate,and collect relevant information during software maintenance tasks.IEEE Trans.on Software Engineering,2006,32(12):971-987.
[22]Li H,Xing Z,Peng X,Zhao W.What help do developers seek,when and how?In:Proc.of the 20th Working Conf.on Reverse Engineering(WCRE).IEEE,2013.142-151
[23]Page L,Brin S,Motwani R,Winograd T.The pagerank citation ranking:Bringing order to the Web.Technical Report,1999-66,Stanford InfoLab,1999.
[24]Lin Y,Liu Z,Sun M,Liu Y,Zhu X.Learning entity and relation embeddings for knowledge graph completion.In:Proc.of the 29th AAAI Conf.on Artificial Intelligence.2015.
[25]Graepel T,Candela JQ,Borchert T,Her-brich R.Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft’s Bing search engine.In:Proc.of the 27th Int’l Conf.on Machine Learning(ICML 2010).2010.
[26]Mishne A,Shoham S,Yahav E.Typestate-based semantic code search over partial programs.ACM SIGPLAN Notices,2012,47(10):997-1016.[doi:10.1145/2398857.2384689]
[27]Bajracharya S,Ngo T,Linstead E,Dou YM,Rigor P,Baldi P,Lopes C.Sourcerer:A search engine for open source code supporting structure-based search.In:Proc.of the Companion to the 21st ACM SIGPLAN Symp.on Object-oriented Programming Systems,Languages,and Applications.ACM Press,2006.681-682.[doi:10.1145/1176617.1176671]
[28]McMillan C,Grechanik M,Poshyvanyk D,Xie Q,Fu C.Portfolio:Finding relevant functions and their usage.In:Proc.of the 33rd Int’l Conf.on Software Engineering.ACM Press,2011.111-120.[doi:10.1145/1985793.1985809]
[29]Stylos J,Myers BA.Mica:A Web-search tool for finding API components and examples.In:Proc.of the 23rd Int’l Conf.on Program.IEEE,2015.[doi:10.1109/VLHCC.2006.32]
[30]Treude C,Robillard MP.Augmenting API documentation with insights from stack overflow.In:Proc.of the 38th Int’l Conf.on Software Engineering.ACM Press,2016.392-403.[doi:10.1145/2884781.2884800]
[31]McMillan C,Grechanik M,Poshyvanyk D,Fu C,Xie Q.Exemplar:A source code search engine for finding highly relevant applications.IEEE Trans.on Software Engineering,2012,38(5):1069-1087.
[32]Li X,Wang ZR,Wang QX,Yan SM,Xie T,Mei H.Relationship-aware code search for JavaScript frameworks.In:Proc.of the24th ACM SIGSOFT Symp.on the Foundations of Software Engineering.ACM Press,2016.690-701.[doi:10.1145/2950290.2950341]
[33]Mandelin D,Xu L,Bodík R,Kimelman D.Jungloid mining:Helping to navigate the API jungle.ACM SIGPLAN Notices,2005,40(6):48-61.[doi:10.1145/1065010.1065018]
[34]Li X,Wang QX,Jin Z.Description reinforcement based code search.Ruan Jian Xue Bao/Journal of Software,2017,28(6):1405-1417(in Chinese with English abstract).http://www.jos.org.cn/1000-9825/5226.htm[doi:10.13328/j.cnki.jos.005226][34]黎宣,王千祥,金芝.基于增强描述的代码搜索方法.软件学报,2017,28(6):1405-1417.http://www.jos.org.cn/1000-9825/5226.htm[doi:10.13328/j.cnki.jos.005226]
[2]Sadowski C,Stolee KT,Elbaum S.How developers search for code:A case study.In:Proc.of the 2015 10th Joint Meeting on Foundations of Software Engineering.Bergamo,2015.191-201.
[3]Xia X,Bao LF,Lo D,Kochhar PS,Hassan AE,Xing ZC.What do developers search for on the Web?Empirical Software Engineering,2017,22(6):3149-3185.
[4]Potvin R,Levenberg J.Why Google stores billions of lines of code in a single repository.Communications of the ACM,2016,59(7):78-87.[doi:10.1145/2854146]
[5]Gu X,Zhang H,Kim S.Deep code search.In:Proc.of the 40th Int’l Conf.New York:ACM Press,2018.933-944.http://doi.org/10.1145/3180155.3180167
[6]Kashyap V,Brown DB,Liblit B,Melski D,Reps TW.Source forager-A search engine for similar source code.arXiv preprint arXiv:1706.02769,2017.
[7]Linstead E,Bajracharya SK,Ngo TC,Rigor P,Lopes CV,Baldi P.Sourcerer:Mining and searching Internet-scale software repositories.Data Mining and Knowledge Discovery,2009,18(2):300-336.
[8]Reiss SP.Semantics-based code search.In:Proc.of the 31st Int’l Conf.on Software Engineering.2009.243-253.
[9]Stollee KT,Elbaum SG,Dobos D.Solving the search for source code.ACM Trans.on Software Engineering and Methodology,2014,23(3):1-45.
[10]Sahavechaphan N,Claypool KT.XSnippet:Mining for sample code.In:Proc.of the 21st Annual ACM SIGPLAN Conf.on Object-oriented Programming Systems,Languages,and Applications.2006.413-430.
[11]Begel A.Codifier:A programmer-centric search user interface.In:Proc.of the Workshop on Human-computer Interaction and Information Retrieval.2007.23-24.
[12]Heydon A,Najork M.Mercator:A scalable,extensible Web crawler.In:Proc.of the World Wide Web 2.4.1999.219-229.
[13]Ahmadi-Abkenari F,Selamat A.An architecture for a focused trend parallel Web crawler with the application of click stream analysis.Information Sciences,2012,184(1):266-281.
[14]Hsieh JM,Gribble SD,Levy HM.The architecture and implementation of an extensible Web crawler.In:Proc.of the InNSDI 2010.2010.329-344.
[15]Furnas GW,Landauer TK,Gomez LM,Dumais ST.The vocabulary problem in human-system communication.Communications of the ACM,1987,30(11):964-971.
[16]Wang S,Lo D,Jiang L.Active code search:Incorporating user feedback to improve code search relevance.In:Proc.of the 29th ACM/IEEE Int’l Conf.on Automated Software Engineering.Vasteras,2014.677-682.
[17]Roldan-Vega M,Mallet G,Hill E,Fails JA.CONQUER:A tool for NL-based query refinement and contextualizing code search results.In:Proc.of the 2013 29th IEEE Int’l Conf.on Software Maintenance(ICSM).2013.512-515.
[18]Lu M,Sun X,Wang S,Lo D,Duan Y.Query expansion via WordNet for effective code search.In:Proc.of the 2015 IEEE 22nd Int’l Conf.on Software Analysis,Evolution and Reengineering(SANER).2015.545-549.
[19]Miller GA.WordNet:A lexical database for English.Communications of the ACM,1995,38(11):39-41.
[20]Lemos OA,de Paula AC,Zanichelli FC,Lopes CV.Thesaurus-based automatic query expansion for interface-driven code search.In:Proc.of the 11th Working Conf.on Mining Software Repositories.Hyderabad,2014.212-221.
[21]Ko AJ,Myers BA,Coblenz MJ,Aung HH.An exploratory study of how developers seek,relate,and collect relevant information during software maintenance tasks.IEEE Trans.on Software Engineering,2006,32(12):971-987.
[22]Li H,Xing Z,Peng X,Zhao W.What help do developers seek,when and how?In:Proc.of the 20th Working Conf.on Reverse Engineering(WCRE).IEEE,2013.142-151
[23]Page L,Brin S,Motwani R,Winograd T.The pagerank citation ranking:Bringing order to the Web.Technical Report,1999-66,Stanford InfoLab,1999.
[24]Lin Y,Liu Z,Sun M,Liu Y,Zhu X.Learning entity and relation embeddings for knowledge graph completion.In:Proc.of the 29th AAAI Conf.on Artificial Intelligence.2015.
[25]Graepel T,Candela JQ,Borchert T,Her-brich R.Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft’s Bing search engine.In:Proc.of the 27th Int’l Conf.on Machine Learning(ICML 2010).2010.
[26]Mishne A,Shoham S,Yahav E.Typestate-based semantic code search over partial programs.ACM SIGPLAN Notices,2012,47(10):997-1016.[doi:10.1145/2398857.2384689]
[27]Bajracharya S,Ngo T,Linstead E,Dou YM,Rigor P,Baldi P,Lopes C.Sourcerer:A search engine for open source code supporting structure-based search.In:Proc.of the Companion to the 21st ACM SIGPLAN Symp.on Object-oriented Programming Systems,Languages,and Applications.ACM Press,2006.681-682.[doi:10.1145/1176617.1176671]
[28]McMillan C,Grechanik M,Poshyvanyk D,Xie Q,Fu C.Portfolio:Finding relevant functions and their usage.In:Proc.of the 33rd Int’l Conf.on Software Engineering.ACM Press,2011.111-120.[doi:10.1145/1985793.1985809]
[29]Stylos J,Myers BA.Mica:A Web-search tool for finding API components and examples.In:Proc.of the 23rd Int’l Conf.on Program.IEEE,2015.[doi:10.1109/VLHCC.2006.32]
[30]Treude C,Robillard MP.Augmenting API documentation with insights from stack overflow.In:Proc.of the 38th Int’l Conf.on Software Engineering.ACM Press,2016.392-403.[doi:10.1145/2884781.2884800]
[31]McMillan C,Grechanik M,Poshyvanyk D,Fu C,Xie Q.Exemplar:A source code search engine for finding highly relevant applications.IEEE Trans.on Software Engineering,2012,38(5):1069-1087.
[32]Li X,Wang ZR,Wang QX,Yan SM,Xie T,Mei H.Relationship-aware code search for JavaScript frameworks.In:Proc.of the24th ACM SIGSOFT Symp.on the Foundations of Software Engineering.ACM Press,2016.690-701.[doi:10.1145/2950290.2950341]
[33]Mandelin D,Xu L,Bodík R,Kimelman D.Jungloid mining:Helping to navigate the API jungle.ACM SIGPLAN Notices,2005,40(6):48-61.[doi:10.1145/1065010.1065018]
[34]Li X,Wang QX,Jin Z.Description reinforcement based code search.Ruan Jian Xue Bao/Journal of Software,2017,28(6):1405-1417(in Chinese with English abstract).http://www.jos.org.cn/1000-9825/5226.htm[doi:10.13328/j.cnki.jos.005226][34]黎宣,王千祥,金芝.基于增强描述的代码搜索方法.软件学报,2017,28(6):1405-1417.http://www.jos.org.cn/1000-9825/5226.htm[doi:10.13328/j.cnki.jos.005226]