持续集成测试用例集优化综述研究
|
摘要
基于互联网的软件开发要求产品快速迭代,同时保持产品的质量,其关键的环节就是持续集成.持续集成通过自动化测试来保证集成到主干的代码质量.持续集成时的测试用例选择是一个非常有挑战性的问题.如果运行所有的测试用例,需要消耗大量的计算资源,并造成测试反馈周期过长.如果选择的测试用例集不合适,又不足以覆盖必要的测试代码以保证待集成代码的质量.持续集成测试用例集优化的目的是平衡测试资源和测试质量,在尽可能不影响测试质量的情况下,减少持续集成的资源需求.对近年来国内外学者在该领域的研究工作进行了系统的分析、提炼和总结.为此,首先从研究主题、影响因子、研究方法、研究对象以及性能评价等方面提出5个研究问题;然后从电子文献数据库搜索最近10年的研究工作,经过仔细审查和筛选后选择39篇文献作为研究对象;最后,从选择文献中收集数据,通过定量分析和可视化展示来回答提出的研究问题.总的来说,回顾了持续集成测试用例集优化的研究进展,为该领域的研究者提供了一些有用的发现,并总结了面临的问题和挑战.
Currently, many software applications are rapidly developed and deployed. Rapid iterations are used to implement new requirements while keeping high quality. Continuous integration(CI) is an important activity to make it work. CI tests the code automatically before it will be integrated into the master branch to ensure its quality. The big challenge is how to select an appropriate test case set for continuous integration. Running all test cases will consume a large amount of resource and get feedback in a long cycle, but selecting an improper test case subset will lose some required test, and therefore increase quality risk. The goal of test set optimization is to select an appropriate test case subset which can satisfy the test requirement for each unit or iteration test while reducing the test resource as much as possible. In this paper, a systematic literature review is performed to reveal the current status of CI-test set optimization based on existing literature. More specifically, the related studies published between 2007 and 2017 from four databases of electronic literature are searched to finally select 39 important studies for careful review. Five research questions are presented in order to achieve the above goal, and data from the selected studies are extracted to analyze and answer the questions. The analysis results contribute some useful findings and new challenges for researches and practitioners.
Currently, many software applications are rapidly developed and deployed. Rapid iterations are used to implement new requirements while keeping high quality. Continuous integration(CI) is an important activity to make it work. CI tests the code automatically before it will be integrated into the master branch to ensure its quality. The big challenge is how to select an appropriate test case set for continuous integration. Running all test cases will consume a large amount of resource and get feedback in a long cycle, but selecting an improper test case subset will lose some required test, and therefore increase quality risk. The goal of test set optimization is to select an appropriate test case subset which can satisfy the test requirement for each unit or iteration test while reducing the test resource as much as possible. In this paper, a systematic literature review is performed to reveal the current status of CI-test set optimization based on existing literature. More specifically, the related studies published between 2007 and 2017 from four databases of electronic literature are searched to finally select 39 important studies for careful review. Five research questions are presented in order to achieve the above goal, and data from the selected studies are extracted to analyze and answer the questions. The analysis results contribute some useful findings and new challenges for researches and practitioners.
引文
[1]Hilton M,Nelson N,Tunnell T,Marinov D,Dig D.Trade-Offs in continuous integration:Assurance,security,and flexibility.In:Proc.of the Joint Meeting on Foundations of Software Engineering.2017.197-207.
[2]Vasilescu B,Yu Y,Wang H,Devanbu P,Filko V.Quality and productivity outcomes relating to continuous integration in GitHub.In:Proc.of the Joint Meeting on Foundations of Software Engineering.2015.805-816.
[3]Hilton M,Tunnell T,Huang K,Marinov D,Dig D.Usage,costs,and benefits of continuous integration in open-source projects.In:Proc.of the IEEE/ACM Int’l Conf.on Automated Software Engineering.2016.426-437.
[4]Martensson T,Stahl D,Bosch J.Continuous integration impediments in large-scale industry projects.In:Proc.of the IEEE Int’l Conf.on Software Architecture.IEEE,2017.169-178.
[5]Virmani M.Understanding DevOps&bridging the gap from continuous integration to continuous delivery.In:Proc.of the 5th Int’l Conf.on Innovative Computing Technology(INTECH).2015.78-82.
[6]Rathod N,Surve A.Test orchestration a framework for continuous integration and continuous deployment.In:Proc.of the Int’l Conf.on Pervasive Computing.2015.1-5.
[7]Kitchenham BA,Charters S.Guidelines for performing systematic literature reviews in software engineering.EBSE Technical Report,EBSE-2007-01,ICSE IEEE Computer Society,2007.1051-1052.
[8]Badampudi D,Wohlin C,Kai P.Experiences from using snowballing and database searches in systematic literature studies.In:Proc.of the Int’l Conf.on Evaluation and Assessment in Software Engineering.2015.1-10.
[9]Kitchenham BA,Dyba T,Jorgensen M.Evidence-Based software engineering.In:Proc.of the Int’l Conf.on Software Engineering.IEEE Computer Society,2004.273-281.
[10]Yoo S,Harman M.Regression testing minimization,selection and prioritization:A survey.Software Testing,Verification and Reliability,2012,22(2):67-120.
[11]Kazmi R,Jawawi DNA,Mohamad R,Ghani I.Effective regression test case selection:A systematic literature review.ACMComputing Surveys,2017,50(2):29.
[12]Nanda A,Mani S,Sinha S,Harrold MJ,Orso A.Regression testing in the presence of non-code changes.In:Proc.of the 4th IEEEInt’l Conf.on Software Testing,Verification and Validation.2011.21-30.
[13]Qiu D,Li B,Ji S,Leung H.Regression testing of Web service:A systematic mapping study.ACM Computing Surveys,2015,47(2):21.
[14]Stratis P,Rajan A.Test case permutation to improve execution time.In:Proc.of the IEEE/ACM Int’l Conf.on Automated Software Engineering.2016.45-50.
[15]Huang CY,Chen CS,Lai CE.Evaluation and analysis of incorporating fuzzy expert system approach into test suite reduction.Information and Software Technology,2016,79(C):79-105.
[16]Qu X,Cohen MB,Woolf KM.Combinatorial interaction regression testing:A study of test case generation and prioritization.In:Proc.of the IEEE Int’l Conf.on Software Maintenance.2007.255-264.
[17]Engstr?m E,Runeson P,Skoglund M.A systematic review on regression test selection techniques.Information and Software Technology,2010,52(1):14-30.
[18]Catal C,Mishra D.Test case prioritization:A systematic mapping study.Software Quality Journal,2013,21(3):445-478.
[19]Soetens QD,Demeyer S,Zaidman A,Pérez J.Change-Based test selection:An empirical evaluation.Empirical Software Engineering,2016,21(5):1990-2032.
[20]Blondeau V,Etien A,Anquetil N,Cresson S,Croisy P,Ducasse S.Test case selection in industry:An analysis of issues related to static approaches.Software Quality Journal,2016,1-35.
[21]Spieker H,Gotlieb A,Marijan D,Mossige M.Reinforcement learning for automatic test case prioritization and selection in continuous integration.In:Proc.of the 26th ACM SIGSOFT Int’l Symp.on Software Testing and Analysis.2017.12-22.
[22]Busjaeger B,Xie T.Learning for test prioritization:An industrial case study.In:Proc.of the 24th ACM SIGSOFT Int’l Symp.on Foundations of Software Engineering.2016.975-980.
[23]Elbaum S,Rothermel G,Penix J.Techniques for improving regression testing in continuous integration development environments.In:Proc.of the 22nd ACM SIGSOFT Int’l Symp.on Foundations of Software Engineering.2014.235-245.
[24]Parsai A,Soetens QD,Murgia A,Demeyer S.Considering polymorphism in change-based test suite reduction.In:Proc.of the Int’l Conf.on Agile Software Development.Cham:Springer-Verlag,2014.166-181.
[25]Shi A,Thummalapenta S,Lahiri SK,Bjorner N,Czerwonka J.Optimizing test placement for module-level regression testing.In:Proc.of the IEEE/ACM Int’l Conf.on Software Engineering.IEEE Computer Society,2017.689-699.
[26]Memon A,Gao Z,Nguyen B,Dhanda S,Nickell E,Siemborski R,Micco J.Taming Google-scale continuous testing.In:Proc.of the 39th Int’l Conf.on Software Engineering.2017.233-242.
[27]Marijan D,Liaaen M,Gotlieb A,Sen S,Ieva C.TITAN:Test suite optimization for highly configurable software.In:Proc.of the IEEE Int’l Conf.on Software Testing,Verification and Validation(ICST).2017.524-531.
[28]Ekelund ED,Engstrom E.Efficient regression testing based on test history:An industrial evaluation.In:Proc.of the IEEE Int’l Conf.on Software Maintenance and Evolution.2015.449-457.
[29]Noor TB,Hemmati H.A similarity-based approach for test case prioritization using historical failure data.In:Proc.of the 26th IEEE Int’l Symp.on Software Reliability Engineering(ISSRE).2015.58-68.
[30]Hemmati H,Fang Z,Mantyla MV.Prioritizing manual test cases in traditional and rapid release environments.In:Proc.of the 8th IEEE Int’l Conf.on Software Testing,Verification and Validation(ICST).2015.1-10.
[31]Herzig K,Greiler M,Czerwonka J,Murphy B.The art of testing less without sacrificing quality.In:Proc.of the IEEE/ACM Int’l Conf.on Software Engineering.2015.483-493.
[32]Bach T,Andrzejak A,Pannemans R.Coverage-Based reduction of test execution time:Lessons from a very large industrial project.In:Proc.of the IEEE Int’l Conf.on Software Testing,Verification and Validation Workshops.IEEE,2017.3-12.
[33]Beszédesá,Gergely T,Schrettner L,Jász J,LangóL,Gyimóthy T.Code coverage-based regression test selection and prioritization in WebKit.In:Proc.of the 28th IEEE Int’l Conf.on Software Maintenance(ICSM).2012.46-55.
[34]Strandberg PE,Sundmark D,Afzal W,Ostrand TJ,Weyuker EJ.Experience report:Automated system level regression test prioritization using multiple factors.In:Proc.of the 27th IEEE Int’l Symp.on Software Reliability Engineering(ISSRE).2016.12-23.
[35]Engstr?m E,Runeson P,Ljung A.Improving regression testing transparency and efficiency with history-based prioritization-an industrial case study.In:Proc.of the 4th IEEE Int’l Conf.on Software Testing,Verification and Validation(ICST).2011.367-376.
[36]Srikanth H,Cashman M,Cohen MB.Test case prioritization of build acceptance tests for an enterprise cloud application:An industrial case study.Journal of Systems and Software,2016,119:122-135.
[37]Nguyen CD,Perini A,Tonella P,Kessler FB.Automated continuous testing of multiagent systems.In:Proc.of the 5th European Workshop on Multi-Agent Systems(EUMAS).2007.
[38]Soetens QD,Demeyer S,Zaidman A.Change-Based test selection in the presence of developer tests.In:Proc.of the 17th IEEEEuropean Conf.on Software Maintenance and Reengineering(CSMR).2013.101-110.
[39]V?st S,Wagner S.Trace-Based test selection to support continuous integration in the automotive industry.In:Proc.of the Int’l Workshop on Continuous Software Evolution and Delivery.ACM,2016.34-40.
[40]Campos J,Arcuri A,Fraser G,Abreu R.Continuous test generation:Enhancing continuous integration with automated test generation.European Journal of Oral Sciences,2010,109(4):241-248.
[41]Marijan D.Multi-Perspective regression test prioritization for time-constrained environments.In:Proc.of the IEEE Int’l Conf.on Software Quality,Reliability and Security(QRS).2015.157-162.
[42]Gligoric M,Eloussi L,Marinov D.Practical regression test selection with dynamic file dependencies.In:Proc.of the 2015 Int’l Symp.on Software Testing and Analysis.ACM,2015.211-222.
[43]Legunsen O,Hariri F,Shi A,Yu YF,Zhang LM,Marinov D.An extensive study of static regression test selection in modern software evolution.In:Proc.of the ACM SIGSOFT Int’l Symp.on Foundations of Software Engineering.2016.583-594.
[44]Mirarab S,Akhlaghi S,Tahvildari L.Size-Constrained regression test case selection using multicriteria optimization.IEEE Trans.on Software Engineering,2012,38(4):936-956.
[45]Vasic M,Parvez Z,Milicevic A,Gligoric M.File-Level vs.module-level regression test selection for.NET.In:Proc.of the Joint Meeting on the Foundations of Software Engineering.ACM,2017.848-853.
[46]Jiang B,Zhang Z,Tse TH,Chen TY.How well do test case prioritization techniques support statistical fault localization.In:Proc.of the 33rd Annual IEEE Int’l Computer Software and Applications Conf.2009,1:99-106.
[47]Do H,Mirarab S,Tahvildari L,Rothermel G.The effects of time constraints on test case prioritization:A series of controlled experiments.IEEE Trans.on Software Engineering,2010,36(5):593-617.
[48]Knauss E,Staron M,Meding W,S?der O,Nilsson A,Castell M.Supporting continuous integration by code-churn based test selection.In:Proc.of the IEEE Int’l Workshop on Rapid Continuous Software Engineering.2015.19-25.
[49]D?singer S,Mordinyi R,Biffl S.Communicating continuous integration servers for increasing effectiveness of automated testing.In:Proc.of the 27th IEEE/ACM Int’l Conf.on Automated Software Engineering.2012.374-377.
[50]Marijan D,Liaaen M.Effect of time window on the performance of continuous regression testing.In:Proc.of the IEEE Int’l Conf.on Software Maintenance and Evolution.2016.568-571.
[51]Cho Y,Kim J,Lee E.History-Based test case prioritization for failure information.In:Proc.of the 23rd IEEE Asia-Pacific Software Engineering Conf.(APSEC).2016.385-388.
[52]Marijan D,Liaaen M.Test prioritization with optimally balanced configuration coverage.In:Proc.of the IEEE Int’l Symp.on High Assurance Systems Engineering.IEEE,2017.100-103.
[53]Yoo S,Nilsson R,Harman M.Faster fault finding at Google using multi objective regression test optimization.In:Proc.of the 8th European Software Engineering Conf.and the ACM SIGSOFT Symp.on the Foundations of Software Engineering(ESEC/FSE2011).Szeged,2011.
[54]Marijan D,Gotlieb A,Sen S.Test case prioritization for continuous regression testing:An industrial case study.In:Proc.of the29th IEEE Int’l Conf.on Software Maintenance(ICSM).2013.540-543.
[55]Biswas S,Blanton RD.Reducing test execution cost of integrated,heterogeneous systems using continuous test data.IEEE Trans.on Computer-Aided Design of Integrated Circuits and Systems,2011,30(1):148-158.
[56]Celik A,Vasic M,Milicevic A,Gligoric M.Regression test selection across JVM boundaries.In:Proc.of the 11th Joint Meeting on Foundations of Software Engineering.2017.809-820.
[57]Mondal D,Hemmati H,Durocher S.Exploring test suite diversification and code coverage in multi-objective test case selection.In:Proc.of the IEEE Int’l Conf.on Software Testing,Verification and Validation.2015.1-10.
[2]Vasilescu B,Yu Y,Wang H,Devanbu P,Filko V.Quality and productivity outcomes relating to continuous integration in GitHub.In:Proc.of the Joint Meeting on Foundations of Software Engineering.2015.805-816.
[3]Hilton M,Tunnell T,Huang K,Marinov D,Dig D.Usage,costs,and benefits of continuous integration in open-source projects.In:Proc.of the IEEE/ACM Int’l Conf.on Automated Software Engineering.2016.426-437.
[4]Martensson T,Stahl D,Bosch J.Continuous integration impediments in large-scale industry projects.In:Proc.of the IEEE Int’l Conf.on Software Architecture.IEEE,2017.169-178.
[5]Virmani M.Understanding DevOps&bridging the gap from continuous integration to continuous delivery.In:Proc.of the 5th Int’l Conf.on Innovative Computing Technology(INTECH).2015.78-82.
[6]Rathod N,Surve A.Test orchestration a framework for continuous integration and continuous deployment.In:Proc.of the Int’l Conf.on Pervasive Computing.2015.1-5.
[7]Kitchenham BA,Charters S.Guidelines for performing systematic literature reviews in software engineering.EBSE Technical Report,EBSE-2007-01,ICSE IEEE Computer Society,2007.1051-1052.
[8]Badampudi D,Wohlin C,Kai P.Experiences from using snowballing and database searches in systematic literature studies.In:Proc.of the Int’l Conf.on Evaluation and Assessment in Software Engineering.2015.1-10.
[9]Kitchenham BA,Dyba T,Jorgensen M.Evidence-Based software engineering.In:Proc.of the Int’l Conf.on Software Engineering.IEEE Computer Society,2004.273-281.
[10]Yoo S,Harman M.Regression testing minimization,selection and prioritization:A survey.Software Testing,Verification and Reliability,2012,22(2):67-120.
[11]Kazmi R,Jawawi DNA,Mohamad R,Ghani I.Effective regression test case selection:A systematic literature review.ACMComputing Surveys,2017,50(2):29.
[12]Nanda A,Mani S,Sinha S,Harrold MJ,Orso A.Regression testing in the presence of non-code changes.In:Proc.of the 4th IEEEInt’l Conf.on Software Testing,Verification and Validation.2011.21-30.
[13]Qiu D,Li B,Ji S,Leung H.Regression testing of Web service:A systematic mapping study.ACM Computing Surveys,2015,47(2):21.
[14]Stratis P,Rajan A.Test case permutation to improve execution time.In:Proc.of the IEEE/ACM Int’l Conf.on Automated Software Engineering.2016.45-50.
[15]Huang CY,Chen CS,Lai CE.Evaluation and analysis of incorporating fuzzy expert system approach into test suite reduction.Information and Software Technology,2016,79(C):79-105.
[16]Qu X,Cohen MB,Woolf KM.Combinatorial interaction regression testing:A study of test case generation and prioritization.In:Proc.of the IEEE Int’l Conf.on Software Maintenance.2007.255-264.
[17]Engstr?m E,Runeson P,Skoglund M.A systematic review on regression test selection techniques.Information and Software Technology,2010,52(1):14-30.
[18]Catal C,Mishra D.Test case prioritization:A systematic mapping study.Software Quality Journal,2013,21(3):445-478.
[19]Soetens QD,Demeyer S,Zaidman A,Pérez J.Change-Based test selection:An empirical evaluation.Empirical Software Engineering,2016,21(5):1990-2032.
[20]Blondeau V,Etien A,Anquetil N,Cresson S,Croisy P,Ducasse S.Test case selection in industry:An analysis of issues related to static approaches.Software Quality Journal,2016,1-35.
[21]Spieker H,Gotlieb A,Marijan D,Mossige M.Reinforcement learning for automatic test case prioritization and selection in continuous integration.In:Proc.of the 26th ACM SIGSOFT Int’l Symp.on Software Testing and Analysis.2017.12-22.
[22]Busjaeger B,Xie T.Learning for test prioritization:An industrial case study.In:Proc.of the 24th ACM SIGSOFT Int’l Symp.on Foundations of Software Engineering.2016.975-980.
[23]Elbaum S,Rothermel G,Penix J.Techniques for improving regression testing in continuous integration development environments.In:Proc.of the 22nd ACM SIGSOFT Int’l Symp.on Foundations of Software Engineering.2014.235-245.
[24]Parsai A,Soetens QD,Murgia A,Demeyer S.Considering polymorphism in change-based test suite reduction.In:Proc.of the Int’l Conf.on Agile Software Development.Cham:Springer-Verlag,2014.166-181.
[25]Shi A,Thummalapenta S,Lahiri SK,Bjorner N,Czerwonka J.Optimizing test placement for module-level regression testing.In:Proc.of the IEEE/ACM Int’l Conf.on Software Engineering.IEEE Computer Society,2017.689-699.
[26]Memon A,Gao Z,Nguyen B,Dhanda S,Nickell E,Siemborski R,Micco J.Taming Google-scale continuous testing.In:Proc.of the 39th Int’l Conf.on Software Engineering.2017.233-242.
[27]Marijan D,Liaaen M,Gotlieb A,Sen S,Ieva C.TITAN:Test suite optimization for highly configurable software.In:Proc.of the IEEE Int’l Conf.on Software Testing,Verification and Validation(ICST).2017.524-531.
[28]Ekelund ED,Engstrom E.Efficient regression testing based on test history:An industrial evaluation.In:Proc.of the IEEE Int’l Conf.on Software Maintenance and Evolution.2015.449-457.
[29]Noor TB,Hemmati H.A similarity-based approach for test case prioritization using historical failure data.In:Proc.of the 26th IEEE Int’l Symp.on Software Reliability Engineering(ISSRE).2015.58-68.
[30]Hemmati H,Fang Z,Mantyla MV.Prioritizing manual test cases in traditional and rapid release environments.In:Proc.of the 8th IEEE Int’l Conf.on Software Testing,Verification and Validation(ICST).2015.1-10.
[31]Herzig K,Greiler M,Czerwonka J,Murphy B.The art of testing less without sacrificing quality.In:Proc.of the IEEE/ACM Int’l Conf.on Software Engineering.2015.483-493.
[32]Bach T,Andrzejak A,Pannemans R.Coverage-Based reduction of test execution time:Lessons from a very large industrial project.In:Proc.of the IEEE Int’l Conf.on Software Testing,Verification and Validation Workshops.IEEE,2017.3-12.
[33]Beszédesá,Gergely T,Schrettner L,Jász J,LangóL,Gyimóthy T.Code coverage-based regression test selection and prioritization in WebKit.In:Proc.of the 28th IEEE Int’l Conf.on Software Maintenance(ICSM).2012.46-55.
[34]Strandberg PE,Sundmark D,Afzal W,Ostrand TJ,Weyuker EJ.Experience report:Automated system level regression test prioritization using multiple factors.In:Proc.of the 27th IEEE Int’l Symp.on Software Reliability Engineering(ISSRE).2016.12-23.
[35]Engstr?m E,Runeson P,Ljung A.Improving regression testing transparency and efficiency with history-based prioritization-an industrial case study.In:Proc.of the 4th IEEE Int’l Conf.on Software Testing,Verification and Validation(ICST).2011.367-376.
[36]Srikanth H,Cashman M,Cohen MB.Test case prioritization of build acceptance tests for an enterprise cloud application:An industrial case study.Journal of Systems and Software,2016,119:122-135.
[37]Nguyen CD,Perini A,Tonella P,Kessler FB.Automated continuous testing of multiagent systems.In:Proc.of the 5th European Workshop on Multi-Agent Systems(EUMAS).2007.
[38]Soetens QD,Demeyer S,Zaidman A.Change-Based test selection in the presence of developer tests.In:Proc.of the 17th IEEEEuropean Conf.on Software Maintenance and Reengineering(CSMR).2013.101-110.
[39]V?st S,Wagner S.Trace-Based test selection to support continuous integration in the automotive industry.In:Proc.of the Int’l Workshop on Continuous Software Evolution and Delivery.ACM,2016.34-40.
[40]Campos J,Arcuri A,Fraser G,Abreu R.Continuous test generation:Enhancing continuous integration with automated test generation.European Journal of Oral Sciences,2010,109(4):241-248.
[41]Marijan D.Multi-Perspective regression test prioritization for time-constrained environments.In:Proc.of the IEEE Int’l Conf.on Software Quality,Reliability and Security(QRS).2015.157-162.
[42]Gligoric M,Eloussi L,Marinov D.Practical regression test selection with dynamic file dependencies.In:Proc.of the 2015 Int’l Symp.on Software Testing and Analysis.ACM,2015.211-222.
[43]Legunsen O,Hariri F,Shi A,Yu YF,Zhang LM,Marinov D.An extensive study of static regression test selection in modern software evolution.In:Proc.of the ACM SIGSOFT Int’l Symp.on Foundations of Software Engineering.2016.583-594.
[44]Mirarab S,Akhlaghi S,Tahvildari L.Size-Constrained regression test case selection using multicriteria optimization.IEEE Trans.on Software Engineering,2012,38(4):936-956.
[45]Vasic M,Parvez Z,Milicevic A,Gligoric M.File-Level vs.module-level regression test selection for.NET.In:Proc.of the Joint Meeting on the Foundations of Software Engineering.ACM,2017.848-853.
[46]Jiang B,Zhang Z,Tse TH,Chen TY.How well do test case prioritization techniques support statistical fault localization.In:Proc.of the 33rd Annual IEEE Int’l Computer Software and Applications Conf.2009,1:99-106.
[47]Do H,Mirarab S,Tahvildari L,Rothermel G.The effects of time constraints on test case prioritization:A series of controlled experiments.IEEE Trans.on Software Engineering,2010,36(5):593-617.
[48]Knauss E,Staron M,Meding W,S?der O,Nilsson A,Castell M.Supporting continuous integration by code-churn based test selection.In:Proc.of the IEEE Int’l Workshop on Rapid Continuous Software Engineering.2015.19-25.
[49]D?singer S,Mordinyi R,Biffl S.Communicating continuous integration servers for increasing effectiveness of automated testing.In:Proc.of the 27th IEEE/ACM Int’l Conf.on Automated Software Engineering.2012.374-377.
[50]Marijan D,Liaaen M.Effect of time window on the performance of continuous regression testing.In:Proc.of the IEEE Int’l Conf.on Software Maintenance and Evolution.2016.568-571.
[51]Cho Y,Kim J,Lee E.History-Based test case prioritization for failure information.In:Proc.of the 23rd IEEE Asia-Pacific Software Engineering Conf.(APSEC).2016.385-388.
[52]Marijan D,Liaaen M.Test prioritization with optimally balanced configuration coverage.In:Proc.of the IEEE Int’l Symp.on High Assurance Systems Engineering.IEEE,2017.100-103.
[53]Yoo S,Nilsson R,Harman M.Faster fault finding at Google using multi objective regression test optimization.In:Proc.of the 8th European Software Engineering Conf.and the ACM SIGSOFT Symp.on the Foundations of Software Engineering(ESEC/FSE2011).Szeged,2011.
[54]Marijan D,Gotlieb A,Sen S.Test case prioritization for continuous regression testing:An industrial case study.In:Proc.of the29th IEEE Int’l Conf.on Software Maintenance(ICSM).2013.540-543.
[55]Biswas S,Blanton RD.Reducing test execution cost of integrated,heterogeneous systems using continuous test data.IEEE Trans.on Computer-Aided Design of Integrated Circuits and Systems,2011,30(1):148-158.
[56]Celik A,Vasic M,Milicevic A,Gligoric M.Regression test selection across JVM boundaries.In:Proc.of the 11th Joint Meeting on Foundations of Software Engineering.2017.809-820.
[57]Mondal D,Hemmati H,Durocher S.Exploring test suite diversification and code coverage in multi-objective test case selection.In:Proc.of the IEEE Int’l Conf.on Software Testing,Verification and Validation.2015.1-10.