基于指令交换的代码混淆方法
|
摘要
软件程序是按一定顺序排列的指令序列,指令的排列组合构成了千变万化的程序语义.指令顺序重排通常会相应地导致程序语义的变化,通过分析相邻指令序列的相对独立性,可以在不影响程序语义的前提下交换相邻指令序列,增大指令距离,改变程序特征,在一定程度上增加逆向分析代价.通过改进程序的形式化定义论证相邻指令交换的充分条件,采用模拟退火算法实现随机化的指令乱序混淆方法,并将指令乱序方法与虚拟机代码保护技术融合,实现基于指令乱序的虚拟机代码保护系统IS-VMP,使用加密算法实例进行系统测试,验证了指令乱序混淆算法的可行性与有效性.
The program is a sequence of instructions in a certain order, and the permutation and combinations of instructions constitute the ever-changing program semantics. Although reordering instructions usually changes the program semantics, it is possible to swap adjacent instruction sequences without changing the program semantics via analyzing the relative independence of adjacent instruction sequences. Instructions swapping increases the distance of instructions and change characteristics of the program, which raises the cost of reverse analysis to a certain extent. Sufficient conditions of instructions swapping are proven by the improvement of the formal definition of the program, upon which the randomize method of instructions reordering based on simulated annealing is proposed in the study.Furthermore, a prototype of IS-VMP(virtual machine protection system based on instructions reordering) is implemented. In addition, the experiments are carried out with a set of encryption algorithms. Experiment results show that instruction reordering is effective and applicable for anti-reversing.
The program is a sequence of instructions in a certain order, and the permutation and combinations of instructions constitute the ever-changing program semantics. Although reordering instructions usually changes the program semantics, it is possible to swap adjacent instruction sequences without changing the program semantics via analyzing the relative independence of adjacent instruction sequences. Instructions swapping increases the distance of instructions and change characteristics of the program, which raises the cost of reverse analysis to a certain extent. Sufficient conditions of instructions swapping are proven by the improvement of the formal definition of the program, upon which the randomize method of instructions reordering based on simulated annealing is proposed in the study.Furthermore, a prototype of IS-VMP(virtual machine protection system based on instructions reordering) is implemented. In addition, the experiments are carried out with a set of encryption algorithms. Experiment results show that instruction reordering is effective and applicable for anti-reversing.
引文
[1]Collberg C,Thomborson CD,Low D.A taxonomy of obfuscating transformations.Technical Report,148,Department of Computer Science the University of Auckland New Zealand,1997.
[2]Wroblewski G.General method of program code obfuscation[Ph.D.Thesis].Wroclaw:Institute of Engineering Cybernetics,Wroclaw University of Technology,2002.
[3]Birrer BD,Raines RA,Baldwin RO,et al.Program fragmentation as a metamorphic software protection.In:Proc.of the Int’l Symp.on Information Assurance and Security.IEEE,2007.369-374.[doi:10.1109/IAS.2007.28]
[4]Li Y,Zuo ZH.An overview of object-code obfuscation technologies.Computer Technology and Development,2007,17(4):125-127(in Chinese with English abstract).[doi:10.3969/j.issn.1673-629X.2007.04.034]
[5]Ghosh S,Hiser J,Davidson JW.Replacement attacks against VM-protected applications.ACM SIGPLAN Notices,2012,47(7):203-214.[doi:10.1145/2365864.2151051]
[6]Coogan KP.Deobfuscation of packed and virtualization-obfuscation protected binaries[Ph.D.Thesis].University of Arizona,2011.
[7]Coogan K,Lu G,Debray S.Deobfuscation of virtualization-obfuscated software:A semantics-based approach.In:Proc.of the ACM Conf.on Computer and Communications Security.ACM Press,2011.275-284.[doi:10.1145/2046707.2046739]
[8]Fang DY,Zhang H,Tang ZY,Chen XJ.DAS-VMP:A virtual machine-based software protection method for defending against semantic attacks.Journal of Sichuan University(Engineering Science Edition),2017,49(1):159-168(in Chinese with English abstract).[doi:10.15961/j.jsuese.2017.01.021]
[9]Banescu S,Collberg C,Ganesh V,et al.Code obfuscation against symbolic execution attacks.In:Proc.of the Conf.on Computer Security Applications.ACM Press,2016.189-200.[doi:10.1145/2991079.2991114]
[10]Xie X,Liu FL,Lu B,et al.Virtual machine protection based on Handler obfuscation enhancement.Computer Engineering and Applications,2016,52(15):146-152(in Chinese with English abstract).[doi:10.3778/j.issn.1002-8331.1410-0299]
[11]Wu WM,Xu WF,Lin ZY,et al.Software protection technique based on improved virtual machine.Computer Engineering&Science,2014,36(4):655-661(in Chinese with English abstract).[doi:10.3969/j.issn.1007-130X.2014.04.015]
[12]Fang DY,Zhao Y,Wang HJ,Gu YX,Xu GL.Software protection based on virtual machine with time diversity.Ruan Jian Xue Bao/Journal of Software,2015,26(6):1322-1339(in Chinese with English abstract).http://www.jos.org.cn/1000-9825/4592.htm[doi:10.13328/j.cnki.jos.004592]
[13]Wang H,Fang D,Li G,et al.TDVMP:Improved virtual machine-based software protection with time diversity.In:Proc.of the ACM SIGPLAN on Program Protection and Reverse Engineering Workshop.2014.1-9.[doi:10.1145/2556464.2556468]
[14]Wang H,Fang D,Li G,et al.NISLVMP:Improved virtual machine-based software protection.In:Proc.of the 2013 9th Int’l Conf.on Computational Intelligence and Security(CIS).IEEE,2013.479-483.[doi:10.1145/2556464.2556468]
[15]Kuang K,Tang Z,Gong X,et al.Exploiting dynamic scheduling for VM-based code obfuscation.In:Proc.of the Trustcom/Bigdatase/Ispa.IEEE,2017.489-496.[doi:10.1109/TrustCom.2016.0101]
[16]Tang Z,Li G,Fang D,et al.Code virtualized protection system with instruction set randomization.Journal of Huazhong University of Science&Technology,2016,44(3):28-33(in Chinese with English abstract).[doi:10.13245/j.hust.160306]
[17]Collberg C,Thomborson C,Low D.Manufacturing cheap,resilient,and stealthy opaque constructs.In:Proc.of the ACMSIGPLAN-SIGACT Symp.on Principles of Programming Languages.1997.184-196.[doi:10.1145/268946.268962]
[18]Pretschner A,Pretschner A,Pretschner A,et al.Code obfuscation against symbolic execution attacks.In:Proc.of the Conf.on Computer Security Applications.ACM Press,2016.189-200.[doi:10.1145/2991079.2991114]
[19]Guillot Y,Gazet A.Automatic binary deobfuscation.Journal of Computer Virology and Hacking Techniques,2010,6(3):261-276.[doi:10.1007/s11416-009-0126-4]
[20]Lemay E,Ford MD,Keefe K,et al.Model-based security metrics using adversary view security evaluation(ADVISE).In:Proc.of the 8th Int’l Conf.on Quantitative Evaluation of Systems.IEEE Computer Society,2011.191-200.[doi:10.1109/QEST.2011.34]
[21]Mavrogiannopoulos N,Kisserli N,Preneel B.A taxonomy of self-modifying code for obfuscation.Computers&Security,2011,30(8):679-691.[doi:10.1016/j.cose.2011.08.007]
[22]Zhao YJ,Tang ZY,Wang N,et al.Evaluation of code obfuscating transformation.Ruan Jian Xue Bao/Journal of Software,2012,23(3):700-711(in Chinese with English abstract).http://www.jos.org.cn/1000-9825/3994.htm[doi:10.3724/SP.J.1001.2012.03994]
[23]Wang H,Fang D,Li J,et al.The research and discussion on effectiveness evaluation of software protection.In:Proc.of the Int’l Conf.on Computational Intelligence and Security.IEEE Computer Society,2016.628-632.[doi:10.1109/CIS.2016.0152]
[4]李勇,左志宏.目标代码混淆技术综述.计算机技术与发展,2007,17(4):125-127.[doi:10.3969/j.issn.1673-629X.2007.04.034]
[8]房鼎益,张恒,汤战勇,等.一种抗语义攻击的虚拟化软件保护方法.四川大学学报(工程科学版),2017,49(1):159-168.[doi:10.15961/j.jsuese.2017.01.021]
[10]谢鑫,刘粉林,芦斌,等.Handler混淆增强的虚拟机保护方法.计算机工程与应用,2016,52(15):146-152.[doi:10.3778/j.issn.1002-8331.1410-0299]
[11]吴伟民,许文锋,林志毅,等.基于增强型虚拟机的软件保护技术.计算机工程与科学,2014,36(4):655-661.[doi:10.3969/j.issn.1007-130X.2014.04.015]
[12]房鼎益,赵媛,王怀军,顾元祥,许广莲.一种具有时间多样性的虚拟机软件保护方法.软件学报,2015,26(6):1322-1339. http://www.jos.org.cn/1000-9825/4592.htm[doi:10.13328/j.cnki.jos.004592]
[16]汤战勇,李光辉,房鼎益,等.一种具有指令集随机化的代码虚拟化保护系统.华中科技大学学报(自然科学版),2016,44(3):28-33.[doi:10.13245/j.hust.160306]
[22]赵玉洁,汤战勇,王妮,等.代码混淆算法有效性评估.软件学报,2012,23(3):700-711.http://www.jos.org.cn/1000-9825/3994.htm[doi:10.3724/SP.J.1001.2012.03994]
[2]Wroblewski G.General method of program code obfuscation[Ph.D.Thesis].Wroclaw:Institute of Engineering Cybernetics,Wroclaw University of Technology,2002.
[3]Birrer BD,Raines RA,Baldwin RO,et al.Program fragmentation as a metamorphic software protection.In:Proc.of the Int’l Symp.on Information Assurance and Security.IEEE,2007.369-374.[doi:10.1109/IAS.2007.28]
[4]Li Y,Zuo ZH.An overview of object-code obfuscation technologies.Computer Technology and Development,2007,17(4):125-127(in Chinese with English abstract).[doi:10.3969/j.issn.1673-629X.2007.04.034]
[5]Ghosh S,Hiser J,Davidson JW.Replacement attacks against VM-protected applications.ACM SIGPLAN Notices,2012,47(7):203-214.[doi:10.1145/2365864.2151051]
[6]Coogan KP.Deobfuscation of packed and virtualization-obfuscation protected binaries[Ph.D.Thesis].University of Arizona,2011.
[7]Coogan K,Lu G,Debray S.Deobfuscation of virtualization-obfuscated software:A semantics-based approach.In:Proc.of the ACM Conf.on Computer and Communications Security.ACM Press,2011.275-284.[doi:10.1145/2046707.2046739]
[8]Fang DY,Zhang H,Tang ZY,Chen XJ.DAS-VMP:A virtual machine-based software protection method for defending against semantic attacks.Journal of Sichuan University(Engineering Science Edition),2017,49(1):159-168(in Chinese with English abstract).[doi:10.15961/j.jsuese.2017.01.021]
[9]Banescu S,Collberg C,Ganesh V,et al.Code obfuscation against symbolic execution attacks.In:Proc.of the Conf.on Computer Security Applications.ACM Press,2016.189-200.[doi:10.1145/2991079.2991114]
[10]Xie X,Liu FL,Lu B,et al.Virtual machine protection based on Handler obfuscation enhancement.Computer Engineering and Applications,2016,52(15):146-152(in Chinese with English abstract).[doi:10.3778/j.issn.1002-8331.1410-0299]
[11]Wu WM,Xu WF,Lin ZY,et al.Software protection technique based on improved virtual machine.Computer Engineering&Science,2014,36(4):655-661(in Chinese with English abstract).[doi:10.3969/j.issn.1007-130X.2014.04.015]
[12]Fang DY,Zhao Y,Wang HJ,Gu YX,Xu GL.Software protection based on virtual machine with time diversity.Ruan Jian Xue Bao/Journal of Software,2015,26(6):1322-1339(in Chinese with English abstract).http://www.jos.org.cn/1000-9825/4592.htm[doi:10.13328/j.cnki.jos.004592]
[13]Wang H,Fang D,Li G,et al.TDVMP:Improved virtual machine-based software protection with time diversity.In:Proc.of the ACM SIGPLAN on Program Protection and Reverse Engineering Workshop.2014.1-9.[doi:10.1145/2556464.2556468]
[14]Wang H,Fang D,Li G,et al.NISLVMP:Improved virtual machine-based software protection.In:Proc.of the 2013 9th Int’l Conf.on Computational Intelligence and Security(CIS).IEEE,2013.479-483.[doi:10.1145/2556464.2556468]
[15]Kuang K,Tang Z,Gong X,et al.Exploiting dynamic scheduling for VM-based code obfuscation.In:Proc.of the Trustcom/Bigdatase/Ispa.IEEE,2017.489-496.[doi:10.1109/TrustCom.2016.0101]
[16]Tang Z,Li G,Fang D,et al.Code virtualized protection system with instruction set randomization.Journal of Huazhong University of Science&Technology,2016,44(3):28-33(in Chinese with English abstract).[doi:10.13245/j.hust.160306]
[17]Collberg C,Thomborson C,Low D.Manufacturing cheap,resilient,and stealthy opaque constructs.In:Proc.of the ACMSIGPLAN-SIGACT Symp.on Principles of Programming Languages.1997.184-196.[doi:10.1145/268946.268962]
[18]Pretschner A,Pretschner A,Pretschner A,et al.Code obfuscation against symbolic execution attacks.In:Proc.of the Conf.on Computer Security Applications.ACM Press,2016.189-200.[doi:10.1145/2991079.2991114]
[19]Guillot Y,Gazet A.Automatic binary deobfuscation.Journal of Computer Virology and Hacking Techniques,2010,6(3):261-276.[doi:10.1007/s11416-009-0126-4]
[20]Lemay E,Ford MD,Keefe K,et al.Model-based security metrics using adversary view security evaluation(ADVISE).In:Proc.of the 8th Int’l Conf.on Quantitative Evaluation of Systems.IEEE Computer Society,2011.191-200.[doi:10.1109/QEST.2011.34]
[21]Mavrogiannopoulos N,Kisserli N,Preneel B.A taxonomy of self-modifying code for obfuscation.Computers&Security,2011,30(8):679-691.[doi:10.1016/j.cose.2011.08.007]
[22]Zhao YJ,Tang ZY,Wang N,et al.Evaluation of code obfuscating transformation.Ruan Jian Xue Bao/Journal of Software,2012,23(3):700-711(in Chinese with English abstract).http://www.jos.org.cn/1000-9825/3994.htm[doi:10.3724/SP.J.1001.2012.03994]
[23]Wang H,Fang D,Li J,et al.The research and discussion on effectiveness evaluation of software protection.In:Proc.of the Int’l Conf.on Computational Intelligence and Security.IEEE Computer Society,2016.628-632.[doi:10.1109/CIS.2016.0152]
[4]李勇,左志宏.目标代码混淆技术综述.计算机技术与发展,2007,17(4):125-127.[doi:10.3969/j.issn.1673-629X.2007.04.034]
[8]房鼎益,张恒,汤战勇,等.一种抗语义攻击的虚拟化软件保护方法.四川大学学报(工程科学版),2017,49(1):159-168.[doi:10.15961/j.jsuese.2017.01.021]
[10]谢鑫,刘粉林,芦斌,等.Handler混淆增强的虚拟机保护方法.计算机工程与应用,2016,52(15):146-152.[doi:10.3778/j.issn.1002-8331.1410-0299]
[11]吴伟民,许文锋,林志毅,等.基于增强型虚拟机的软件保护技术.计算机工程与科学,2014,36(4):655-661.[doi:10.3969/j.issn.1007-130X.2014.04.015]
[12]房鼎益,赵媛,王怀军,顾元祥,许广莲.一种具有时间多样性的虚拟机软件保护方法.软件学报,2015,26(6):1322-1339. http://www.jos.org.cn/1000-9825/4592.htm[doi:10.13328/j.cnki.jos.004592]
[16]汤战勇,李光辉,房鼎益,等.一种具有指令集随机化的代码虚拟化保护系统.华中科技大学学报(自然科学版),2016,44(3):28-33.[doi:10.13245/j.hust.160306]
[22]赵玉洁,汤战勇,王妮,等.代码混淆算法有效性评估.软件学报,2012,23(3):700-711.http://www.jos.org.cn/1000-9825/3994.htm[doi:10.3724/SP.J.1001.2012.03994]