Download Abstract Stack Grap to Detect Obfuscated Calls In Binaries - Notes | CS 6463 and more Papers Computer Science in PDF only on Docsity!
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
Abstract Stack Graph to Detect Obfuscated Calls in Binaries
Arun Lakhotia and Eric Uday Kumar
Center for Advanced Computer Studies, University of Louisiana at Lafayette, LA
{ arun, euk4141 } @cacs.louisiana.edu
Abstract
Information about calls to the operating system (or kernel libraries) made by a binary executable may be used to determine whether the binary is malicious. Being aware of this approach, malicious programmers hide this information by making such calls without using the call instruction. For instance, the ‘call addr’ instruction may be replaced by two push instructions and a return instruction, the first push pushes the address of instruction after the return instruction, and the second push pushes the address addr. The code may be further obfuscated by spreading the three instructions and by splitting each instruction into multiple instructions. This paper presents a method to statically detect obfuscated calls in binary code. The notion of abstract stack is introduced to associate each element in the stack to the instruction that pushes the element. An abstract stack graph is a concise representation of all abstract stacks at every point in the program. An abstract stack graph, created by abstract interpretation of the binary executables, may be used to detect obfuscated calls and other stack related obfuscations.
1. Introduction
Programmers obfuscate their code with the intent of making it difficult to discern information from the code. Programs may be obfuscated to protect intellectual property and to increase security of code (by making it difficult for others to identify vulnerabilities) [8, 13, 19]. Programs may also be obfuscated to hide malicious behavior and to evade detection by anti-virus scanners [6, 14, 17]. The primary goal of obfuscation is to increase the effort involved in manually or automatically analyzing a program. In the context of anti-virus scanning, the context of our study, automated analysis may be
performed at the desktop, at quarantine servers in an enterprise, or on back-end machines of an anti-virus company’s laboratory [16]. In contrast, manual analysis is performed by engineers in Emergency Response Teams of anti-virus companies and research laboratories. The goal of obfuscation in malicious programs—virii, worms, Trojans, spy wares, backdoors—is to escape detection by automated analysis and significantly delay detection by manual analysis. A common obfuscation technique that is found in viruses, henceforth used generically to mean malicious programs, is that they obfuscate call instructions [17]. For instance, the call addr instruction may be replaced by two push instructions and a ret instruction, the first push pushing the address of instruction after the ret instruction, the second push pushing the address addr. The code may be further obfuscated by spreading the three instructions and by further splitting each instruction into multiple instructions. Obfuscation of call instructions breaks most static analysis based methods for detecting a virus since these methods depend on recognizing call instructions to (a) identify the kernel functions used by the program and (b) to identify procedures in the code. The obfuscation also takes away important cues that are used during manual analysis. We are then left only with dynamic analysis, i.e., running a suspect program in an emulator and observing the kernel calls it makes. Such analysis can easily be thwarted by what is termed as “picky” virus—one that does not always execute its malicious payload. In addition dynamic analyzers must use some heuristic to determine when to stop analyzing a program, for it may not terminate without user input. Virus writers can bypass stopping heuristics by introducing a delay loop that simply wastes cycles. It is therefore important to detect obfuscated calls both for static and dynamic analysis of viruses. This paper presents a method to statically detect This work was supported in part by funds from Louisiana obfuscated calls when the obfuscation is performed by Governor's Information Technology Initiative.
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
using other stack (-related) instructions, such as push and pop , ret , or instructions that can statically be mapped to such stack operations. The method uses abstract interpretation [10] wherein the stack instructions are interpreted to operate on an abstract stack. Instead of keeping actual data elements, an abstract stack keeps the address of the instruction that pushes an element on the stack. The infinite set of abstract stacks resulting from all possible executions of a program, a la, static analysis, is concisely represented in an abstract stack graph. Our method may be used to improve manual and automated analysis tools, thereby raising the level of difficulty for a virus writer. Our method can help by removing some common obfuscation techniques from the toolkit of a virus writer. However, we do not claim that the method can deobfuscate all stack related obfuscations. Indeed, writing a program that detects all obfuscations is not achievable for the general problem maps to detecting program equivalence, which is undecidable. The method we present is a partial solution. It addresses only the evaluation of operations that can be mapped to stack push and pop instructions, where each is performed as a unit operation. It does not model situation where the push and pop instructions themselves may be decomposed into multiple instructions, such as one to move the stack pointer and one to move data in/out of the stack. Further, our solution does not model other memory areas, the content of the stack, and the content of registers. This deficiency may be overcome by combining our stack model with the Balakrishnan and Reps’ method for analyzing the content of memory locations [3]. Section 2 presents related work in this area. Section 3 presents the notion of an abstract stack and the abstract stack graph. Section 4 presents our algorithm to construct the abstract stack graph. Section 5 describes how the abstract stack graph may be used to detect various obfuscations. Section 6 outlines future work to develop a complete solution for detecting obfuscations. Section 7 concludes this paper.
2. Background and Related Work
To extract meaningful information from a binary it is first disassembled, i.e., translated to assembly instructions [5, 11, 13, 15]. The disassembled code is usually analyzed further, often following steps similar to those performed for decompilation [7]. Vinciguerra et al. have compiled a survey of disassembly and decompilation techniques [18]. Lakhotia and Singh [12] discuss how a virus writer could attack the various
stages in the decompilation of binaries by taking advantage of the limitation of static analysis. Indeed, Linn et al. [13] present code obfuscation techniques for disrupting the disassembly phase, making it difficult for static analysis to even get started. The art of obfuscation is very advanced. Collberg et al. [9] present “a taxonomy of obfuscating transformations” and a detailed theoretical description of such transformations. There exist obfuscation engines that may be linked to a program to create a metamorphic virus, a virus that creates a transformed copy of itself before propagation. The transformations are such that they change the byte sequence of the executable but do not disrupt the functionality of the program. Two such engines are Mistfall (by z0mbie), which is a library for binary obfuscation [2], and Burneye (by TESO), which is a Linux binary encapsulation tool [1]. Metamorphic viruses are particularly insidious because two copies of the virus do not have the same signature. Hence, they escape signature based anti- virus scanners [6]. Such viruses can sometimes be detected if the operating system calls made by the program can be determined. For example Symantec’s Bloodhound technology uses classification algorithms to compare the set against a database of calls made by known viruses and clean programs [16]. The challenge, however, is in detecting the operating system calls made by a program. The PE and ELF format for binaries include mechanism to inform the linker about the libraries used by a program. But there is no requirement that this information be included in the file headers. In Windows, the entry point address of various system functions may be computed by a program at runtime using a Kernel function called GetProcAddress. Win32.Evol worm uses precisely this method for getting addresses of kernel functions and further obfuscates the method it uses to call these functions. There is hope, however. A recent result by Barak et al. [4] proves that in general program obfuscation is impossible, i.e., there are certain program properties that cannot be obfuscated. This is likely to have an effect on the pace at which new metamorphic transformations are introduced. Lakhotia and Singh [12] observe that though metamorphic viruses pose a serious challenge to anti-virus technologies, the virus writers too are confronted with the same theoretical limits and have to address some the same challenges that the anti-virus technologies face. Indeed research results in detecting obfuscated viruses are beginning to emerge. Christodrescu and Jha use abstract patterns to detect malicious patterns in executables [6]. Mohammed has developed a technique
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
B10: pop edx B11: beq B
5
E: // entry
B3: push ebx B8: pop ebx
B12: call abc
B4: push ecx B5: dec ecx B6: beqz B B7: jmp B
B0: push eax B1: sub ecx, 1h B2: beqz B
1
4
3 7
6
2
(b)
B9: push esi 8
Sample Program
E: //entry point B0: push eax B1: s ub ecx, 1h B2: beqz B B3: push ebx B4: push ecx B5: dec ecx B6: beqz B B7: jmp B B8: pop ebx B9: push esi B10: pop edx B11: beq B B12: call abc
(a)
Figure 2. Abstract Stack Graph for a Sample Program
B
B12 B
B
E
2
1
6 8
5
7
5 7
3
B4^4
E
B
6
Stack Growth
B B B E
B B B B B E
2 4
8
E
B12 B B B E
B B B B B E
B B B B B B B E
B B B B B B B E
1 2 3 1 2 3
(c) (d)
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
with the set of program points P = {3, 5, 7}. Program points in P receive abstract stacks with top B3. Two possible abstract stacks, when traversed from asp = B are, B3|B0|E and B3|B4|B3|B0|E.
4. Constructing an abstract stack graph
Figure 3a presents the evaluation function ℰ for constructing an abstract stack graph. The domain INST is the abstract syntax domain, representing the set of instructions. Each instruction is annotated with its address in the program. Thus, [ m: call addr ] is the instruction ‘ call addr’ at address m. The abstract domain ASG represents the domain of abstract syntax graphs. An element of ASG is a three-tuple ( N , AE , ASP ), where N and AE have the same meaning as in the definition of abstract syntax graph. However, the set ASP is not the same as ASPR. ASP ⊆ ADDR is the set of stack tops. ASP is a projection of ASPR. Loosely speaking, ASP and ASPR are related as follows: Let ℰ [ m: inst ] asg = ( N , AE , ASP ) , then ( m , a ) ∈ ASPR where a ∈ ASP. Figure 3b gives the abstract operations used to define the evaluation function. The operations are abspush , abspop , absret , reset , and i that operate on
the domain ASG. The evaluation function and the abstract operations depend on the following primitive operators: next : ADDR → ADDR , gives the address of the next instruction. inst : ADDR → INST , gives the instruction at an address. validcall : ADDR → Boolean, checks whether the instruction at a given address is a call instruction. Operation abspush pushes a new address on the abstract stack. It is used in the evaluation of the call and push instructions. These two instructions are representative of instructions that perform the push operation. Other instructions may be modeled similar to these instructions. For example, the INT (software interrupt) instruction may be modeled like the call instruction. Instructions that increase the content of stack by directly manipulating the stack pointer, such as sub esp , 8 h , are modeled using the push instruction. Operation abspop pops an element from the abstract stack resulting in a new set of top of stack. The operator is used in the evaluation of ret and pop instructions. Operation absret supports the evaluation of the ret instruction. It checks whether the address at the top of stack represents the address of a call instruction. If so, it returns the address of instruction after the call. Since the abstract stack does not maintain actual return address, the address to return to when a call is made by
abspush : ADDR → ASG → ASG abspush m ( N , AE , ASP ) = ( N ∪ { m }, AE ∪ { m → asp | asp ∈ ASP }, { m } ) abspop : ASG → ASG abspop m ( N , AE , ASP ) = ( N , AE , { x | a ∈ ASP , (a x) ∈ AE } ) absret: ASG → ℘ ADDR absret ( N , AE , ASP ) = { next ( x ) | a ∈ ASP , ( a x ) ∈ AE , validcall ( x )} reset : ADDR → ASG → ASG reset m ( N , AE , ASP ) = ( N ∪ { m }, AE , { m } ) i : ASG → ASG i ( N , AE , ASP ) = ( N , AE , ASP )
Figure 3a. Abstract Operations
ℰ: INST → ASG → ASG
ℰ [ m: push ] asg = ℰ next (m) ( abspush m asg )
ℰ [ m: call addr ] asg = ℰ inst ( addr ) ( abspush m asg )
ℰ [ m: ret ] asg =
∪ ℰ n ( abspop m asg )
n ∈ absret asg ℰ [ m: pop ] asg = ℰ next (m) ( abspop m asg )
ℰ [ m: jnz addr ] asg = (ℰ inst ( addr ) asg ) ∪ (ℰ next (m) asg )
ℰ [ m: jmp addr ] asg = ℰ inst ( addr ) ( i asg )
ℰ [ m: mov esp x ] asg = ℰ next (m) ( reset m asg )
Figure 3b. Evaluation Function
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
In the examples discussed below each program point of interest receives a single abstract stack. Hence, the discussion focuses on the specific stack. This should not be construed to imply that the methods require that the program points of interest receive a single abstract stack. The method discussed may simply be applied to every abstract stack received by a program point.
5.1. Detecting obfuscated calls
The semantics of a call addr instruction may be defined operationally as follows:
- Push the address of the next instruction on the stack
- Assign the address addr to the instruction pointer ( eip ). Figure 4 contains several examples of obfuscated calls. Each example achieves the semantics of a call using a different sequence of instructions. Figure 4(b) shows a program that performs the semantics of a call using a combination of push and jmp instructions. That the jmp instruction actually
performs a call becomes known from the abstract stack graph at the entry point of the call. When an address is not known to be the entry point of a procedure, the abstract stack graph at the ret instruction, program point 6, discloses the obfuscation. During normal execution the top of the stack at this program point contains the return address L3 pushed by the push instruction at label L0. In the abstract stack graph, the top of stack at program point 6 contains is E. That the ret instruction is returning from an obfuscated call is detected because E is not the address of a call instruction. Figure 4(c) shows two obfuscations of call. The two differ in how control is transferred to the target address. In the first, the target address is pushed on the stack and a ret instruction pops this address from the stack and transfers control. In the second, the target address is pushed on the stack, it is then popped into a register, and an indirect jump is performed to the address in the register. The labels and program points in the two examples have been chosen such that both examples have the same stack and abstract stack graph. In both examples the actual transfer of control is done
Figure 5. Possible ways of obfuscating parameters to calls
E: push E ;entry L0: push eax L1: push ebx L2: call fun L3: … … L8: ;fun here
Normal passing
E: push E ;entry 1 L0: push eax 2 L1: push edx 3 L2: pop eax 4 L3: push ebx 5 L4: call fun 6 L5: … L8: ;fun here
Redundant push / pop
E: push E ;entry 1 L0: push eax 2 L1: push ebx 3 L2: call fun1 4 L3: call fun 6 L4: ;fun1 here L5: ret 5 L8: ;fun here
Out of turn push
5(a) 5(b) 5(c)
eip = L
Actual Stack
6
E
L
L
1
2
4
3 5
L
L
Abstract Stack Graph
esp
E
eax ebx L
4
E
L
L
1
2
6
3
L
L
5
Abstract Stack Graph
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
at instruction labeled L3. In the first example this is a ret instruction and in the second example it is pop ebx. These instructions are designated as program point 5. The top of the abstract stack at program point 5 contains L0 , the address of the instruction that pushed the target address on the stack. Thus, once again when a ret statement is encountered it can be determined that it was reached due to an obfuscated call.
5.2. Detecting obfuscated parameters
When analyzing a program for malicious behavior it is often useful to know the parameters being passed to a function. A program may be deemed malicious depending on the parameter. For instance, opening a file for read may be considered acceptable, but opening for write may indicate malicious intent. Parameters to a function are most often passed on the stack or in registers. Abstract stack graph can aid in determining the parameters that are passed on the stack. If a call takes n instructions, the top n elements
on the abstract stacks at a program point before the call instruction represent the locations where those parameters were pushed. The i th^ parameter corresponds to the i th^ element on the stack (starting from the top). This is assuming the first parameter is pushed last. If the last parameter is pushed first, we change it around. At the entry point, the parameters are determined after compensating for the return address. Figure 5 presents example programs that obfuscate where parameters are pushed. Figure 5(a) contains a sample normal code. In this program, the arguments to the function are pushed immediately before the call instruction. Figure 5(b) contains an example of out-of turn push. In this program instructions at L0 and L push the parameters in registers eax and ebx onto the stack. These are parameters intended to be parameters to call fun , but they are pushed before the instruction call L8. This gives the incorrect appearance that the parameters are being passed to the function at L. Thus, a push instruction need not pass parameters to the first call instruction. The abstract stack graph for
E 3
L
1
2
Abstract Stack Graph
E: push E ;entry L0: call fun L1: … … L15: ;fun here … L20: ret
Normal call / ret E: push E ;entry 1 L0: call fun 2 L1: … … L15: ;fun here L15: pop eax 3 L20: jmp eax 4
Using pop to return E: push E ;entry 1 L0: call fun 2 ; n junk bytes L5: … … L15: ;fun here L15: add [ esp ], n 3 ; modify [ esp ] L20: ret 4
Return elsewhere E: push E ;entry 1 L0: call fun 2 L1: … … L15: ;fun here L15: pop eax 3 ; loose return address ; calc new ret address L19: push Addr 4 L20: ret^5
Abusing call
6(a) 6(b) 6(c) 6(d)
Figure 6. Possible ways of obfuscating the ret
eip = L
esp
Actual Stack
E eip = L
esp
Actual Stack
E eip = L
esp
Actual Stack
E eip = Addr
esp
Actual Stack
E
E^1
L0 2 3
Abstract Stack Graph
4
Abstract Stack Graph
E^3
L
2
1
5
4
L
In Proceedings of Fourth IEEE International Workshop on Source Code Analysis and Manipulation , Chicago, IL,
a specific abstract stack. A node is annotated with the statements that receive an abstract stack with that node at the top. An abstract stack graph may be used to support disassembly of obfuscated code and to detect obfuscations related to stack operations. Examples of how to use the abstract stack graph to detect few obfuscated calls, returns and parameters were presented. The construction of an abstract stack graph offers a step towards analyzing obfuscated binaries for malicious behavior.
8. References
[1] "Teso, Burneye Elf Encryption Program," https://teso.scene.at, Last accessed July 1, 2004. [2] "Z0mbie," http://z0mbie.host.sk, Last accessed July 1, 2004. [3] G. Balakrishnan and T. Reps, "Analyzing Memory Accesses in X86 Executables," in International Conference on Compiler Construction (CC) 2004, Barcelona, Spain, 2004. [4] B. Barak, O. Goldreich, R. Impagliazzo, S. Rudich, A. Sahai, S. Vadhan, and K. Yang, "On the (Im)Possibility of Obfuscating Programs," in Advances in Cryptology (CRYPTO'01), Santa Barbara, California, USA, 2001. [5] S. Cho, "Win32 Disassembler," http://www.geocities.com/~sangcho/disasm.html, Last accessed July 1, 2004. [6] M. Christodrescu and S. Jha, "Static Analysis of Executables to Detect Malicious Patterns," in The 12th USENIX Security Symposium (Security '03), Washington DC, USA, 2003. [7] C. Cifuentes and K. J. Gough, "Decompilation of Binary Programs," Software Practice and Experience , vol. 25, pp. 811 - 829, 1995. [8] C. Collberg and C. Thomborson, "Watermarking, Tamper-Proofing, and Obfuscation - Tools for Software Protection," IEEE Transactions on Software Engineering , vol. 28, pp. 735-746, 2002. [9] C. Collberg, C. Thomborson, and D. Low, "A Taxonomy of Obfuscating Transformations," Department of Computer Science, The University of Auckland, 148, July 1997. [10] N. D. Jones and F. Nielson, "Abstract Interpretation: A Semantics-Based Tool for Program Analysis," in Handbook of Logic in Computer Science: Semantic Modelling , vol. 4, S. Abramsky, D. M. Gabbay, and T. S. E. Maibaum, Eds. Oxford, UK: Oxford University Press, 1995, pp. 527-636.
[11] C. Kruegel, W. Robertson, F. Valeur, and G. Vigna, "Static Disassembly of Obfuscated Binaries," in USENIX Secuirty 2004, San Diego,
[12] A. Lakhotia and P. K. Singh, "Challenges in Getting Formal with Viruses," Virus Bulletin 2003, http://www.virusbtn.com/magazine/archives/ 09/formal.xml. [13] C. Linn and S. Debray, "Obfuscation of Executable Code to Improve Resistance to Static Disassembly," in Proceedings of the 10th ACM Conference on Computer and Communication Security 2003, Washington D.C., USA, 2003. [14] M. Mohammed, "Zeroing in on Metamorphic Viruses," The Center for Advanced Computer Studies, University of Louisiana at Lafayette, M.S. Thesis, 2003. [15] B. Schwarz, S. Debray, and G. Andrews, "Disassembly of Executable Code Revisited," in Ninth Working Conference on Reverse Engineering (WCRE'02), Richmond, Virginia,
[16] Symantec, "Understanding Heuristics: Symantec's Bloodhound Technology," http://www.symantec.com/avcenter/reference/heur istc.pdf, Last accessed July 1, 2004. [17] P. Szor and P. Ferrie, "Hunting for Metamorphic," in Virus Bulletin Conference, Prague, 2001. [18] L. Vinciguerra, L. Wills, N. Kejriwal, P. Martino, and R. Vinciguerra, "An Experimentation Framework for Evaluating Disassembly and Decompilation Tools for C++ and Java," in 10th Working Conference on Reverse Engineering,
[19] G. Wroblewski, "General Method of Program Code Obfuscation," Institute of Engineering Cybernetics, Wroclaw University of Technology, PhD. Thesis, 2002.
