Software Reverse Engineering: Techniques, Tools, and Challenges, Slides of Computer Science

Download Software Reverse Engineering: Techniques, Tools, and Challenges and more Slides Computer Science in PDF only on Docsity!

Computer Science:

Software Reverse Engineering

Software Reverse Engineering

(SRE)

SRE

• We assume that
  • Reverse engineer is an attacker
  • Attacker only has exe (no source code)
• Attacker might want to
  • Understand the software
  • Modify the software
• SRE usually focused on Windows
• So we’ll focus on Windows

SRE Tools

• Disassembler
  • Converts exe to assembly  as best it can
  • Cannot always disassemble correctly
  • In general, it is not possible to assemble disassembly into working exe
• Debugger
  • Must step thru code to completely understand it
  • Labor intensive  lack of automated tools
• Hex Editor
  • To patch (make changes to) exe file
• Regmon, Filemon, VMware, etc.

Why is a Debugger Needed?

• Disassembler gives static results
  • Good overview of program logic
  • But need to “mentally execute” program
  • Difficult to jump to specific place in the code
• Debugger is dynamic
  • Can set break points
  • Can treat complex code as “black box”
  • Not all code disassembles correctly
• Disassembler and debugger both required for any serious SRE task

SRE Necessary Skills

• Working knowledge of target assembly code
• Experience with the tools
  • IDA Pro  sophisticated and complex
  • SoftICE  large two-volume users manual
• Knowledge of Windows Portable Executable (PE) file format
• Boundless patience and optimism
• SRE is tedious and labor-intensive process!

SRE Example

• Program requires serial number
• But Trudy doesn’t know the serial number!

Can Trudy find the serial number?

SRE Example

• IDA Pro disassembly

Looks like serial number is S123N

SRE Example

• Again, IDA Pro disassembly

And hex view…

SRE Example

test eax,eax gives AND of eax with itself

• Result is 0 only if eax is 0
• If test returns 0, then jz is true

Trudy wants jz to always be true!

Can Trudy patch exe so that jz always true?

SRE Example

• Edit serial.exe with hex editor

serial.exe

serialPatch.exe

Save as serialPatch.exe

SRE Example

• Any “serial number” now works!

• Very convenient for Trudy!

SRE Attack Mitigation

• Impossible to prevent SRE on open system
• But can make such attacks more difficult
• Anti-disassembly techniques
  • To confuse static view of code
• Anti-debugging techniques
  • To confuse dynamic view of code
• Tamper-resistance
  • Code checks itself to detect tampering
• Code obfuscation
  • Make code more difficult to understand

Anti-disassembly

• Anti-disassembly methods include
  • Encrypted object code
  • False disassembly
  • Self-modifying code
  • Many others
• Encryption prevents disassembly
  • But still need code to decrypt the code!
  • Same problem as with polymorphic viruses