Detecção e identificação de comportamento evasivo em executáveis para Windows usando instrumentação binária dinâmica

Abstract

In the current technological landscape, characterized by the continuous evolution and sophistication of digital threats, malware analysis has become fundamental for developing new defense strategies capable of efficiently protecting information systems. For this purpose, analysis via reverse engineering is a crucial technique. It is imperative to understand the evasive strategies adopted by malware to optimize the analysis process, as these are designed to evade conventional analysis methods, demanding more sophisticated approaches for identification and neutralization. This research focuses on the understanding, detection, and mitigation of the primary evasive techniques used by malware on Windows systems, as well as on identifying the protection tools employed by malware creators, such as Themida and Obsidium. These tools aim to obstruct and complicate analysis by experts and security tools, allowing for a more precise delimitation of the analysis scope and ensuring a more effective response against malicious behaviors. The main objective of this work is to develop strategies to detect evasive tactics used by malicious software and to identify the protection tools used, in order to enhance and streamline malware detection and analysis processes. To achieve this objective, this work also presents Contradef, a dynamic binary instrumentation (DBI) tool built on Intel Pin, specifically designed to observe, log, and mitigate the evasive behavior of Windows binaries during execution. The tool logs indicators of evasive behavior while preserving the transparency of the environment. The evaluation was conducted with real, protected samples executed in a virtualized environment similar to that used by analysts, demonstrating that Contradef is capable of revealing the hidden execution flow, reducing the impact of environment probing, and producing analysis traces at the API, instruction, and memory levels. The main contributions are: (i) the systematization of contemporary evasive techniques; (ii) a practical approach to support researchers and security professionals in the analysis of evasive malware; and (iii) an analysis of some of the anti-analysis strategies of the Themida and Obsidium protectors.