TECHNIQUES FOR MALWARE DETECTION AND ANALYSIS: A MACHINE LEARNING APPROACH

The rapid evolution of malware variants and the manual bottlenecks in reverse engineering demand automated, scalable, and resilient detection mechanisms. This dissertation investigates the application of machine learning and semantic feature extraction to mitigate human-in-the-loop dependency and address the degradation of classification models due to concept drift. The research is structured around three core contributions. First, it introduces SAUCY SPICE and its scalable evolution, Saucy Express, an unsupervised framework for automated, generalized signature generation and real-time kernel-mode blocking, enabling signature generation in a viable timeframe for rapid response. Second, it proposes the Bifocal Agent, a dual-granularity anomaly detection system that identifies malicious code regions at both the function and basic-block levels. By employing a novel 'Tails' aggregation strategy and a consensus mechanism that establishes a targeted triage "Gray Zone," the framework significantly reduces false-positive rates (from 45% to 16% compared to baselines) and streamlines the security analyst's workload. Third, the study evaluates the robustness of representation learning models (including Asm2Vec, Trex, and jTrans) against concept drift and structural modifications such as compiler optimizations. The findings reveal that current approaches fail to effectively capture code semantics, resulting in representations (embeddings) that are strongly biased toward structural artifacts. In a practical experiment, it is shown how the lack of true functional semantics limits the application of code embedding techniques for malware classification. Ultimately, this thesis provides actionable frameworks to accelerate malware characterization and exposes critical gaps in contemporary code embedding, pointing toward the necessity of domain-specific, semantic-driven approaches in cybersecurity defense