AI-Based Malware Behavior Classification and Mitigation System Using Machine Learning

Abstract

The continuous evolution and technical sophistication of modern cyber threats present severe challenges to enterprise networks, endpoint devices, and personal data privacy. Conventional security measures that rely predominantly on static file signatures and hardcoded indicators of compromise frequently fail against advanced polymorphic, packed, and fileless malware strains that dynamically alter their binary structures upon replication. To overcome these critical defensive limitations, this study proposes an advanced, integrated behavior-based malware classification and automated mitigation framework specifically engineered for Windows operating environments. Instead of examining static file hashes on disk prior to execution, the proposed system captures and analyzes high-velocity, multi-variable runtime telemetry in real time. This comprehensive telemetry monitoring encompasses low-level process creation paths, parent-child execution hierarchies, CPU and memory resource utilization percentages, file system modification counts, active network socket connections, and system event triggers. These dynamic behavioral attributes are processed through optimized supervised machine learning algorithms, specifically Decision Trees, Random Forests, Support Vector Machines, and Logistic Regression, to accurately classify software execution patterns as benign or malicious while predicting specific threat types. Furthermore, the framework introduces a multi-class risk-scoring mechanism (categorizing threats into Low, Medium, High, and Critical tiers) to streamline Security Operations Center (SOC) incident triage. Upon identifying malicious behavior exceeding predefined risk thresholds, an automated mitigation pipeline initiates immediate defensive protocols, including real-time security alerts and the instantaneous termination of hazardous processes. All monitored telemetry streams, classification outputs, risk evaluations, and mitigation actions are dynamically rendered through a responsive, interactive React.js web dashboard. Comprehensive experimental benchmarking reveals that the Random Forest model achieves the highest classification accuracy at 89.2%, outperforming alternative algorithms and demonstrating strong efficacy for proactive endpoint protection and automated threat containment.