A Dual-Model AI System for Linux Malware Detection Using Static ELF Analysis and Network Flow Behavior

Abstract

This study introduces a two-part deep learning setup designed to catch malware in Linux systems and IoT network flows. One model examines file structures before execution, while the other watches live network activity to spot suspicious patterns. Because they work together—using insights from code layout along with traffic timing—they respond faster to unknown threats. After fixing problems like overlapping training data and uneven sample counts, both parts now handle new examples more reliably. Results show stronger accuracy across different devices, making it practical for real-world use. We built two custom models: first, a bidirectional LSTM plus attention setup that sorts network traffic—hit 88% precision on 13 million pieces from the IoT-23 collection using split-by-group sampling and strict time splits, ran it for 19 rounds; second, a mix of convolutional layers stacked with LSTM, also including attention, focused on fixed binary checks—reached 66% correct calls across 1,815 Linux ELF files once fully retrained, went through 45 cycles. It runs across multiple GPUs at once, uses half-and-full precision during learning, sharpens results via focal loss tuning, groups data by type—all aimed at tackling extreme label skew (nearly 89% bad, just under 11% clean examples). Outcomes—Around 88% accuracy showed up after tough retraining using clear time splits to block data leaks, backed by similar F1 and precision numbers. Instead, the fixed setup hit just 66%, which lines up with its narrow sample pool—only 1,815 yes-or-no cases limiting variety. Across tests, results matched those starting points pretty well. Each system ran on an NVIDIA RTX 3060 Ti, making smart use of RAM when learning. Better scores? Still possible—if we grab more ELF files or fine-tunes settings longer. So, here is the deal—our two-part system works well at spotting malware, especially on Linux and IoT devices. Instead of just one method, we used a combo that leans heavily on the network side, which does better thanks to the huge IoT-23 set with around 13 million entries. On the flip side, the static part hits 66%, limited by less data, but still useful in real use cases. We also fixed leaks and skewed labels that were made early. The numbers look too good, so now it is more honest. That cleanup sets a tighter bar for how these security models should train.