Machine Learning-based Detection of Advanced Persistent Threat Attacks Using Network Traffic Analysis

Authors

  • P. Balaganesh
  • S. Nalayiramuthu
  • G. Sudhakar
  • C. Vinothkumar

Keywords:

Advanced persistent threat, Cybersecurity, Entropy analysis, Intrusion detection system, Machine learning, Network traffic analysis

Abstract

Complicated cyber threats known as advanced persistent threats target big organizations and vital systems. These attacks differ from regular ones because they unfold in phases, stay hidden, and then slowly steal information. Because such intrusions change often and look like normal activity, older security tools that rely on fixed patterns fail to catch them. Instead of using outdated methods, this work explores a new way of applying machine learning to examine how data moves across networks, studying both numbers and behaviours to spot danger. Flow patterns like timing gaps, data size shifts, and uneven sessions help map how advanced threats behave. Instead of relying on old methods, machine learning tools, random forest, SVM, and XG boost are tested against standard attack records. Results show fewer mistakes in spotting intrusions, catching more real attacks than usual setups. Built to grow with network demands, the system adapts easily, fitting large business environments while cutting down how long hidden threats stay active.

References

M. K. Vishnu Priya and V. S. Shankar Sriram, “An incisive analysis of advanced persistent threat detection using machine learning techniques,” in Computational Intelligence in Data Mining: Proc. ICCIDM 2021, pp. 59–74, 2022.

J. Al Saraireh and A. Masarweh, “A novel approach for detecting advanced persistent threats,” Egyptian Informatics Journal, vol. 23, no. 4, pp. 45–55, 2022.

Y. Mei, W. Han, S. Li, K. Lin, Z. Tian, and S. Li, “A novel network forensic framework for advanced persistent threat attack attribution through deep learning,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 9, pp. 12131–12140, 2024.

A. O. Ishaya, A. Aminat, B. Hashim, and A. A. Adekunle, “Improved detection of advanced persistent threats using an anomaly detection ensemble approach,” Advances in Science, Technology and Engineering Systems Journal, vol. 6, no. 2, pp. 295–302, 2021.

P. S. Charan, P. M. Anand, and S. K. Shukla, “DMAPT: Study of data mining and machine learning techniques in advanced persistent threat attribution and detection,” in Data Mining – Concepts and Applications, C. Thomas, Ed. London, U.K.: IntechOpen, 2021.

H. N. Eke, A. Petrovski, and H. Ahriz, “The use of machine learning algorithms for detecting advanced persistent threats,” in Proc. 12th Int. Conf. Security of Information and Networks (SIN), 2019, pp. 1–8.

A. S. Basnet, M. C. Ghanem, D. Dunsin, H. Kheddar, and W. Sowinski Mydlarz, “Advanced persistent threats (APT) attribution using deep reinforcement learning,” Digital Threats: Research and Practice, vol. 6, no. 3, pp. 1–23, 2025.

L. Shang, D. Guo, Y. Ji, and Q. Li, “Discovering unknown advanced persistent threats using shared features mined by neural networks,” Computer Networks, vol. 189, p. 107937, 2021.

I. Nadim, N. R. Rajalakshmi, and K. Hammadeh, “A novel machine learning model for early detection of advanced persistent threats utilizing semi-synthetic network traffic data,” Journal of VLSI Circuits and Systems, vol. 6, no. 2, pp. 31–39, 2024.

A. H. Khan, “Machine learning-driven behavioural analysis of high volume network traffic for advanced persistent threat detection,” International Journal of Emerging Research in Engineering and Technology, vol. 5, no. 2, pp. 85–94, 2024.

M. Imran, H. U. R. Siddiqui, A. Raza, M. A. Raza, F. Rustam, and I. Ashraf, “A performance overview of machine learning based defence strategies for advanced persistent threats in industrial control systems,” Computers & Security, vol. 134, p. 103445, 2023.

H. N. Eke and A. Petrovski, “Advanced persistent threats detection based on a deep learning approach,” in Proc. IEEE 6th Int. Conf. Industrial Cyber Physical Systems (ICPS), 2023, pp. 1–10.

U. Sakthivelu and C. N. S. Vinoth Kumar, “Advanced persistent threat detection and mitigation using a machine learning model,” Intelligent Automation & Soft Computing, vol. 36, no. 3, pp. 3691–3707, 2023.

V. C. Jadala, S. K. Pasupuleti, C. M. Sai Baba, S. Hrushikesava Raju, and N. Ravinder, “Analyzing and detecting advanced persistent threat using machine learning methodology,” in Sustainable Communication Networks and Application: Proc. ICSCN 2021, Singapore: Springer Nature, pp. 497–506, 2022.

C. D. Xuan, D. Duong, and H. X. Dau, “A multi-layer approach for advanced persistent threat detection using machine learning based on network traffic,” Journal of Intelligent & Fuzzy Systems, vol. 40, no. 6, pp. 11311–11329, 2021.

J. H. Joloudari, M. Haderbadi, A. Mashmool, M. Ghasemigol, S. S. Band, and A. Mosavi, “Early detection of the advanced persistent threat attack using performance analysis of deep learning,” IEEE Access, vol. 8, pp. 186125–186137, 2020.

Z. Chen, J. Liu, Y. Shen, M. Simsek, B. Kantarci, H. T. Mouftah, and P. Djukic, “Machine learning enabled IoT security: Open issues and challenges under advanced persistent threats,” ACM Computing Surveys, vol. 55, no. 5, pp. 1–37, 2022.

A. Vance, “Flow-based analysis of advanced persistent threats detecting targeted attacks in cloud computing,” in Proc. Int. Scientific Practical Conf. Problems of Info Communications Science and Technology, Ukraine, 2014, pp. 173–176.

S. S. Karim, M. Afzal, W. Iqbal, D. A. Abri, and Y. Abbas, “SLF-ADM: Securing Linux frontiers: Advanced persistent threat (APT) detection using machine learning,” Neural Computing and Applications, vol. 37, no. 21, pp. 16715–16734, 2025.

K. A. Akbar, Y. Wang, G. Ayoade, Y. Gao, A. Singhal, L. Khan, and K. Jee, “Advanced persistent threat detection using data provenance and metric learning,” IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 5, pp. 3957–3969, 2022.

A. Dijk, “Detection of advanced persistent threats using artificial intelligence for deep packet inspection,” in Proc. IEEE Int. Conf. Big Data, 2021, pp. 2092–2097.

K. Ali and D. Boomsma, “Machine learning in cyber security: Predicting and preventing advanced persistent threats (APTs),” preprint, 2024.

Published

2026-03-30

How to Cite

Balaganesh, P., Nalayiramuthu, S., Sudhakar, G., & Vinothkumar, C. (2026). Machine Learning-based Detection of Advanced Persistent Threat Attacks Using Network Traffic Analysis. Journal of Cyber Security in Computer System, 5(1), 31–39. Retrieved from https://matjournals.net/engineering/index.php/JCSCS/article/view/3311