Big Data-Driven Public Transport Analytics and Delay Prediction Using Apache Spark
Keywords:
Apache spark, Big data analytics, Data processing, HDFS, Hadoop, Public transportation, Spark SQLAbstract
Public transportation systems generate large volumes of data from daily operations such as route schedules, trip timings, and service performance records. With the rapid growth of urban populations, analysing this data efficiently has become critical for improving transportation planning and service quality. However, traditional data processing systems are not suitable for handling the scale and complexity of such data. Big Data technologies provide scalable solutions for storing and processing large datasets. Platforms such as Hadoop and Apache Spark enable distributed data storage and parallel processing, making them suitable for transportation analytics. Despite this, many transportation datasets are still analysed using limited or isolated methods without fully leveraging Big Data frameworks. This project presents a Big Data–Driven Public Transport Analytics System using the Hadoop and Apache Spark ecosystem. Public transport datasets are stored in the Hadoop Distributed File System (HDFS) and processed using Apache Spark. Analytical queries are performed using Spark SQL and Hive to extract insights such as route performance, peak-hour traffic patterns, and delay trends. The proposed system demonstrates how Big Data technologies can be effectively used for transportation analytics. It helps transport authorities improve planning and decision-making while providing a practical implementation of distributed data processing concepts.
References
M. Hassan, H. D. Mahin, A. Al Nafees, A. Paul, and S. S. Shraban, “Big data applications in intelligent transport systems: a bibliometric analysis and review,” Discover Civil Engineering, vol. 2, Mar. 2025.
I. B. Selvaraj, B. Vadivelu, and C. S. J. Robert Paulraj, “Data reduction for big data application in IOT,” OPSEARCH, Dec. 2025.
E. Bellini et al., “An IoE and big multimedia data approach for urban transport system resilience management in smart cities,” Sensors, vol. 21, no. 2, Jan. 2021.
M. Xiaokaiti, J. Jia, and L. Yang, “Intelligent fault identification system integrating artificial intelligence and power big data,” Procedia Computer Science, vol. 261, pp. 313–319, 2025.
T. Arjunan, “Real-time detection of network traffic anomalies in big data environments using deep learning models,” International Journal for Research in Applied Science and Engineering Technology, vol. 12, no. 3, pp. 844–850, Mar. 2024.
S. Nesmachnow, R. Massobrio, S. Guridi, S. Olmedo, and A. Tchernykh, “Big data analysis for travel time characterization in public transportation systems,” Sustainability, vol. 15, no. 19, Jan. 2023.
C. Ma, M. Zhao, and Y. Zhao, “An overview of hadoop applications in transportation big data,” Journal of Traffic and Transportation Engineering (English Edition), vol. 10, no. 5, pp. 900–917, Sep. 2023.
D. Ushakov, E. Dudukalov, E. Mironenko, and K. Shatila, “Big data analytics in smart cities’ transportation infrastructure modernization,” Transportation Research Procedia, vol. 63, pp. 2385–2391, 2022.
D.-K. Tran, D.-Q. Hoang, V.-T. Le, M.-D. Nguyen Thai, and T.-H. Do, “Real-time traffic flow prediction using big data analytics,” Intelligence of Things: Technologies and Applications, Jan. 2022, pp. 398–405.
M. Miftah, D. I. Desrianti, N. Septiani, A. Y. Fauzi, and C. Williams, “Big data analytics for smart cities: Optimizing urban traffic management using real-time data processing,” Journal of Computer Science and Technology Application, vol. 2, no. 1, pp. 14–23, Feb. 2025.
S. Ahmed, Y. Abdel-Hamid, and H. A. Hefny, “Traffic flow prediction using big data and GIS: A survey of data sources, frameworks, challenges, and opportunities,” International Journal of Computing and Digital Systems, vol. 14, no. 1, pp. 613–632, Aug. 2023.
X. Zou, E. Chung, H. Ye, and H. Zhang, “Deep learning for traffic prediction and trend deviation identification: A case study in Hong Kong,” Data Science for Transportation, vol. 6, Oct. 2024.
L. Monteiro-Fialho, E. Cueto-Rubio, C. Granell and S. Trilles, “Transportation analytics using smart card data: A systematic review,” in IEEE Transactions on Intelligent Transportation Systems, vol. 26, no. 7, pp. 11010–11033, Jul. 2025.
M. B and D. K. Lavanya, “Big data architecture for intelligent transport systems,” International Journal of Innovative Technology and Exploring Engineering, vol. 8, no. 9, pp. 1281–1286, Jul. 2019.
B. Medina-Salgado, E. Sánchez-DelaCruz, P. Pozos-Parra, and J. E. Sierra, “Urban traffic flow prediction techniques: A review,” Sustainable Computing: Informatics and Systems, vol. 35, Sep. 2022.
