Big Data in the Modern Era: Technological Advances, Management Practices, and Research Challenges
Keywords:
Big data, Big data analytics, Data management, Hadoop, Noise accumulationAbstract
In the modern digital era, big data has evolved from a technical buzzword into a fundamental shift in how people understand the world. Big data is more than just large data; it is about gaining valuable insights from complex and voluminous data sources. Big data is more than just “large files”, as it encompasses the vast amounts of complex data created from social media interactions to medical sensors. This study discusses the movement of big data and the significance of the large amount of information in identifying subtle population patterns that small-scale data often misses. However, from a logical perspective, bigger is not always better. A researcher must navigate significant computational and statistical hurdles. These include storage bottlenecks and the “noise accumulation” trap, where high dimensionality leads to spurious correlations—mathematical patterns that look real but are actually accidental. To bridge the gap between theory and practical application, this study examines the shift toward “Analytics 3.0.” This paradigm requires traditional IT infrastructures to coexist with flexible, open-source technologies like Hadoop. By analysing current management styles and privacy concerns, this review emphasises that the future of big data depends on a new statistical paradigm. To achieve a competitive advantage, organisations must transcend the traditional approach of gathering information.
References
G. George, M. R. Haas, and A. Pentland, “Big data and management,” Academy of Management Journal, vol. 57, no. 2, pp. 321–326, Apr. 2014.
J. Fan, F. Han, and H. Liu, “Challenges of big data analysis,” National Science Review, vol. 1, no. 2, pp. 293–314, Jun. 2014.
A. Labrinidis and H. V. Jagadish, “Challenges and opportunities with big data,” Proceedings of the VLDB Endowment, vol. 5, no. 12, pp. 2032–2033, Aug. 2012.
S. Sagiroglu and D. Sinanc, “Big data: A review,” 2013 International Conference on Collaboration Technologies and Systems (CTS), San Diego, CA, USA, 2013, pp. 42–47.
A. McAfee and E. Brynjolfsson, “Big data: The management revolution,” Harvard Business Review, vol. 90, no. 10, pp. 60–128, Oct. 2012.
T. H. Davenport and J. Dyché, “Big data in big companies” International Institute for Analytics, May 2013.
Apache Software Foundation, “Apache Hadoop documentation,” 2023.
C.-W. Tsai, C.-F. Lai, H.-C. Chao, and A. V. Vasilakos, “Big data analytics: a survey,” Journal of Big Data, vol. 2, Oct. 2015.
N. Elgendy and A. Elragal, “Big data analytics: A literature review paper,” in Advances in Data Mining, Applications and Theoretical Aspects, St. Petersburg, Russia: Springer, 2014, pp. 214–227.
D. Fisher, R. DeLine, M. Czerwinski, and S. Drucker, “Interactions with big data analytics,” Interactions, vol. 19, no. 3, pp. 50–59, May 2012.
D. Maltby, “Big data analytics,” ASIST, Oct. 2011.
V. Rajaraman, “Big data analytics,” Resonance, vol. 21, pp. 695–716, 2016.
P. Russom, Big data analytics. Renton, WA, USA: TDWI Research, TDWI Best Practices Report, 2011.
J. Zakir, T. Seymour, and K. Berg, “Big data analytics,” Issues in Information Systems, vol. 16, no. 2, pp. 81–90, 2015.
D. Tosi, R. Kokaj, and M. Roccetti, “15 years of big data: A systematic literature review,” Journal of Big Data, vol. 11, May 2024.
A. Jamarani, S. Haddadi, R. Sarvizadeh, M. H. Kashani, M. Akbari, and S. Moradi, “Big data and predictive analytics: A systematic review of applications,” Artificial Intelligence Review, vol. 57, Jun. 2024.
A. Badshah, A. Daud, R. Alharbey, A. Banjar, A. Bukhari, and B. Alshemaimri, “Big data applications: Overview, challenges and future,” Artificial Intelligence Review, vol. 57, Sep. 2024.
A. Siddique, A. Gupta, J. T. Sawyer, T.-S. Huang, and A. Morey, “Big data analytics in food industry: a state-of-the-art literature review,” npj Science of Food, vol. 9, Mar. 2025.
