A Multimodal Deep Learning Framework for Predicting Social Demeanor through Sentiment Analysis of Social Media Videos
DOI:
https://doi.org/10.46610/RTAIA.2026.v05i01.004Keywords:
DeepFace, EfficientNet, Multi-output neural network, Social demeanor prediction, Social Media sentiment analysis, Video emotion recognitionAbstract
Social media videos capture abundant social and emotional interactions that traditional sentiment analysis does not focus on, as it usually focuses on text or image. This paper introduces a novel model of predicting social demeanor on the basis of video-based sentiment analysis on social media. It uses EfficientNet in deep visual feature extraction and DeepFace in rich facial emotion recognition. They are inputted into a multi-output neural network which categorizes both coarse sentiments (Positive, Negative, Neutral) and subtle social demeanors (Friendly, Supportive, Respectful, Hostile, Dismissive, Rude, Neutral) together. The model is trained and tested with a balanced dataset of over 3000 videos and exceeding 60000 manually extracted frame and an in-depth sentiment and social demeanor annotations. Strong metrics are used to evaluate the system i.e. precision, recall, F1-score, and ROC-AUC. Experimenting with real-world videos confirms the system to be working as far as its ability to identify small social signals is concerned, making it possible to develop an in-depth analysis of social interactions over the internet. The project fills in the sentiment analysis and social behavior modeling gap with a cognitive science-based hierarchical methodology, providing a scalable, interpretative, and ethical way of social media analysis, content moderation, and mental health monitoring.
References
L. Minocha, P. Jain, A. Singh, and P. Pandey, “Social Media’s Impact on Business and Society: A Study,” 2022 8th International Conference on Advanced Computing and Communication Systems (ICACCS), vol. 1, Mar. 2022.
J. Liu, Y. Zhang, and Z. Ye, “DeformableIR: Text Image Super Resolution Using Deformable Attention Transformer*,” 2024 36th Chinese Control and Decision Conference (CCDC), pp. 349–353, May 2024.
G. Li, Q. Zheng, L. Zhang, S. Guo, and L. Niu, “Sentiment Infomation based Model For Chinese text Sentiment Analysis,” IEEE Xplore, Nov. 01, 2020.
M. Pathak, D. Chaudhary, K. Sharma, A. K. Sharma, A. Gupta, and B. K. Sharma, “A Robust EfficientNet Architecture for Brain Tumor Classification and Identification Using MRI Image,” In2023 11th International Conference on Intelligent Systems and Embedded Design (ISED), pp. 1–5, Dec. 2023.
A. M. Kuzmina and N. S. Khlebnikov, “Visual Inclusive Communication of Socially-Oriented Nonprofit Organizations on Social Media,” In2025 Communication Strategies in Digital Society Seminar (ComSDS), pp. 258–261, Apr. 2025.
L. Lévêque, F. Villoteau, E. V. B. Sampaio, M. Perreira Da Silva, and P. Le Callet, “Comparing the Robustness of Humans and Deep Neural Networks on Facial Expression Recognition,” Electronics, vol. 11, no. 23, p. 4030, Dec. 2022.
T. Kong, J. Shao, J. Hu, X. Yang, S. Yang, and R. Malekian, “EEG-Based Emotion Recognition Using an Improved Weighted Horizontal Visibility Graph,” Sensors, vol. 21, no. 5, p. 1870, Mar. 2021.
Y. Himeur, A. Alsalemi , and A. A. - Kababji , “A survey of recommender systems for energy efficiency in buildings: Principles, challenges and prospects,” Information Fusion, vol. 72, Feb. 2021.
Fernández and S. Awinat, “Multimodal Sentiment Analysis based on Video and Audio Inputs,” Procedia Computer Science, vol. 251, pp. 41–48, 2024.
F. Chao and G. Yu, “Causal inference using regression-based statistical control: Confusion in Econometrics,” Journal of data and information science/Journal of Data and Information Science, vol. 8, no. 1, pp. 21–28, Feb. 2023.
L. Martino, R. San Millán-Castillo, and E. Morgado, “Spectral information criterion for automatic elbow detection,” Expert Systems with Applications, vol. 231, p. 120705, Nov. 2023.
U. K. Das et al., “Enhancing sentiment analysis accuracy on social media comments using a tuned BERT model,” Discover Computing, vol. 28, no. 1, Sep. 2025.
Karim, R. A. Rasul, and T. Sultana, “Sentiment Analysis of Social Media Data for Predicting Consumer Behavior Trends Using Machine Learning,” arXiv preprint arXiv:2510.19656. 2025 Oct 22.
D. Chen, B.-Y. Zhang, X.-H. Li, and Z.-T. Liu, “Human-Robot Emotional Interaction Model Based on Reinforcement Learning,” 2024 43rd Chinese Control Conference (CCC), pp. 4767–4771, Jul. 2024.
K. Zheng, W. Li, and K. Liu, “Real-Time Multimodal Student Behavior Analysis in Smart Classroom Environments,” 2025 4th International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC), pp. 558–563, Aug. 2025.
Aya, M. A. Alkhafaji, R. Hussam, N. H. Haroon, M. G. Majeed, and T. R. Al-Shaikhli, “Predicting Consumer Behaviour and Results Using Social Media and Deep Learning,” In2023 Annual International Conference on Emerging Research Areas: International Conference on Intelligent Systems (AICERA/ICIS), Nov. 2023.
Z. Zhan, “Construction of a Psychological Health Monitoring System Based on Social Text Sentiment Analysis,” In 2024, 4th International Conference on Mobile Networks and Wireless Communications (ICMNWC), pp. 1–5, Dec. 2024.
Z. Blažević, M. Škiljo, M. Bartulović, A. Znaor, L. Kranjac, and A. Ljubica, “Measurements of Frequency Splitting Phenomenon in Resonant Wireless Power Transfer System of Inductively Fed Antennas,” 2024 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), pp. 1–4, Sep. 2024.
A. Md. Sattar, M. Kr. Ranjan, and G. Singh, “Social, Emotional and Ethical (SEE) Attributes, Which Configures Our Bioinformatics Systems to Activate the Hidden Forces to Shape Human Decisions,” Nature-Inspired Intelligent Computing Techniques in Bioinformatics, vol. 1066, pp. 175–202, Nov. 2022.
E. Gautama, T. K. A. Rahman, and L. Kamelia, “Measurement of Optimizer Performance on The EfficientNet Architecture in Convolutional Neural Network for Classification of Matoa Maturity Levels,” 2024 10th International Conference on Wireless and Telematics (ICWT), pp. 1–6, Jul. 2024.
P. K. Das, S. Sreevatsav, and P. R. Sagar, “OSLTBDNet: Orthogonal softmax layer-based tuberculosis detection network with small dataset,” Biomedical Signal Processing and Control, vol. 105, pp. 107584–107584, Feb. 2025.
N. Zhou, Q. Deng, W. Luo, X. Huang, Y. Du, B. Chen, and W. Pedrycz, “Correntropy meets cross-entropy: A robust loss against noisy labels,” SSRN Electronic Journal, 2024.
Y. J. Lee, C. W. Jeong, and H. J. Kim, “Forensic feature extraction of document paper using periodic marks: PCA and t-SNE for manufacturer discrimination and document dating,” Forensic Science International, vol. 367, no. 112348, Feb. 2025.
M. Maeda, H. Nomiya, S. Sakaue, T. Hochin, and Y. Nishizaki, “Emotional video scene retrieval system for lifelog videos based on facial expression intensity,” in Proc. 18th IEEE/ACIS Int. Conf. Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), Jun. 2017, pp. 551–556.
K. Alkanan, Y. Zhong, X. Luo, and R. Dong, “Advanced multimodal biometric authentication for IoT-enabled smart home security: Integrating deep learning-based face recognition with behavioral patterns,” Kuwait Journal of Science, Art. no. 100548, Jan. 2026.
A. Patil, M. Ranjan, K. Deore, P. Sonje, K. Patil, and R. Patil, “Hybrid automata-based control framework for real-time optimization in space-based solar power transmission,” in EPJ Web of Conferences, vol. 328, no. 01057, 2025.
X. Chi, J. Wang, J. He, W. Wang, Y. Xi, L. Zhao, Y. Su, A. An, B. Sun, and N. Ai, “Consumer perception of milk with different fat content: Integrating check-all-that-apply, quantitative descriptive analysis, rate-all-that-apply, and facial emotion analysis,” Journal of Dairy Science, Oct. 2025.
