Evaluating Transformer and Graph Aggregation for Explainable Retail Product Growth Prediction
Keywords:
Explainability, Graph representation, Knowledge discovery, Multimodal learning, TransformerAbstract
Multimodal prediction should demonstrate complementary information rather than assume that a more complex architecture discovers better knowledge. This study evaluates a small Transformer and fixed graph aggregation hybrid on linked UCI Online Retail records. Historical product descriptions form the text input, six sales-history measurements form the numerical input, and shared customers define a product graph. Using 90-day histories at the September, October, and November 2011 cutoffs produces 1,983 training, 2,137 validation, and 2,273 test products. The target records whether positive units sold in the next 30 days exceed those in the preceding 30 days. Graphs, vocabulary, and inputs respect the relevant observation cutoff, while validation average precision selects among three training checkpoints. Across five seeds, the numerical model obtains test average precision of 0.6923 and ROC AUC of 0.6881. The full hybrid obtains average precision of 0.5658 and ROC AUC of 0.5331, while the numerical-plus-graph model obtains average precision of 0.6793. Modality occlusion examines fitted-model sensitivity but is not treated as causal explanation. The results do not support hybrid superiority in this limited temporal benchmark and show why a linked multimodal representation requires empirical comparison with simple historical features.
References
T. Baltrušaitis, C. Ahuja, and L. P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, Feb. 2019.
A. Vaswani et al., “Attention is all you need,” Proceedings of the 31st International Conference on Neural Information Processing Systems,” Dec. 2017, pp. 6000–6010.
D. Chen, “Online Retail,” UCI Machine Learning Repository, 2015.
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun. 2019, pp. 4171–4186.
N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Nov. 2019, pp. 3982–3992.
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv, Sept. 2016.
W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp. 1025–1035.
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” arXiv, Oct. 2017.
S. Brody, U. Alon, and E. Yahav, “How attentive are graph attention networks?,” International Conference on Learning Representations, 2022.
C. Ying et al., “Do transformers really perform bad for graph representation?,” Proceedings of the 35th International Conference on Neural Information Processing Systems, Dec. 2021, pp. 28877–28888.
L. Rampášek, M. Galkin, V. P. Dwivedi, A. T. Luu, G. Wolf, and D. Beaini, “Recipe for a general, powerful, scalable graph transformer,” Advances in Neural Information Processing Systems, 2022.
S. Jain and B. C. Wallace, “Attention is not explanation,” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun. 2019, pp. 3543–3556.
M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” Proceedings of the 34th International Conference on Machine Learning, vol. 70. Aug. 2017, pp. 3319–3328.
J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, “Sanity checks for saliency maps,” Proceedings of the 32nd International Conference on Neural Information Processing Systems, Dec. 2018, pp. 9525–9536.
Z. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec, “GNNExplainer: Generating explanations for graph neural networks,” Advances in Neural Information Processing Systems, 2019.
D. Luo et al., “Parameterized explainer for graph neural network,” Advances in Neural Information Processing Systems,” 2020.
S. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp. 4768–4777.
M. T. Ribeiro, S. Singh, and C. Guestrin, ““Why should i trust you?”: Explaining the predictions of any classifier,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp 1135–1144.
M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” The Semantic Web, 2017, pp. 593–607.
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang and P. S. Yu, “A comprehensive survey on graph neural networks,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 4–24, Jan. 2021.
Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jul. 2019, pp. 6558–6569.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 1597–1607.
A. Radford et al., “Learning transferable visual models from natural language supervision,” arXiv, Feb. 2021.
T. Gebru et al., “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, Nov. 2021.
M. Mitchell et al., “Model cards for model reporting,” Proceedings of the Conference on Fairness, Accountability, and Transparency, Jan. 2019, pp. 220–229.
S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in ML-based science,” Patterns, vol. 4, no. 9, Sept. 2023.
R. Geirhos et al., “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, pp. 665–673, 2020.
P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” Proceedings of the 38th International Conference on Machine Learning, 2020, pp. 5637–5664.
A. Hogan et al., “Knowledge graphs,” ACM Computing Surveys, vol. 54, no. 4, pp. 1–37, Jul. 2020.
T. Saito and M. Rehmsmeier, “The Precision-Recall Plot is more informative than the ROC Plot when evaluating binary classifiers on imbalanced datasets,” PLoS ONE, vol. 10, no. 3, 2015.
C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, “On calibration of modern neural networks,” Proceedings of the 34th International Conference on Machine Learning, vol. 70, Aug. 2017, pp. 1321–1330.
T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 785–794.
A. N. Angelopoulos and S. Bates, “A gentle introduction to conformal prediction and distribution-free uncertainty quantification,” arXiv, Dec. 2022.
