A Unified Disparate Impact Ratio (DIR) Framework for Data Science: Assessing Fairness, Model Fit, and Predictive Risk
Keywords:
Data science, Disparate Impact Ratio, Framework, Model fit, Multidimensional vectorAbstract
The study focuses on how the increasing adoption of data science, machine learning, and artificial intelligence in decision-making has created a need for evaluation methods that address not only predictive performance but also fairness and prediction-related risk. This study proposes a Unified Disparate Impact Ratio (DIR) Framework for Data Science that integrates three complementary dimensions: fairness, model fit, and predictive risk. The framework uses DIR to examine differences in favorable outcome rates between a focal group and a reference group, while incorporating conventional model-fit measures, including accuracy, precision, recall, F1-score, AUC, and calibration. Predictive risk is assessed through false-positive rates, false-negative rates, error disparities, and uncertainty-related indicators. The proposed framework represents model evaluation as a multidimensional vector (UDIR = [DIR, MF, PR]), enabling simultaneous examination of outcome disparities, predictive effectiveness, and subgroup-specific prediction risks. A performance life-cycle model is further proposed to support data preparation, model development, fairness assessment, validation, deployment, and continuous monitoring. The framework emphasizes that DIR should be interpreted alongside model performance and predictive-risk measures rather than treated as a standalone determination of fairness. The proposed approach provides a structured foundation for fairness-aware, performance-aware, and risk-aware data science and can be extended across domains including finance, healthcare, education, employment, insurance, cybersecurity, and public-sector analytics. Future empirical research can validate the framework using real-world datasets, multiple algorithms, dynamic monitoring, explainable AI, and causal fairness analysis.
References
S. Das, R. Stanton, and N. Wallace, “Algorithmic fairness,” Annual Review of Financial Economics, vol. 15, pp. 565–593, 2023.
S. Caton and C. Haas, “Fairness in machine learning: A survey,” ACM Computing Surveys, vol. 56, no. 7, pp. 1–38, Apr. 2024.
D. Pessach and E. Shmueli, “A review on fairness in machine learning,” ACM Computing Surveys, vol. 55, no. 3, pp. 1–44, Feb. 2022.
Z. Tang, J. Zhang, and K. Zhang, “What-is and how-to for fairness in machine learning: A survey, reflection, and perspective,” ACM Computer Survey, vol. 55, no. 13s, pp. 1–37, Jul. 2023.
L. Giuliani, E. Misino, and M. Lombardi, “Generalized disparate impact for configurable fairness solutions in ML,” in Proceedings of the 40th International Conference Machine Learning, vol. 202, pp. 11443–11458, 2023.
M. Jorgensen, H. Richert, E. Black, N. Criado, and J. Such, “Not so fair: The impact of presumably fair machine learning models,” in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, Society, Aug. 2023, pp. 297–311.
Y. Roh, K. Lee, S. E. Whang, and C. Suh, “Improving fair training under correlation shifts,” in Proceedings of the 40th International Conference on Machine Learning, vol. 202, pp. 29179–29209, 2023.
H. Weerts, R. Xenidis, F. Tarissan, H. P. Olsen, and M. Pechenizkiy, “Algorithmic unfairness through the lens of EU non-discrimination law: Or why the law is not a decision tree,” in Proceedings of the 2023 ACM Conference on Fairness, Accountability, Transparency, 2023, pp. 805–816.
D. Dablain, B. Krawczyk, and N. Chawla, “Towards a holistic view of bias in machine learning: Bridging algorithmic fairness and imbalanced learning,” Discover Data, vol. 2, 2024.
S. Teuber and B. Beckert, “An information-flow perspective on algorithmic fairness,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, pp. 15337–15345, 2024.
D. Plecko and E. Bareinboim, “Reconciling predictive and statistical parity: A causal approach,” in Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, Feb. 2024, pp. 14625–14632.
J. P. Lalor, A. Abbasi, K. Oketch, Y. Yang, and N. Forsgren, “Should fairness be a metric or a model? A model-based framework for assessing bias in machine learning pipelines,” ACM Transactions on Information Systems, vol. 42, no. 4, pp. 1–41, Mar. 2024.
A. Shome, L. Cruz, and A. van Deursen, “Data vs. model machine learning fairness testing: An empirical study,” in Proceedings of the 5th IEEE/ACM International Workshop on Deep Learning Testing Testing Deep Learning, Jul. 2024, pp. 1–8.
S. Min, H. Asif, and J. Vaidya, “Exploring the inequitable impact of data missingness on fairness in machine learning,” IEEE Intelligent Systems, vol. 40, no. 3, pp. 28–38, 2025.
M. S. Ah Lee, K. Padh, D. Watson, N. Kilbertus, and J. Singh, “Fairness under uncertainty in sequential decisions,” in 2026 ACM Conference on Fairness, Accountability, and Transparency, Jun. 2026, pp. 1784–1809.
