A Framework for Reliable AI Serving Infrastructure in Large-Scale Cloud and Edge Environments
Keywords:
AI serving infrastructure, Fault tolerance, MLOps, Observability, Qualitative framework refinement, Service-level objectivesAbstract
Reliable AI serving depends on coordinated decisions about workloads, heterogeneous resources, placement, serving state, recovery, change management, and operational responsibility. Research offers specialized serving mechanisms, but their integration with practitioner decision-making remains underdeveloped. This study develops and qualitatively refines the Reliable AI Serving Infrastructure (RASI) Framework for cloud, edge, and hybrid environments. A structured literature synthesis provided the preliminary eight-dimensional framework, followed by three semi-structured expert interviews using a common five-question guide. Each transcript was coded separately before hybrid deductive-inductive thematic comparison and researcher-led framework refinement. Five themes connected usable capacity and budgets, constraint-dependent placement, representative release testing, contextual observability and response ownership, and justification of architectural complexity. Expert accounts supported established reliability controls while extending attention to shared accelerator access, provider responsibility, domain-sensitive evaluation, and explicit acceptance evidence. The refined framework retains eight dimensions and adds conditional readiness reviews and reassessment across release and operational feedback loops. Serving-state controls and edge degradation remain primarily literature-supported because the interviews provided limited direct evidence. The contribution is an auditable decision structure linking technical controls with operational constraints, rather than a validated architecture or effectiveness claim. Three self-reported interviews and an incompletely documented systematic-search process limit the strength and transferability of the findings; operational evaluation and completion of the literature-review record remain necessary.
References
F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “INFaaS: Automated model-less inference serving,” in Proceedings of the 2021 USENIX Annual Technical Conference (USENIX ATC 21), 2021, pp. 397–411.
K. Razavi, M. Luthra, B. Koldehofe, M. Mühlhäuser and L. Wang, “FA2: Fast, accurate autoscaling for serving deep learning inference with SLA guarantees,” 2022 IEEE 28th Real-Time and Embedded Technology and Applications Symposium (RTAS), Milano, Italy, 2022, pp. 146–159.
H. Zhang, Y. Tang, A. Khandelwal, and I. Stoica, “SHEPHERD: Serving DNNs in the wild,” in Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation, 2023, pp. 787–808.
G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-based generative models,” in Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2022), 2022, pp. 521–538.
W. Kwon et al., “Efficient memory management for large language model serving with PagedAttention,” Proceedings of the 29th Symposium on Operating Systems Principles, Oct. 2023, pp. 611–626.
Z. Li et al., “AlpaServe: Statistical multiplexing with model parallelism for deep learning serving,” in Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2023), 2023, pp. 663–679.
A. Agrawal et al., “Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,” in Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2024), 2024, pp. 117–134.
Y. Zhong et al., “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2024), 2024, pp. 193–210.
P. Patel et al., "Splitwise: Efficient generative LLM inference using phase splitting,” 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), Buenos Aires, Argentina, 2024, pp. 118–132.
B. Sun et al., “Llumnix: Dynamic scheduling for large language model serving,” in Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2024), 2024, pp. 173–191.
Y. Fu et al., “ServerlessLLM: Low-latency serverless inference for large language models,” in Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2024), 2024, pp. 135–153.
R. Qin et al., “Mooncake: Trading more storage for less computation—A KVCache-centric architecture for serving LLM chatbot,” in Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST 2025), 2025, pp. 155–170.
J. Stojkovic et al., “TAPAS: Thermal- and power-aware scheduling for LLM inference in cloud platforms,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), vol. 2, 2025, pp. 1266–1281.
L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy, “Punica: Multi-tenant LoRA serving,” Proceedings of Machine Learning and Systems, vol. 6, 2024.
Y. Sheng et al., “S-LoRA: Serving thousands of concurrent LoRA adapters,” Proceedings of Machine Learning and Systems, vol. 6, 2024.
D. R. Torres, C. Martín, B. Rubio, and M. Díaz, “An open source framework based on Kafka-ML for distributed DNN inference over the Cloud-to-Things continuum,” Journal of Systems Architecture, vol. 118, Sept. 2021.
B. Zhang, Y. Li, S. Zhang, Y. Zhang, and B. Zhu, “An adaptive task migration scheduling approach for edge-cloud collaborative inference,” Wireless Communications and Mobile Computing, vol. 2022, no. 1, Jan. 2022.
W.-Q. Ren et al., “A survey on collaborative DNN inference for edge intelligence,” Machine Intelligence Research, vol. 20, pp. 370–395, 2023.
Y. Hao, N. Ding, W. Xia, H. Ge, and L. Xu, “DNN partitioning for cooperative inference in edge intelligence: Modeling, solutions, toolchains,” ACM Computing Surveys, vol. 58, no. 8, pp. 1–34, Feb. 2026.
D. Kreuzberger, N. Kühl, and S. Hirschl, “Machine learning operations (MLOps): Overview, definition, and architecture,” IEEE Access, vol. 11, pp. 31866–31879, 2023.
E. Breck, S. Cai, E. Nielsen, M. Salib, and D. Sculley, “The ML test score: A rubric for ML production readiness and technical debt reduction,” in Proceedings of the 2017 IEEE International Conference on Big Data, 2017
M. J. Page et al., “The PRISMA 2020 statement: An updated guideline for reporting systematic reviews,” The BMJ, vol. 372, 2021
B. A. Kitchenham, L. Madeyski, and D. Budgen, “SEGRESS: Software Engineering Guidelines for REporting Secondary Studies,” IEEE Transactions on Software Engineering, vol. 49, no. 3, pp. 1273–1298, 2023.
V. Braun and V. Clarke, “One size fits all? What counts as quality practice in (reflexive) thematic analysis?” Qualitative Research in Psychology, vol. 18, no. 3, pp. 328–352, 2021.
A. Tong, P. Sainsbury, and J. Craig, “Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups,” International Journal for Quality in Health Care, vol. 19, no. 6, pp. 349–357, Dec. 2007.
