Transition from Chatbots to Computer-Using AI Agents: A Systematic Review of Autonomous Workflows, Evaluation, Security, and Human Oversight
Keywords:
Agentic AI, Artificial intelligence, Autonomous workflows, Computer-using agents, Human-computer interaction, Human oversight, Large language modelsAbstract
These are enormous language models (LLMs) that enable not only the generation of information, but also acting within external software environments through tools, browsers, application interfaces, code execution, and computer-use mechanisms. This is a qualitative change in behaviour, introducing an interesting research problem since the ability to make a believable response goes hand-in-hand with creating, mutating, or deleting external state. This systematic review explores the required transition from conversational LLMs to computer-using AI agents, and studies autonomous workflows, evaluation, safety, trustworthiness, and human oversight. B. Role of background on reasoning-and-acting and tool use with evidence from agent, web, desktop, software-engineering, and safety benchmarks (vs React/Tool former vs OSWorld 2.0/swe-bench/AgentDojo w AI-comp-towards-24k/big-k-c) using-agent safety and security. This review uses a candidate eligibility protocol, a screening log at the source level, explicit pre-defined analytical categories, and systematic data extraction along categories including environment, action authority, planning horizon in space and time, verification of execution from intent to movement, as well as efficiency, safety, and human oversight. This synthesis demonstrates that agentic capability consists of multiple dimensions. Simply completing tasks is not enough, as an agent could arrive at a correct final state performing irrelevant actions, breaking through authorization boundaries, losing track of concealed state, or executing commands in untrusted content. Realistic interaction still proves much more difficult than isolated question answering, and long-horizon evaluation reveals additional limitations in constraint tracking, cross-source reasoning, recovery, and verification, as shown by web and desktop benchmarks. A security study shows that granting access to the tools is opening completely new attack surfaces that cannot be covered by model-level refusal behavior. The review thus identifies five interacting deployment dimensions: capability, environmental access, action autonomy, monitoring and verification, and human oversight. They are organized in a proposed Human-Oversight and Agentic Control (HOAC) conceptual framework, not yet a validated quantitative model for empirical evaluation. An existing frontier-model case study, GPT-6 Astra (LLM), is presented separately based on its academic evidence, as its benchmark and safety claims are developer-reported and therefore do not constitute independent validation. The review also concludes: permission-aware architectures, long-horizon evaluation, independent benchmarking, stronger defences against prompt injection and Bias amplification, risk-based human intervention, and interoperable audit trails. Computer-using agents should be evaluated as socio-technical control systems where having the capability of modelling is only one component of safety and reliability.
References
S. Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” arXiv.org, Mar. 2023.
T. Schick et al., “Toolformer: Language models can teach themselves to use tools,” Proceedings of the 37Th International Conference on Neural Information Processing Systems, Dec. 2023, pp. 68539-68551.
X. Liu et al., “AgentBench: Evaluating LLMs as Agents,” arXiv.org, 2023.
L. Wang et al., “A Survey on Large Language Model Based Autonomous Agents,” Frontiers of Computer Science, vol. 18, no. 6, Mar. 2024.
X. Deng et al., “Mind2Web: Towards a Generalist Agent for the Web,” arXiv.org, 2023.
S. Zhou et al., “WebArena: A Realistic Web Environment for Building Autonomous Agents,” arXiv.org, Oct. 24, 2023.
T. Xie et al., “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments,” arXiv.org, 2024.
R. Abhyankar, Q. Qi, and Y. Zhang, “OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents,” arXiv.org, 2025.
M. Yuan et al., “OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks,” arXiv.org, 2026.
C. E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” arXiv.org, Oct. 2023.
E. Debenedetti, J. Zhang, M. Balunović, L. Beurer-Kellner, M. Fischer, and F. Tramèr, “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” arXiv.org, 2024.
Z. Zhang et al., “Agent-SafetyBench: Evaluating the Safety of LLM Agents,” arXiv.org, 2024.
A. Chen et al., “JARVIS or Ultron? A Survey on the Safety and Security Threats of Computer-Using Agents,” Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 45407–45441, 2026.
OpenAI, “GPT-6 Astra: A new generation of intelligence,” OpenAI, Sept. 3, 2026.
OpenAI, “Safety overview: GPT-6 Astra,” OpenAI, Sept. 3, 2026.
OpenAI, “Model guidance: GPT-6 Astra,” OpenAI Developer Documentation, 2026.
M. J. Page et al., “The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews,” British Medical Journal, vol. 372, no. 71, 2021.
