OS Agents: A Survey on MLLM-based Agents
for Computer, Phone and Browser Use

Xueyu Hu1,†, Tao Xiong1,‡, Biao Yi1,‡, Zishu Wei1,‡
Ruixuan Xiao1, Yurun Chen1, Jiasheng Ye2, Meiling Tao3, Xiangxin Zhou4,5,
Ziyu Zhao1, Yuhuai Li1, Shengze Xu6, Shawn Wang7, Xinchen Xu1 Shuofei Qiao1 Zhaokai Wang8
Kun Kuang1, Tieyong Zeng6, Liang Wang4,5, Jiwei Li1, Yuchen Eleanor Jiang3,
Wangchunshu Zhou3, Guoyin Wang9 Keting Yin1, Zhou Zhao1,
Hongxia Yang10, Fan Wu8, Shengyu Zhang1,*, Fei Wu1,
1Zhejiang University 2Fudan University 3OPPO AI Center
4University of Chinese Academy of Sciences
5Institute of Automation, Chinese Academy of Sciences
6The Chinese University of Hong Kong 7Tsinghua University 8Shanghai Jiao Tong University
901.AI 10The Hong Kong Polytechnic University

†Project Lead ‡Core Contributor *Corresponding Author
{huxueyu, sy_zhang}@zju.edu.cn

Paper Repository Zhihu Icon Zhihu Zhihu Icon OpenReview X Icon Twitter
❗Why is there no arXiv link for this paper?
This paper was rejected by arXiv with the justification: "Our moderators determined that your submission does not contain sufficient original or substantive scholarly research and is not of interest to arXiv." This reasoning appears to be inconsistent with the content and contribution of the paper. We attempted an appeal, but unfortunately, this was unsuccessful, and no further explanation was provided. A resubmission did not resolve the issue either. As a result, the ONLY way to access the paper at the moment is through our GitHub Repository or via OpenReview Archive. We are disappointed by the lack of transparency in arXiv’s moderation process.

Overview of Survey

This project conducts a comprehensive survey on OS Agents, which are (M)LLM-based Agents using computers, phones and browsers by operating within the environments and interfaces (e.g., Graphical User Interface (GUI) and Command Line Interface (CLI)) provided by operating systems (OS) to automate tasks. The survey begins by elucidating the fundamentals of OS Agents, exploring their key components including the environment, observation space, and action space, and outlining essential capabilities such as understanding, planning, and grounding. Methodologies for constructing OS Agents are examined, with a focus on domain-specific foundation models and agent frameworks. A detailed review of evaluation protocols and benchmarks highlights how OS Agents are assessed across diverse tasks. Finally, current challenges and promising future research directions, including safety and privacy, personalization and self-evolution, are discussed. Ultimately, we hope this study will serve as a catalyst for innovation, driving meaningful progress in both academia and industry.

Contact

Please let us know if you find out a mistake or are interested in contributing by e-mail: huxueyu.zju@gmail.com.

Citation

If you find our work valuable for your research or applications, we would greatly appreciate a star ⭐ and a citation using the BibTeX entry provided below.

@article{huagents,
  title={OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use},
  author={Hu, Xueyu and Xiong, Tao and Yi, Biao and Wei, Zishu and Xiao, Ruixuan and Chen, Yurun and Ye, Jiasheng and Tao, Meiling and Zhou, Xiangxin and Zhao, Ziyu and others}
}