OpenBMB/AppCopilot

OpenBMB/AppCopilot

Description: A General, Accurate, Long-Horizon, and Efficient Mobile Agent driven by Multimodal Foundation Models

Language: Python

License: Apache-2.0

Stars: 293

Forks: 12

Open issues: 1

Created: 2025-08-27T07:47:20Z

Pushed: 2025-11-18T06:18:21Z

Default branch: main

Fork: no

Archived: no

README:

AppCopilot: Toward General, Accurate, Long‑Horizon, and Efficient Mobile Agent

【English | 中文】

📖 Overview

With the rapid evolution of large language models and multimodal foundation models, the mobile-agent landscape has proliferated without converging on the fundamental challenges. This paper identifies four core problems that must be solved for mobile agents to deliver practical, scalable impact: (1) generalization across tasks, modalities, apps, and devices; (2) accuracy, specifically precise on-screen interaction and click targeting; (3) long-horizon capability for sustained, multi-step goals; and (4) efficiency, specifically high-performance runtime on resource-constrained devices.

We present AppCopilot, a multimodal, multi-agent, general-purpose on-device assistant that operates across applications and constitutes a full-stack, closed-loop system from data to deployment. AppCopilot operationalizes this position through an end-to-end autonomous pipeline spanning data collection, training, deployment, high-quality and efficient inference, and PC/mobile application development. At the model layer, it integrates multimodal foundation models with robust Chinese–English support. At the reasoning and control layer, it combines chain-of-thought reasoning, hierarchical task planning and decomposition, and multi-agent collaboration. At the execution layer, it enables user personalization and experiential adaptation, voice interaction, function/tool calling, cross-app and cross-device orchestration, and comprehensive mobile app support. The system design incorporates profiling-driven optimization for latency, memory, and energy across heterogeneous hardware. Empirically, AppCopilot achieves significant improvements along all four dimensions: stronger generalization. higher-precision on-screen actions, more reliable long-horizon task completion, and faster, more resource-efficient runtime.

By articulating a cohesive position and a reference architecture that closes the loop from “data collection—training and deployment—high-quality, efficient inference—application development”, this paper offers a concrete roadmap for general-purpose digital assistants and provides actionable guidance for both academic research and industrial adoption.

🎉 News

At 2025-8-15, we are excited to announce the release of AppCopilot. AppCopilot is a general-purpose, on-device intelligent assistant that understands text and images, coordinates agents to complete complex tasks, works seamlessly across apps, and supports secure, real-time, cross-device collaboration.

⚡️ Quickstart

Click to expand

AppCopilot Local Run

This section mainly introduces how to connect to the model trained on the server through the API and run AppCopilot locally.

Local Environment Basic Requirements

The following table shows the relevant dependency requirements for the local environment:

| Dependency | Specific Requirements | |-----------------|---------------------------------------------------------------------| | Operating System| An operating system that supports Android Studio | | Software | Install Android Studio | | Python Environment| Install Python environment, recommended Python version 3.12 | | Network | Disable local VPN to ensure proper connection to the server's vllm API |

##### Install Android Studio

Android Studio is an integrated development environment (IDE) for Android platform development. It can be downloaded from the official Android Studio website.

Server Environment Basic Requirements

The following table introduces the relevant dependency requirements for the server-side environment:

| Dependency | Specific Requirements | |-----------------|---------------------------------------------------------------------| | Operating System| An operating system that supports Conda and vLLM | | Software | Install Conda, create a vLLM environment, and install vLLM dependencies|

##### Conda Installation

Conda is an open-source, cross-platform package manager and environment manager that helps users quickly install, run, and manage software packages and their dependencies. You can download it from the official Conda website.

After installing Conda, configure the Python virtual environment with the recommended Python version 3.12:

conda create --name vllm_env python=3.12

##### vLLM Installation

vLLM is an open-source high-performance library for large language model inference and services, providing faster responses for generative AI applications at a lower cost and higher efficiency. Here, configure the vLLM-related dependencies and install vLLM version 0.9.1 with the following command:

pip install vllm==0.9.1

##### Other Configuration To connect to the server API and run AppCopilot, the other configuration requirements for the server environment are as follows:：

pip install git+https://github.com/huggingface/transformers@f3f6c86582611976e72be054675e2bf0abb5f775
pip install accelerate
pip install qwen-vl-utils
pip install openai
git clone https://huggingface.co/Qwen/Qwen-VL-7B

Clone the Code

First, clone the folder from the remote repository to the local machine and add the necessary files:

mkdir AppCopilot
cd AppCopilot
git clone https://github.com/OpenBMB/AppCopilot.git .

To enhance the agent's ability to operate on Android phones, this project also requires the installation of the YADB tool to improve the native ADB functionality. It addresses the limitations of ADB in text input, screenshot capture, and UI layout extraction, providing more efficient and precise operations. Run the following command:

git clone https://github.com/ysbing/YADB.git ./YADB

Local System Environment Variable Configuration

##### Configure ADB Environment Variable 1.Windows System ADB Environment Variable Configuration:

On Windows, right-click on This PC, select Properties, and then click Advanced System Settings.

In the pop-up window, click Environment Variables,…