S
ShizuPortal
ShizuStore

Unofficial ShizuStore Web Portal

This is an unofficial web portal for ShizuStore self-hosted by rdevz-ph. Visit the official portal at https://shizustore.com/.

CatalogAI agentsAgent-VScreen
Agent-VScreen

Agent-VScreen

v1.3.0
by kilolonion•AI agents•MIT
Need ShizuStore app? Download APK

About Application

Agent-VScreen

Agent-VScreen

release license Android 11+ no root MCP

A background second screen for any phone agent.

中文 · English

Agent-VScreen opens and drives other apps on an invisible virtual display, so your own screen stays yours. All you see is a small floating window showing what the AI is doing, which you can take over or stop at any moment.

Any agent on the phone can use it: agents with a terminal use the vscreen CLI, agents that speak MCP use MCP, and your own code can use plain HTTP.

An agent solving sudoku in Chrome and playing Roco Kingdom on the background screen, mirrored live in the floating window

Left: an agent solves sudoku in Chrome on the background screen (8x speed). Right: an agent plays Roco Kingdom in landscape, walking with the joystick, dragging the camera and handling battle pop-ups (2.6x speed). In both, the user's own screen stays on the chat app.

Minis, Eta, RikkaHub and Muse all driving a forum app on the background screen through vscreen

One install command: four different Android agents (Minis, Eta, RikkaHub, Muse) all connect and drive the background screen.

Agent-VScreen Home, Connect, Records and Settings screens

The app: language switch, first-time setup, connecting an agent (this phone / Tailscale, CLI / MCP / HTTP), settings, app rules and diagnostics.

  • Version: 1.3.0 (versionCode 33) · Download APK
  • Package: com.kilon.agentvscreen
  • Android 11+ (Android 14+ recommended), arm64
  • Privileges: Shizuku only, no root

What it does

  • Drives apps in the background: launch apps, tap, swipe, type, press keys and wait for the screen to change on the virtual display, verifying every step.
  • Reports the screen back: after each action it returns the current screen, visible text and tappable elements, plus a screenshot. Built-in offline OCR (PaddleOCR, on device). The app calls no LLM itself; your agent's own model decides what the screen means.
  • Watches: keep an eye on the background screen for a condition (say, some text appearing) and notify the agent when it is met.
  • Floating window:
    • The border color shows who has the screen: rainbow while the AI works, blue when idle, green when you have taken over, red when you stopped it;
    • Drag to move, pinch to resize, fling to throw it; drag it half off the screen or fling it sideways and it slides into the edge as a tab; tap the tab and it slides back out;
    • Press and hold the picture for 0.5 s (a ring fills) to take over; the title, buttons and question card take their own touches and never pass them to the background screen;
    • × closes it with one tap: while the AI is working it only hides the window and the AI keeps going; with no task it closes the background screen;
    • While the AI acts or waits, "Stop AI" sits at the bottom centre and is the only thing that stops it; then the same place shows "Hand back", and a persistent notification stays until you release it;
    • Full screen: the picture stays put while the keyboard, typing box and question cards float over it; answer pages (draw a box, show me) move the floating window out of the way until you answer.
  • Leaves your agent's own access alone: it does not take adb, touch wireless debugging or restart adbd. One-tap setup only adds this app and never changes other apps' settings.
  • App restrictions: block apps entirely or make them read-only for the AI. Payment, delete and similar buttons still need your confirmation.

More than a background screen: the scaffold around it

Agent-VScreen is more than a virtual screen an agent can use: it builds a whole scaffold around the background screen so that your own agent operates it directly, sees it clearly, keeps up with time, and knows whether each step really worked.

  • Time awareness: while the agent works, the screen is looked at about 5 times a second and its text read about once a second, with timestamps. So every result's [Meanwhile] block tells what text appeared, disappeared or flashed from the last result until now (including the seconds the agent spent thinking or calling other tools), where the picture changed and for how long, with a contact sheet when it changed. Countdowns, loading, animations and pop-ups are not missed.
  • Waiting and watching: after any action add an until condition (text appears, text disappears, picture changes, picture is still, a pixel's color, a given app is in front) with a time limit; if it never comes, the result says so. --stop is watched for the whole stretch, --watch keeps looking after the steps. --keep notify|wake keeps watching after the turn ends and notifies the user or wakes the agent when the condition shows up. For realtime games, saved pause and resume steps make every stretch resume first and pause at the end, so the game becomes turn-based.
  • Seeing clearly: the accessibility tree and on-device OCR (bundled ML Kit, optionally the more accurate PP-OCRv6 small, both offline on the phone) are merged into numbered elements with coordinates, so tap #N and tap <text on screen> both work. OCR is split per character, so tabs on one line are separate targets and a part of a line can be tapped at its own center. There is also local color-spot finding (spots), zoom with a coordinate ruler (zoom), and scroll-and-collect of a whole page's text with deduplication (collect).
  • Every step checked: a step that misses (until never came, the text is not in the box, focus is not where you tapped) stops the stretch there, skips the rest and describes the screen now; a missed tap lists the closest candidates.
  • Working with others: sessions let several agents take turns on one screen (BUSY says whose it is and when it frees up); handoff in/out takes over an app the user has open on the main screen or hands it back; the user can take over or stop the AI at any time; per-app rules (blocked or read-only) and a confirmation stop for payment buttons (CONFIRM_REQUIRED). Each app can also keep notes and saved step segments (--save / use) that the next launch lists.
  • No model inside, no fight over privileges: the phone side calls no model; the caller's model decides what the screen means. Only Shizuku is used: no wireless debugging pairing, no adb.

Install and set up

The app UI is in Chinese for now; button names below are translated.

  1. Install the APK and open Agent-VScreen.
  2. Make sure Shizuku is running, then tap Grant Shizuku.
  3. Tap One-tap setup for the overlay, notifications, read-only accessibility, battery optimization and background running. On Honor, Huawei, Xiaomi, OPPO, vivo and similar phones, also open System background management and allow auto-start and background activity.
  4. In Connect agent, tap Copy message for agent and send it to your agent as is. It installs the vscreen command and saves a short guide into its own long-term memory, so new conversations know the tool is there. The home screen shows whether it remembered. Then:
vscreen launch Settings
vscreen act "tap Search; type About phone; key enter until text:About phone"
vscreen help all

Use with Operit

Operit is just an example here; nothing is tailored to it. Other Android agents on the same phone work with the default install from "Connect an agent" (command line, MCP or HTTP), and cloud or desktop agents connect over Tailscale (see the next section).

There are two ways; pick one:

  • ToolPkg (recommended): import agent-vscreen.toolpkg built from integrations/operit (also in Releases), send the token to Operit's AI to save, and it works; screenshots go straight to the model. See that folder's README.
  • Remote MCP: Operit and Operit 2 can both connect to remote MCP servers with no plugin, as follows.
  1. Finish the first-time setup in Agent-VScreen and make sure the service is running.
  2. Copy the token under Connect an agent → MCP.
  3. Open Operit's MCP management page, tap Import/Connect → Connect remote, and fill in:
    • Name (plugin ID): vscreen
    • URL: http://127.0.0.1:18765/mcp (18765 is the default port; use yours if you changed it in the app)
    • Connection type: httpStream
    • Bearer Token: the token you just copied
  4. Save and turn its switch on. Remote plugins need no "Deploy"; once the status shows active it is ready.
  5. Try in a chat: "Use vscreen to open Settings on the virtual screen and tell me the system version."

You can also use Config import: paste the JSON below and tap Merge config:

{
  "mcpServers": {
    "vscreen": {
      "url": "http://127.0.0.1:18765/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}

Notes:

  • Operit and Agent-VScreen must be on the same phone. If Operit runs on another device, replace the URL with the Tailscale address (see the next section).
  • Operit's own screen tools and vscreen can coexist. To keep it on the background screen only, add to the character prompt: "Always use the vscreen tool to operate other apps; never touch the main screen."
  • The token equals control of this phone: don't paste it into public character cards, shared configs or screenshots.

Connecting from other devices (optional, off by default)

The service only listens on 127.0.0.1:18765. To call it from a computer or another phone, choose Another device (Tailscale) in step 1 of "Connect an agent", then Built-in: no Tailscale app needed and no system VPN slot taken; sign in to Tailscale once inside the app and only devices on your own tailnet can reach it. Tailscale app mode and SSH port forwarding also work.

A self-hosted Headscale works too: enter its address under Advanced (for example hs.example.com; https:// is added for you), then open the registration page or use a pre-auth key. Each server keeps its own login. The built-in node only carries what other devices send to Agent-VScreen, not the phone's own internet traffic, so it has no exit node. If the phone needs an exit node to get online, keep the Tailscale app on and choose Tailscale app mode.

The Tailscale setup sits right under that choice. Once connected, the message for your agent, the install command, the MCP config and the ability note all switch to the Tailscale address.

The app speaks Chinese or English (the Language card on the home page: System / 中文 / English), and so does everything it tells the agent.

The token equals control of the phone, so the app never offers a public link.

What's new in 1.3.0

One step language for scripts, saved segments and lessons

  • * in words is any characters: tap *Bluetooth* taps the item holding it, wait text:Total*items; words@2 is the second place from the top, @-1 the last. When the same words are in several places, the reply marks each with its @N.
  • Flow: try <step> or try { … } (not found or not as expected: skip and go on); if text:Close { tap Close } else { back }; repeat 3 { scroll down }, repeat { scroll down } until text:End. Braces may span lines and nest. A mistake runs nothing and says how to write it.
  • note save <name> "steps" stores or rewrites a segment without running it, taught ones too.
  • Human pacing: act --human, or note human on to keep it for an app. A random pause before each action, taps slightly off-centre inside the element with uneven holds, varied swipe and drag durations and ends, typing 1–3 characters at a time. Waits and conditions are unchanged.

You teach it, it asks you

  • Teach it by doing it: do it once on the background screen and it is saved as a segment agents can replay. Take over the screen, tap the record key in the bottom bar, work as usual and tap Done; the review page says where each step came from, lets you untick slips and name it. A tap on words becomes tap <words> (found again when the layout moves); only when no words name the place is it a point. Words that appear, or the app that comes up, after a step become its until, so the replay waits for the same result. Typed words become a blank {1} that agents fill with use <name> <values>…; passwords are never kept. Agents can ask for a lesson with vscreen teach "what to show"; "Show it" on the phone starts recording right away.
  • Show me: vscreen ask "Which one do you mean?" puts the question on the phone; you tap the place on a picture of the background screen and the agent gets the point and the element there. Boxes, choices and text answers work too; with --async it waits until you answer.
  • Lessons are smoother: "Show it" takes over and records at once, "Done" also hands the screen back. A tap on a search result after typing is kept as "the N-th item holding the typed value" (tap *{1}*@2), so replaying with another value taps by content, not by place. A saved lesson keeps one contact sheet of its key steps (red rings mark taps); agents get it with the teach reply and again with note show <name>. The app's Schedules & lessons page searches lessons, deletes several at once and shows blocks indented.
  • The floating window tells you when it has a question: the folded tab turns into a violet question mark; expand it or go full screen and answer yes/no and choices right there.

Seeing clearly, tapping right

  • Checks judge only what came after the action: until text:… counts words that appeared after the action, once the app reacted and the picture settled; when nothing changed, the reply says the app did not react.
  • Numbers belong to one page: on a new page an old #N is STALE_ELEMENT and never lands on the same words there.
  • Areas that keep moving (videos, live streams, carousels) are recognized: they no longer slow every step or list bullet comments as changes.
  • Seeing your own taps: the [Meanwhile] sheet has a frame before each action with where it touched circled, and spare places show what actions led to. vscreen frames looks back at the recording: longer, denser or a given stretch.
  • One element per place: OCR no longer repeats words the app's own elements carry; collect gathers by cards and keeps two columns apart, leaves tab bars out, and confirms the end with a second swipe elsewhere.
  • Word taps are faster: tap <words> looks in the app's own elements first and needs no full-screen OCR when they settle it.
  • Time as it was spent: every reply says how long the phone took and on what; the command and the liveness check go out together, and pictures are smaller.

On call, or fully out of the way

  • Schedules: vscreen schedule "weekdays 07:00" books when an agent will use the phone; before each time the service is started and checked, and you are told in time if something blocks it. One-off bookings too (today 10:30). ready at the top of status says whether it can be used.
  • Always on / hidden / blocked: the switch on keeps it running; off hides it (no notification, nothing in the background) while a local agent or a booking can still wake it, and it hides again afterwards; "Block agent access" is really off.
  • The background screen is muted: its apps make no sound on the phone.
  • Main-screen notes only when we disturbed it: you switching apps on the main screen is no longer reported.
  • Agents learn about app updates: an agent's next use of vscreen shows what changed for it and asks it to refresh its note.

Remote access that holds

  • Built-in Tailscale: one node at a time that lives and dies with the app's process, so a restart by the system no longer leaves the phone unreachable for minutes; the saved login is kept while the server cannot be reached ("dropped, reconnecting"), so you are not asked to log in again; "Connected" is the real link state, with the relay and how often it dropped.
  • A token is enough over the built-in Tailscale: no signing, and MCP clients connect remotely as they are.
  • The CLI no longer hangs: an unresponsive local app is woken and tried again; when the app restarts or stops answering for a minute during a command, the CLI stops and says why; a remote phone that does not answer is explained (hidden, blocked, offline, logged out). VSCREEN_PROXY sets a proxy.
  • The log and status record why the previous app process ended, and when the app went to the background, was frozen or the node stalled.

The app

  • The home page holds the switch, the screen, anything waiting for you and a set of tiles (Connect, Permissions, Screen & window, Schedules & lessons, Activity log, Setup), each with its one key fact; explanations sit behind "?". The app, the floating window and full screen share one violet glass look.
  • Check for updates: "Check for updates" on the home list looks at the GitHub releases and shows what's new with a download link; the app also checks quietly at most once a day.
  • The Shizuku grant button always answers: already granted, Shizuku not running, refused before or no dialog shown — each is said and leads you to the right place; the grant state is read again when you come back to the app.

Not verified on every phone: whether vendor background managers let exact alarms and the wake broadcast through for a frozen app; whether muting with PLAY_AUDIO works for every player on MagicOS.

What's new in 1.2.0

This version rebuilds the command gate, the reply format, the security model and the guide for agents, and went through two real-phone regression rounds with docs/TASKS.md (Honor Magic 7, MagicOS 11). The upgrade table from 1.1 is in section 9 of API.md.

  • Structured results: every reply carries ok, code and next (look at the screen / fix the command / ask the user / retry later), plus each step's status, the screen's elements and the notes. A stretch that stops at a step is a failure; no reading text to find out. The new API is /v2; /v1 is unchanged and stays for one more version.

  • One command table: option checks, help, MCP tool schemas and the docs are generated from it. MCP has one tool per command besides vscreen.

  • More precise taps: text that appears several times is listed with coordinates instead of tapping the first; a #number whose element moved away is not tapped at its old place; an action that changed nothing is flagged.

  • Shorter waits: a page left with only a spinner or blinking cursor counts as settled. Perception is driven by new frames.

  • One input path: taps, swipes and keys are injected by the display service; dumpsys parsing and shell input are gone.

  • A token per agent: look-only / operate / everything, revoked separately, with separate sessions.

  • Signed requests: from another device the token never travels, requests cannot be replayed or altered, and replies are verified. The CLI does it by itself.

  • The user confirms on the phone: payment, irreversible steps, force-stop, taking an app off the main screen, and apps set to "ask every time". A flag added by the agent no longer counts. A trusted agent can be given 30 minutes without confirmations.

  • Activity log: who ran which command, when, with what result. Typed text is recorded as a length only.

  • Harder to knock over: connection cap, request size and time limits; a command stops when its caller disconnects; a resent request never runs twice; long work can run --async and be collected with task; commands queue.

  • Events are not lost: kept until the next fetch, re-fetchable, stored on disk; watches resume after an app restart; the event broadcast goes only to an app you name.

  • A rewritten guide for agents: built around one loop (look → act → check), with what to do next for each result code; vscreen guide is one page.

  • Hands-off by default: Permissions in the app has two levels. The default, "hands-off", asks you on the phone only for the step that actually spends money (pay, place order, transfer, top up) and for apps you set to "ask every time"; "careful" also asks for irreversible buttons such as delete, steps toward a purchase, force-stop and taking an app off the main screen. The "clear" button next to an input box is no longer treated as irreversible.

  • A new page layout: the home page goes Connect → Permissions → Settings → Activity log; permissions live in one place (level, app rules, tokens) and settings are grouped (floating window / background screen / reading and recording).

  • A smoother floating window: its picture is scaled to the size shown at 20 frames a second (30 while you control the screen) instead of the agent's full-size frames; moving the window updates its position once per display frame, so it stays under the finger; after taking over, your finger reaches the screen as it moves, so lists follow it. A landscape screen opens almost as wide as the phone, and each orientation keeps its own size.

  • Look-only tokens can look: their act (no steps or only waits) looks at the screen in use; stop --force and handoff in need the "everything" scope.

  • Numbers never point at the wrong element: on one screen a number names one element; after a scroll the same element keeps its number and new ones get new numbers. A number whose element left the screen gives STALE_ELEMENT instead of tapping something else.

  • Floating window: a landscape screen opens at about three quarters of the phone's width, and the whole window stays on screen when it appears or turns.

Docs

  • API.md (Chinese): the full CLI, MCP and HTTP reference, sessions, watches, permissions and security, the floating window, remote access, known limits and upgrading from 1.1.
  • docs/REFERENCE.md: every command option and result code (generated from the source).
  • docs/TASKS.md: the fixed task set to run on a real phone before a release.

Build from source

JDK 17 and the Android SDK (compileSdk 36).

echo "sdk.dir=/path/to/android-sdk" > local.properties
./gradlew :app:assembleRelease

Built-in Tailscale is a small Go program (tsbridge/, Go 1.21+). Run tsbridge/build.sh once before building to produce app/src/main/jniLibs/arm64-v8a/libtsbridge.so; without it the APK still builds, just without built-in Tailscale.

Release signing is optional: copy keystore.properties.example to keystore.properties and fill it in. Otherwise release builds are signed with the debug key.

License

MIT. The bundled OCR models (app/src/main/assets/vscreen-ocr/) come from PaddleOCR under their own Apache-2.0 license.

Thanks

Release Notes & Changelog

可直接覆盖安装 1.2.x,签名不变。

新功能

  • 示教:在副屏上亲手做一遍,存成 agent 能重放的段;agent 也能用 teach 请你演示。存下的示教带关键步骤缩略图。
  • 指给我看:ask 把问题放到手机上,你在画面上点一下、框一下或选一项就能回答。
  • 步骤语言:* 通配、字@2 第 N 处、try / if / repeat,脚本、段和示教共用一套。
  • 定时预约:schedule 预约 agent 来用手机的时间,到点前自动准备好。
  • 仿人节奏:act --human 或 note human on。
  • 检查更新:App 里一键检查 GitHub 新版本。

改进

  • 核对只看动作之后的画面;换页后旧编号失效,不会点错。
  • 按文字点更快;视频、轮播等动态区域不再拖慢每一步。
  • 服务分为常驻、收起、禁止访问三种状态;副屏静音。
  • 内置 Tailscale 更稳:单节点、断线保留登录、显示真实连接状态。
  • 悬浮窗和 App 统一为紫色玻璃风格;悬浮窗可滑进边缘折叠,回答页和全屏退出后都回到正常悬浮窗。

下载

  • agent-vscreen-1.3.0.apk:Android 11+,arm64,需要 Shizuku。
  • agent-vscreen.toolpkg:Operit 工具包 1.3.0。

SHA-256 439ee24cb1ee5049279d6483b877b5434143e66368447ee06f06129a8e3f742d

Declared Android Permissions

15 total

Shizuku API Permissions

moe.shizuku.manager.permission.API_V23

This application connects to the Shizuku service to execute system-level operations with ADB elevated privileges.

Standard Android Permissions

INTERNET
ACCESS_NETWORK_STATE
FOREGROUND_SERVICE
FOREGROUND_SERVICE_SPECIAL_USE
WAKE_LOCK
SYSTEM_ALERT_WINDOW
POST_NOTIFICATIONS
REQUEST_IGNORE_BATTERY_OPTIMIZATIONS
RECEIVE_BOOT_COMPLETED
USE_EXACT_ALARM
SCHEDULE_EXACT_ALARM
PACKAGE_USAGE_STATS
QUERY_ALL_PACKAGES
com.kilon.agentvscreen.DYNAMIC_RECEIVER_NOT_EXPORTED_PERMISSION

Specifications

CategoryAI agents
Packagecom.kilon.agentvscreen
LicenseMIT
GitHub Stars118
Installs407
Target SDKAndroid 14+ (API 35)
Last UpdatedNovember 15, 2026
Release DateNovember 17, 2026
Root RequirementRootless (Shizuku)
ShizuStore Installation

Install via ShizuStore to enable automatic updates and silent installations using Shizuku.

Agent-VScreen - Shizuku App | ShizuPortal