🧭 Decision Guide
Why trending now: Cannot be determined from the materials
Try it if you
-
You need an AI agent to click, type, and verify through the accessibility tree of Finder or Slack.The README Quick start uses snapshot, click, and snapshot for an observe-act-observe loop; the Dense apps example includes Slack.
-
You need one CLI across macOS, Windows, and Linux.The README Installation section says npm installs it on macOS, Windows, or Linux; the Platform support table lists all three.
-
You want actions that do not move the pointer or take foreground focus by default.The README Quick start says Headless by default and that ref actions go through the accessibility API.
-
You want in-process integration instead of starting the CLI for every command.The README Language bindings section says every release ships libagent_desktop_ffi for in-process calls.
Skip it if you
-
Your automation depends on pixel matching and the target app does not expose a usable accessibility tree.The README says agent-desktop works through the OS accessibility tree and explicitly uses no pixel matching.
-
You do not have a graphical session or at-spi2-core on Linux.The README Installation section says Linux needs the user's graphical session with at-spi2-core.
-
You need complete system window-control capabilities on Linux Wayland.The README Platform support table describes Linux App & window management as Launch/list; X11 window control, while Wayland is mentioned only for screenshot portal support.
-
You cannot allow desktop snapshots, clipboard data, or debug artifacts to contain private content.The README privacy note says snapshots, clipboard reads, screenshots, command output, and debug artifacts may contain private desktop content.
Requirements
- Requires the npm installation path: npm install -g agent-desktop.
- macOS requires Accessibility permission.
- Linux requires the user's graphical session and at-spi2-core.
- Linux Wayland clipboard requires wl-clipboard.
- Native binaries support x64 and ARM64.
First step (verbatim from README)
npm install -g agent-desktop
Watch out
-
Do not use a bare @e8; a ref must retain the full snapshot ID.The README says bare @e8 is rejected with INVALID_ARGS; the example ref is @s8f3k2p9:e8.
-
An old ref may return STALE_REF after the UI changes.The README Quick start says UI changes return STALE_REF or AMBIGUOUS_TARGET and require a new snapshot.
-
In PowerShell, quote every ref because @token is the splatting operator.The README explicitly requires quoting refs in PowerShell, such as '@s8f3k2p9:e8'.
-
For dense apps, start with snapshot --skeleton and then drill down with --root.The README Dense apps example uses snapshot --skeleton --app Slack -i --compact and snapshot --root.
-
Headless by default does not mean physical input is available; use the global --headed flag.The README says ref actions do not move the pointer or take focus, while --headed allows physical input.
Not stated in the README
- The README does not specify the minimum supported Node.js version.
- The README does not provide a compatibility list or success rate for individual desktop applications.
- The README does not provide performance data for commands such as snapshot or click.
- The README does not describe the specific changes from the previous release to v1.0.2.
- The README does not explain how the 9 contributors and 10 recent commits relate to maintenance work.
💡 Deep Analysis
6
No
I want to integrate agent-desktop into an enterprise agent and upload snapshots, screenshots, and clipboard results to a model or public logs. Does the README support this data flow?
No—not for unconditional upload or public logging: the README explicitly treats these outputs as potentially private desktop data.
- The introduction directly warns that snapshots, clipboard reads, screenshots, command output, and debug artifacts may contain private desktop content and says to keep them out of public logs and uploads.
- Accessibility snapshots can include window titles, control names, and text; clipboard and screenshots may expose broader user content. The project does not promise automatic redaction.
- The optional JEV helper sends screen descriptions to a configured model provider and is separate from the native CLI, so that data path requires its own enterprise review.
- If the requirement is to upload complete desktop context without filtering, the project conflicts with that constraint. The README does not provide built-in guarantees for redaction, retention, access control, or compliance.
- README introduction: snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content
- README introduction: keep them out of public logs and uploads
- README introduction: the optional JEV helper sends screen descriptions to a configured model provider
- Project insights, “usage_limitations”: snapshots, screenshots, clipboard, and debug artifacts create data-leakage risk
Yes
I need my agent to operate macOS Finder and avoid clicking the wrong target when using dense applications such as Slack or when windows change. Are snapshots and refs sufficient?
Yes: the project binds targets to a snapshot and turns stale references and window ambiguity into explicit, machine-readable failures instead of silently clicking another object.
- The Quick start demonstrates
snapshot --app Finder -i, clicking the returned ref, and taking another snapshot for verification. - The README says a bare
@e8returnsINVALID_ARGS; UI changes produceSTALE_REForAMBIGUOUS_TARGET, requiring a new snapshot. - When multiple windows exist, the error details contain candidate window IDs, which can be narrowed with
--window-id. - For dense applications such as Slack,
--skeletonand--compactcan map the UI before--rootdrills into a region. However, the target application must still expose a usable Accessibility tree; custom-drawn or virtualized controls are not automatically made addressable.
- README, “Quick start”: Observe, act on a ref, observe again
- README, “Quick start”: refs belong to their snapshot; UI changes produce STALE_REF or AMBIGUOUS_TARGET
- README, “Quick start”: several windows produce AMBIGUOUS_TARGET; pass one with --window-id
- README, “Quick start”: dense apps can use --skeleton, --compact, and --root
agent-desktop snapshot --app Finder -i # interactive elements, each with a ref
Yes
I use PowerShell on Windows to control multiple native application windows and want the agent to decide safely whether to retry from error states. Is agent-desktop a better fit than coordinate-based scripts?
Yes: Windows Accessibility support, window scoping, and the uniform JSON error contract match these constraints better than coordinate-based scripts.
- The platform table lists Windows support for the Accessibility tree, click/type/keyboard, mouse input, clipboard, and application/window management. Windows needs no additional grant for applications at the terminal’s integrity level.
- For multiple windows, the README says the tool returns
AMBIGUOUS_TARGET; the agent can use candidate IDs from the error details with--window-id, expressing target context more reliably than fixed coordinates. - Every command returns
okplus eitherdataorerror. Errors includecode,disposition, and usuallysuggestion; automatic retry is appropriate only whenerror.disposition.retryissafe. - In PowerShell, every ref must be quoted because a bare
@tokenis parsed as the splatting operator. Window selection, ref lifetime, and retry behavior can therefore be explicit in the script.
- README, “Platform support”: Windows supports the Accessibility tree, click/type/keyboard, mouse input, clipboard, and app/window management
- README, “Installation”: Windows needs no grant for apps at the terminal’s integrity level
- README, “For AI agents”: errors carry code, disposition, and suggestion; retry only when retry is safe
- README, “Quick start”: quote every ref in PowerShell; use --window-id for multiple windows
It depends
I need an agent on a Linux GNOME environment to operate applications, read the clipboard, and handle notifications. My environment may use Wayland; is agent-desktop suitable for direct deployment?
It depends: it can cover the main capabilities in a real Linux graphical session with AT-SPI and the required desktop conditions, but “Linux support” does not imply feature parity everywhere.
- The Installation section requires a user graphical session and
at-spi2-core, so a generic headless server is not the deployment form directly documented by the README. - The platform table lists the Accessibility tree, application launch/listing, and notifications, but mouse input is marked X11-dependent, while screenshots distinguish X11 windows from the Wayland portal.
- Wayland clipboard support requires
wl-clipboard; notification support is identified as GNOME Shell, and shell surfaces list GNOME tray and Action Center differences. - Semantic snapshots, clicking, typing, and GNOME notifications are plausible targets. Workflows requiring physical mouse control or a uniform system-panel API remain insufficiently specified.
- README, “Installation”: Linux needs the user's graphical session with at-spi2-core
- README, “Platform support”: mouse input uses X11 and --headed; screenshots use X11 windows or a Wayland portal
- README, “Platform support”: Wayland clipboard requires wl-clipboard; notifications are GNOME Shell
- README, “Platform support”: Linux shell surfaces include GNOME tray and Action Center
agent-desktop skills get platform # what differs on the OS you run on
It depends
I am building a Rust-based AI agent that must control native applications on macOS, Windows, and Linux. Can I replace three platform-specific mouse-and-keyboard script stacks with agent-desktop?
It depends: the project provides a common CLI, JSON contract, and C library, but platform capabilities are not identical.
- The README lists macOS, Windows, and Linux support and exposes structured snapshots through the Accessibility API or Linux AT-SPI; ref actions do not rely on pixel matching.
- macOS requires Accessibility permission. Linux requires the user’s graphical session and
at-spi2-core; mouse input depends on X11, while Wayland adds conditions for screenshots and clipboard access. - Shell surfaces are still marked Planned on macOS, and Linux window, notification, and physical-input features vary across X11, Wayland, and GNOME.
- A shared abstraction is therefore reasonable for semantic observation, clicking, and typing, but system surfaces, physical mouse workflows, and Linux desktop-specific flows must remain platform-aware.
- README, “Platform support”: Accessibility tree is supported on macOS, Windows, and Linux
- README, “Installation”: macOS needs Accessibility permission; Linux needs the user's graphical session with at-spi2-core
- README, “Platform support”: shell surfaces are Planned on macOS
- README, “Quick start”: headless by default; ref actions use the accessibility API
npm install -g agent-desktop
It depends
My agent core is also written in Rust and will execute click, type, and find frequently. Since the README says every CLI call is a separate process, should I integrate the C FFI directly?
It depends: C FFI is worth evaluating for high-frequency or latency-sensitive workloads, but it is not a drop-in replacement without additional engineering constraints.
- The README explicitly says every CLI call runs in its own process. This improves isolation but can add process-startup and JSON-serialization overhead.
- The project also provides a C library for in-process calls; the project insights identify ABI versioning, memory ownership, and thread rules as FFI concerns.
- The CLI has a uniform envelope with
okand eitherdataorerror, making it easier to validate the agent loop first. FFI requires stricter lifecycle and ABI handling. - For a Rust agent still validating the workflow, the CLI is the simpler integration surface. If call frequency or latency becomes a measured bottleneck, use the FFI path after checking the project’s ABI and threading documentation; the README does not provide performance numbers.
- README introduction: “Every CLI call runs in its own process”
- README, “Language bindings” as summarized in the project insights: a C FFI library is provided
- README, “For AI agents”: every command prints an envelope with ok and either data or error
- Project insights, “usage_limitations”: FFI requires ABI-major, memory-ownership, and threading rules
agent-desktop skills get desktop # the loop, refs, verification and retry rules
✨ Highlights
-
snapshot returns a JSON accessibility tree with refs
-
Headless ref actions do not take focus by default
-
Supports macOS, Windows, and Linux
-
Provides 60 commands and a C FFI
-
macOS requires Accessibility permission
🔧 Engineering
-
Use snapshot, click, and JSON refs to read and operate desktop apps
-
Each CLI call runs in its own process and supports 60 commands
-
libagent_desktop_ffi enables in-process tool calls
⚠️ Risks
-
UI changes can cause STALE_REF and require a new snapshot
-
Multiple windows return AMBIGUOUS_TARGET and require --window-id
-
Linux requires a graphical session and at-spi2-core
-
Snapshots and screenshots may contain private desktop content
👥 For who?
-
Teams that need AI agents to operate Finder or Slack
-
Developers automating desktops on macOS, Windows, or Linux
-
Integrators needing the Rust CLI or libagent_desktop_ffi