Vy by Vercept

Vy's defining trait was reading the screen rather than the code behind it. The agent took screenshots, ran them through a vision model, and identified buttons, input boxes and menus by how they looked. That let it work with any software that has a visible interface, including programs with no API at all. The approach also meant it adapted as layouts changed, since it judged the screen fresh each time instead of relying on fixed coordinates.

Because Vy worked from the screen, it did not need an integration for every app it touched. That broadens what you can automate, since many desktop and internal tools never shipped a public API. The trade-off is that screen-based control is slower and fussier than a clean API call. For software that has no other hook, seeing pixels is the only option.

The pitch rested on how well the model spotted on-screen controls. According to Vercept, its model scored 92% on the ScreenSpot v1 benchmark, a test that measures whether an agent can locate a named element on a real interface. That number mattered because computer-use agents live or die on clicking the right thing. A wrong click on a checkout or delete button is more than an inconvenience.

Vercept ran Vy as a cloud computer-use agent that could operate a remote machine, a MacBook being the common example. You didn't have to keep your own laptop open and attended, because the agent worked on a machine it controlled. That setup is convenient for long or repeated tasks, and it raises obvious questions about who can see what the agent does and what data passes through.

You drove Vy by describing what you wanted in plain language rather than scripting each step. The agent turned that description into a sequence of clicks and keystrokes and carried it out. That lowers the skill bar for automation, since you don't need to know a scripting language to set it going. It also means vague instructions go wrong, so clear task phrasing matters.

Real screens move. Menus expand, dialogs pop up, and content shifts as the page loads. Vy leaned on spatial reasoning to keep track of where things were and to follow an interface as it changed during a task. That's the hard part of screen-based automation and the reason vision-first agents exist at all. It's also where these agents still stumble, which is why supervision stayed part of the job.