UFO²

UFO² hooks into Windows at the system level rather than through the screen alone. It combines UI Automation, Win32 and COM to read the real structure of an interface, including which control is a button and which is a text box, instead of guessing from a picture. Microsoft points to Excel as the example: turning table data into a chart can be one API call, where a traditional RPA tool simulates several mouse clicks. That depth is what earns the AgentOS label.

The platform splits the work across two kinds of agent. A central HostAgent reads the goal, breaks it into ordered subtasks, launches the programs it needs and coordinates the run through a state machine. Each running program gets its own AppAgent, a specialist that knows that app's interface, APIs and domain knowledge. This split keeps the system modular: a new program means a new AppAgent, not a rewrite of the core.

UFO² decides how to act, not just what to do. When an app exposes a native API, the agent calls it for speed and accuracy. When it doesn't, the agent falls back to mouse and keyboard actions. This mixed approach covers both standard and non-standard interfaces, which is where screen-only agents tend to fail. It's the difference between a tool that works on well-behaved apps and one that keeps working when an interface is rough.

Reading an interface reliably is the hard part of desktop automation, so UFO² fuses two methods. It uses Windows UI Automation for structured controls and adds vision-based parsing for anything UIA misses, like custom or canvas-drawn elements. The hybrid pipeline supports both standard and non-standard interfaces, and Microsoft credits it with better stability when the layout isn't tidy. It's the layer that decides what's actually on screen before the agent acts.

Every step in a typical agent means another round trip to the language model, and that adds up. UFO² bundles several predicted actions into a single LLM call and validates them against the live interface state before running them. Microsoft reports up to 51 percent fewer queries with this method, which cuts both latency and cost. For anyone paying per token, that saving is a real part of the pitch, not a footnote.

UFO² keeps a knowledge base it can draw on while it works. It blends offline app documentation, online search, user demonstrations and past execution traces into a vector store and retrieves the relevant pieces on the fly. The result is an agent that leans on what it has seen before rather than starting cold each time, in the same way a person learns a program by using it. Microsoft frames this as the foundation for agents that improve over time.

Running an agent on your own desktop usually means losing control of the mouse. On Windows, UFO² addresses this with a Picture-in-Picture desktop: automation runs in an isolated virtual desktop while you keep using your main screen and input devices. Microsoft lists this as coming soon in the current documentation, so it's on the roadmap rather than shipping today. Once it lands, it removes one of the biggest reasons people hesitate to let an agent drive their machine.