Artificial intelligence is evolving beyond text generation and question-answering to execute tasks directly within websites and software applications through a capability known as ‘computer use’ — enabling AI systems to interpret screen content, click, type, and navigate digital interfaces autonomously.
The technology allows AI systems to carry out repetitive operations such as filling in forms, querying records, and interacting with platforms that lack application programming interfaces (APIs) — the mechanisms that allow different software programmes to exchange data directly.
The functionality gained wider visibility in October 2024, when Anthropic released a public beta version of the feature within its Claude 3.5 Sonnet model. OpenAI, Google, and several Chinese technology companies have since developed solutions with comparable capabilities.
Computer use operates through an observation-and-action cycle. The system first receives a screenshot and identifies the relevant elements on screen. It then determines the operation to perform — which may involve a click, typing text, scrolling through a page, or pressing specific keys.
Although the AI interacts with the interface in a manner similar to a human user, it does not physically control a mouse or keyboard. Instead, it sends commands that are executed by the computing environment — typically through a browser or a virtual machine. After each action, a new screenshot allows the system to verify the outcome and adjust subsequent steps accordingly.
This capability is particularly useful for automating tasks within legacy enterprise systems, public portals, and websites that do not offer direct integration with other applications. Rather than relying solely on technical connections between platforms, an AI agent can interact with services through the user-facing interface.
The feature is typically embedded within AI agents — systems capable of planning and executing sequences of tasks with a degree of autonomy. Unlike a conventional chatbot, which is limited to providing instructions, an agent equipped with computer use can carry out the operations required to complete a task, verify results, and correct certain errors.
However, granting an AI system control over digital interfaces raises security concerns. Misinterpretations can lead to incorrect operations being executed, while malicious instructions embedded in web pages may attempt to manipulate system behaviour. For this reason, deploying this technology requires access restrictions, human oversight, and confirmation mechanisms for sensitive operations such as payments, data modifications, or file deletion.
Source: Diário Económico
Original article: https://www.diarioeconomico.co.mz/?p=556365
