Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders.<\/em> Subscribe Now<\/em><\/p>\n\n\n\n A new framework from researchers at The University of Hong Kong (HKU) and collaborating institutions provides an open source foundation for creating robust AI agents that can operate computers. The framework, called OpenCUA, includes the tools, data, and recipes for scaling the development of computer-use agents (CUAs).<\/p>\n\n\n\n Models trained using this framework perform strongly on CUA benchmarks, outperforming existing open source models and competing closely with closed agents from leading AI labs like OpenAI and Anthropic.<\/p>\n\n\n\n Computer-use agents are designed to autonomously complete tasks on a computer, from navigating websites to operating complex software. They can also help automate workflows in the enterprise. However, the most capable CUA systems are proprietary, with critical details about their training data, architectures, and development processes kept private.<\/p>\n\n\n\n \u201cAs the lack of transparency limits technical advancements and raises safety concerns, the research community needs truly open CUA frameworks to study their capabilities, limitations, and risks,\u201d the researchers state in their paper.<\/p>\n\n\n\n Power caps, rising token costs, and inference delays are reshaping enterprise AI. Join our exclusive salon to discover how top teams are:<\/p>\n\n\n\n Secure your spot to stay ahead<\/strong>: https:\/\/bit.ly\/4mwGngO<\/p>\n\n\n\n At the same time, open source efforts face their own set of hurdles. There has been no scalable infrastructure for collecting the diverse, large-scale data needed to train these agents. Existing open source datasets for graphical user interfaces (GUIs) have limited data, and many research projects provide insufficient detail about their methods, making it difficult for others to replicate their work.<\/p>\n\n\n\n According to the paper, \u201cThese limitations collectively hinder advances in general-purpose CUAs and restrict a meaningful exploration of their scalability, generalizability, and potential learning approaches.\u201d<\/p>\n\n\n\n OpenCUA is an open source framework designed to address these challenges by scaling both the data collection and the models themselves. At its core is the AgentNet Tool for recording human demonstrations of computer tasks on different operating systems.<\/p>\n\n\n\n The tool streamlines data collection by running in the background on an annotator\u2019s personal computer, capturing screen videos, mouse and keyboard inputs, and the underlying accessibility tree, which provides structured information about on-screen elements. This raw data is then processed into \u201cstate-action trajectories,\u201d pairing a screenshot of the computer (the state) with the user\u2019s corresponding action (a click, key press, etc.). Annotators can then review, edit, and submit these demonstrations.<\/p>\n\n\n\n Using this tool, the researchers collected the AgentNet dataset, which contains over 22,600 task demonstrations across Windows, macOS, and Ubuntu, spanning more than 200 applications and websites. \u201cThis dataset authentically captures the complexity of human behaviors and environmental dynamics from users\u2019 personal computing environments,\u201d the paper notes.<\/p>\n\n\n\n Recognizing that screen-recording tools raise significant data privacy concerns for enterprises, the researchers designed the AgentNet Tool with security in mind. Xinyuan Wang, co-author of the paper and PhD student at HKU, explained that they implemented a multi-layer privacy protection framework. \u201cFirst, annotators themselves can fully observe the data they generate\u2026 before deciding whether to submit it,\u201d he told VentureBeat. The data then undergoes manual verification for privacy issues and automated scanning by a large model to detect any remaining sensitive content before release. \u201cThis layered process ensures enterprise-grade robustness for environments handling sensitive customer or financial data,\u201d Wang added.<\/p>\n\n\n\n To accelerate evaluation, the team also curated AgentNetBench, an offline benchmark that provides multiple correct actions for each step, offering a more efficient way to measure an agent\u2019s performance.<\/p>\n\n\n\n The OpenCUA framework introduces a novel pipeline for processing data and training computer-use agents. The first step converts the raw human demonstrations into clean state-action pairs suitable for training vision-language models (VLMs). However, the researchers found that simply training models on these pairs yields limited performance gains, even with large amounts of data.<\/p>\n\n\n\n The key insight was to augment these trajectories with chain-of-thought (CoT) reasoning. This process generates a detailed \u201cinner monologue\u201d for each action, which includes planning, memory, and reflection. This structured reasoning is organized into three levels: a high-level observation of the screen, reflective thoughts that analyze the situation and plan the next steps, and finally, the concise, executable action. This approach helps the agent develop a deeper understanding of the tasks.<\/p>\n\n\n\n \u201cWe find natural language reasoning crucial for generalizable computer-use foundation models, helping CUAs internalize cognitive capabilities,\u201d the researchers write.<\/p>\n\n\n\n This data synthesis pipeline is a general framework that can be adapted by companies to train agents on their own unique internal tools. According to Wang, an enterprise can record demonstrations of its proprietary workflows and use the same \u201creflector\u201d and \u201cgenerator\u201d pipeline to create the necessary training data. \u201cThis allows them to bootstrap a high-performing agent tailored to their internal tools without needing to handcraft reasoning traces manually,\u201d he explained.<\/p>\n\n\n\n The researchers applied the OpenCUA framework to train a range of open source VLMs, including variants of Qwen and Kimi-VL, with parameter sizes from 3 billion to 32 billion. The models were evaluated on a suite of online and offline benchmarks that test their ability to perform tasks and understand GUIs.<\/p>\n\n\n\n The 32-billion-parameter model, OpenCUA-32B, established a new state-of-the-art success rate among open source models on the OSWorld-Verified benchmark. It also surpassed OpenAI\u2019s GPT-4o-based CUA and significantly closed the performance gap with Anthropic\u2019s leading proprietary models.<\/p>\n\n\n\n For enterprise developers and product leaders, the research offers several key findings. The OpenCUA method is broadly applicable, improving performance on models with different architectures (both dense and mixture-of-experts) and sizes. The trained agents also show strong generalization, performing well across a diverse range of tasks and operating systems.<\/p>\n\n\n\n According to Wang, the framework is particularly suited for automating repetitive, labor-intensive enterprise workflows. \u201cFor example, in the AgentNet dataset, we already capture a few demonstrations of launching EC2 instances on Amazon AWS and configuring annotation parameters on MTurk,\u201d he told VentureBeat. \u201cThese tasks involve many sequential steps but follow repeatable patterns.\u201d<\/p>\n\n\n\n However, Wang noted that bridging the gap to live deployment requires addressing key challenges around safety and reliability. \u201cThe biggest challenge in real deployment is safety and reliability: the agent must avoid mistakes that could inadvertently alter system settings or trigger harmful side effects beyond the intended task,\u201d he said.<\/p>\n\n\n\n The researchers have released the code, dataset, and weights for their models.<\/p>\n\n\n\n As open source agents built on frameworks like OpenCUA become more capable, they could fundamentally evolve the relationship between knowledge workers and their computers. Wang envisions a future where proficiency in complex software becomes less important than the ability to clearly articulate goals to an AI agent.<\/p>\n\n\n\n He described two primary modes of work: \u201coffline automation, where the agent leverages its broader software knowledge to pursue a task end-to-end,\u201d and \u201conline collaboration, where the agent responds in real-time and works side by side with the human, much like a colleague.\u201d Basically, the humans will provide the strategic \u201cwhat,\u201d while increasingly sophisticated AI agents handle the operational \u201chow.\u201d<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>The challenge of building computer-use agents<\/h2>\n\n\n\n
\n\n\n\n\n
\n<\/div>Introducing OpenCUA<\/h2>\n\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/08\/image_7fdde4.png?w=800\")
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/08\/image_05f5eb.png\")
A new recipe for training agents<\/h2>\n\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/08\/image_fb4236.png?w=800\")
Putting OpenCUA to the test<\/h2>\n\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/08\/image_e2e708.png?w=800\")
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n