New AI Benchmark, OS-Marathon, Tackles Real-World Automation Challenges
A new standardized test, dubbed OS-Marathon, is challenging the limits of artificial intelligence in automating complex, repetitive digital workflows – the kind that bog down office workers in tasks like expense report processing and data entry. Researchers unveiled the benchmark alongside a novel teaching method that dramatically improves an agent’s ability to learn from limited examples and scale to handle large datasets.
The need for a rigorous testing ground arose from a critical gap in evaluating Computer-Use Agents (CUAs), according to the team behind OS-Marathon. Jing Wu, Daphne Barretto (Microsoft), and Yiye Chen (Georgia Institute of Technology), along with colleagues Nicholas Gydé, Yanan Jian, and Yuhang He, recognized that existing benchmarks largely focused on short-term tasks, failing to address the complexities of extended, iterative processes common in professional settings.
“Current agents struggle with tasks that require sustained reasoning and consistent execution over a long period,” a senior official stated. “OS-Marathon is designed to expose these weaknesses and drive innovation.”
OS-Marathon: A Rigorous Test of Long-Horizon AI
OS-Marathon comprises 242 tasks spanning two key domains: expense reporting and transcript processing. These scenarios were deliberately chosen to mirror real-world workflows that are both tedious for humans and ideally suited for automation. The benchmark utilizes seven distinct execution environments, including web-based systems and spreadsheet applications, creating a diverse and realistic testing ground.
The core innovation lies in the benchmark’s focus on long-horizon tasks – those requiring an agent to maintain context and execute a series of interconnected steps over an extended period. This approach circumvents limitations imposed by the context length restrictions of current CUAs. Researchers discovered that existing agents frequently exhibit logical incoherence in task ordering, experience hallucinations during action planning, and struggle to maintain consistency across repetitive sub-workflows.
For example, agents often attempted to populate system fields without first extracting relevant data from source documents, leading to errors. This highlights a fundamental challenge: agents need to not only understand what to do but also when and why.
A Breakthrough in AI Teaching
Alongside the benchmark, the research team developed a remarkably efficient teaching method. This method allows agents to learn effectively from just a few examples, then apply that knowledge to much larger, previously unseen datasets. This represents a significant step towards truly scalable automation.
The strategy employs dual-level instruction, guiding agents in both global planning – orchestrating the overall repetitive loop – and sub-workflow execution, mastering the logic of each individual step. By abstracting workflows into key steps, the method enables state-of-the-art agents to adapt efficiently to larger, unseen data collections.
To quantify performance beyond simple success rates, the team introduced Sub-Workflow Accuracy (SWA), a novel metric that measures an agent’s reliability over extended action sequences. SWA is calculated as the number of correctly executed sub-workflows divided by the total number of sub-workflows (n/N).
Current AI Struggles and Future Directions
Initial experiments revealed significant difficulties for leading CUAs on the OS-Marathon tasks. The primary failure modes identified were logical incoherence, hallucination, and inconsistency. However, the application of the new demonstration method, particularly when combined with the AgentS2.5 framework and GPT-5, yielded substantial performance improvements.
“The results demonstrate the effectiveness of focused demonstration techniques in enhancing performance in repetitive, structured tasks,” one analyst noted.
The researchers acknowledge that the current benchmark focuses on Levels 1 and 2 tasks, and that full-scale evaluation can be computationally expensive. Future work will focus on expanding OS-Marathon to include Levels 3 and 4, representing more complex challenges. They also plan to explore methods for reducing the cost of demonstration creation and improving the generalizability of agents across diverse workflows.
The project website, accessible at https://os-marathon.github.io/, provides further details and resources for the research community. This work represents a crucial step forward in the development of practical, reliable CUAs capable of automating tedious workflows and unlocking new levels of productivity in professional settings.
Related reading
