New SOTA Web Agent beats even Operator with human intervention

1. The Rise of AI Web Agents and the Need for Robust Benchmarks

AI web agents are transforming digital interaction by automating complex online tasks. However, the web's dynamic nature poses significant challenges for evaluating their performance reliably. This has created a critical demand for robust, standardized benchmarks like Halluminate's Web Bench, ensuring verifiable capabilities and accelerating trust in this emerging technology.

2. Understanding Halluminate's Web Bench: A New Standard for Agent Evaluation

Halluminate's Web Bench offers a rigorous, comprehensive standard for evaluating AI browser agents by distinguishing between "READ" and "WRITE" tasks. Explore its methodology and current results at halluminate.ai/blog/benchmark.

3. Introducing rtrvr.ai: The Local-First, DOM-Powered Agent

rtrvr.ai distinguishes itself in the crowded field of AI web agents through a fundamental architectural difference: its commitment to local operation. Unlike many leading agents that rely on remote cloud browsers, rtrvr.ai operates directly within the user's own browser on their device as a Chrome Extension. This design philosophy offers a suite of significant advantages that directly address common pain points in web automation.

By originating network requests from the user's local IP address, rtrvr.ai largely bypasses prevalent bot detection mechanisms and CAPTCHA challenges that frequently impede cloud-based agents. Furthermore, it avoids the use of Debugger permissions, which can expose browsers to exploits and trigger bot detection, and crucially, it can reuse local signed-in profiles and subscriptions. This means rtrvr.ai can seamlessly interact with authenticated content, such as signed-in Instagram accounts or paywalled articles, without requiring users to share sensitive credentials with a third-party provider, a capability often blocked for cloud browsers.

Beyond its local operational paradigm, rtrvr.ai employs a Document Object Model (DOM)-based approach to "see" and interact with web pages. Rather than relying solely on visual cues or screenshots, rtrvr.ai leverages the underlying HTML structure of a webpage, providing a deeper and more robust understanding of content and elements. This technical foundation enables several powerful capabilities: it allows for seamless operation in background tabs, facilitating powerful multi-tab workflows and efficient parallel task execution; it yields highly accurate scrapers by directly interacting with the page's structure; and it accelerates agentic workflows by utilizing full DOM information to skip redundant actions and distribute subtasks onto new tabs.

A particularly compelling benefit of this DOM-based approach is its ability to mitigate the "exponential failure rate" problem inherent in multi-step web automation. In complex workflows, the probability of overall success can decrease dramatically with each additional step if individual steps have independent failure rates. By parallelizing steps across multiple tabs, rtrvr.ai fundamentally re-architects this problem, making sophisticated, multi-step tasks significantly more robust and reliable. This is a profound technical advantage for enterprise-level automation, where complex sequences of actions are common.

rtrvr.ai further empowers users through its "AI Function Calling" capability, allowing them to define and supply their own custom code or functions that the AI agent can autonomously invoke. This feature provides immense flexibility and extensibility, enabling users to tailor the agent's capabilities to virtually any external tool, API, or custom workflow, effectively shifting control and customization power to the user. From a practical standpoint, rtrvr.ai operates as a client-side Chrome extension, vetted and tested by Google, ensuring a secure and sandboxed execution environment.

4. rtrvr.ai's Performance on the Halluminate Web Bench

rtrvr.ai's performance on the Halluminate Web Bench sets a new benchmark for AI web agent capabilities, demonstrating a remarkable 81.79% overall success rate that significantly surpasses other leading models. Access the complete data here.

Overall Success: A Comprehensive Look

On the Halluminate Web Bench, rtrvr.ai achieved an impressive 81.79% overall success rate, placing it at the forefront of evaluated agents and significantly outperforming models from prominent organizations. For context, the previously top-ranked Anthropic Sonnet 3.7 CUA achieved 66.0%, followed by Skyvern 2.0 at 64.4%, and OpenAI CUA at 59.8%. rtrvr.ai even beats Open AI Operator with Human Supervision at 76.5%. This performance suggests a redefinition of the current state-of-the-art for web agents on this specific benchmark.

Unparalleled Speed: Agent Runtime Duration

Beyond accuracy, the speed at which an AI agent completes tasks is a crucial performance metric, especially for latency-sensitive applications. Our evaluation showed that rtrvr.ai achieved an average execution time of 0.9 minutes (less than 1 minute) per task, making rtrvr.ai the fastest of all agents observed in this context.

This dramatically outperforms other agents benchmarked by Halluminate, which show average execution times ranging from several minutes to over twenty minutes per task:

This exceptional speed is a direct benefit of rtrvr.ai's local, DOM-based architecture. By operating as a Chrome Extension, it largely avoids network latencies associated with remote cloud browsers and leverages direct, efficient interaction with the DOM. This allows it to navigate and process information much faster than vision-based or remote alternatives, reinforcing its previously reported capability of being up to 4x faster than alternatives like OpenAI Operator.

Deep Dive: Excelling in Read Tasks

rtrvr.ai demonstrated exceptional proficiency in read-heavy tasks, achieving an 89.45% success rate. This performance underscores rtrvr.ai's strength in accurate data extraction and information retrieval, which are core capabilities for numerous automation needs. This high success rate directly correlates with rtrvr.ai's DOM-based approach, which enables it to "see" and interpret the underlying structure of web pages with high fidelity.

Deep Dive: Navigating Write Tasks

While write-heavy tasks are generally more challenging for all AI agents due to their interactive nature and need for precise state management, rtrvr.ai demonstrated strong performance with a 62.89% success rate. This score is notably higher than other agents on the benchmark. For instance, Skyvern 2.0, the next highest, achieved 46.6%, and Anthropic Sonnet 3.7 CUA scored 39.4%. This lead is significant, especially considering the general industry trend of lower performance on write tasks.

Our Evaluation Methodology and Key Learnings

Our evaluation of rtrvr.ai on the Halluminate Web Bench involved specific setup procedures and yielded valuable insights into the agent's capabilities and areas for future improvement:

5. Comparing rtrvr.ai to the Field: A Leader in Its Own Right

rtrvr.ai's performance on the Halluminate Web Bench is not merely a quantitative win; it represents a qualitative validation of its unique architectural approach. The consistently superior scores across all categories, especially when contrasted with agents that rely on cloud-based infrastructure, highlight a fundamental advantage derived from its local operation and DOM-based design.

Most AI web agents operating in cloud environments frequently encounter "Infrastructure issues" such as bot detection, CAPTCHA challenges, and difficulties with authenticated logins. These external barriers can significantly impede an agent's ability to complete tasks, regardless of its underlying AI intelligence. rtrvr.ai's local operation directly addresses these common pitfalls by functioning within the user's browser and leveraging their local IP address and existing signed-in sessions, inherently circumventing many of these obstacles.

Furthermore, rtrvr.ai's lead in the challenging "write task" category is particularly noteworthy. While the industry generally observes lower performance for write-heavy tasks due to their inherent complexity and the need for precise interaction and state management, rtrvr.ai's significantly higher success rate demonstrates its advanced capabilities in these interactive scenarios. This strong performance in a notoriously difficult area underscores the robustness of its DOM-based approach, which aids in navigating complex web elements and managing multi-step processes more effectively.

A crucial qualitative aspect of rtrvr.ai's performance lies in the nature of its failures. The overwhelming majority of rtrvr.ai's errors are "agent errors" rather than "infrastructure errors." This distinction is profoundly important for development and product teams. An "agent error" signifies that the AI itself failed to understand or execute a task, a problem that can be directly addressed through improvements to the model, refinement of prompt structures, or enhancement of its internal logic. Conversely, an "infrastructure error," such as being blocked by a CAPTCHA or IP detection, is often external to the agent's core intelligence and much harder for the agent developer to control.

6. Unpacking Failure Modes: Insights from rtrvr.ai's Performance

A critical component of the Halluminate Web Bench evaluation is the detailed breakdown of failure modes, which provides granular insight into where and why agents falter. For rtrvr.ai, this analysis reveals a highly favorable distribution of errors, underscoring the effectiveness of its unique architectural design.

The Breakdown: Agent vs. Infrastructure Errors

rtrvr.ai's failure analysis shows a striking imbalance between the two primary categories of errors:

This extremely low percentage of infrastructure errors is a direct and powerful testament to rtrvr.ai's local, browser-extension design. Unlike cloud-based agents that frequently encounter obstacles such as bot detection, CAPTCHAs, and login authentication issues due to their remote nature, rtrvr.ai's operation within the user's own browser effectively bypasses these common external barriers.

Focusing on Agent-Specific Improvements

The overwhelming majority of rtrvr.ai's failures fall into the "agent error" category. These types of failures typically relate to the AI's internal logic, such as issues with picking the right web trajectory for some niche tasks, fewer/missing tools to interact with tail-end websites, incomplete execution of tasks, sometimes can be attributed to less intelligent Gemini Flash Model used unlike Pro, more intelligent for complex tasks, and timeouts due to inefficient processing or model unavailability. rtrvr.ai doesn't use any reserved quota yet. While any failure indicates an area for improvement, having nearly all failures attributable to the agent itself is, paradoxically, a desirable characteristic for an AI agent.

This distribution means that rtrvr.ai's development team can concentrate almost entirely on enhancing the core AI's intelligence, reasoning, and robustness. Efforts can be precisely targeted at refining prompt structures, improving model configurations, and bolstering the agent's ability to understand and interact with diverse web elements. This contrasts sharply with agents that face a high proportion of infrastructure errors, where development resources must often be diverted to managing external factors like proxy rotations or CAPTCHA solving services.

7. The rtrvr.ai Advantage: Why Local and DOM-Based Matters

rtrvr.ai's exceptional performance on the Halluminate Web Bench is a direct consequence of its foundational design principles: local operation and a DOM-based approach. These are not merely technical specifications but strategic choices that translate into profound real-world advantages for users and businesses.

The local operation of rtrvr.ai, running as a Chrome Extension directly within the user's browser, is a game-changer for web automation. This architecture inherently bypasses the pervasive challenges of bot detection and CAPTCHAs, which are common hurdles for cloud-based agents. By originating network requests from the user's local IP address and reusing existing signed-in profiles and subscriptions, rtrvr.ai gains unparalleled access to authenticated and paywalled content that other agents struggle with.

Complementing this, rtrvr.ai's DOM-based approach provides a deeper, more robust understanding of web pages compared to agents that rely on visual parsing. By interacting directly with the Document Object Model, rtrvr.ai can operate seamlessly in background tabs, enabling efficient multi-tab workflows and parallel task execution. This technical capability is crucial for achieving accurate data scraping and faster agentic workflows.

8. Conclusion & The Future of Web Automation with rtrvr.ai

The Halluminate Web Bench has provided a critical, standardized framework for evaluating the burgeoning field of AI web agents, and rtrvr.ai's performance on this benchmark sets a new standard for the industry. With an impressive 81.79% overall success rate, rtrvr.ai has not only surpassed leading models from established organizations but has also demonstrated exceptional proficiency in both read-heavy tasks (89.45% success) and the notoriously challenging write-heavy tasks (62.89% success). Crucially, its average task execution time of 0.9 minutes positions it as the fastest among its peers, offering a significant advantage for latency-sensitive applications.

The core of rtrvr.ai's advantage lies in its unique architectural design: a local, browser-extension operation combined with a DOM-based approach. This design fundamentally addresses the pervasive "infrastructure issues" that plague many cloud-based agents, as evidenced by rtrvr.ai's remarkably low 5.26% infrastructure error rate. This means rtrvr.ai is less susceptible to bot detection, CAPTCHAs, and login challenges, ensuring consistent access and reliable task execution.

The overwhelming majority of its failures are attributable to "agent errors," which, while requiring attention, represent a "good problem" for development teams. This allows rtrvr.ai to focus its efforts almost entirely on enhancing the core AI's intelligence, reasoning, and execution capabilities, rather than expending resources on external environmental workarounds. This systematic evaluation provides a clear roadmap for future development, ensuring that product enhancements are data-driven and yield the most significant improvements in user experience.

rtrvr.ai's performance on the Halluminate Web Bench is not just a testament to its current capabilities but also a strong indicator of its potential to lead the next generation of AI web agents. Its ability to operate reliably within the user's environment, handle complex multi-step workflows, and integrate with personalized, authenticated content positions it as a powerful, secure, and efficient solution for a wide array of real-world web automation needs. As the demand for robust and dependable AI automation continues to grow, rtrvr.ai's proven architectural advantages and benchmark-validated performance make it a compelling choice for individuals and enterprises seeking to unlock the full potential of autonomous web interaction.