IBM Research introduced VAKRA, an executable benchmark featuring over 8,000 enterprise APIs across 62 domains to diagnose AI agent execution failures. The framework evaluates autonomous agents across multi-step API chaining, document retrieval, and policy adherence tasks where standard models consistently struggle. By analyzing full tool-execution trajectories rather than output text alone, VAKRA reveals the exact structural failure modes crippling modern agentic workflows.
Four Core Tasks Exposing Agent Weaknesses
Traditional benchmark datasets often evaluate AI models on isolated questions or standalone code execution. IBM Research's VAKRA benchmark reveals that existing LLMs perform poorly when required to coordinate multi-step tool calls across complex enterprise APIs and domain documents.
To systematically identify where agents fall short, VAKRA organizes tasks into four distinct operational capabilities:
- Capability 1 (API Chaining): Comprises 2,077 test instances across 54 domains using the SLOT-BIRD and SEL-BIRD tool collections, requiring 1 to 12 tool calls to process structured data via Model Context Protocol (MCP) servers.
- Capability 2 (Tool Selection): Tests 1,597 instances across 17 REST-BIRD domains with toolsets averaging 116 APIs, forcing agents to manage API selection under OpenAI's 128-tool limit.
- Capability 3 (Multi-Hop Reasoning): Measures 869 test instances across 38 domains requiring 1 to 5 logical hops across dashboard endpoints to synthesize answers.
- Capability 4 (Multi-Source & Policy Adherence): Evaluates 644 instances across 41 domains combining API calls, RAG document retrieval, multi-turn dialogs, and plain-text tool-usage rules.
Execution-Centric Waterfall Evaluation Architecture
To prevent inefficient data transfers over protocol layers, VAKRA utilizes a special get_data tool that initializes databases server-side while supplying lightweight data previews to the agent. This design ensures that agents interact with realistic database environments without incurring massive payload overhead during execution cycles.
Evaluation follows a multi-stage waterfall pipeline that inspects both reasoning paths and final answers. First, Capability 4 tasks undergo programmatic policy verification to confirm that agents respected tool usage constraints. Next, predicted tool-execution trajectories are executed in the benchmark environment to verify intermediate state outputs against ground truth responses. Finally, an LLM judge evaluates output factual consistency only for samples that successfully pass trajectory verification.