A new enterprise IT evaluation from IBM and Artificial Analysis shows that top AI models fail to achieve a 50% success rate when resolving real-world infrastructure failures. The ITBench-AA benchmark evaluated frontier systems on 59 complex Kubernetes incident response tasks to test autonomous Site Reliability Engineering (SRE) capabilities. The results reveal a significant gap between conversational AI performance and reliable agentic execution in enterprise settings.
Quantifying the Frontier Gap
Claude Opus 4.7 led the leaderboard with a score of 47% at a cost of $5.38 per task, followed closely by GPT-5.5 at 46% and Qwen3.7 Max at 42%. Among open-weights models, GLM-5.1 reached 40% at $1.23 per task, matching Gemini 3.5 Flash while operating at a lower cost. Smaller open models also demonstrated high cost-efficiency, with Gemma 4 31B scoring 37% at just $0.14 per task, outperforming Gemini 3.1 Pro Preview's 30% score at $2.23 per task.
Every tested frontier AI model scored below 50% on the benchmark, exposing critical challenges in agentic troubleshooting. The testing framework gave agents shell access to sandboxed environments containing alerts, metrics, logs, and application topology. Tasks required models to issue terminal commands and return a structured JSON report specifying the precise Kubernetes entities responsible for root causes such as network blockages or quota exhaustion.
The Pitfalls of Over-Investigation
The evaluation revealed that longer agent execution paths often decreased accuracy due to strict scoring criteria. Because performance was measured using precision gated by full recall, models were heavily penalized for identifying co-occurring symptoms or upstream chaos mechanisms alongside true root causes.
For example, GPT-5.5 averaged 31 turns per task to achieve 46%, whereas Gemini 3.1 Pro Preview averaged 83 turns but dropped to 30% after generating excessive false positives. Models that over-investigate tend to flag secondary fault-injection mechanisms, which harms their overall precision score.
Key Performance and Cost Breakdown
According to IBM and Artificial Analysis, performance varied widely across model families and pricing tiers:
- Claude Opus 4.7 (Adaptive Reasoning): 47% accuracy at $5.38 per task
- GPT-5.5 (xhigh): 46% accuracy averaging 31 turns per task
- GLM-5.1 (Reasoning): 40% accuracy at $1.23 per task
- Gemini 3.5 Flash (high): 40% accuracy at $1.70 per task
- DeepSeek V4 Pro (Reasoning): 38% accuracy
- Gemma 4 31B (Reasoning): 37% accuracy at $0.14 per task
- Gemini 3.1 Pro Preview: 30% accuracy averaging 83 turns at $2.23 per task