NVIDIA's NeMo Retriever team has developed an agentic retrieval pipeline that secured the #1 spot on the ViDoRe v3 leaderboard with a 69.22 NDCG@10 score. The architecture replaces standard semantic similarity search with an iterative reasoning loop, while also claiming the #2 position on the reasoning-focused BRIGHT benchmark. By moving away from domain-specific heuristics, the design demonstrates how autonomous agent workflows can handle complex enterprise visual and multi-step reasoning queries.
How the ReACT Retrieval Loop Functions
Standard Retrieval-Augmented Generation (RAG) relies on single-pass dense retrieval, which struggles with complex multi-part queries or visually dense layouts. The NeMo Retriever pipeline addresses this limitation by deploying a ReACT (Reasoning and Acting) architecture that enables the underlying language model to actively plan, search, and refine search terms.
The model utilizes dedicated function calls including think for strategy planning, retrieve(query, top_k) for corpus exploration, and final_results to package ranked output. When an agent hits context boundaries or step limits, the workflow executes a Reciprocal Rank Fusion (RRF) fallback mechanism to aggregate document rankings across all previous search iterations.
To eliminate the network overhead associated with external Model Context Protocol (MCP) servers, NVIDIA re-engineered the backend infrastructure. The team substituted the separate server setup with an in-process, thread-safe singleton retriever protected by reentrant locks, drastically lowering latency and maximizing GPU utilization across concurrent search tasks.
Benchmark Performance and Model Trade-offs
NVIDIA's agentic pipeline captured the top spot on ViDoRe v3 with a 69.22 NDCG@10 score, outperforming standalone dense retrieval baselines that scored 64.36.
Ablation experiments conducted across closed and open-weights foundation models highlighted significant trade-offs between reasoning performance and latency:
- Opus 4.5 + nemotron-colembed-vl-8b-v2: Reached 69.22 NDCG@10 on ViDoRe v3, averaging 9.2 retrieval calls and 136.3 seconds per query.
- gpt-oss-120b + nemotron-colembed-vl-8b-v2: Achieved 66.38 NDCG@10 on ViDoRe v3 with lower latency (78.6 seconds per query) and 2.4 average retrieval calls.
- Opus 4.5 + llama-embed-nemotron-reasoning-3b: Ranked #2 on the BRIGHT benchmark with an NDCG@10 of 50.90 across multi-step logical tasks.
- gpt-oss-120b + llama-embed-nemotron-reasoning-3b: Scored 41.27 NDCG@10 on BRIGHT, showing a wider gap between frontier and open models on deep reasoning workflows.