IBM Research has introduced ALTK-Evolve, an open-source memory subsystem that allows AI agents to learn from past execution attempts instead of simply re-reading historical transcripts. In benchmark tests on AppWorld, the system boosted Scenario Goal Completion on hard multi-step tasks by 14.2 percentage points. The framework addresses the common issue where autonomous agents fail to generalize principles from prior errors.
Distilling Principles From Raw Trajectories
Traditional agent architectures attempt long-term adaptation by re-injecting raw chat transcripts into the prompt window. ALTK-Evolve replaces this approach with a continuous dual-flow loop that extracts, refines, and injects actionable guidelines at the moment of action.
In the downward flow, an Interaction Layer powered by observability tools like Langfuse captures complete agent trajectories, including user commands, thoughts, API calls, and execution results. Pluggable extractors mine these traces for structural patterns and save them as candidate entities. In the upward flow, a background consolidation job merges duplicates, prunes weak strategies, and indexes high-value rules for just-in-time retrieval.
Benchmark Performance on AppWorld
IBM Research evaluated ALTK-Evolve using a ReAct agent on the AppWorld benchmark, where agents execute complex multi-step tasks across simulated APIs. Evaluated under strict Scenario Goal Completion (SGC) metrics on unseen tasks, the system delivered consistent improvements across all difficulty tiers:
- Easy Tasks: SGC increased from 79.0% to 84.2% (+5.2 points)
- Medium Tasks: SGC increased from 56.2% to 62.5% (+6.3 points)
- Hard Tasks: SGC increased from 19.1% to 33.3% (+14.2 points)
- Aggregate Score: SGC improved from 50.0% to 58.9% (+8.9 points)
The hard task category achieved a 74% relative increase in scenario goal completion by surfacing concise, learned guidelines at execution time. By retrieving only the top five relevant rules per execution step, the system prevents prompt bloat while reducing flaky behavior across task variants.
Integration Options and Current Limitations
Developers can integrate ALTK-Evolve across multiple workflow complexity levels. For rapid testing, a Lite mode plugin is available for environments like Claude Code, OpenAI Codex, and IBM Bob, allowing local trajectory extraction via filesystem hooks.
For enterprise setups, low-code ReAct integrations connect via Arize Phoenix tracing, while pro-code integrations connect through the Model Context Protocol (MCP) using tools like get_guidelines and save_trajectory.