이미지: Hugging Face 화면 갈무리
Summary
- IBM Research measured the effectiveness of agent memory across 8 models using ALTK-Evolve and found that the optimal amount of memory varied by model
- DeepSeek-V3.2 (671B) saw task completion rise 9.5 percentage points and scenario completion rise 16.1 percentage points with the full guideline set, while gpt-oss-120b (117B) achieved a 16.1 percentage point completion rate improvement using compressed core guidelines
- GLM-5 (745B) showed no measurable performance improvement even with added memory
- 발표
- IBM Research, Hugging Face 블로그, 2026-08-18
- 기법명
- ALTK-Evolve (가이드라인 추출·통합·검색 파이프라인, 가중치 업데이트 없음)
- 평가 벤치마크
- AppWorld, 585개 과제(test_normal 168 + test_challenge 417), 9개 시뮬레이션 앱
- DeepSeek-V3.2(671B MoE)
- 전체 가이드라인 세트 적용 시 TGC +9.5%p, SGC +16.1%p
- gpt-oss-120b(117B MoE)
- 압축 검색 방식 적용 시 TGC +16.1%p, 토큰 증가는 +5%에 그침
- GLM-5(745B MoE)
- 기억 추가에도 측정 가능한 성능 향상 없음, '포화' 패턴으로 분류
- GPT-5.5·Opus
- TGC는 포화 상태였지만 SGC는 각각 +7.2%p, +7.1%p 추가 상승
- 실험 모델 규모
- 30B 밀집모델부터 최상급 독점 모델까지 총 8개
It might seem obvious that feeding an AI agent all of its past experience would boost performance, but reality isn't that simple. In tests across 8 models, IBM Research found that giving the same amount of "memory" caused some models to improve dramatically, others to actually get worse, and others to show no change at all.
Memory needs a prescription too
ALTK-Evolve, released by IBM Research, is a tool that extracts lessons from an agent's own past task trajectories. It pulls out reusable behavioral guidance — strategies that worked, mistakes to avoid, exception cases — from both successful and failed tasks, consolidates them into a single guideline set, and feeds it back to the agent at inference time. Model weights are never changed, and there's no human labeling process involved. The research team says this keeps adoption costs low and makes the approach portable across different models.
The team validated this memory technique on the AppWorld benchmark. Agents performed 585 multi-step tasks (168 in test_normal, 417 in test_challenge) across 9 simulated apps covering calendars, messaging, and payments, scored on two metrics. TGC (Task Goal Completion) is the standard completion rate measuring whether an individual task was fully finished. SGC (Scenario Goal Completion) is a much stricter metric that only awards points if all variants of the same scenario — versions differing in data, phrasing, or exception conditions — are passed.
Eight models, three distinct responses
Testing 8 models ranging from 30B-class dense models to top-tier proprietary models, the researchers found models split into one of three patterns.
| Model | Scale | Response pattern | Completion rate (TGC) change | Scenario completion rate (SGC) change |
|---|---|---|---|---|
| DeepSeek-V3.2 | 671B MoE | Preferred full set | +9.5pp | +16.1pp |
| gpt-oss-120b | 117B MoE | Preferred compressed retrieval | +16.1pp | Completed with +5% tokens |
| GLM-5 | 745B MoE | Saturated (no change) | No measurable improvement | No measurable improvement |
| GPT-5.5, Opus | - | TGC saturated, SGC still rising | Near ceiling | +7.2pp, +7.1pp respectively |
Stronger models with more headroom could handle the full guideline set, including rare edge cases. DeepSeek-V3.2 fell into this category. Conversely, relatively weaker models were overwhelmed by extensive instructions. gpt-oss-120b performed best when given only a few task-specific instructions selected from a set of high-confidence core guidelines, using roughly half the tokens compared to the full-set approach. GLM-5 showed no noticeable change under either approach — a pattern the researchers termed "saturation," though they didn't pin down a definitive cause. They left open several possibilities: the model may have already been near its ceiling on these tasks, the guidelines may not have addressed the remaining failure points, or the model may have failed to properly apply the instructions.
The research team noted that which pattern a model falls into isn't determined simply by parameter count. Benchmark headroom, context window size, architecture, guideline quality, and task distribution all appear to play a role, and disentangling these factors remains ongoing work.
SGC bites harder than completion rate
The stricter SGC metric generally moved more than TGC. For DeepSeek, while TGC rose 9.5 percentage points, SGC jumped 16.1 percentage points — the research team interpreted this as good guidelines contributing more to passing every variant of a scenario than to average task performance. This effect held even for top-tier models. GPT-5.5 and Opus were already near their ceiling on TGC, but their SGC still rose by 7.2 and 7.1 percentage points respectively. As long as some failure types remained, memory continued to have an effect.
Token costs and prompt caching
Re-injecting the full guideline set at every step inflates input tokens. The compressed retrieval approach, by contrast, kept costs close to baseline and was advantageous for weaker models on both accuracy and cost. For DeepSeek, the number of ReAct steps stayed similar whether or not memory was used, averaging 18-19 steps. This means the additional cost came from increased input tokens rather than longer reasoning chains. The research team identified prompt caching as the key lever for reducing costs in production. Since portions of the guideline set repeat identically at every step, they can be cached — and designing the system to keep this prefix stable can bring costs down to a practical level even for strong models using the full guideline set.
Limitations and next steps
The research team stressed that these results are a starting point. They confirmed that the current retrieval method, which ranks guidelines by cosine similarity, doesn't perfectly predict which guideline will actually help with a specific task. They also noted that for very weak models, self-distillation from their own trajectories alone provides insufficient signal, and said they're exploring a separate teacher-model distillation approach. Validation is also currently limited to the single AppWorld benchmark, with broader agent benchmarks and real-world deployment testing said to be underway. Controlled experiments to disentangle context window size from a model's inherent capability also remain unfinished.
Around the same time, Nous Research also released a bot mode for its agent app Hermes Desktop, taking the approach of giving each bot its own role, model, and memory. This shows that how to divide up memory for agents is a topic multiple companies are tackling simultaneously.
Editor's take
What makes this research interesting isn't that it overturns the conventional wisdom that "giving memory helps" — it's that it attaches a condition to that wisdom. It's intuitive that stronger models digest more information better, but the fact that the 745B-scale GLM-5 showed no response at all demonstrates that parameter count alone can't predict memory capacity. Anyone who has designed agent systems in practice will recognize this pattern — apply the same prompt engineering technique across multiple models, and some respond, some show no reaction, and some actually get worse. This study puts hard numbers on that experience.
Teams operating agent pipelines domestically should check two things right away. First, guideline injection strategy needs to differ depending on whether the model in use is a top-tier model with context headroom or a small-to-mid-sized model. The common practice of dumping the entire manual into a small-to-mid-sized model can inflate token costs while actually hurting performance. Second, judging memory's effectiveness by completion rate (TGC) alone risks underestimating its real benefit. In this study, top-tier models showed no change in TGC but still improved on SGC. Services where reliability matters should also track stricter metrics like full scenario pass rate.
Over the coming weeks, this issue will likely push toward "advancing retrieval algorithms." The research team itself acknowledged the limits of cosine-similarity-based retrieval and flagged a selector trained on outcome signals as the next step. As results from deployment testing beyond AppWorld emerge, the current three-pattern classification could be refined further or even overturned.
