Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents

Removing the current task from the curator input lowers untrained JITMEM-base by up to 3.1 ALFWorld and 4.6 WebShop success-rate points; after RL training, the maximum drops widen to 11.4 and 10.4 points, which the paper interprets as evidence that RL learns to exploit the task signal.

Appendix examples show task-specific payload synthesis on all three benchmarks: combining partially relevant episodes into an ALFWorld clean-and-place procedure, converting unrelated product searches into WebShop search and attribute-selection advice, and distilling MMS cases into an ordered tau2-bench diagnostic that retains the policy requirement for explicit approval before a mutating call.

On tau2-bench with GPT-5.4 executor, prompted JITMEM-gpt reports 75.6 micro-average success rate, 3.9 points above the strongest baseline, driven by 72.6 on Telecom; on Airline and Retail, the paper says no memory method improves over no memory beyond variance, and JITMEM remains on par with baselines.

With Gemini-2.5-Pro as executor, transferred RL-trained JITMEM reports 86.2 ALFWorld success rate, 61.0 WebShop score, and 50.5 WebShop success rate; prompted JITMEM-gemini reports the highest WebShop values in the block, 72.1 score and 61.0 success rate.

On ALFWorld, a curator trained with Qwen3-8B and transferred to GPT-5.4 reaches 86.7 success rate, 1.4 points below a curator trained directly with GPT-5.4; with Qwen3-8B test executor the same training raises success from 60.5 to 77.4.

Changing the number of retrieved WebShop trajectories from k=3 to k=5 changes success rate by less than two points for every executor; the differences are within standard deviation for Qwen3-8B and GPT-5.4, and the main experiments use k=3.

For untrained JITMEM-base, storing all trajectories with correctness labels rather than filtering to successful trajectories lowers success by 1.5-2.9 points on ALFWorld and 2.3-3.4 on WebShop across executors; replacing raw traces with write-time ReasoningBank-style distillations lowers success by 1.7-2.9 on ALFWorld and 6.8-8.2 on WebShop.

With Qwen3-8B as executor, RL-trained JITMEM reports 77.4 ALFWorld success rate, 61.1 WebShop score, and 32.8 WebShop success rate, exceeding the strongest listed write-time baseline by 16.2, 20.5, and 16.3 points respectively; the untrained JITMEM-base is competitive on ALFWorld but has lower WebShop score and success than SkillOS-base.

On ALFWorld with GPT-5.4 executor, JITMEM uses 9.8K input tokens, 0.87K output tokens, and 11.6 steps per task; relative to JITMEM-base, the paper reports reductions of 10.1%, 13.0%, and 12.1%, while all memory methods add input context but generally reduce output tokens and steps versus no memory.

On WebShop, rebuilding the training bank after 100 GRPO steps and training 50 more raises success by 2.8 points for Qwen3-8B and 0.9 for GPT-5.4 but leaves Gemini-2.5-Pro unchanged; warm-starting the test bank with 100 training trajectories changes success by at most 1.3 points and remains within standard deviation.

Forcing the RL-trained curator to receive no retrieved trajectories degrades it to or below untrained JITMEM-base across all executors, with maximum reported success-rate drops of 14.8 points on ALFWorld and 15.2 on WebShop; the paper takes this as evidence that RL learns to distill retrieved experience rather than only emit hints from parametric knowledge.

The authors' reproduced no-memory baselines are at or below the SkillOS-reported values. For Qwen3-8B, thinking mode is closer on ALFWorld, non-thinking is closer on WebShop, and WebShop additionally uses calibrated search guidance because the reported no-memory result could not otherwise be fully reproduced.

For an identical input, the untrained curator emits a generic action sequence, while the RL-trained curator supplies an ALFWorld-specific workflow: move to the desklamp and then examine the bowl with it; the paper characterizes this as emergence of environment-specific procedural semantics.

Two ALFWorld tasks retrieving the same past experience receive different payloads: the hot-potato task foregrounds heat/cool state-transition guidance, whereas the newspaper task foregrounds placement and target-location verification.

With GPT-5.4 as executor, transferred RL-trained JITMEM reports 86.7 ALFWorld success rate, 53.8 WebShop score, and 45.4 WebShop success rate, while prompted JITMEM-gpt reports 83.3, 57.0, and 47.5 respectively; JITMEM-base with the weaker Qwen3-8B curator exceeds the listed stronger-curator write-time baselines on ALFWorld.

Main evaluation uses 140 ALFWorld test tasks, 500 WebShop test instances, and tau2-bench airline, retail, and telecom domains; banks start empty, tasks are streamed in batches, ground-truth verifiers score tasks, and the executor-as-judge controls bank admission.

Across 100 GRPO steps on ALFWorld and WebShop, the plotted training and validation success rates generally rise while executor turns fall; the paper describes training as stable under task reward alone, without auxiliary content-quality reward, task grouping, or return shaping.