Memory reconstruction over memory replay: a better pattern for giving agents useful past experience

A new paper called MemHarness names and formalizes a pattern that a lot of teams building agent memory systems have been informally bumping into: most memory-augmented agents today just retrieve a similar past experience and paste it verbatim into the current context, trusting the model to figure out what's still relevant. The paper shows this "replay" approach frequently backfires — the researchers call it negative transfer — because a stored experience was shaped by a different state of the world, and injecting it unchanged can actively mislead the agent when the current situation differs even slightly. The fix they propose is architectural: instead of retrieve-then-paste, the agent runs retrieved memories through a policy model that critiques and rewrites them to fit the present context before acting on them, essentially asking "given what actually worked last time, and given how this situation differs, what should I actually do here" rather than "what did I do last time." This reconstructive step is trained end-to-end with reinforcement learning (GRPO) rather than bolted on as a fixed rule, so the model learns when to trust, adapt, or discard a retrieved memory. Tested on ALFWorld and WebShop, agents using this approach beat both plain RL agents with no memory and agents using static memory replay, with the biggest gains showing up in out-of-distribution scenarios — exactly the situations where verbatim replay is most likely to mislead. For developers building anything with long-term memory (coding agents that recall past fixes, support agents that recall past tickets, personal assistants that recall past preferences), the practical takeaway is a design pattern: don't just retrieve and inject memory into context — build a critique/reconstruction step that actively adapts retrieved experience to the current state before the agent acts on it, since raw replay is a known source of silent, hard-to-debug failures.

Source

View on ShipDigest