Agent learning & self-improvement

A support agent resolves a difficult case after a specialist corrects it. A coding agent finds a reliable way to reproduce a failure. A document agent discovers why it keeps extracting the wrong field. Each experience contains something worth learning. The business benefit arrives when that discovery improves later work: fewer repeated corrections, less investigation, more reliable completion.

That creates a practical question for companies building agents: what should change because of experience, and how will we know that change helped? A customer preference, a reusable procedure, a tool repair and a trained model each answer a different problem. They also require different evidence, maintenance and release decisions.

Our reading of the research and operational accounts points to a useful direction: treat learning as a product capability with explicit inputs, persistent artifacts and measurable consequences. This field note develops that view through the main design patterns, lessons from deployed systems, a reference architecture and a way to choose between approaches. RRSI and the furniture walkthrough later in the article illustrate one part of that wider landscape.

1. Follow the experience all the way to later work

An agent combines a model with a harness: the instructions, context, memory, skills, tools and control flow that shape its work. The runtime executes those components. A learning process decides what experience should change; a release process decides when the change becomes active.

An agent can use feedback at several timescales. During a task, it can inspect a tool error, revise its plan and try again. Between tasks, it can retrieve a remembered preference or an earlier solution. Between releases, an optimizer can revise its prompt, skills or code. A training process can update the model's parameters.

These mechanisms can work together. A successful retry might reveal a procedure worth retaining; repeated failures might justify a code repair; a large collection of verified trajectories might become training data. The important design decision is which information crosses from one timescale to the next.

We use four links to assess a learning claim: experience → persistent change → later use → measured consequence. A stored lesson demonstrates retention. A later trace shows whether the agent retrieved and followed it. A comparison on eligible tasks establishes whether it was useful. Breaking down the chain makes failures diagnosable: the lesson may be wrong, unavailable, ignored, or relevant without improving the outcome.

Updating a live inventory record keeps business data current. Learning a better way to check inventory changes future behavior. Similarly, a conversation history can preserve what happened without extracting a reusable lesson. Both capabilities are useful; they need different product contracts.

The distinction between writing an improvement and benefiting from it also appears in research. Harness Updating Is Not Harness Benefit studies these capabilities separately and finds that stronger task-solving models do not uniformly benefit more from updated harnesses. Our engineering interpretation is to test the complete pairing: the learned artifact, its retrieval mechanism and the agent that must use it.

2. Five design patterns for durable improvement

Experience memory: carry the right lesson forward

Memory is useful when later work lacks information already learned elsewhere. An agent might retain a user's preference, the outcome of a previous investigation, or a general lesson about a recurring failure. These correspond to facts, experiences and procedures; storing them together without distinguishing their purpose makes retrieval and correction harder.

ReasoningBank explores extracting reasoning lessons from successful and unsuccessful trajectories. ExpeL combines insights from experience with retrieval of relevant past cases. Both illustrate improvement through external knowledge while model weights remain fixed.

The reusable pattern is distil, scope, retrieve, verify. Preserve the evidence behind a lesson, specify where it applies, select it for a later task, then measure its effect. For a research agent, an earlier search strategy may be reusable while the facts it found require fresh verification. For a support agent, a customer preference belongs to that customer; a general troubleshooting procedure needs broader evidence before becoming shared guidance.

The trade-off is retrieval quality. More retained information can introduce irrelevant, contradictory or stale advice. Measure useful retrieval and later behavior alongside memory writes. A system that keeps fewer, better lessons may outperform one that remembers every transcript.

Reusable skills: turn a discovery into a procedure

A skill describes how to do a class of work. It can include instructions, examples, scripts and checks. For a coding agent, a skill might explain how to reproduce a service failure; for a document agent, how to reconcile fields across sources. Compared with an isolated memory, it has a clearer invocation condition, expected result and maintenance owner.

Voyager demonstrates an executable skill library in Minecraft, alongside curriculum generation and environmental feedback. That research setting establishes a mechanism for accumulating reusable behavior. SkillOpt instead studies bounded edits to external skill documents, with validation controlling which changes remain.

Our proposed enterprise pattern is discover once, package, test on another eligible task. A new skill should explain when to invoke it and when to stop. Validate its dependencies and whether the agent actually chooses it. If the procedure always applies and correctness is mechanically checkable, implement the critical steps in ordinary software and use the skill to explain their use.

Skills introduce instruction maintenance: duplicated rules, broken references and increasingly narrow exceptions. Replacing an obsolete procedure can be more valuable than adding another one.

Prompt and context optimization: improve how evidence is used

This pattern fits an agent that has the necessary tools and information but applies them inconsistently. A prompt change can clarify the decision rule. A context change can improve which evidence is selected, summarized or presented. The persistent artifact may be a system instruction, a retrieval policy or a maintained playbook.

GEPA uses execution feedback and reflection to search for better prompts. Agentic Context Engineering develops contexts through generation, reflection and curation. Their shared lesson is that concrete feedback about a failure provides a more useful improvement signal than a score alone.

Our design rule is to separate missing evidence from misused evidence. If a product's dimensions disappear inside a tool adapter, prompt tuning cannot reliably restore them. If the dimensions reach the model but it ignores a clearance requirement, the instructions or context organization become plausible intervention points.

This approach offers inspectable, reversible changes without retraining. Its limitation is dependence on the examples used during search. A prompt can learn particular answers, accumulate fragile exceptions, or improve one component while hurting the whole workflow. Evaluate the downstream task as well as the optimized component.

Workflow and code evolution: repair the system around the model

Some failures live in tool contracts, argument mapping, routing, retries or output schemas. Here the learned artifact is executable behavior. A document agent might need a missing extraction field added; a support agent might need a different escalation path; a coding agent might need a reliable test command.

AFlow searches code-represented workflows using execution feedback. The Darwin Gödel Machine explores agent code changes and maintains an archive of candidates. These approaches broaden the editable surface beyond instructions.

The transferable pattern is diagnose the failing boundary, propose a bounded patch, preserve working behavior. More editing freedom can uncover repairs that prompt-only search cannot express. It also increases the number of ways a candidate can alter permissions, compatibility and failure handling.

Our recommendation is to grant edit access according to the diagnosed problem. Keep source evidence readable, isolate candidate execution, and protect the acceptance conditions. A change to a tool result schema deserves compatibility checks; a new retry policy deserves failure injection and latency checks. The evaluation should follow the behavior being changed.

Model training: put learned behavior into the weights

Training becomes relevant when repeated work produces useful examples or rewards and the limitation persists across good tools, instructions and context. The durable artifact is a model checkpoint or adapter. Learning may be less visible in each request because the behavior is encoded in parameters rather than a retrieved procedure.

Agent Lightning separates agent execution from a reinforcement-learning training process. SEAL explores models generating self-edits that are used for adaptation through fine-tuning. Both require a training loop; retaining a memory or making a normal API call does not perform that update.

This is a different operational commitment: data preparation, reward quality, training infrastructure, versioned serving and retention testing. It can be appropriate for high-volume, stable task families. Rapidly changing company policies and customer-specific facts often benefit from separately editable sources.

The choice is therefore about where the behavior should live. Keep facts that must be corrected or revoked in an accessible store. Consider training for recurring capabilities supported by enough reliable evidence. Test the resulting model with the harness it will actually use.

3. What operational systems teach us

The production evidence is most useful when it exposes the change mechanism. Company adoption, agent activity and benchmark performance answer different questions. We selected the following accounts for the engineering lessons they reveal; the comparisons below are our synthesis of those accounts.

Correction becomes valuable when its cause is recoverable

In the Tax AI pilot built for Crete, OpenAI and Thrive describe capturing practitioner corrections alongside source documents, extracted fields, mappings and final outputs. Repeated, reviewed failure patterns become targeted evaluations and bounded Codex engineering tasks. Engineers retain responsibility for product decisions and shipping. The team's account also explains why a changed field can represent workflow noise or expert judgment rather than an extraction error.

The design lesson is to make correction part of the product's evidence model. “The specialist changed the answer” is too ambiguous for automatic learning. “This source field was present, this adapter omitted it, and the specialist restored it” identifies a repairable cause. A business can reduce repeated diagnostic work while preserving expert judgment where the signal remains uncertain.

Shared procedures need an owner and a scope

Shopify describes using shared River interactions to improve skills, prompts and defaults, supported by a reproducible engineering environment. Asana describes a different authority model: anyone can provide task feedback, while admins and editors control what enters permanent shared memory. These are mechanisms described by Shopify and Asana, rather than controlled comparisons of their learning benefit.

Together they suggest two distinct design decisions: make useful work discoverable, and decide who can turn it into future guidance. Shared visibility helps a team avoid rediscovery. Controlled promotion protects a team from treating one person's preference as universal policy. In a multi-client product, the equivalent boundary is between customer-specific learning and changes to the shared agent.

Learning creates a maintenance workload

Warp describes scorers identifying failed runs and a scheduled improvement process proposing skill and configuration diffs for review. It explicitly discourages piling on addenda, favoring redrafts and removal of outdated guidance. Warp's account is a concrete example of turning failure analysis into maintained artifacts.

The design lesson is that a learning loop needs an editing policy, not just an adding policy. Every retained rule competes for context and may contradict an older rule. Ask whether each change adds, replaces, narrows or removes guidance. Track review effort and rejected changes alongside successful updates. Increased proposal volume can otherwise hide increased operating work.

The evaluator must preserve the business question

Dropbox reports optimizing its Dash relevance judge with DSPy. Early candidates copied example-specific content or changed the rating scale. The team constrained those behaviors and measured agreement against human labels; the account reports normalized mean squared error improving from 8.83 to 4.86 for one comparison. This measures that judge's agreement, rather than end-to-end customer resolution. Dropbox's engineering account makes both the mechanism and the failure modes visible.

The broader lesson is that optimization pressure exposes weaknesses in the objective. A learner can produce a better number by changing the problem. Protect the scale, success conditions and reference evidence. When the judge itself needs improvement, treat that as a separate change with its own acceptance test.

Decagon's DuetBench-2 account examines agent-building capability, production outcomes and the improvement process itself, including changes to simulation and judging machinery. These vendor-reported dimensions are useful to distinguish, but they do not establish unlimited recursive growth. Improving a support procedure and improving the system that discovers procedures are separate claims, each needing later evidence.

Across these accounts, the most reusable asset is the chain connecting a failure to an understandable change. Public reports provide uneven detail about rejected candidates, total cost and long-term retention. They support specific mechanisms and reported outcomes; they cannot establish an industry-wide return on investment or a ranking of vendors.

4. An architecture that connects learning to delivery

Our reference design separates four responsibilities: executing work, preserving evidence, proposing improvements and activating validated versions. They can share infrastructure while retaining clear permissions and budgets.

The execution path uses a known configuration: model, instructions, selected skills, scoped memory and tools. It returns an answer or action result and records the evidence needed to explain it. It remains usable while an improvement job runs elsewhere.

The evidence path captures source versions, selected context, tool calls, outcomes and corrections. Grouping similar failures can reveal a recurring cause, but labels require a clear interpretation. An abandoned checkout is not automatically a bad recommendation. A support ticket closing is not automatically a correct resolution. Where the environment cannot establish success, use sampled expert review and record uncertainty.

The improvement path selects a failure class, forms a hypothesis and edits an allowed artifact. Its output is a candidate: the original version, the diff, the triggering evidence, the expected effect and evaluation results. Logging rejected candidates helps avoid repeating failed ideas and makes the cost of search visible. The optimizer can use a different model and more time than the customer-facing agent.

The activation path decides which version later tasks use. Candidate generation and release are separate responsibilities. A low-risk personal preference may enter scoped memory after an appropriate admission check. A shared procedure needs broader evidence and an owner. Tool code or model changes require their own release process. These channels let a product learn at different speeds without giving every update the same authority.

Version the complete behavior boundary: prompts, skills, tools, model configuration, context policy, evaluator and relevant data snapshots. Otherwise, an improvement attributed to a prompt might actually come from a model or source change. Keep a rollback target and inspect post-release outcomes. Restoring a version changes future behavior; external actions already taken may require separate compensation.

5. Measure improvement at three levels

First ask whether the artifact works. Then whether the agent uses it effectively. Finally whether later business work improves. This prevents a technically correct skill or a better component score from becoming an unsupported claim about customer value.

Use deterministic evidence where the task allows it: totals can be recalculated, tool results inspected, database writes read back, and tests executed. Model judges can help evaluate open-ended quality, but should be calibrated against expert examples and audited for disagreement. Human review also needs a rubric and a way to resolve inconsistent labels.

Separate development feedback from acceptance evidence. The optimizer can inspect failures while searching. Selection data compares candidates. Regression cases protect previously working behavior. A release holdout tests generalization after selection. Repeatedly using the same holdout to guide changes turns it into development data; fresh evidence and limits on adaptive reuse become necessary.

Compare candidates under matched conditions: the same eligible tasks, model settings, tool access and data versions, with repeated trials where execution varies. To understand the cause, compare the proposed repair with a direct manual fix and test removal of the changed component. Report sample sizes and failure categories. A single blended score can conceal a valuable gain accompanied by an unacceptable regression.

EvoHarnessBench finds that expanding a harness can reduce performance on previously solved tasks, and that adaptation and retention do not always improve together. Our practical response is to maintain old-task checks as new tools, skills and agents are added. Capability count is a poor substitute for retained capability.

Static replay is useful for diagnosis but has limits: a different tool call may produce an observation absent from the original trace. Use executable environments, controlled simulations or bounded live trials when outcomes depend on new actions. After release, continue measuring eligible tasks and adverse cases; an offline win can fade as policies, APIs and customer needs change.

6. Choose the intervention before the framework

A learning method is useful when its editable surface matches the failure. The same customer-visible mistake can originate at several boundaries. Diagnose with the trace, then choose the smallest complete repair.

Parts of the machinery are already available. LangMem provides tools for memory and prompt optimization. Google's ADK optimization interface offers experimental optimizers with distinct edit scopes: its root prompt optimizer changes root instructions; its root agent optimizer also changes skill instructions, while excluding tools and sub-agents. The scope matters more than the generic label “agent optimizer.”

Managed services also package persistence and extraction: Google Memory Bank and Amazon Bedrock AgentCore Memory document memory capabilities. A platform can supply infrastructure while the application still decides what should be remembered, where it applies and how to verify benefit.

Our build-versus-buy boundary is straightforward: reuse storage, tracing, search and evaluation machinery when it fits; retain ownership of the business objective, evidence labels, scope, non-negotiable constraints and release accountability. Request an exportable change history from any packaged system. A convincing demonstration should connect the original experience, the accepted artifact and later outcomes.

Keep these artifacts portable. A stronger base model may make a workaround unnecessary, and a framework upgrade may invalidate a skill. Compare a new model with and without accumulated guidance before migrating it. Learning should make upgrades easier to assess, rather than lock the product into its past assumptions.

7. Accumulated learning needs an operating budget

The cost of improvement includes proposal generation, evaluation, storage, human review, release work and maintenance. Serving cost can also rise if the learned harness retrieves more text or makes more calls. Measure cost per valid completion and the burden of repeated correction, rather than tokens alone.

A simple planning estimate is: eligible future tasks × expected net benefit per task − improvement and maintenance cost. Use the population that can actually benefit. A lesson relevant to one customer should not be credited across every request. Compare the estimate with a direct code fix, better source data or a model upgrade; automated search is one option among them.

Set a stopping rule before running optimization: a budget, a minimum worthwhile gain, a regression tolerance and a saturation condition. More rounds create more opportunities to fit the evaluation examples. A rejected candidate can represent a useful outcome when it prevents an expensive or fragile release. Round count is an experiment setting, not a measure of learning.

Memory and instructions need lifecycle management. Each retained lesson should have provenance, scope, an owner and a way to resolve conflicts or remove it. Review it when the underlying policy, tool or model changes. Controlled deletion preserves relevance; indiscriminate retention grows context cost and instruction debt.

Persistent learning also changes the trust boundary. AgentPoison demonstrates attacks through poisoned memory or knowledge bases. Our design response is to keep retrieved material as evidence, protect permissions and release rules outside the learner's editable surface, and authorize promotion into behavioral guidance. Read access to a customer trace does not imply permission to turn it into a rule for other customers.

Privacy and applicability are separate concerns. A lesson may contain no confidential information and still be wrong for another team. Widening its scope requires a decision about relevance, not merely a check for secrets.

8. Where RRSI fits: disciplined harness evolution

Regularized Recursive Self-Improvement applies to the system around a fixed model: its harness. The RRSI project and released Google Research code constrain both proposal generation and candidate selection. The approach includes shrinking edit budgets, a history of evidence, leakage screening, noise-aware acceptance, inference-cost rules and pruning.

Its useful contribution is a discipline for accumulating changes. A proposal should identify a transferable mechanism. Its measured gain should survive the uncertainty of the baseline and justify added cost. Earlier changes can be reconsidered when they stop earning their place. A critic can screen a proposal, while executable boundaries and protected evaluation remain necessary for a production adaptation.

This fits recurring failures that may require edits across instructions, context, skills or tool code. It requires meaningful feedback and a reproducible evaluation environment. It cannot infer an unavailable business truth merely by inspecting an answer. Corrections, source evidence and verifiable outcomes supply that truth, with expert review where interpretation remains uncertain.

The furniture walkthrough makes the mechanism visible. The same five customer requests encounter changing harness components, then produce responses that are validated. Select a change to inspect its code diff. Watch whether the repair makes a relevant fact available, changes a decision rule, or improves how the agent completes the task.