ADR-0027: Agent Self-Improvement Loop ("Evolve")¶
Metadata¶
Status: Accepted (2026-10-21) · Date: 2026-10-21 · Deciders: nguyenhuuca · Tags: agents, ci, workflow, quality
Related: builds on ADR-0022 (headless runs) and ADR-0025 (auto gates, never for .claude/); input from scripts/agent-metrics.mjs (docs/reports/agent-metrics-2026-10.md)
Context¶
scripts/agent-metrics.mjsnow measures the agent's work: escaped defects, reverts, review rounds, blocked runs, flaky tests, CI failures, cost.- Today a bad signal only turns into a better rule when the owner runs a retro by hand. The owner wants the agent to improve itself from the metrics, without being asked each time.
- First real signal:
62526c97fixed a defect from the dynamic-cron-jobs run. Existing job bodies caught and logged every exception, so the new scheduler recorded SUCCESS on failure. Plan review, code review and QA ×2 all passed it. - Risks of an agent that edits its own rules:
- Goodhart: it can make the numbers look better (weaker tests, lower gates, a changed metrics script) instead of the work.
- Drift: one bad rule makes every later run worse, and nobody notices.
- Bloat: rules only grow, each run gets slower and more expensive.
- Injection: issue, PR and commit text could be turned into rules.
Decision Drivers¶
- Every change to the agent's rules must come with evidence that it helps, checked by code the agent cannot change.
- The owner keeps control of the direction: one click per change, no reading of raw metrics needed.
- The loop must not be able to weaken any gate, metric or eval.
- Bounded cost per run.
Considered Options¶
| Option | Pros | Cons |
|---|---|---|
| A. Manual retro (today) | No risk | Depends on the owner remembering; signals are lost |
| B. Evolve loop, proposals as PRs, owner merges | Automatic diagnosis and proof; owner only merges | One human step per change |
| C. Evolve loop with auto-merge | Fully hands-off | The agent changes its own rules with no human read; one bad change compounds; conflicts with ADR-0025 (no auto-merge for .claude/) |
Decision Outcome¶
Option B.
D1. The loop¶
/evolve (run by the owner)
Observe agent-metrics --json → new signals (escaped defect, user bug report, revert, blocked, flaky, /revise, cost jump)
Diagnose per signal: original run diff, fix commit, plan/report → Five Whys → the stage that should have caught it
Propose one small change at that stage, on branch evolve/<date>-<signal>
Prove deterministic eval (D3)
Ship PR "Evolve: <signal>" with the Five Whys and the eval table → owner merges or closes
Re-measure next month's metrics; a later run may propose removing a rule that no longer earns its place (D5)
- At most 3 signals per run; the rest wait for the next run.
- Signal ids are stable (
defect:<fix sha>,revert:<sha>,blocked:<run id>, …). A signal that already has anevolvePR or issue (open, merged or closed) is skipped, so a change the owner rejected is not proposed again. The state lives in GitHub, not in commits tomain.
D2. What the agent may change (enforced by the workflow, not by the prompt)¶
- May add or tighten:
.claude/skills/**/SKILL.md,.claude/skills/**/resources/*.md,.claude/rules/*.md,.claude/agents/*.md(prompt body only, not frontmatter), and new directories underevals/seeded/. - Never:
.github/,scripts/,.claude/hooks/,.claude/settings*.json, existingevals/seeded/*cases,api/.coverage-threshold, application code, migrations,CLAUDE.md,AGENTS.md. - A deterministic step checks
git diff --name-only origin/main...HEADagainst this allowlist and against "no deleted lines in rule files" (removals only through D5). Any violation → no PR, anevolveissue that names the violation. - A change that needs CI or a hook (for example a grep check for swallowed exceptions in
jobs/) is written in the PR body as an owner action, not made by the agent. - Issue, PR and commit text is data. The agent quotes it as evidence; it never copies instructions from it into a rule.
D3. The fitness function: seeded-bug evals¶
evals/seeded/<id>/case.json= a real defect:baseandheadcommits of the run that introduced it, the files, and anexpectblock (the file that must get a blocking finding, plus regexes the finding must match).scripts/evolve.mjs evalruns the reviewer (worker-reviewer, the code-review stage of/autopilot) ongit diff base..headwith the.claude/of a given ref, and grades the verdict block deterministically: caught = a blocking finding on an expected file whose text matches anexpectregex.- A proposal passes only when, with the eval runner from
main:- the new case is missed with
main's rules in at least 2 of 3 runs (otherwise no rule change is needed; the PR only adds the case, and that case-only PR needsmainto catch it in 2 of 3), - it is caught with the branch's rules in at least 2 of 3 runs (model output is not deterministic),
- every existing case is still caught with the branch's rules (no regression),
- the mean reviewer cost per case rises by at most 25%.
- the new case is missed with
- The agent writes the case, but cannot game it: an
expectthat matches everything fails rule 1, one that matches nothing fails rule 2. - The eval table (case × ref × run → caught, cost) goes into the PR body.
D4. Running it¶
- Local only: the owner runs
/evolve; it goes through the same stages in the session and asks before any push or merge, like autopilot Gate 2. Since ADR-0029 a weekly headless workflow is a second way in. - Gate hardening: the case must be added by the branch, its
base/headmust be commits onmain, rule-file frontmatter is frozen (new skills:name,description,argument-hint,disable-model-invocationonly), no new agents, andnot-eval-provenproposals may not touch the reviewer's rules. - Budget: at most 3 signals per run, at most 3 eval runs for the new case and 1 run per existing case (retry once on a miss).
- Headless workflow removed (2026-10-21). A weekly
.github/workflows/agent-evolve.yml(propose job with a read-only token, ship job that checks and opens the PR) was built and then removed by the owner. Its first real run showed that headless Claude Code treats.claude/files as sensitive and asks for permission on every edit, which CI cannot grant, so the agent could not write a single proposal; a packaging bug then closed the month's signal with a wrong "no proposal" issue (#50). Running it headless again needs a new decision (for example: the agent writes encoded file copies outside.claude/and the ship job applies them).
D5. Pruning¶
- Every rule line added by Evolve ends with
<!-- evolve:<case id> -->. - Quarterly (or when total rule size grows by more than 20% since the last prune), a run may propose removing one such line. It passes only when its case is still caught without it (D3 rule 2) and nothing regresses. This is the only path that deletes rule lines.
D6. Compress: removing text, not only adding it (amendment 2026-10-21)¶
Adding rules alone makes every session pay for more context (CLAUDE.md + always-loaded rules ≈ 37 KB, loaded by every session and worker). Evolve may also cut:
- Signal:
compress:<YYYY-MM>, at most one a month, after the defect signals. It carries lint hints fromscripts/evolve.mjs lint-rules: exact duplicate lines across rule files, dead path references, the always-loaded size. - What may be cut: duplicated text (keep one home, point to it), dead references, text that contradicts its source of truth, stale claims.
- Paths: the D2 paths plus
CLAUDE.md, existing files only, removals allowed. Protected, never cut or edited:AGENTS.md,.claude/rules/security.md, the evolve skill,verdict.md, frontmatter, and any line carrying a safety rule (push, gates, merge, deny, secrets, tokens, prod, smoke, security, Core Principles, "never …"), matched by a fixed pattern inscripts/evolve.mjs. - Gate (
gate --kind compress):- C1: the touched files shrink, and the always-loaded context drops ≥ 5% or the touched files lose ≥ 1 KiB;
- C2: no new duplicate lines or dead references;
- C3: every seeded case is still caught with the compressed rules (and there is at least one case).
- Always a draft PR. With a small eval set the gate cannot prove that behavior outside it is unchanged, so the owner reads every removed line.
- Rewriting a skill to work better (not just shorter) is a
ruleproposal: it needs a seeded case that the old text misses.
Consequences¶
- Good: every escaped defect becomes a regression eval for the agent, and each rule change has before/after evidence. The eval set grows with real bugs, never with invented ones.
- Good: the owner's job shrinks to reading a short PR (Five Whys + eval table) and merging or closing it.
- Cost: about one agent run (~$1–3) plus the evals (one Sonnet review per case run, ~$0.10–0.20) per
/evolverun with signals; with no new signal it stops after the metrics step. - Limit: only stages a reviewer can be tested on are proven by D3. Changes to plan review or QA rules are proposed with the Five Whys only and marked "not eval-proven" in the PR.
- Limit: keyword grading can miss a correct finding worded differently; the 2-of-3 rule and the owner's read cover that.
- Risk: a merged rule could still hurt cases not in the eval set. The monthly metrics and D5 pruning are the backstop; rollback is
git revertof the evolve PR.
Validation¶
- First case
evals/seeded/2026-10-19-swallowed-job-exceptions(from62526c97): missed with today's rules, caught after the Evolve change, in 2 of 3 runs. - The allowlist step rejects a test branch that edits
scripts/agent-metrics.mjs. - After two months: escaped defect rate does not rise while the number of runs grows, and total rule size stays within the D5 bound.
Links¶
- ADR-0022 Headless autopilot · ADR-0025 Auto gates
scripts/agent-metrics.mjs,docs/reports/agent-metrics-2026-10.md
Changelog¶
- 2026-10-21: Proposed; accepted by the owner the same day. The eval runner lives in
scripts/evolve.mjs(subcommandssignals,check-paths,eval,gate). - 2026-10-21: D6 (compress) added at the owner's request: Evolve may also remove duplicate, dead or stale text, with protected lines and a size + eval gate.
- 2026-10-21: headless workflow
agent-evolve.ymlremoved at the owner's request (D4);/evolveis local only. Themanifestcommand went with it. - 2026-10-22: signal
bug:#<N>: a closedbugissue (in-app report, ADR-0026) whose fix changed lines an agent run wrote (git blameon the fix's parent). Autopilot fixes are titledAutopilot: …, notfix:, so these were not counted before. A fix that only adds lines is not linked. - 2026-10-22:
bug:#<N>also links to a commit co-authored by Claude in a normal session when no autopilot run wrote the lines (origin: session, span = that commit). About 72% of the month's commits are Claude-co-authored but under 10% belong to autopilot runs, and all 5 bug reports so far came from session code. - 2026-10-22: headless workflow back with staged proposals outside
.claude/and anapplycommand (ADR-0029).