RSIBench

RSIBench documentation

Measure whether an Agent can improve its own code.

RSIBench evaluates recursive harness self-improvement with frozen model weights. The coding Agent executes benchmark tasks, reads train trajectories, edits the code that governs future runs, and inherits only candidates accepted by the registered train-side gate.

Core boundary. RSIBench changes the Agent harness, not the foundation model. Model weights, tasks, verifiers, sandbox policy, and budgets remain fixed within a comparison.

Evaluation loop

One RSI run begins from an initial Agent state and ends with a final inherited Agent. No component may inspect test trajectories during evolution.

1 · Roll outRun every frozen training task three times and archive tool calls, outputs, and verifier consequences.
2 · DiagnoseWrite per-task counterfactual findings from mixed and failed train trajectories.
3 · EditModify registered shared-resource or Pi source modules while model weights stay frozen.
4 · SelectBuild and smoke the candidate, run its declared targets plus cumulative anchors at pass@3, then require trajectory post-review.
5 · ReportAfter five rounds, score A0 and A_last on the untouched test split at avg@3.

An accepted candidate becomes the starting state of the next RSI round. A rejected candidate is archived and the previous Agent state is retained.

Data isolation

Train · 84 tasks

The Agent may inspect train trajectories from six frozen domain splits. Candidate acceptance uses only predeclared train targets and regression anchors.

Test · 84 tasks

Test trajectories are never mounted for the Meta-agent and test scores never select a candidate. After evolution ends, A0 and A_last are each evaluated three times.

RSIBench has no validation split. Causal target/anchor checks and post-review make train-side selection explicit while preserving test for final reporting.

Metrics

Initial performance

A0 is the avg@3 test score of the original, unmodified Agent.

Final performance

A_last is the avg@3 test score of the final inherited Agent after five optimization rounds.

Held-out RSI lift

ΔRSI = test(A_last) − test(A0). Positive lift means persistent harness changes generalized beyond train-side evidence and selection.

Each of the six benchmark domains is reported independently; incompatible scales are not collapsed into an “Overall” score.

Attribution

RSIBench records six cross-agent shared modules—Prompt, Rules, Skills, Hooks, MCP, and Workflow—and twelve physical Pi source modules. Git ownership, rebuilds, rollback, and keep-one ablation all use these registered physical boundaries.

Every domain requires A0 and A_last avg@3. Terminal additionally requires every accepted-generation test score and changed-module keep-one ablation; those enrichments are optional for other domains.

Submissions

The public submission workflow accepts a frozen model identifier, an initial Agent implementation, the final inherited checkpoint, complete training trajectories, code diffs for every candidate, and cost/token accounting.

  • Use the official task snapshots and verifiers.
  • Do not expose sealed test content to the Agent.
  • Submit A0 and A_last test scores and trajectories at avg@3.
  • For Terminal, also submit every accepted-generation score and changed-module keep-one ablation at avg@3.
  • Report results separately for every benchmark category.
  • Preserve accepted and rejected code changes for trajectory-level audit.