Leaderboard · Main results
Does self-modification make the Agent better?
For every model–Agent pair, each completed domain cell reports the final score and RSI lift achieved by the inherited Agent over its original checkpoint.
Leaderboard · Terminal + Pi ablations
Where does recursive improvement come from?
Each cell is a separate five-round experiment. Verified submissions report the final held-out score in black and the matched lift in parentheses.
Editable surface
Prompt, source subsystem, and conditional-resource treatments isolate which intervention surface gives the Meta-agent useful leverage.
| Model | Prompt | Agent loop | Tool runtime | Observation | Context | Compaction | Skills + loader | Hooks |
|---|
Cognitive controller
Findings, self-selected validation targets, causal review, and their complete composition are evaluated as independent controller treatments.
| Model | Findings only | Targets only | Review only | Full causal |
|---|
rsibench-data registry · awaiting verified ablation submissions
Why RSIBench matters
Five insights organize the evidence below.
The leaderboard answers whether performance moves. The rest of the site asks what kind of capability moved, under which starting point, through which mechanism, and toward what larger form of adaptation.
Measure RSI beyond task competence.
A strong fixed agent and an agent that can improve its own harness are not the same capability. The controlled loop isolates the latter.
02 · SEPARATE HEADROOM → SUBSTRATESCompare where evolution starts.
Minimal Pi versus mature Claude Code and Codex separates initial engineering strength, adaptive capacity, and final performance.
03 · REVERSE ADAPTATION → EDITABLE SURFACELet the harness learn the model.
Six shared resources and twelve Pi source systems make the harness a portable compatibility and improvement layer.
04 · LOCATE CAUSE → ABLATIONSExplain the gain, not only the score.
Trajectories, diffs, checkpoint lineage, rollback attribution, safety, robustness, and transfer expose how improvement occurs and what it costs.
05 · ACCUMULATE EXPERIENCE → LINEAGEMove from competence toward self-evolution.
Exploration becomes executable structure; selection and inheritance make reflection consequential—without claiming open-ended evolution or ASI.
Method · Design principles
Every control rules out a different false claim of self-improvement.
The editable harness is a path-owned two-layer surface. Layer 1 exposes six shared resources to Claude Code, Codex, and Pi; Layer 2 exposes twelve rebuild-required source-code modules for Pi. Only modules with a real accepted diff are attributed.
| Module | What it does | Owned files / paths | Scope |
|---|---|---|---|
| LAYER 1 6 shared resources | |||
| Prompt | Defines standing role, objectives, constraints, and project context. | Claude: CLAUDE.md; Codex: AGENTS.md; Pi: .pi/APPEND_SYSTEM.md | All agents |
| Rules | Encodes persistent policies, strategies, and decision constraints. | .claude/rules/**; .codex/rules/**; .pi/rules/** | All agents |
| Skills | Packages reusable domain knowledge and operating procedures. | .claude/skills/**; .agents/skills/**; .pi/skills/** | All agents |
| Hooks | Runs checks, transformations, or recovery actions at lifecycle events. | .claude/hooks.json, .claude/hooks/**; .codex/hooks.json, .codex/hooks/**; .pi/extensions/rsibench/m07_hooks/**, .pi/hooks/** | All agents |
| MCP | Adds declared external capabilities through servers and adapters. | .mcp.json, mcp/**; .codex/mcp.toml, mcp/**; .pi/extensions/rsibench/m10_mcp_adapters/**, .pi/mcp/** | All agents |
| Workflow | Orchestrates subagents, commands, and repeatable multi-step sequences. | .claude/agents/**, .claude/commands/**; .codex/agents/**, .codex/prompts/**; .pi/prompts/**, .pi/agents/** | All agents |
| LAYER 2 12 source code | |||
| Agent Loop | Controls rounds, tool-call cycles, stopping, cancellation, and retries. | packages/agent/src/agent-loop.ts, agent.ts, stream-fn.ts | Pi · rebuild |
| Observation Processing | Preserves tool results, errors, shell output, truncation, and observation fidelity. | packages/agent/src/harness/messages.ts, utils/shell-output.ts, truncate.ts; packages/coding-agent/src/core/messages.ts, tools/output-accumulator.ts, truncate.ts | Pi · rebuild |
| Context Construction | Builds provider-call context and preserves correct message handoff. | packages/coding-agent/src/core/model-runtime.ts, provider-composer.ts | Pi · rebuild |
| Tool Runtime | Owns tool schemas, execution, timeouts, mutation serialization, and low-level behavior. | packages/agent/src/harness/tools/**; packages/coding-agent/src/core/tools/**, bash-executor.ts, exec.ts | Pi · rebuild |
| Coding Session Runtime | Coordinates session events, recovery, runtime state, and the SDK boundary. | packages/coding-agent/src/core/agent-session.ts, agent-session-*.ts, sdk.ts | Pi · rebuild |
| Compaction | Controls context-overflow triggers, summaries, branch recovery, and overflow handling. | packages/agent/src/harness/compaction/**; packages/coding-agent/src/core/compaction/** | Pi · rebuild |
| Session Storage | Owns durable state, working-directory tracking, restoration, and recovery. | packages/agent/src/harness/session/**; packages/coding-agent/src/core/session-manager.ts, session-cwd.ts | Pi · rebuild |
| Agent Harness Runtime | Connects harness assembly, environment adaptation, shared types, and lifecycle entry points. | packages/agent/src/harness/agent-harness.ts, env/**, types.ts, index.ts, node.ts, proxy.ts | Pi · rebuild |
| Prompt Loader | Discovers, escapes, orders, and assembles prompt resources before execution. | packages/agent/src/harness/system-prompt.ts, prompt-templates.ts; packages/coding-agent/src/core/system-prompt.ts, prompt-templates.ts; cli/initial-message.ts | Pi · rebuild |
| Skill & Resource Loader | Discovers, filters, prioritizes, parses, and loads skills and resources. | packages/agent/src/harness/skills.ts; packages/coding-agent/src/core/skills.ts, resource-loader.ts | Pi · rebuild |
| Extension & Event Runtime | Owns extension loading and execution, event bus, and internal dispatch. | packages/coding-agent/src/core/extensions/**, event-bus.ts; packages/coding-agent/src/extensions/** | Pi · rebuild |
| Provider Request & Stream Runtime | Maps request and reasoning parameters and parses streams, tool calls, usage, finish reasons, and provider errors. | packages/ai/src/api/openai-completions.ts, openai-responses.ts, anthropic-messages.ts, simple-options.ts, transform-messages.ts | Pi · rebuild |
What we measure
168 Tasks Across Six Categories
Every category tests a materially different way an agent must work: operate a terminal, learn an unfamiliar environment, repair code, conduct scientific analysis, complete expert digital work, or reconstruct hidden behavior. Each domain keeps its own held-out score—there is no aggregate total to hide a narrow failure.
Terminal
30 tasksLong-horizon system administration, debugging, compilation, data processing, security, and tool-use problems inside a real terminal.
Example taskquery-optimize · rewrite a slow SQL query without changing its outputEnvironment Learning
30 tasksOpen-ended optimization and adaptation tasks where the agent must infer how a new environment behaves from interaction and feedback.
Example taskBorden Source Inversion · infer a 3-D contaminant source from monitoring wellsSWE
30 tasksPurpose-built, long-horizon software-engineering tasks that require understanding and repairing an entire generated repository.
Example taskRepository repair · diagnose a failing generated codebase and deliver a verified patchScience
24 tasksComputational research tasks that combine scientific reasoning, analysis code, reproducibility, and quantitative verification.
Example taskRaman spectrum fitting · recover physically valid peaks from measured spectraFrontier
24 tasksDiverse expert workflows spanning CAD, document processing, layout recreation, music, medicine, and other frontier computer-use tasks.
Example taskcad-model · construct and verify a constrained 3-D partRepo Reconstruction
30 tasksClean-room software reconstruction from executable behavior and documentation: infer the hidden contract, implement it, and survive official tests.
Example taskBinary behavior reconstruction · rebuild a hidden Rust/Go/C/C++ program from observations
Experiments · Evidence after the main table
Mechanism, scaling, transfer, and efficiency.
The experiments follow the paper: first locate the source of harness improvement, then measure scaling, test cross-domain transfer, and finally study the performance–cost operating point.
How does RSI scale?
We separate within-run improvement from model intelligence: one analysis tracks harness lift over recursive rounds; the other compares model-level RSI capability across model generations.
Must establish: whether newer or stronger frozen models learn better harnesses faster, whether the relationship recurs across domains, and where additional RSI rounds stop producing reliable marginal lift.
Where does harness improvement come from?
Start from the selected harness, restore one changed physical module to H0, retain every other selected edit, and rerun the matched evaluation.
| # | Condition | Counterfactual intervention | Causal held-out drop | Test | Drop |
|---|---|---|---|---|---|
| 00 | Full Hk* | selected harness · reference | — | — | |
| L1 | Shared resources | Prompt · Rules · Skills · Hooks · MCP · Workflow | — | — | |
| L2 | Pi core source | 12 path-owned runtime subsystems · see paper appendix | — | — |
Must establish: which surface causally carries the gain, and whether any less-obvious module dominates ordinary instruction changes. First run an integrity and dependency check: if an edit in one module requires another module, the single-module effect is not identifiable; restore the coupled modules together and report the result as an interaction, never as a naive individual contribution.
Can a harness optimized in one domain transfer to another?
Evolve Hk* on one source domain, freeze it, and evaluate the same checkpoint on every target domain without further updates or reselection.
| Source \ Target | Term. | Env. | C-SWE | Sci. | Front. | Repo |
|---|---|---|---|---|---|---|
| Terminal | +.15 | +.06 | +.05 | +.03 | +.04 | +.07 |
| Environment | +.05 | +.13 | +.03 | +.07 | +.06 | +.04 |
| SWE | +.07 | +.03 | +.17 | +.02 | +.04 | +.08 |
| Science | +.03 | +.07 | +.02 | +.16 | +.08 | +.03 |
| Frontier | +.04 | +.06 | +.03 | +.07 | +.18 | +.04 |
| Repo Reconstruction | +.07 | +.04 | +.08 | +.03 | +.04 | +.16 |
Must establish: whether accepted changes encode reusable agent mechanisms or narrow source-benchmark tricks.
How does the validation objective select a Pareto-efficient harness?
First measure the system-level movement from H0 to Hk*. Then sweep the cost coefficient in Jλ(H)=Acc(H)−λCost(H) while holding the model, tasks, horizon, and rollout budget fixed.
| # | Objective | Operating point | Accuracy–cost frontier | Acc. | Cost | Pareto |
|---|---|---|---|---|---|---|
| 01 | λ = 0 | accuracy-only endpoint | — | — | — | |
| 02+ | λ > 0 sweep | cost-aware selected endpoints | — | — | — |
Must establish: whether harness adaptation improves pass rate at comparable cost, and how the validation coefficient selects among non-dominated accuracy–cost operating points.
Scroll to inspect each edit
One scroll. One concrete harness upgrade.
Every row now states both the installed mechanism and its intended effect. Guidance adds artifact-first literal checks; State retains a compact failure ledger; Control installs verify–retry–stop gates; Action adds tools, APIs, and MCP adapters. The orange path carries evidence into the next validation-gated revision. Guidance is observed; later rows remain explicitly schematic.
Start with no extracted improvement modules.
The foundation model and Pi runtime are fixed. The editable harness boundary exposes four empty sockets—Guidance, State, Control, and Action—but none of the trajectory-derived upgrades has been installed yet.
Require the literal deliverable before declaring success.
Across eight training trajectories, the Agent found a missing artifact, a relaxed structural proxy, and the wrong test scenario. It edited only workspace/AGENTS.md: deliver the requested artifact first, check its exact path and structure, then test the literal edge case instead of a convenient proxy.
Keep a compact ledger of failures, decisions, and evidence.
The proposed State module writes a structured failure ledger, retains the best artifact and verifier evidence, and compacts stale dialogue into a short task summary. The next round receives the evidence needed to avoid repeating failed approaches without carrying the full trajectory.
Gate inheritance with explicit verify, retry, and stop rules.
The proposed Control module calls the verifier after material edits, retries with a different plan after a diagnosed failure, stops loops that add no evidence, and restores the best-known artifact before validation. A candidate cannot become inherited state merely because it looks plausible.
Make new operations callable.
The proposed Action module registers task-specific tools, typed API wrappers, and MCP adapters; validates arguments before dispatch; and normalizes tool results back into the Agent context. The harness can then execute operations that prompting alone cannot perform.
Citation
Cite This Work
@misc{rsibench-2026,
title = {RSIBench},
author = {{RSIBench Team}},
year = {2026},
url = {https://harness-rsibench.com/}
}
If you use RSIBench in your research, please cite the BibTeX entry above. Paper metadata will be updated when the manuscript is released.