` - each with a Hide/Show toggle. Anything else (`file`: docx, pptx, ...) shows as a download chip. Paths under ~2 MB are inlined into `report.html`; larger ones stay as download links so the report doesn't balloon. **One slot per artifact**: when a turn has `attachments`, the viewer suppresses its inline-render toggle for fenced blocks in that turn's text - so put the output in `attachments` once and the response text stays plain source. The lite report renders none of this - it links each trace file, and the analyzer and the user open the `ref`'d files directly - so keep every `ref` relative to the flow root either way. If a flow needs something the prebuilt viewer doesn't cover, edit `report/frontend/atoms.jsx` (`ArtifactView`) and rebuild via `frontend/scripts/build.sh` (EAP install only - the CLI doesn't extract the frontend).

完整查看器内联渲染 `image`/`svg`，`html` 放入沙箱化的可滚动 iframe，`pdf` 用浏览器原生查看器，`json`/`text` 放入 `
`——每个都带隐藏/显示切换。其余（`file`：docx、pptx 等）显示为下载芯片。约 2 MB 以下的路径内联进 `report.html`；更大的保持为下载链接，以免报告膨胀。**每个产物一个槽位**：当某个轮次带 `attachments` 时，查看器会取消该轮次文本中围栏代码块的内联渲染开关——所以把输出放一次到 `attachments`，响应文本保持纯源码。精简报告不渲染这些——它链接每个轨迹文件，由分析器和用户直接打开被 `ref` 引用的文件——因此无论哪种方式都要让每个 `ref` 相对流程根目录。如果流程需要预置查看器未覆盖的功能，编辑 `report/frontend/atoms.jsx`（`ArtifactView`）并通过 `frontend/scripts/build.sh` 重建（仅限 EAP 安装——CLI 不解包前端）。

**Split the prompt set so the analyzer can't overfit to the number you report.** The analyzer reads transcripts to propose changes - it will, by design, fix the specific cases it sees. The score you report must come from cases it never read. How you carve that depends on how many cases you have; pick the lightest structure that gives a held-out number you can trust:

**切分提示词集合，使分析器无法对你报告的数字过拟合。**分析器阅读对话记录来提出改动——按设计，它会修复它看到的特定用例。你报告的分数必须来自它从未读过的用例。如何切分取决于你有多少用例；选择能给出可信保留数字的最轻量结构：

- **Default - train / test.** *Train* is the set whose transcripts the analyzer reads each round. *Test* is everything else: scored every round alongside train, never opened by the analyzer; its score picks the winning round and is the headline. **Draw the split at random, stratified by `tags[0]` - never by baseline score.** Train needs enough failures to show a pattern (a handful to a couple dozen), but get them by making train big enough, not by hand-picking the worst cases: a train slice selected for low scores means the analyzer only ever sees pathological cases and tunes for the tail, and those cases regress toward the mean on re-run anyway - a healthy train gain with a flat test set is the signature. After the baseline, check that train and test means agree within noise; if they don't, re-draw before round 1. The analyzer can still *focus* on the failures within train.
  **默认——训练 / 测试。***Train* 是分析器每轮阅读其对话记录的集合。*Test* 是其余全部：每轮与训练集一起评分，但从不被分析器打开；其分数选出获胜轮并构成头条。**随机抽取切分，按 `tags[0]` 分层——绝不按基线分数。**训练集需要足够多的失败来呈现模式（几个到两三个），但要通过把训练集做得足够大来获得，而不是手工挑选最差的用例：专挑低分用例的训练切片意味着分析器只会看到病态用例并为尾部调优，而这些用例重跑时本就会向均值回归——训练集健康提升而测试集持平正是其特征。基线之后，检查训练与测试均值是否在噪声范围内一致；如果不一致，在第 1 轮之前重新抽取。分析器仍可以*聚焦*训练集内的失败用例。

- **Small set, or a cross-case metric** (pairwise ordering, ranking - anything that fragments inside a slice). Don't split. Score the whole set every round, lean on **reps** to tighten the noise, and have the analyzer read a few targeted failure transcripts rather than a fixed slice. Label per-round scores in the report as **directional**: an improvement that holds across reps is real signal, but with no held-out set the headline is an iterate-on number, not a publish number.
  **小集合，或跨用例指标**（成对排序、排名——任何在切片内部会碎片化的东西）。不要切分。每轮对整个集合评分，依靠**重复次数**收紧噪声，让分析器阅读若干有针对性的失败对话记录而非固定切片。在报告中把逐轮分数标注为**方向性**：跨重复保持的改进是真实信号，但在没有保留集的情况下，头条是一个用于迭代的数字，不是可发布的数字。

- **Large set** (~150+ cases) where you want a final number untouched by round selection: optionally carve a third *validation* slice - scored each round to pick the winner - and hold *test* back until the end. For most evals the extra bookkeeping isn't worth it.
  **大集合**（约 150+ 用例）且你想要一个不受轮次选择影响的最终数字：可选地再切出第三个*验证*切片——每轮评分用于挑选获胜者——并把*测试*留到最后。对大多数评测来说，额外的簿记不值得。

Be honest with the user about what their split can and can't tell them. For a binary pass-rate the 95% CI half-width is roughly `1/sqrt(n·reps)` - 25 test cases at 2 reps is about ±14 points, 50 at 2 reps about ±10 - so show that number for their set and let them choose; reps and test-size are two knobs on the same dial. Whatever they pick, record the IDs in `_state.json` exactly as they appear in `baseline/results.jsonl`'s `prompt_id` (a runner that sanitizes ids for file paths keys its rows on the sanitized form), fix the split once, and don't change it.

就切分能告诉他们什么、不能告诉什么，对用户保持诚实。对二元通过率，95% 置信区间半宽约为 `1/sqrt(n·reps)`——25 个测试用例 2 次重复约 ±14 分，50 个用例 2 次重复约 ±10——所以把这个数字展示给他们的集合让他们选择；重复数和测试集大小是同一个旋钮上的两个刻度。无论他们选什么，把 ID 按其在 `baseline/results.jsonl` 的 `prompt_id` 中的原样记录到 `_state.json`（会把 id 清洗成文件路径的运行器以清洗后的形式作为行的键），一次性固定切分，并且不再更改。

**Reps per prompt** is the other knob. Model outputs vary run-to-run, and repeating each prompt tightens the estimate - especially worth it on a small set, or whenever the Step 2 budget has room. Offer it and let the user pick the count; if they choose more than one, record results as "k of N reps passed" rather than a single pass/fail. Don't default this silently in either direction. Reps tighten only the noise they re-sample - a runner that reuses a once-built generated artifact across reps leaves its build variance untouched (see the Step 0.5 rebuild check).

**每个提示词的重复次数**是另一个旋钮。模型输出逐次运行各不相同，重复每个提示词能收紧估计——在小集合上或步骤 2 预算有富余时尤其值得。提供该选项并让用户选数量；如果他们选择多于一次，把结果记录为"N 次重复中通过 k 次"而不是单一的通过/失败。不要在任何方向上悄悄设默认值。重复只收紧它们重新采样的噪声——跨重复复用一次性构建产物的运行器不会触及构建方差（见步骤 0.5 的重建检查）。

**Isolate ground truth from the model under test.** If the eval has reference answers, rubrics, or expected outputs, make sure they are **not reachable from the model's context** - not in its system prompt, not in a tool result it can read, not in a file its code-execution or bash tool can open, not in a fixture its container mounts. Keep them in the grader only. This has to be structural; "don't look at the answers" in a prompt is not a defense, and an agent under optimization pressure will eventually find a `cat evals.json` that wins the game without playing it. If the user's existing eval stores prompts and answers in the same file, split it before the first round. And if the eval derives from a public benchmark and the agent has web or network tools, the answers are also reachable *through the internet* - public mirrors of the benchmark often include solutions. Screen every passing transcript for fetches of the benchmark's repo or solution mirrors (and consider blocking egress to them); treat a pass accompanied by a solution-hosting fetch as invalid, and don't assume any automated contamination check will catch it.

**把标准答案与被测模型隔离开。**如果评测有参考答案、评分规则或期望输出，确保它们**无法从模型上下文到达**——不在其系统提示词中，不在其可读取的工具结果中，不在其代码执行或 bash 工具能打开的文件中，不在其容器挂载的夹具中。把它们只保存在评分器里。这必须是结构性的；提示词里写"不要看答案"不是防线，处于优化压力下的代理最终会找到一条 `cat evals.json` 的路径，不玩游戏就赢下游戏。如果用户现有的评测把提示词和答案存在同一个文件里，在第 1 轮之前拆开。如果评测源自公开基准且代理有网页或网络工具，答案也能*通过互联网*到达——基准的公开镜像常常包含解答。筛查每个通过的对话记录中对基准仓库或解答镜像的抓取（并考虑阻断到它们的外联）；把伴随解答镜像抓取的通过视为无效，并且不要指望任何自动化污染检查会抓住它。

【评论】该段强调用结构性隔离而非提示词约束来保护标准答案，并覆盖公开基准经互联网泄漏的途径——针对的是优化压力下代理"绕过任务、直接读取答案"的作弊路径。

**Handling non-text payloads in traces.** Don't embed raw base64 or binary blobs in the trace text - they bloat the file and render as noise. Instead, write each image or binary payload to a sidecar file and put a markdown reference at the point in the turn where it appeared - `![]()` for an image, a link for anything else - so the report shows a thumbnail and the analyzer can still open the real file. Only fall back to a bare placeholder like `<[binary: 12kB]>` when the payload genuinely isn't worth keeping.

**处理轨迹中的非文本负载。**不要在轨迹文本中嵌入原始 base64 或二进制块——它们会让文件膨胀并渲染成噪声。改为把每个图像或二进制负载写到侧车文件，并在轮次中它出现的位置放一个 markdown 引用——图像用 `![]()`，其他用链接——这样报告显示缩略图，分析器仍能打开真实文件。只有当负载确实不值得保留时，才退回使用 `<[binary: 12kB]>` 这类裸占位符。

**Baseline.** Run the unmodified app across **the full set** at the chosen rep count, and record it in `baseline/` and in `_state.json`. Record an environment fingerprint with it - repo commit, lockfile hash, local patches or stubs, disabled tools, model id - and re-verify the fingerprint before each round's run: a changed fingerprint means re-baseline, because the comparison is broken either way. If time or an environment change later separates the baseline from a round, score a no-change control alongside the candidate and gate against the control, not the stale baseline number - identical-code drift of a few points between measurement days is common; where a stochastic build sits between the lever and the score, the control must re-run the build, not just the scorer. The control re-run also estimates the same-environment flip rate for free; don't set a keep gate inside that noise floor (Step 2's threshold sizing). (If cost matters and the pinned reps/split push the total above the Step 2 ceiling, mention it before this run.) You can run the report builder (full or lite - Step 4 has the invocation) on the flow directory now if you want a look: with only a single variant on disk (just `baseline/`, or no `_state.json` yet) the full viewer shows only the Evals and Transcripts tabs - the Summary tab auto-hides until there's a second variant to compare.

**基线。**让未修改的应用在选定重复次数下跑**全集**，记录到 `baseline/` 和 `_state.json`。同时记录环境指纹——仓库提交、lockfile 哈希、本地补丁或存根、禁用的工具、模型 id——并在每轮运行前重新核验指纹：指纹变了就意味着重新采集基线，因为无论哪种情况比较都已失效。如果时间或环境变化后来把基线与某一轮隔开，就在候选旁边加评一个无改动对照，并以对照而非过期的基线数字做门控——相同代码在不同测量日之间漂移几分很常见；当随机构建位于杠杆与分数之间时，对照必须重跑构建，而不只是评分器。对照重跑还顺带估计了同环境翻转率；不要在那个噪声底之内设置保留门（步骤 2 的阈值定标）。（如果成本重要且固定的重复数/切分使总额超出步骤 2 上限，在本次运行前先说明。）如果你想先看看，现在就可以在流程目录上运行报告构建器（完整版或精简版——调用方式见步骤 4）：磁盘上只有一个变体时（只有 `baseline/`，或还没有 `_state.json`），完整查看器只显示 Evals 和 Transcripts 两个标签页——Summary 标签页会自动隐藏，直到有第二个变体可比。

---

## Step 4: The loop / 步骤 4：循环

Each round is: **analyze -> apply -> run -> record**. Two rules are non-negotiable throughout: only the train split's transcripts are ever read, and neither the eval set nor the budget changes without going back to the user.

每一轮是：**分析 -> 应用 -> 运行 -> 记录**。两条规则全程不可协商：只读训练切分的对话记录，且不回头找用户就绝不改动评测集合或预算。

**Analyze.** If the next change is already clear - the user named a specific fix, or the last round's result points straight at one - skip the analyzer and write `vN/change.md` directly. Otherwise spawn **one fresh analyzer subagent** (the Claude Code Task tool) and give it: the previous round's **train-split trace files** - the runner writes a trace for every case, so don't hand over the `traces/` directory; list the exact `vN/traces/_rep.json` paths whose `` is in `_state.json.train_ids` (or copy them to a scratch directory) and tell the analyzer not to read anything else under `traces/` - the **train rows of `results.jsonl`** so it sees every metric's per-case score, the **train rows of `trajectory/scores.tsv`** so it sees how each case has moved across every prior round, the current version of the artifact being iterated on, the scope and off-limits list from Step 1, and **which metric this round is targeting** - on round 1 that's the Step 1 goal, along with the guardrail metrics it must hold. The analyzer scans the scores to pick which transcripts to read, reads however it likes - sort by the target metric, diff low vs high scorers, correlate across metrics, spot cases stuck flat across rounds - and returns a proposed change with a short rationale that cites the specific traces motivating it. Save that rationale to `vN/change.md`. A fresh subagent each round keeps the outer session from accumulating transcript content in its own context.

**分析。**如果下一个改动已经明确——用户点名了具体修复，或上一轮的结果直指某个改动——跳过分析器，直接写 `vN/change.md`。否则派生**一个全新的分析器子代理**（Claude Code Task 工具）并给它：上一轮的**训练切分轨迹文件**——运行器为每个用例都写轨迹，所以不要直接交出 `traces/` 目录；列出 `` 在 `_state.json.train_ids` 中的那些确切路径 `vN/traces/_rep.json`（或把它们复制到临时目录），并告诉分析器不要读 `traces/` 下的其他任何内容——`results.jsonl` 的**训练行**，让它看到每个指标的逐用例分数；`trajectory/scores.tsv` 的**训练行**，让它看到每个用例在之前每一轮如何变动；被迭代产物的当前版本；步骤 1 的范围与禁区清单；以及**本轮针对哪个指标**——第 1 轮即步骤 1 的目标，连同其必须保持的护栏指标。分析器扫描分数以挑选要读的对话记录，阅读方式随它——按目标指标排序、对比高低分者、跨指标做相关、找出多轮持平的用例——然后返回一项提议改动及引用了佐证轨迹的简短理由。把该理由保存到 `vN/change.md`。每轮全新的子代理避免外层会话把对话记录内容积累进自己的上下文。

The outer session **does not read transcripts itself** - for choosing changes it sees scores only. That separation is the data-isolation guarantee: nothing from the held-out test set can leak into a proposed change, because the thing proposing changes never sees anything but train.

外层会话**自己不读对话记录**——在选择改动时它只看分数。这种分离就是数据隔离保证：保留测试集的任何内容都不可能泄入提议的改动，因为提出改动的东西除训练集外什么也看不到。

【评论】训练/测试隔离借鉴自机器学习实践，这里把它扩展到提出改动的模型自身的阅读行为：只让分析器读训练轨迹，从流程上杜绝保留集信息回流进改动提议。

Three steers to pass the analyzer. First, **generalize, don't memorize**: the change should describe the failure *behavior*, not the failure *content* - pasting specific nouns or phrases from train cases into the prompt is the fastest route to an overfit change that helps train and does nothing held-out. Second, transcripts show what a block makes the model *do*, not what it prevents or enables without visible action - when proposing to remove something, name which cases you expect to regress, not just which recover. Third, it's fine - especially early on - for the analyzer to propose trying a different lever entirely (a tool description, an API parameter) rather than another wording of the same sentence; exploring where the headroom is can be worth a round.

给分析器三条引导。第一，**泛化，不要背题**：改动应描述失败的*行为*，而非失败的*内容*——把训练用例中的具体名词或短语粘贴进提示词是制造过拟合改动的最快路径，它帮训练集、对保留集毫无作用。第二，对话记录展示的是一段文字让模型*做什么*，而不是它在没有可见动作时阻止或启用了什么——提议删除某内容时，要指出你预期哪些用例会回退，而不只是哪些会恢复。第三，分析器提议彻底换一个杠杆（工具描述、API 参数）而不是同一句话的另一种措辞，这完全没问题——尤其是在早期；探明提升空间在哪里值得一轮。

**Spend a round only on a change the eval can see**, whether the analyzer proposes it or you write it directly. (A fix the user asked for by name still gets its round; say so in that round's status message if you expect it to land inside the noise floor.) A change whose effect is smaller than the Step 0.5 noise floor is kept or reverted largely by chance. So fix the targeted behavior at its root (rewrite the section that causes it, add the missing rule or capability) rather than rewording a line; effect is the measure, not diff length - one missing fact, a new tool or a different `effort` setting can be the whole fix. A fix can gain at most what the cases showing the behavior now lose: if no single behavior loses enough to clear the floor, treat it as a stall instead of inflating the change - run Step 4.5's categorization now, and in that round's status message offer more reps or cases, which is what lowers the floor. A root fix is still one change - one hypothesis about why cases fail, in one patch, kept or reverted whole, however many places it touches - not unrelated fixes bundled together (only Step 4.5's breadth pass does that, for gaps each too small to measure alone).

**只为评测看得见的改动花一轮**，无论它是分析器提出的还是你直接写的。（用户点名的修复同样有资格占一轮；如果你预计它会落在噪声底之内，在该轮状态消息里说明。）效应小于步骤 0.5 噪声底的改动，保留或回退基本靠运气。所以要从根源修复目标行为（重写导致它的那一节，补上缺失的规则或能力），而不是改写一行的措辞；效应才是尺度，不是 diff 长度——一个缺失的事实、一个新工具或不同的 `effort` 设置就可能构成整个修复。一项修复最多能赢得当前展现该行为的用例所损失的总和：如果没有任何单一行为的损失足以越过噪声底，就把它当作停滞处理而不是夸大改动——现在就执行步骤 4.5 的归类，并在该轮状态消息中提供增加重复或用例的选项，那才是降低噪声底的办法。根源修复仍是一项改动——关于用例为何失败的一个假设、一个补丁、整体保留或整体回退，无论它触及多少处——而不是把不相关的修复捆绑在一起（只有步骤 4.5 的广度遍历才这样做，针对每个都小到无法单独度量的缺口）。

A sketch of the analyzer's prompt:

分析器提示词的草稿：

> Here is the current `` we are iterating on. Below are the train-split rows from `results.jsonl` (each case's full `grade` dict) and the corresponding transcripts. **This round's target is `` (`` is better); `` must not regress.** The off-limits list is: `<...>`. Find the cases doing worst on ``, plus a couple doing best for contrast; name the single behavior that most often costs it, and propose **one** concrete change to the artifact as a unified diff; it may touch several places, but every hunk must serve that one behavior. Say how many cases show the behavior and how much `` it costs across them - a fix can gain no more than that. The noise floor on `` is about `<±X>`: if a full fix would still land inside it, say so instead of proposing a change; otherwise fix the behavior at its root (rewrite the section that causes it, add the missing rule or capability) rather than rewording a line. For every trace you cite as evidence, **quote the relevant lines verbatim** and append its trace file path (`/traces/_rep.json`) - and, if `report.html` was built by the full viewer, the deep link `report.html#tab=transcript&ex=&cmp=&rrep=` - so the user can open that exact rep with one click. Do not reference test cases.

> 这是我们正在迭代的当前 ``。下面是 `results.jsonl` 的训练切分行（每个用例的完整 `grade` 字典）及对应的对话记录。**本轮目标是 ``（以 `` 为更优方向）；`` 不得回退。**禁区清单为：`<...>`。找出 `` 上表现最差的用例，外加几个表现最好的作对照；指出最常导致失分的单一行为，并以统一 diff 的形式对产物提出**一项**具体改动；它可以触及多处，但每个 hunk 都必须服务于这一个行为。说明有多少用例展现该行为、它在这些用例上共损失多少 ``——修复最多只能赢回这么多。`` 的噪声底约为 `<±X>`：如果完整修复仍会落在噪声底之内，就直接说明而不是提议改动；否则从根源修复该行为（重写导致它的那一节，补上缺失的规则或能力），而不是改写一行措辞。你引用为证据的每条轨迹都要**逐字引用相关行**，并附上其轨迹文件路径（`/traces/_rep.json`）——如果 `report.html` 由完整查看器构建，还要附上深链 `report.html#tab=transcript&ex=&cmp=&rrep=`——让用户一键打开确切的那个重复。不要引用测试用例。

When reading `trajectory/scores.tsv` to pick focus cases, filter to **train rows only** - feeding test-row movement back into the change proposal is a leak of the held-out signal.

阅读 `trajectory/scores.tsv` 挑选聚焦用例时，只筛**训练行**——把测试行的变动反馈进改动提议就是对保留信号的泄漏。

**Apply.** (If `approve_each_round` is set, show the diff and one-line rationale and wait for a yes before running; a no counts as a reverted round with the user's reason recorded in `change.md`.) Before applying, do a **de-fluff pass** on the proposed change: cut anything that's a platitude, a restatement of default behavior, or advice with no operational content ("be careful," "think step by step"). Do this every round - fluff accumulates one reasonable-sounding sentence at a time. Then edit the actual files in the user's codebase and save the diff to `vN/change.patch` so it can be reverted cleanly (for a brand-new artifact with no prior version, diff against `/dev/null`). That patch is the round's record of what changed (and what the full viewer's diff drawer renders - click a variant row in Summary), so cut it against the user's real source paths - not a scratch copy under `.claude/hillclimb/` - so each hunk reads as an edit they can apply directly to their repo; if the loop is iterating on a temporary copy, diff the original file instead. Also snapshot the full post-change artifact - the system prompt, skill file, or whatever you're iterating on - to `vN/` (e.g., `vN/skill.md`) so each round is inspectable on its own without replaying patches. If the patch touches a path in `_state.json.harness_paths`, the next run will stop for `--approve-harness`: show the user the diff and wait for their OK rather than running the round unattended.

**应用。**（若设置了 `approve_each_round`，展示 diff 和一行理由，等待同意后再运行；拒绝则计为一轮回退，用户原因记录在 `change.md`。）应用之前，对提议的改动做一次**去水分处理**：删掉任何属于陈词滥调、重述默认行为或没有可操作内容的建议（"小心一点"、"一步步思考"）。每轮都做——水分会以一次一句貌似合理的话的速度累积。然后编辑用户代码库中的真实文件，并把 diff 保存到 `vN/change.patch` 以便能干净回退（对没有先前版本的全新产物，对 `/dev/null` 做 diff）。该补丁是本轮的变更记录（也是完整查看器 diff 抽屉渲染的内容——在 Summary 中点击某个变体行即可看到），所以要针对用户的真实源码路径生成——而不是 `.claude/hillclimb/` 下的临时副本——让每个 hunk 都像是能直接应用到其仓库的编辑；如果循环迭代的是临时副本，就改为 diff 原文件。同时把改动后的完整产物快照——系统提示词、技能文件或任何你正在迭代的东西——保存到 `vN/`（例如 `vN/skill.md`），让每一轮无需重放补丁即可单独检视。如果补丁触及 `_state.json.harness_paths` 中的路径，下一次运行会因 `--approve-harness` 而停止：向用户展示 diff 并等待其确认，而不是无人值守地跑完这一轮。

**Run.** Run the full eval - every case, at the chosen rep count. Running the entire set every round keeps every score in the history directly comparable. Keep the runner's concurrency maxed out (up to the rate limit) - the loop's cadence is gated on how fast each pass finishes - provided retries back off with jitter (Step 0.5); in a shared-quota environment, maxed-out concurrency with a hot retry loop converts someone else's burst into your zero-scores. Write per-case results to `vN/results.jsonl`, the aggregate to `vN/summary.json`, and every case's transcript to `vN/traces/_rep.json` (the train/test wall is enforced at hand-off - the Analyze step passes only the train files - not by withholding test traces from disk, which the report and Step 5's before/after pairs need). (`trajectory/scores.tsv` is regenerated by the report builder - full or lite - from those files; the runner doesn't write it.)

**运行。**跑完整评测——所有用例，按选定的重复次数。每轮跑整个集合让历史中的每个分数都直接可比。让运行器的并发保持打满（直至速率上限）——循环的节奏取决于每遍完成的速度——前提是重试带抖动退避（步骤 0.5）；在共享配额环境中，打满的并发加上高频重试循环会把别人的突发流量变成你的零分。把逐用例结果写入 `vN/results.jsonl`，汇总写入 `vN/summary.json`，每个用例的对话记录写入 `vN/traces/_rep.json`（训练/测试之墙在交接环节执行——分析步骤只传训练文件——而不是靠扣住测试轨迹不落盘，报告和步骤 5 的前后配对需要它们）。(`trajectory/scores.tsv` 由报告构建器——完整版或精简版——从这些文件重新生成；运行器不写它。)

Three gates before a round's full pass - each costs at most one case, against a pass that costs all of them:

一轮完整跑批之前的三道闸门——每道最多耗费一个用例，与之相对，一次失败的全量跑批会耗费全部用例：

- **Premise-probe config levers.** When the round's change is a config-surface lever (a model parameter, a tool config, `effort` - anything validated server-side at create or call time) rather than prompt content, run one case first and confirm the lever is accepted - and, where the response exposes it, echoed back. A loud validation error is the cheap outcome; the expensive one is a runner that degrades the config error into retries or scored zeros and runs the full set anyway.
  **对配置杠杆先做前提探针。**当本轮改动是配置层面的杠杆（模型参数、工具配置、`effort`——任何在创建或调用时由服务端校验的东西）而非提示词内容时，先跑一个用例确认该杠杆被接受——并且在响应暴露它的场合被回显。响亮的校验错误是便宜的结局；昂贵的是把配置错误降级成重试或零分、却照样跑完整集的运行器。

- **Print the resolved scope.** Before the pass, have the runner print what it actually resolved - case count × reps × model × estimated cost - and compare it to the approved plan. A dry run that resolves a different case count than the plan is a stop, not a warning.
  **打印解析后的范围。**跑批之前，让运行器打印它实际解析到的内容——用例数 × 重复数 × 模型 × 估计成本——并与已批准的计划比对。试运行解析出的用例数与计划不同就是停止信号，不是警告。

- **Canary before an unattended or expensive pass** - run one case and compare its error and latency profile to baseline's; if degraded, hold rather than burn the round.
  **无人值守或昂贵的跑批之前先放金丝雀**——跑一个用例，将其错误和延迟特征与基线比较；若已劣化，宁可暂停也不要烧掉这一轮。

Infra health per round - retries, timeouts, served-model mismatches - is in `errors.jsonl`; a round whose error profile differs grossly from baseline's is void-and-rerun, not a comparable data point.

每轮的基础设施健康状况——重试、超时、实际服务模型不符——记录在 `errors.jsonl` 中；错误特征与基线截然不同的一轮应作废重跑，而不是当作可比数据点。

**Record and report.** Update `_state.json` (round number, and `best` if this round's test score beats it). A number you'll report - including privately-authored held-out cases and their raw results - must land in the recorded `vN/` layout, never only in a run log. If the session or machine is ephemeral (a CI runner, a remote session), also copy each round's `vN/` to storage that survives it. Don't report a round's score until every case has landed - partial reads can show a sign that flips when the batch finishes; if you must report mid-run, flag it as `N/total`. After every round, write the status table (row layout below) at the top of `narrative.md` and keep it current - it is the per-round record, on disk even when the run is headless - then report in chat a one-line headline ("v3 test 0.71 -> 0.74, change: ``"). With the full viewer, follow the headline with a pointer to `report.html#tab=summary` instead of the table - the Summary tab is exactly this table, sortable and with the diff drawer one click away. With the lite report, which shows only the primary metric, paste the markdown table under the headline, since it is what carries the guardrail, perf, `$/run` and `spend` columns. `$/run` and `spend` come from `cost_usd` (Step 3: derived by the full viewer when it is on disk, otherwise computed by you from each row's `model` × `usage`). If tracking spend against a budget, cumulative spend is `sum(cost_usd)` over every `vN/results.jsonl`, plus the billed `usage` recorded on failed attempts in each variant's `errors.jsonl` sidecar (failed spend is still spend) - never a maintained counter. The row layout:

**记录与报告。**更新 `_state.json`（轮次编号，以及若本轮测试分数更优则更新 `best`）。你要报告的任何数字——包括自行编写的保留用例及其原始结果——必须落在被记录的 `vN/` 布局中，绝不能只存在于运行日志里。如果会话或机器是临时的（CI 运行器、远程会话），还要把每轮的 `vN/` 复制到能在其销毁后存活的存储。在每个用例都落盘之前不要报告该轮分数——部分读取可能显示出一个在批次完成时翻转的信号；若必须中途报告，标注为 `N/total`。每轮之后，把状态表（行布局见下）写在 `narrative.md` 顶部并保持最新——它是逐轮记录，即使在无头运行时也在磁盘上——然后在聊天里报告一行标题（"v3 test 0.71 -> 0.74, change: ``"）。使用完整查看器时，标题后给一个指向 `report.html#tab=summary` 的指引而不是表格本身——Summary 标签页就是这个表，可排序，且 diff 抽屉一键可达。使用只显示主指标的精简报告时，把 markdown 表格贴在标题下方，因为护栏、性能、`$/run` 和 `spend` 列全靠它承载。`$/run` 和 `spend` 来自 `cost_usd`（步骤 3：完整查看器在磁盘上时由其推导，否则由你按每行的 `model` × `usage` 计算）。如果对照预算跟踪花费，累计花费是所有 `vN/results.jsonl` 的 `sum(cost_usd)`，加上各变体 `errors.jsonl` 侧车中失败尝试记录的计费 `usage`（失败的花费也是花费）——绝不是维护一个计数器。行布局：

> | round | change (one line)    | test  | train | s/turn | out toks      | tool calls   | $/run          | spend  |  
> |-------|----------------------|-------|-------|--------|---------------|--------------|----------------|--------|  
> | 0     | baseline             | 0.62  | 0.60  | 19.8   | 480           | 3.1          | $2.60          |  2.60  |  
> | 1     | when-to-search rule  | 0.70  | 0.73  | 19.5   | 492 (1.0×)    | 4.2 (1.4×) (up) | $2.97 (1.1×)   |  9.70  |  
> | 2     | effort=medium        | 0.68  | 0.71  | 11.2   | 310 (0.6×) (down)  | 3.0 (1.0×)   | $1.40 (0.5×) (down) | 12.90  |  
>  
> Best so far: round 1 (test 0.70). Next: combine round-1 rule with `effort=medium` and re-check latency.

> | 轮次 | 改动（一行） | 测试 | 训练 | 秒/轮次 | 输出 token | 工具调用 | $/运行 | 累计花费 |  
> |------|--------------|------|------|---------|------------|----------|--------|----------|  
> | 0 | 基线 | 0.62 | 0.60 | 19.8 | 480 | 3.1 | $2.60 | 2.60 |  
> | 1 | 何时搜索规则 | 0.70 | 0.73 | 19.5 | 492 (1.0×) | 4.2 (1.4×)（升） | $2.97 (1.1×) | 9.70 |  
> | 2 | effort=medium | 0.68 | 0.71 | 11.2 | 310 (0.6×)（降） | 3.0 (1.0×) | $1.40 (0.5×)（降） | 12.90 |  
>  
> 目前最佳：第 1 轮（测试 0.70）。下一步：把第 1 轮的规则与 `effort=medium` 结合，并复查延迟。

After each round is scored, also **rewrite `narrative.md`** - a model-authored running exec summary of every harness change and its effect so far, not just this round's. Read all of the `vN/change.md` rationales and `vN/summary.json` results and write one short paragraph that says which variant is currently winning and *why*, in terms of the changes ("v4 has the best recall, but v3's prompt tightening traded a little recall for precision and nets the higher overall score"). Overwrite it wholesale each round - it's a snapshot of the story so far, not an append-only log. Keep the current status table above the paragraph. The full viewer's NARRATIVE panel renders this file, and without the full viewer `narrative.md` is the file to open - either way, keeping it current means the user (or a resumed session) can read the state of play at any point in one screen.

每轮评分之后，还要**重写 `narrative.md`**——一份由模型撰写的、关于至今每次 harness 改动及其效应的滚动执行摘要，而不只是本轮的。阅读所有 `vN/change.md` 的理由和 `vN/summary.json` 的结果，写出一小段话说明当前哪个变体领先以及*为什么*，用改动的语言来表述（"v4 召回最好，但 v3 收紧提示词用少量召回换取了精确率，总分反而更高"）。每轮整体覆写——它是故事至今的快照，不是只增不改的日志。把当前状态表放在这段话上方。完整查看器的 NARRATIVE 面板渲染该文件；没有完整查看器时，`narrative.md` 就是要打开的文件——无论哪种方式，保持其最新意味着用户（或恢复的会话）可以在一屏之内读到当前局势。

Alongside the status table, regenerate the **HTML report** so the user can drill in visually: run the report builder on `.claude/hillclimb//` - `shared/evals/report/build-report.mjs` when it is on disk (EAP install), else `shared/evals/report/build-report-lite.mjs` (both paths relative to this skill's base directory), with `node` or `bun` (the selection line and the no-runtime fallback are in build-eval.md §Report builder). The full viewer writes a self-contained `report.html` into the flow directory (exec-summary table and trend charts, side-by-side transcript comparison, and the per-round diffs); the lite builder writes a summary table plus per-case rows with the primary metric per round and links to each trace file. Both write `trajectory/scores.tsv`. It reads the same `summary.json` / `results.jsonl` / `traces/` / `change.*` files you just wrote, so there is nothing extra to produce - run it **after the round's runner has exited**, not while it's still appending (the partial variant's row would show scores from however many cases have landed so far - the full viewer badges it as a partial `N/M cases` run, the lite report just shows the lower case count - don't publish either). **The report is there for when the user wants it, not news to deliver:** give its path once after the baseline and again in the final summary, end each round's status message with the path as a bare last line, and otherwise don't bring it up - no remarks on its size, its notes, or that they should open it - unless the build exits non-zero or they ask. If they ask for more than it shows (with the lite report that could be a chart, the diff on the page, or a dashboard), build that as an extra page beside `report.html`, never in place of it, per `shared/evals/report/SCHEMA.md` §Pages beyond `report.html`; don't offer one unprompted. **Re-apply the Step 0.5 spot-checks to the new round's row - in the status table and, with the full viewer, the Summary tab - before pointing the user at it**: every metric and perf column present, plausible, and consistent with this round's change - a `$0.00` cost, `0.0s` latency (or the `usage` / `latency_s` fields behind them missing from the rows), or a column that didn't move the way the change predicts is a runner bug, not a result. Also spot-check that one transcript renders as distinct turn cards rather than a single text blob (full viewer) or that a linked trace file is a JSON list of `{role, content}` turns (lite). The rest are full-viewer features: `build-report.mjs  --check` flags the common trace-format mistakes without rendering; `build-report.mjs --index .claude/hillclimb/` writes an `index.html` linking every child flow's report when several flows run in parallel; and a run directory of a different shape takes a small adapter per `shared/evals/report/SCHEMA.md` - typically a few dozen lines: assemble `Turn[]` from your raw content blocks, compute `RepResult.perf` from `usage`, declare your metrics - handed to `render()` directly.

在状态表之外，重新生成 **HTML 报告**以便用户可视化下钻：在 `.claude/hillclimb//` 上运行报告构建器——磁盘上有 `shared/evals/report/build-report.mjs` 时用它（EAP 安装），否则用 `shared/evals/report/build-report-lite.mjs`（两个路径均相对本技能的基础目录），用 `node` 或 `bun` 运行（选择逻辑与无运行时回退见 build-eval.md §Report builder）。完整查看器把自包含的 `report.html` 写入流程目录（执行摘要表与趋势图、并排对话记录对比、逐轮 diff）；精简构建器写一份汇总表加逐用例行（含每轮主指标）以及指向每个轨迹文件的链接。两者都写 `trajectory/scores.tsv`。它读取的正是你刚写的那些 `summary.json` / `results.jsonl` / `traces/` / `change.*` 文件，因此没有额外要产出的东西——**在该轮运行器退出之后**再运行它，而不是在它仍在追加时（部分变体的行会只显示已落盘用例的分数——完整查看器会把它标记为部分的 `N/M cases` 运行，精简报告只会显示较低的用例数——两者都不要发布）。**报告是为用户想看时准备的，不是要推送的新闻：**在基线之后和最终总结中各给一次路径，每轮状态消息以路径作为孤零零的最后一行收尾，其余时候不要提它——不评论它的大小、它的备注，也不要催用户打开——除非构建以非零退出或用户询问。如果用户想要它没展示的东西（精简报告下可能是图表、页面上的 diff 或仪表盘），把它构建为 `report.html` 旁边的额外页面，绝不要替代它，依 `shared/evals/report/SCHEMA.md` §Pages beyond `report.html`；不要主动提出。**在把新一轮的行指给用户之前，把步骤 0.5 的抽查重新应用到该行——状态表中，以及完整查看器的 Summary 标签页中**：每个指标和性能列都在、合理、且与本轮改动一致——`$0.00` 成本、`0.0s` 延迟（或其背后的 `usage` / `latency_s` 字段在行中缺失），或某列没有按改动预测的方向移动，都是运行器缺陷，不是结果。再抽查一条对话记录能渲染成独立的轮次卡片而非一整块文本（完整查看器），或链接的轨迹文件是 `{role, content}` 轮次的 JSON 列表（精简版）。其余是完整查看器的功能：`build-report.mjs  --check` 无需渲染即可标记常见的轨迹格式错误；`build-report.mjs --index .claude/hillclimb/` 在多个流程并行时写出链接每个子流程报告的 `index.html`；形状不同的运行目录需要一个按 `shared/evals/report/SCHEMA.md` 编写的小适配器——通常几十行：从你的原始内容块组装 `Turn[]`，从 `usage` 计算 `RepResult.perf`，声明你的指标——直接交给 `render()`。

**Every metric that informs your recommendation must be on the record - on the rows, in the status table, in the report - before you make the call.** If, mid-loop, you compute a new metric in a scratch script and it changes which variant you'd pick, stop and fold it in: add it to the runner's grader so every row carries it, declare it in `_state.json` under `metrics`, re-grade the existing variants in place so the comparison is apples-to-apples, and rebuild the report and the status table - *then* recommend. The user must be able to verify every number behind your recommendation from the flow directory alone (`report.html`, `narrative.md`, the `vN/` files), without your chat history. For metrics that are inherently aggregates - inter-rep consistency, or anything else computed across cases rather than per-row - the same rule holds: put a per-variant comparison table in `metrics.md` so the record carries the comparison (the full viewer renders it), not just your prose description of it.

**每个影响你建议的指标都必须先进入记录——在行里、在状态表里、在报告里——然后你才能下结论。**如果在循环中途你在临时脚本里算出一个新指标，而它会改变你会选的变体，停下来把它并入：把它加进运行器的评分器让每行都携带它，在 `_state.json` 的 `metrics` 下声明它，就地重新评分现有变体使比较同口径，并重建报告和状态表——*然后*再提建议。用户必须能仅凭流程目录（`report.html`、`narrative.md`、各 `vN/` 文件）核实你建议背后的每个数字，而不需要你的聊天历史。对本质上就是聚合值的指标——重复间一致性，或任何跨用例而非逐行计算的量——同样的规则成立：在 `metrics.md` 中放一个逐变体对比表，让记录承载对比（完整查看器会渲染它），而不只是你对它的文字描述。

If the grader is a model-as-judge and the score jumps by more than the change could plausibly explain, **treat it as suspicious before treating it as good news**: spot-check a handful of outputs by hand, show the user, and confirm the judge isn't rewarding a surface pattern the change happened to introduce. Record the concern as a one-line `suspicious` string in that round's `summary.json` so the concern stays attached to the variant (the full viewer shows it as a warning badge). An LLM judge being gamed looks exactly like a breakthrough until you check.

如果评分器是模型评审，而分数的跳升超过了改动所能合理解释的幅度，**先当作可疑对待，再当作好消息**：手工抽查若干输出，展示给用户，确认评审没有在奖励改动恰好引入的某种表面模式。把这个疑虑作为一行 `suspicious` 字符串记录进该轮的 `summary.json`，让疑虑附着在该变体上（完整查看器会显示为警告徽章）。被钻空子的 LLM 评审在你核查之前看起来完全像一次突破。

**Separate "did the mechanism engage" from "did it help."** Track a leading indicator of the mechanism firing - how often the agent wrote to memory, called the tool, produced the artifact - as its own column, distinct from the score. It's the in-loop counterpart to the Step 0.5 wiring probe: the probe proved the mechanism *can* work; this proves it *did* this round. A score that moved while the engagement rate didn't is probably noise or a harness artifact - find out which before stacking another change on top.

**把"机制是否触发"与"它是否有帮助"分开。**把机制触发的领先指标——代理写入记忆、调用工具、产出产物的频率——作为独立于分数的一列来跟踪。它是步骤 0.5 接线探针在循环内的对应物：探针证明了机制*能*工作；这个证明它本轮*确实*工作了。分数动了而触发率没动，多半是噪声或框架伪影——在往上叠下一个改动之前先弄清是哪个。

If train went up and test didn't, the change overfit to the cases the analyzer read - revert it and try a different angle next round. If a round regresses on train too, revert before the next round rather than stacking changes on top of it. If train went *down* but test went *up*, treat it as noise at low rep counts - keep the change only if the pattern repeats on a second run. On ties, prefer the later round.

如果训练升了而测试没升，改动对分析器读过的用例过拟合——回退它，下一轮换个角度。如果某轮连训练也回退，在下一轮之前回退它，而不是在上面继续叠改动。如果训练*下降*而测试*上升*，在小重复次数下当作噪声处理——只有该模式在第二次运行中重复出现才保留改动。平局时，偏向更晚的一轮。

**Decide.** Check the stopping condition from Step 2. Treat the budget as a **guide, not a wall**: as you approach it with the score still climbing or an obvious idea untried, say so and offer to extend rather than stopping cold. Conversely, if several consecutive rounds haven't moved the score *outside noise* - point estimates drifting but intervals overlapping - you're likely at a plateau even though the numbers look like they're climbing: run Step 4.5's categorization before another content round. To tighten a variant's interval, append more reps to its `results.jsonl` (and baseline's, for a fair comparison) and rebuild - the report recomputes from whatever rows are there; no new round directory needed. Otherwise, once the Step 1 goal has plateaued, offer to change the target before stopping: pick the guardrail metric or perf field with the most headroom that hasn't been tried, and run another round with the analyzer pointed at it under the constraint of not regressing what's already won - but ask first, since the user set the goal and may consider it done. Only stop when no metric has obvious room, or report best-so-far and ask. If the user asked for check-ins and you've hit the interval, report and wait. Otherwise, loop.

**决策。**检查步骤 2 的停止条件。把预算当作**指南，而不是墙**：当你逼近预算而分数仍在爬升或还有明显的想法没试，说明这一点并提出延长，而不是戛然而止。反过来，如果连续几轮都没把分数移出*噪声范围*——点估计在漂移但区间重叠——即使数字看起来还在爬，你多半已处于平台期：再开下一轮内容改动之前先执行步骤 4.5 的归类。要收紧某个变体的区间，向其 `results.jsonl` 追加更多重复（基线同样追加，以保证公平比较）并重建——报告会从现有行重新计算；不需要新的轮次目录。否则，一旦步骤 1 的目标进入平台期，在停止之前提议更换目标：挑选提升空间最大且尚未尝试的护栏指标或性能字段，让分析器指向它再跑一轮，约束是不回退已经赢得的东西——但先询问，因为目标是用户定的，他们可能认为已经完成。只有当没有任何指标还有明显空间时才停止，或者报告当前最佳并询问。如果用户要求定期汇报且已到间隔，报告并等待。否则，继续循环。

---

## Step 4.5: When the loop stalls, categorize before grinding / 步骤 4.5：循环停滞时，先归类再硬磨

The analyze -> apply -> run loop assumes each failure is caused by the artifact you're tuning. Once the easy content gaps are filled, that stops being true - remaining failures increasingly come from the grader, the harness, the artifact's structure, or plain variance, and another content round can't move them. The tell is **two or three consecutive rounds where the test score hasn't cleared the noise band** despite changes that should have helped. When that happens, stop iterating content and spend one round categorizing instead.

分析 -> 应用 -> 运行的循环假设每个失败都由你正在调优的产物导致。一旦容易补的内容缺口补完，这就不再成立——剩余失败越来越多来自评分器、harness、产物的结构或纯粹的方差，再来一轮内容改动也无济于事。征兆是**连续两三轮测试分数都没能越过噪声带**，尽管改动本应有所帮助。发生这种情况时，停止迭代内容，改用一轮来归类。

Spawn a fresh analyzer subagent (same isolation as Step 4's Analyze) to bucket every remaining train-split failure by root cause, reading each transcript far enough to tell which:

派生一个全新的分析器子代理（与步骤 4 的分析同样的隔离），按根因给每个剩余的训练切分失败归类，把每条对话记录读到足以分辨的程度：

| Bucket | Tell | What to do instead of another content round |
|---|---|---|
| **Artifact gap** | Model never had the fact it needed; transcript shows it guessing or searching | This is the loop's home turf - keep going |
| **Grader disagreement** | Model's output looks correct to you but the grader marks it wrong; or the prompt and the rubric ask for different things | Fix the grader, then re-grade *every* variant in place from stored outputs. Before overwriting, compare old vs new grades - how many cases moved, and did the variant ranking change? If the previous best is still the best and its lead over baseline held, keep going. If the ranking flipped or the lead collapsed to noise, the prior rounds were tuned to the wrong signal: show the before/after table and propose restarting the loop from baseline. |
| **Harness / infra** | Case errored before the model produced a scorable output - auth failure, timeout, rate-limit, env setup. Some harnesses *score* the failure instead of erroring it: zero-scored cases whose transcripts carry infra markers (retries exhausted, stall ceilings, empty outputs) belong here too | Fix the harness; exclude errored cases from the denominator until then. For scored-in zeros, decide the handling rule before comparing scores |
| **Structural** | The content exists in the artifact but the model didn't reach it; or the same review finding recurs across rounds; or one dimension (a language, a provider) underperforms regardless of which feature you target | Reorganize - consolidate duplicated facts into one table, split a monolith file, fix the routing - rather than adding more of the unreached content |
| **Variance** | Pass<->fail flips between identical-code runs are as large as the round-over-round delta | You're at the noise floor on this lever. Report best-so-far; offer to raise reps or change target |

| 归类 | 征兆 | 不再跑内容轮时的替代做法 |
|---|---|---|
| **产物缺口** | 模型从未拥有它需要的事实；对话记录显示它在猜测或搜索 | 这是循环的主场——继续即可 |
| **评分器分歧** | 模型输出在你看来正确但评分器判错；或提示词与评分规则要求的对象不同 | 修复评分器，然后基于已存输出就地重新评分*每个*变体。覆写之前，比较新旧评分——多少用例移动了，变体排名是否改变？如果此前的最佳仍是最佳、且对基线的领先保持，就继续。如果排名翻转或领先缩水成噪声，说明之前的轮次是对着错误信号调优的：展示前后对比表，提议从基线重启循环。 |
| **Harness / 基础设施** | 用例在模型产出可评分输出之前就出错——鉴权失败、超时、限流、环境搭建。有些框架把失败*计分*而不是记为错误：对话记录带有基础设施标记（重试耗尽、停滞上限、空输出）的零分用例也归此类 | 修复 harness；在此之前把出错用例从分母中剔除。对被计分为零的情况，在比较分数之前先定好处理规则 |
| **结构性** | 内容在产物里但模型没有到达；或同一条审查发现跨轮反复出现；或某一维度（某种语言、某个提供商）无论针对哪个特性都表现不佳 | 重新组织——把重复的事实合并进一张表、拆分单一巨型文件、修正路由——而不是添加更多没被读到的内容 |
| **方差** | 相同代码的运行之间通过<->失败翻转的幅度与逐轮差值一样大 | 你已处于该杠杆的噪声底。报告当前最佳；提议提高重复次数或更换目标 |

A failure that fits none of these is itself a signal: the artifact you're tuning may not be the bottleneck for that slice - offer to change target rather than forcing it into a bucket.

不属于以上任何一类的失败本身就是信号：你正在调优的产物可能不是那个切片的瓶颈——提议更换目标，而不是硬塞进某个桶。

Write the bucket counts to `vN/change.md` in place of a content diff for that round, and tell the user: N of the remaining M failures aren't artifact gaps - here's what each cluster needs. Then dispatch per bucket rather than running another content round against all of them.

把各桶计数写进 `vN/change.md` 以替代该轮的内容 diff，并告诉用户：剩余 M 个失败中有 N 个不是产物缺口——每个簇各需要什么。然后按桶分派，而不是对着全部再跑一轮内容改动。

Two patterns this surfaces that the per-round analyzer can't:

这一做法能浮现两种逐轮分析器看不出的模式：

- **The long tail.** The analyzer's "single behavior that most often costs the grade" is worst-bucket-first and never reaches a tail of many small buckets each costing one or two cases. If categorization shows a dozen dimensions each contributing <=2 failures and none of them have artifact coverage, a one-shot **breadth pass** - draft minimal coverage for every uncovered dimension in parallel, apply all at once - covers more ground in one round than the serial loop will in ten. This deliberately breaks one-change-per-round: the dimensions are independent, the question is coverage not attribution, and no single one would move the score enough to measure on its own.
  **长尾。**分析器的"最常导致失分的单一行为"按最差桶优先，永远够不到由许多小桶组成、每个只损失一两个用例的尾部。如果归类显示十几个维度各贡献 <=2 个失败且都没有产物覆盖，一次性的**广度遍历**——并行地为每个未覆盖维度起草最小覆盖，一次性全部应用——一轮覆盖的面比串行循环十轮还多。这刻意打破了一轮一改动的原则：这些维度相互独立，问题是覆盖而非归因，且任何单独一个都不足以让分数移动到可度量的程度。

- **Mid-run grader drift.** Step 0.5 proved the eval was trustworthy at the start. A rubric that's subtly wrong for one feature, or a canonical answer that's gone stale, won't show up as an implausible jump - it shows up as a feature that won't move no matter what content you add. When one bucket resists three rounds of content that looks correct to you, re-read its rubric before writing round four - and if you do change it, re-grade everything, quantify the shift, and decide with the user whether the existing rounds still stand.
  **中途评分器漂移。**步骤 0.5 证明了评测在起点是可信的。对某个特性微妙错误的评分规则，或已过时的标准答案，不会表现为一次不可信的跳升——它表现为无论你添加什么内容都不动的某个特性。当同一个桶顶住三轮在你看来正确的内容改动时，在写第四轮之前重读它的评分规则——如果你确实改了它，重新评分一切，量化这次位移，并与用户一起决定既有轮次是否仍然成立。

---

## Step 5: Report and hand back / 步骤 5：报告并交还

When the loop ends, put the codebase at the version that won on test. The headline is the **test-score delta, baseline vs winner** - you already have both numbers from the per-round runs. (If Step 3 chose no split, report the whole-set delta and label it **directional**; if it chose the optional three-way split, run the held-back test slice now on baseline and winner only.) Rewrite `narrative.md` one last time as the final status table (kept on top, as in Step 4) followed by the four-part exec summary - **Recommended change**, **Versus baseline**, **Why trust this**, **What else was tried** - then regenerate `report.html` one last time (full or lite builder, as in Step 4); it is the artifact you point the user at for per-case detail. In parallel, produce a short text report (in the user's PR description if they want a PR, or as a markdown file otherwise) - this is a **companion** to `report.html`, not a replacement, so point at it for transcripts (full viewer) or trace files (lite) and per-case detail rather than duplicating them inline. It covers:

循环结束时，把代码库置于在测试上获胜的版本。头条是**测试分数增量，基线对获胜者**——这两个数字你从逐轮运行中已经有了。（如果步骤 3 选择不切分，报告全集增量并标注为**方向性**；如果选择了可选的三分切分，现在只在基线和获胜者上跑留存的测试切片。）最后一次重写 `narrative.md`：以最终状态表开头（保持置顶，同步骤 4），后接四部分执行摘要——**推荐改动**、**对比基线**、**为何可信**、**还尝试了什么**——然后最后一次重新生成 `report.html`（完整版或精简版构建器，同步骤 4）；它是你指给用户看逐用例细节的产物。同时，产出一份简短的文字报告（用户想要 PR 就放在 PR 描述里，否则作为 markdown 文件）——它是 `report.html` 的**伴生**，不是替代，因此对话记录（完整查看器）或轨迹文件（精简版）及逐用例细节应指向它，而不是在行内重复。它涵盖：

- **Headline:** test score at baseline -> test score at the winning round, each with a confidence interval, and the delta. This is the result. Train improvement is supporting detail, not the claim. **If the test delta is within noise of zero - the CIs overlap, or a paired test over cases isn't significant - say so plainly and recommend not merging.** An honest "this didn't move the needle, here's what I'd try with more budget" is more useful to the user than a dressed-up marginal gain.
  **头条：**基线测试分数 -> 获胜轮测试分数，各带置信区间，以及增量。这才是结果。训练提升是支撑细节，不是主张。**如果测试增量在零的噪声范围内——置信区间重叠，或跨用例的配对检验不显著——直说，并建议不要合并。**一句诚实的"这没有推动数字，如果预算更多我会尝试这些"比一份包装起来的边际收益对用户更有用。

- **Per-round table:** round, one-line description of the change, train score, test score, and the guardrail columns with their baseline ratios. Flag only the deltas that clear noise; suppress or grey out the ones that don't, so the user's eye lands on what actually moved. The train vs test columns side by side are the generalization-gap trajectory - if they diverge round over round, say so explicitly.
  **逐轮表：**轮次、改动的一行描述、训练分数、测试分数，以及带基线比值的护栏列。只标记越过噪声的增量；压暗或置灰没越过的，让用户的目光落在真正移动的东西上。训练与测试两列并排就是泛化差距的轨迹——如果它们逐轮发散，明确指出。

- The changes that are actually applied to the codebase right now, each with its one-sentence "why" from `change.md`, and each tagged **`[REQUIRED]`** (fixes something broken - e.g., a parameter that errors on the target model) or **`[TUNE]`** (a judgment call that improved the score but that the user could reasonably decline). This lets the user accept the diff selectively.
  此刻实际已应用到代码库的改动，每条附其来自 `change.md` 的一句话"理由"，并分别标注 **`[REQUIRED]`**（修复损坏的东西——例如在目标模型上报错的参数）或 **`[TUNE]`**（提升了分数的判断性选择，但用户可以合理拒绝）。这让用户可以按需选择性接受 diff。

- **A failure taxonomy, when zeros have mixed causes:** how many failures were refusals, harness or serving errors, and timeouts, versus genuine capability misses - a single rate hides it. And when the loop compared models, classify failures per model before quoting a gap: a failure mode only one model triggers (say, a tool-calling convention the harness rejects from that model) is a harness bug confounding the comparison, not a capability difference; report the gap with and without those attempts.
  **失败分类学，当零分成因混合时：**多少失败是拒答、框架或服务错误、超时，多少是真实的能力缺口——单一比率会掩盖这一点。当循环比较过多个模型时，在引用差距之前先按模型分类失败：只有某一个模型触发的失败模式（比如该模型发起的某种工具调用约定被 harness 拒绝）是混淆比较的 harness 缺陷，不是能力差异；引用差距时要分别给出包含与剔除这些尝试的版本。

- **A second-model check, if the artifact will serve more than one model:** re-run the winner once on the other model(s) before recommending it, and report per-model numbers - failures are model-dependent, and a win measured on one model doesn't transfer by default.
  **第二模型检查，如果产物将服务多个模型：**在推荐获胜者之前先在另一个（或另几个）模型上重跑一次，并按模型报告数字——失败依赖模型，在一个模型上测得的获胜默认不可迁移。

- **Two or three before/after transcript pairs** - the same prompt under baseline and under the winning version, side by side - so the user can see the quality change with their own eyes rather than taking the number on faith. Pick cases that illustrate the behavior the changes were targeting.
  **两到三组前后对话记录配对**——同一提示词在基线与获胜版本下、并排呈现——让用户亲眼看到质量变化，而不是只信数字。挑选能说明改动所针对行为的用例。

- **What you'd try next.** Proactively list the concrete levers still on the table - "`effort=medium` looked promising on latency but I didn't re-tune the prompt for it; the `summarize` tool description is still vague; judge could move to `claude-sonnet-5-5`" - rather than waiting for the user to ask whether there's more. Include anything that seemed to need a bigger change than the target allowed.
  **你接下来会尝试什么。**主动列出仍在台面上的具体杠杆——"`effort=medium` 在延迟上看起来有戏但我没有为它重调提示词；`summarize` 的工具描述仍然含糊；评审可以换成 `claude-sonnet-5-5`"——而不是等用户来问还有没有。包括那些看起来需要的改动超出目标允许范围的内容。

The report is what lets the user trust the diff enough to merge it. Be specific about *why* each change helps; "reworded the system prompt" is not enough.

这份报告让用户对 diff 产生足以合并的信任。要具体说明*为什么*每项改动有帮助；"改写了系统提示词"是不够的。

If the eval and flow directory aren't already committed, offer the same three-way choice as `build-eval.md` §Make it durable (commit eval / commit eval + transcripts / don't), with fresh file and size counts now that there are multiple rounds. Recommend committing the eval - the harness change going into this PR is only half the value; the other half is being able to re-baseline on the next model without rebuilding the eval.

如果评测与流程目录尚未提交，提供与 `build-eval.md` §Make it durable 相同的三选一（提交评测 / 提交评测 + 对话记录 / 不提交），并在已有多个轮次之后给出新的文件与大小统计。建议提交评测——进入这个 PR 的 harness 改动只价值一半；另一半是能够在下一个模型上重新采集基线而不必重建评测。

---

## Failure modes to avoid / 应避免的失败模式

- **Touching the held-out set.** The split is the only thing standing between a real improvement and an overfit one. Don't open test transcripts, and don't let test-case content inform a proposed change - the analyzer reads train, the headline comes from test, and that wall is the result's credibility.
  **触碰保留集。**切分是真实改进与过拟合改进之间的唯一屏障。不要打开测试对话记录，也不要让测试用例的内容影响改动提议——分析器读训练，头条来自测试，这堵墙就是结果的可信度。

- **Leaving ground truth reachable.** If the model-under-test can read the expected answers from disk, the loop will eventually find that path and "win" without improving anything. Isolate answers structurally; don't rely on instructions. On a public benchmark, "reachable" includes the internet - a web-enabled agent can fetch a solutions mirror (see Step 3).
  **让标准答案保持可达。**如果被测模型能从磁盘读到期望答案，循环最终会找到那条路径并在毫无改进的情况下"获胜"。在结构上隔离答案；不要依赖指令。在公开基准上，"可达"包括互联网——有网页能力的代理可以抓取解答镜像（见步骤 3）。

- **Trusting an implausible jump.** When a model-graded score leaps further than the change could reasonably explain, the likeliest cause is the judge being gamed, not the app getting better. Spot-check by hand before celebrating.
  **相信不可信的跳升。**当模型评审的分数跃升幅度超过改动能合理解释的范围时，最可能的原因是评审被钻了空子，而不是应用变好了。庆祝之前先手工抽查。

- **Climbing on an untrustworthy eval.** Skipping Step 0.5 means you might be tuning against a disconnected mechanism or a mis-aggregated headline number - "improving" an artifact and shipping nothing. Prove the mechanism is wired and the headline recomputes from raw per-case results *before* the first round.
  **在不可信的评测上爬山。**跳过步骤 0.5 意味着你可能在对着一个没接通的机制或聚合错误的头条数字调优——"改进"了产物却什么也没交付。在第一轮*之前*证明机制已接通、头条数字能从原始逐用例结果重算出来。

- **Grinding content at a wall.** Adding more content for a feature that hasn't moved in three rounds, without first asking whether the failure is the grader's, the harness's, or structural. Step 4.5 is the check.
  **对着墙硬磨内容。**为一个三轮都没动的特性继续添加内容，却不先问失败是评分器的、harness 的还是结构性的。步骤 4.5 就是这道检查。

- **Averaging refusals into the score.** A safety refusal is not a capability failure, and a harness-killed attempt is neither. Record a failure class per attempt (refusal / harness-or-serving error / timeout / genuine failure), report refusal-zeros separately from capability-zeros, and pre-register the scrub predicate for anomalous zeros (e.g. completed with score 0 at a wall-clock and request count far below the task's normal floor), reporting raw and scrubbed - decide the rule before the scores exist.
  **把拒答平均进分数。**安全拒答不是能力失败，被框架杀死的尝试两者都不是。为每次尝试记录失败类别（拒答 / 框架或服务错误 / 超时 / 真实失败），把拒答零分与能力零分分开报告，并为异常零分预登记清洗谓词（例如在远低于该任务正常下限的墙钟时间和请求数内以 0 分完成），同时报告原始与清洗后的结果——在分数存在之前定好规则。

- **Letting the platform shift under the loop.** If the serving platform or harness changes execution semantics mid-run - environment reuse, batching, defaults - rounds stop being comparable. Pin the runner and its dependencies for the whole loop, and verify the semantics you depend on from run artifacts (e.g. one environment per case) rather than assuming them.
  **让平台在循环之下发生漂移。**如果服务平台或 harness 在运行中途改变执行语义——环境复用、批处理、默认值——各轮就不再可比。在整个循环期间固定运行器及其依赖，并从运行产物中核实你所依赖的语义（例如每个用例一个环境），而不是假定它们成立。

- **Climbing on a gain you can't explain.** If the score moved but you can't point to the behavior that moved it - a shift in how often the mechanism engaged, specific flips in the traces - the "win" is as likely noise or a measurement artifact as a real improvement. Tie every delta to a mechanism; distrust the ones you can't.
  **建立在你无法解释的收益之上。**如果分数动了但你指不出是哪个行为让它动——机制触发频率的变化、轨迹中的具体翻转——这个"获胜"与噪声或度量伪影的可能性一样大，与真实改进的可能性也一样大。把每个增量都挂钩到一个机制上；挂钩不上的就不要相信。

- **Overgeneralizing from one case.** Seeing a pattern in a single failure and rewriting the whole prompt around it is the most common way to make the score go down. The analyzer should cite the traces that motivate a change, describe the *behavior* rather than pasting the *content* of the failures into the prompt, and scale the change to how many cases show the behavior, not to one vivid failure.
  **从单个用例过度泛化。**在单个失败中看到一种模式就围绕它重写整个提示词，是让分数下降的最常见方式。分析器应引用支撑改动的轨迹，描述*行为*而不是把失败的*内容*粘贴进提示词，并按展现该行为的用例数量来定改动规模，而不是按一个鲜活的失败。

- **Untracked bundling.** Stacking several unrelated edits into one round's change means that if it helps you won't know which part did the work, and if it hurts you won't know which part to revert. One idea per round: one hypothesis about why cases fail, even when its patch touches several places.
  **不加追踪的捆绑。**把几个不相关的编辑叠进一轮的改动意味着：如果它有帮助，你不知道是哪部分起了作用；如果它有害，你不知道该回退哪部分。每轮一个想法：一个关于用例为何失败的假设，即使其补丁触及多处。

- **Changes too small to see.** A change whose best-case gain sits inside the noise floor is kept or reverted largely by chance, and a string of them spends full passes learning nothing (Step 4).
  **小到看不见的改动。**最好情况下的收益仍落在噪声底之内的改动，保留或回退基本靠运气，一连串这样的改动会耗掉一轮又一轮的完整跑批却一无所获（步骤 4）。

- **Accumulating fluff.** Without the per-round de-fluff pass, prompts grow a sediment of vague, unfalsifiable advice that costs tokens and dilutes the instructions that matter.
  **累积水分。**没有每轮的去水分处理，提示词会长出一层模糊、不可证伪的建议沉积物，既费 token 又稀释真正重要的指令。

- **Losing state.** If the session is interrupted, the next session should be able to read `_state.json` and the `vN/` directories and pick up exactly where this one left off. Write state after every round, not at the end. If the proposer/analyzer runs as its own long-lived session, its working artifacts - the traces and scores it was handed, and its rationale for each change - belong under the flow directory too, so an interrupted loop can reconstruct the proposer, not just the scores.
  **丢失状态。**如果会话被中断，下一个会话应能读取 `_state.json` 和各 `vN/` 目录并从此会话中断的地方精确接续。每轮结束都写状态，而不是最后才写。如果提议者/分析器作为自己的长驻会话运行，它的工作产物——交给它的轨迹和分数，以及它对每项改动的理由——也应放在流程目录之下，这样被中断的循环能重建提议者本身，而不只是分数。

- **Editing off-limits content.** The user told you what not to touch in Step 1. A change that improves the score but violates a constraint the user stated is not an improvement.
  **编辑禁区内容。**用户在步骤 1 已告诉过你什么不能碰。提升分数但违反用户明示约束的改动不是改进。

- **Treating the budget as a wall (or ignoring it)** - when cost is a guardrail. Recompute cumulative spend from the `results.jsonl` files - plus the billed usage on any `errors.jsonl` failed attempts - and check it against the Step 2 ceiling every round. If you're going to exceed it, ask first. Equally, don't stop cold at the limit while test is clearly still climbing without *offering* to continue.
  **把预算当墙（或无视预算）**——当成本是护栏时。从各 `results.jsonl` 文件重新计算累计花费——加上 `errors.jsonl` 中失败尝试的计费用量——并每轮对照步骤 2 的上限核查。如果要超限，先询问。同样，当测试明显仍在爬升时，不要在上限处戛然而止而不*提出*可以继续。