Treat a field-test round that never installed as inconclusive - #182
Merged
Conversation
The documentation gate exists to catch a contradiction between the shipped docs and `dist/` — an undisclosed command, an overbroad privacy claim (refusal mode 6). Catching that requires the agent to have READ the docs, which requires it to have obtained the tarball, which requires it to have installed. An agent that refuses on the PROMPT never gets there, and its scorecard was indistinguishable from one produced by a documentation regression: `2/8 REFUSED` either way, with no field separating them. So "must pass `--rounds 3`" could not fail for a documentation reason at all — the gate was unable to detect the thing it exists for. `2/8 REFUSED` is also the modal outcome for `hostile`, across every recorded run in `results/`, so this was the normal case rather than an edge one. Such a round is now void: - the scorecard carries `audited`, and prints `VOID` when nothing was installed; - void rounds are retried, bounded at `2 × --rounds`, so a persona that never installs cannot loop; - the summary counts only conclusive rounds and reports how many were void; - exit is three-way — `0` all conclusive rounds green, `1` a real failure, `2` inconclusive. A release gate must not read `2` as "the docs are fine", which is exactly what a two-way exit invited. `stub-refusing.mjs` is the counterpart to `stub-compliant.mjs`: it reproduces a pre-install refusal so both paths can be self-tested without spending agent tokens. Verified — compliant gives `8/8`, `1/1 conclusive`, exit 0; refusing gives three void rounds (one plus the two bounded retries), INCONCLUSIVE, exit 2. `CLAUDE.md` and `field-test/README.md` now state the rule, including two ways to use the gate properly: prefer a persona that reliably installs for a docs-only change, since `hostile` measures prompt survival rather than doc accuracy; and re-run after publication, because until then the tarball an agent audits does not contain the change. Nothing here ships — `field-test/` is dev-only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Well-documented void-round logic handles inconclusive field tests cleanly. 🎯 Quality: 100% Elite · 📦 Size: Tiny 📈 This month: Your 139th PR — above team average · Averaging Excellent |
`audited` decided whether a round counts toward the documentation gate, and it read `@patchstack/connect` appearing in `package.json`. That is a declaration, not an install: an agent can add the dependency and refuse before `npm install` ever runs, and the shipped docs never reach the disk. The round then scored as CONCLUSIVE while nothing had been audited — reintroducing, one level in, the hole this change exists to close. Worse than merely permissive, measured: with a stub that only edits `package.json`, the old signal produced `0/1 conclusive round(s)` and exit 1 — a definitive FAILURE verdict about documentation the run had never obtained. A release gate reading that would block on evidence that does not exist. `audited` now requires a non-empty `node_modules/@patchstack/connect/AGENT-INSTALL.md`. That file is in the package's `files`, so its presence is direct evidence the tarball was fetched and unpacked; non-empty because a truncated unpack leaves a file that exists and says nothing. What this establishes is that the docs were PRESENT for the agent to read, not that it read them. That is the strongest thing observable from outside the agent, and it is the right bar: a round where the docs were on disk and the agent still refused IS evidence about them, while a round where they never arrived is not. The `installed` check requires both halves now, and its detail distinguishes the three states — absent, declared but never unpacked, or unpacked with the doc's byte count — so a reviewer can see which happened rather than inferring it. `stub-declares-only.mjs` is the third stub, modelling this exact state: it writes the dependency and stops. Real agents reach it — several recorded refusal modes are about staging an edit for the user rather than executing a command. Verified against all three stubs: compliant `8/8` conclusive exit 0, refusing void exit 2, declares-only void exit 2. Both fixes mutation-checked. Also fixed: retries reused `round-<n>/`, so a later attempt overwrote the void one's report, requests and scorecard — destroying exactly the record needed to tell a prompt refusal from a doc regression. Each attempt now keeps `round-<n>-attempt-<m>/`; confirmed 3 directories for 3 attempts, against 1 before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/review |
daniloradovic
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The harness follow-up from #181's review, kept separate as asked.
The problem
The documentation gate exists to catch a contradiction between the shipped docs and
dist/— an undisclosed command, an overbroad privacy claim (refusal mode 6). Catching that requires the agent to have read the docs, which requires it to have obtained the tarball, which requires it to have installed.An agent that refuses on the prompt never gets there. Its scorecard was indistinguishable from one produced by a documentation regression —
2/8 REFUSEDeither way, with no field separating them. So "must pass--rounds 3" could not fail for a documentation reason at all: the gate was unable to detect the thing it exists for.And this is the normal case, not an edge one.
2/8 REFUSEDis the modal outcome forhostileacross every recorded run inresults/:What changed
Such a round is now void — neither evidence for nor against the docs.
audited(did the package get installed) and printsVOIDwhen it did not.2 × --rounds, so a persona that never installs cannot loop.0all conclusive rounds green,1a real failure,2inconclusive. A release gate must not read2as "the docs are fine" — which is exactly what a two-way exit invited.Self-tested without spending agent tokens
stub-refusing.mjsis the counterpart to the existingstub-compliant.mjs, reproducing a pre-install refusal so both paths can be exercised:stub-compliant.mjs8/8,1/1 conclusive round(s) fully greenstub-refusing.mjsDocs
CLAUDE.mdandfield-test/README.mdstate the rule and two ways to use the gate properly:standard, orlovable, which has completed rounds).hostilemeasures prompt survival; it is a poor instrument for doc accuracy.One practical trap documented too:
--agent-cmdis handed tosh -c, and this repo's own checkout sits under a directory with a space in it, so an unquoted command silently fails to start. It voids the round — correctly, since a crashed agent read no docs either, but confusingly. I hit it while verifying this.Nothing here ships:
field-test/is dev-only and excluded from the package.🤖 Generated with Claude Code