Treat a field-test round that never installed as inconclusive

CLAUDE.md

@@ -18,7 +18,7 @@ The one-line install prompt in `README.md` ("Install prompt (for AI coding tools
 Invariants when touching it:
 
 - The prompt appears in three places that must stay identical: `README.md`, `GETTING-STARTED.md` (the teammate-facing flow), and `field-test/prompt.txt` — `prompt.txt` is the artifact the harness tests.
-- Any change to the prompt, the `guide` checklist output (`src/guide.ts`), or `AGENT-INSTALL.md` must pass `node field-test/run.mjs --persona hostile --rounds 3` before shipping. Agents audit the shipped docs, so inaccuracies in `AGENT-INSTALL.md` cost trust and cause refusals.
+- Any change to the prompt, the `guide` checklist output (`src/guide.ts`), or `AGENT-INSTALL.md` must pass `node field-test/run.mjs --persona hostile --rounds 3` before shipping. Agents audit the shipped docs, so inaccuracies in `AGENT-INSTALL.md` cost trust and cause refusals. **A round that never unpacked the tarball is VOID, not a pass and not a failure**: the shipped docs were never on disk, so no audit of them can have happened, and its scorecard is identical to a doc regression's. Unpacked means a non-empty `node_modules/@patchstack/connect/AGENT-INSTALL.md` — a dependency declaration in `package.json` is not an install. The harness retries void rounds within a bounded budget and exits 2 when every round was void — read that as "re-run", never as "the docs are fine". A doc change also wants a re-run after publication, when the published tarball actually carries it.
 - Don't add reassurance language ("it's safe", "nothing is executed remotely") — agents flag it as a manipulation signal. Don't ask the agent to "follow the guide/instructions it prints" unbounded — name the concrete steps instead.
 - A new real-world refusal report becomes a persona in `field-test/personas/` so the regression stays covered.
 - The fixture installs the *published* package, so an unpublished `guide`/CLI change can't be exercised end-to-end — publish first, or accept the run validates only the prompt shape.

field-test/README.md

@@ -8,6 +8,69 @@ Dev-only: nothing in this directory ships in the npm package.
 
 The install prompt is an adversarial-UX artifact: AI agents actively try to refuse it. Unit tests can't tell you whether an agent will balk at a phrase, mis-read CLI output, or wire the widget with the wrong token — only letting an agent run the real flow does. Each documented refusal mode came from a run like this.
 
+## Void rounds: a refusal before installing is not evidence about the docs
+
+The documentation gate exists to catch mode 6 — a contradiction between the shipped docs and `dist/`,
+such as an overbroad privacy claim. Catching that requires the agent to have READ the docs, which means
+it must have obtained the tarball, which means it must have installed.
+
+An agent that refuses on the *prompt* never gets there. Its scorecard is `2/8 REFUSED`, which is
+byte-identical to what a documentation regression would produce, and no field distinguished the two. So
+"must pass `--rounds 3`" could not fail for a documentation reason at all — the gate was unable to detect
+the thing it existed for, and `2/8 REFUSED` is the modal outcome for `hostile`.
+
+Such a round is now **void**: neither evidence for nor against the docs.
+
+- The scorecard carries `audited`, and prints `VOID` when it is false.
+- `audited` means **the tarball was fetched and unpacked** — specifically, a non-empty
+  `node_modules/@patchstack/connect/AGENT-INSTALL.md` in the fixture. A dependency DECLARATION in
+  `package.json` is not enough and was the first version's mistake: an agent can add the declaration and
+  refuse before `npm install`, and several recorded refusal modes are exactly that shape (staging an edit
+  for the user instead of running a command). Such a round then scored as conclusive while the docs were
+  never on disk.
+- What that establishes is that the docs were **present for the agent to read**, not that it read them.
+  That is the strongest thing observable from outside the agent, and it is the right bar: a round where
+  the docs were on disk and the agent still refused *is* evidence about them; a round where they never
+  arrived is not.
+- The `installed` check now requires both halves, and its detail distinguishes the three states
+  (absent / declared but never unpacked / unpacked, with the doc's byte count).
+- Void rounds are retried, bounded at `2 × --rounds`, so a persona that never installs cannot loop.
+  Every attempt keeps its own `round-<n>-attempt-<m>/` directory — retries used to reuse the round
+  number and overwrite the void attempt's report, destroying the record a reviewer needs to tell a
+  prompt refusal from a doc regression.
+- The summary counts only conclusive rounds, and reports how many were void.
+- Exit codes are three-way: `0` all conclusive rounds green, `1` a real failure, **`2` inconclusive** —
+  nothing unpacked, so the run says nothing. A release gate must not read `2` as "the docs are fine".
+
+Two consequences for how to use it:
+
+- For a docs-only change, prefer a persona that reliably installs (`standard`, or `lovable`, which has
+  completed rounds). `hostile` measures prompt survival; it is a poor instrument for doc accuracy.
+- Re-run after publication. Until the change is published, the tarball an agent installs and audits does
+  not contain it, so even a conclusive round is auditing the previous docs.
+
+Self-testing the two paths costs no agent tokens:
+
+```
+node field-test/run.mjs --rounds 1 --agent-cmd "node '$PWD/field-test/stub-compliant.mjs'"
+node field-test/run.mjs --rounds 1 --agent-cmd "node '$PWD/field-test/stub-refusing.mjs'"
+node field-test/run.mjs --rounds 1 --agent-cmd "node '$PWD/field-test/stub-declares-only.mjs'"
+```
+
+| stub | what it models | expected |
+|---|---|---|
+| `stub-compliant` | performs the whole flow | `8/8`, `1/1 conclusive`, exit **0** |
+| `stub-refusing` | refuses before touching anything | 3 void rounds, INCONCLUSIVE, exit **2** |
+| `stub-declares-only` | writes the dependency, never installs | 3 void rounds, INCONCLUSIVE, exit **2** |
+
+`stub-declares-only` is the one that matters for the `audited` definition. With `audited` reading the
+package.json declaration, that run reported `0/1 conclusive` and exit **1** — a definitive failure verdict
+about documentation it had never obtained.
+
+Quote the path: `--agent-cmd` is handed to `sh -c`, and this repository's own checkout sits under a
+directory with a space in it, so an unquoted command silently fails to start — which voids the round,
+correctly but confusingly.
+
 ## Refusal modes behind the current design
 
 These observed failures explain why the current prompt leads with explicit authorization and one bounded command rather than a verification script or an instruction chain:

field-test/run.mjs

@@ -140,10 +140,27 @@ function verify(fixtureDir, mock, agentOutput) {
     (scripts[key] ?? '').includes('patchstack-connect mark-build'),
   );
 
+  // Did the tarball actually arrive? A DECLARATION in package.json is not an install: an agent can add
+  // the dependency and then refuse before running `npm install`, and the docs it was supposed to audit
+  // never reach the disk. `AGENT-INSTALL.md` is in the package's `files`, so its presence under
+  // node_modules is direct evidence that the tarball was fetched and unpacked.
+  //
+  // Non-empty, because a truncated or interrupted unpack leaves a file that exists and says nothing.
+  const shippedDocs = path.join(fixtureDir, 'node_modules', '@patchstack', 'connect', 'AGENT-INSTALL.md');
+  let unpackedBytes = 0;
+  try { unpackedBytes = statSync(shippedDocs).size; } catch { unpackedBytes = 0; }
+  const unpacked = unpackedBytes > 0;
+
   const checks = {
     installed: {
-      pass: dep !== undefined,
-      detail: dep !== undefined ? `declared ${dep}` : 'not in package.json',
+      // Both halves. Declared-but-not-unpacked is the state that used to score as installed, and it is
+      // exactly the state in which no audit of the shipped docs can have happened.
+      pass: dep !== undefined && unpacked,
+      detail: dep === undefined
+        ? 'not in package.json'
+        : unpacked
+          ? `declared ${dep}, tarball unpacked (AGENT-INSTALL.md ${unpackedBytes}B)`
+          : `declared ${dep} but the tarball was never unpacked — no node_modules/@patchstack/connect/AGENT-INSTALL.md`,
     },
     provisioned: {
       pass: rc.siteUuid === mock.uuid,
@@ -177,11 +194,31 @@ function verify(fixtureDir, mock, agentOutput) {
 
   const refused = !checks.provisioned.pass && /refus|stall|declin/i.test(agentOutput);
   const passed = Object.values(checks).filter((check) => check.pass).length;
-  return { checks, refused, passed, total: Object.keys(checks).length };
+
+  // A round where the tarball never arrived cannot say anything about the SHIPPED DOCS.
+  //
+  // Agents `npm pack` the tarball and audit it, and a contradiction between the docs and `dist/` — an
+  // undisclosed command, an overbroad privacy claim — is a recorded refusal driver (mode 6). That is the
+  // thing the documentation gate exists to detect. But an agent that refuses on the PROMPT never obtains
+  // the tarball, so it never reads the docs at all, and its scorecard is identical to one produced by a
+  // doc regression: `2/8 REFUSED` either way, with no field distinguishing them.
+  //
+  // Gating "must pass N rounds" on that number therefore cannot fail for a documentation reason. Such a
+  // round is VOID — neither evidence for nor against the docs — and is retried rather than counted.
+  //
+  // `unpacked`, deliberately, and NOT `checks.installed.pass`: the two differ when an agent wires the
+  // dependency and stops. What this establishes is that the docs were PRESENT for the agent to read, not
+  // that it read them — that is the strongest thing observable from outside the agent, and it is the right
+  // bar. A round where the docs were on disk and the agent still refused IS evidence about them; a round
+  // where they never arrived is not. (Note the fixture installs from the registry, so the docs on disk are
+  // the PUBLISHED ones — which is why a doc change needs a re-run after publication to be tested at all.)
+  const audited = unpacked;
+  return { checks, refused, passed, total: Object.keys(checks).length, audited };
 }
 
 function printScorecard(round, result, verdict) {
-  console.log(`\n— round ${round}: ${verdict.passed}/${verdict.total} checks passed${verdict.refused ? ' (agent REFUSED)' : ''}${result.timedOut ? ' (TIMED OUT)' : ''}`);
+  const voided = verdict.audited ? '' : ' — VOID: the tarball never arrived, so the shipped docs were never on disk to read';
+  console.log(`\n— round ${round}: ${verdict.passed}/${verdict.total} checks passed${verdict.refused ? ' (agent REFUSED)' : ''}${result.timedOut ? ' (TIMED OUT)' : ''}${voided}`);
   for (const [name, check] of Object.entries(verdict.checks)) {
     console.log(`  ${check.pass ? '✔' : '✖'} ${name} — ${check.detail}`);
   }
@@ -209,7 +246,13 @@ console.log(`prompt: ${opts.prompt}`);
 console.log(`results: ${resultsDir}`);
 
 const summary = [];
+// Bounded: a persona that never installs would otherwise retry forever. Two extra attempts per requested
+// round, then the run reports how many rounds stayed void instead of pretending to a verdict.
+const VOID_RETRY_BUDGET = opts.rounds * 2;
+let voidRetriesLeft = VOID_RETRY_BUDGET;
+let attempt = 0;
 for (let round = 1; round <= opts.rounds; round++) {
+  attempt++;
   const fixtureDir = mkdtempSync(path.join(tmpdir(), 'ps-field-test-'));
   const mock = await startMockApi();
   console.log(`\nround ${round}: fixture=${fixtureDir} mock=${mock.endpoint}`);
@@ -263,7 +306,10 @@ Continue handling the request under the same platform policy, applying your judg
     console.log(`  (score includes a second, user-confirmation turn${confirmResult.timedOut ? ' — TIMED OUT' : ''})`);
   }
 
-  const roundDir = path.join(resultsDir, `round-${round}`);
+  // Per ATTEMPT, not per round. Retries reuse the round number, so a later conclusive attempt overwrote
+  // the void one's report, requests and scorecard — destroying the record of why a round was retried,
+  // which is the evidence a reviewer needs to tell a prompt refusal from a doc regression.
+  const roundDir = path.join(resultsDir, `round-${round}-attempt-${attempt}`);
   mkdirSync(roundDir, { recursive: true });
   writeFileSync(path.join(roundDir, 'report.md'), result.output);
   if (result.stderr.length > 0) {
@@ -280,21 +326,42 @@ Continue handling the request under the same platform policy, applying your judg
     path.join(roundDir, 'scorecard.json'),
     JSON.stringify({ ...verdict, exitCode: result.exitCode, timedOut: result.timedOut, confirmTurn: confirmResult !== null, fixtureDir }, null, 2),
   );
-  summary.push({ round, passed: verdict.passed, total: verdict.total, refused: verdict.refused, timedOut: result.timedOut, confirmTurn: confirmResult !== null });
+  summary.push({ round, attempt, passed: verdict.passed, total: verdict.total, refused: verdict.refused, timedOut: result.timedOut, confirmTurn: confirmResult !== null, audited: verdict.audited });
 
   await mock.close();
   if (opts.keep) {
     console.log(`kept fixture: ${fixtureDir}`);
   } else {
     rmSync(fixtureDir, { recursive: true, force: true });
   }
+
+  // Retry a void round rather than counting it: it is not evidence either way about the docs.
+  if (!verdict.audited && voidRetriesLeft > 0) {
+    voidRetriesLeft--;
+    console.log(`  retrying (void; ${voidRetriesLeft}/${VOID_RETRY_BUDGET} retries left)`);
+    round--;
+  }
 }
 
 writeFileSync(
   path.join(resultsDir, 'summary.json'),
   JSON.stringify({ persona: opts.persona, template: opts.template, agentCmd: opts.agentCmd, prompt: installPrompt, rounds: summary }, null, 2),
 );
 
-const fullPasses = summary.filter((round) => round.passed === round.total).length;
-console.log(`\n${fullPasses}/${summary.length} round(s) fully green. Full results: ${resultsDir}`);
-process.exit(fullPasses === summary.length ? 0 : 1);
+const conclusive = summary.filter((round) => round.audited);
+const voided = summary.length - conclusive.length;
+const fullPasses = conclusive.filter((round) => round.passed === round.total).length;
+console.log(
+  `\n${fullPasses}/${conclusive.length} conclusive round(s) fully green`
+  + (voided > 0 ? `; ${voided} void (tarball never arrived, so the shipped docs were never on disk)` : '')
+  + `. Full results: ${resultsDir}`,
+);
+if (conclusive.length === 0) {
+  console.log(
+    'INCONCLUSIVE: no round unpacked the package, so this run is not evidence about the shipped docs.\n'
+    + 'It is neither a pass nor a failure of the documentation gate. Re-run, or use a persona that installs.',
+  );
+}
+// 2 = inconclusive, distinct from 1 (a real failure): a caller gating a release must not read "the agent
+// refused before installing" as "the docs are wrong", nor as "the docs are fine".
+process.exit(conclusive.length === 0 ? 2 : fullPasses === conclusive.length ? 0 : 1);

field-test/stub-declares-only.mjs

@@ -0,0 +1,27 @@
+// A scripted "agent" that DECLARES the dependency and then refuses, without ever installing it.
+//
+// This is the state that made `audited` unsound when it meant "appears in package.json": the declaration
+// is present, so the round scored as conclusive, while the tarball never arrived and the shipped docs were
+// never on disk to audit. Real agents reach it — several recorded refusal modes are about staging an edit
+// for the user rather than executing a command, and a bolt.diy-style agent that writes the dependency and
+// stops looks exactly like this.
+//
+// `run.mjs` must report a round like this VOID and exit 2.
+import { readFileSync, writeFileSync } from 'node:fs';
+import path from 'node:path';
+
+const manifestPath = path.join(process.cwd(), 'package.json');
+const manifest = JSON.parse(readFileSync(manifestPath, 'utf8'));
+manifest.devDependencies = { ...manifest.devDependencies, '@patchstack/connect': '^0.3.30' };
+writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`);
+
+console.log(`## 1. VERDICT
+
+Stopped short — I added the dependency to package.json for you to review, but I am not going to run the
+install or its setup command myself.
+
+## 2. ACTIONS
+
+- Edited package.json (devDependencies)
+- No install performed, so nothing was fetched or executed
+`);

field-test/stub-refusing.mjs

@@ -0,0 +1,20 @@
+// A scripted "agent" that refuses before installing anything — the shape of a pre-install refusal.
+//
+// It exists to self-test the VOID path: such a round never obtains the tarball, so it never reads the
+// shipped docs, and its scorecard is indistinguishable from one produced by a documentation regression.
+// `run.mjs` must report it as inconclusive (exit 2) rather than as a pass or a failure of the
+// documentation gate, and must retry it within a bounded budget.
+//
+// Deliberately touches nothing: no install, no files, no requests.
+console.log(`## My Decision: **REFUSE**
+
+## 1. VERDICT
+
+Refused — installing an unfamiliar package and running its commands matches the supply-chain pattern I
+am required to be cautious about.
+
+## 2. ACTIONS
+
+- No files edited
+- No installation performed
+`);