Skip to content

Say what the field-test harness establishes, and what it does not - #193

Merged
patchstackdave merged 1 commit into
mainfrom
docs/scope-the-field-test-claim
Aug 28, 2026
Merged

Say what the field-test harness establishes, and what it does not#193
patchstackdave merged 1 commit into
mainfrom
docs/scope-the-field-test-claim

Conversation

@patchstackdave

Copy link
Copy Markdown
Contributor

Docs only. Three files, no behaviour change.

What changed

field-test/README.md opens with what the harness is: structural regression coverage for the install prompt. A green run means the prompt still survives pressures that once broke it — and the value in that is real, because every pressure encoded there produced an actual refusal, and every clause in the prompt exists because a run like it failed without one.

What a green run is not is evidence that a live platform accepts the prompt. The personas are synthetic, written from our own analysis rather than from any platform's policy text, and a hosted product is (prompt × model × runtime × UI) — this reproduces one of the four. The README said as much already; the two documents that tell a contributor to rely on the gate did not, so MAINTAINING.md and CONTRIBUTING.md now say it in the same terms.

The pointer to a future private evaluation repository is gone. High-fidelity evaluation against real policy text would need one, nobody has a standing need for it, and an unbuilt plan referenced in shipped docs reads as a commitment.

Why bother

The harness is a merge gate for prompt changes, so how strong it is taken to be decides how much weight a green run carries. A gate described as validating the prompt against platforms invites treating a field refusal as surprising, when the correct response is to add coverage — which is what the docs now say to do.

The harness is structural regression coverage for the install prompt: a green run
means the prompt still survives pressures that once broke it. The personas are
synthetic, written from our own analysis rather than from any platform's policy
text, so a green run is not evidence that a live platform accepts the prompt and
must not be reported as though it were. A hosted product is
(prompt x model x runtime x UI) and this reproduces one of the four.

That scope was stated in `field-test/README.md` and missing from the two documents
that tell a contributor to rely on the gate. It now appears in all three, in the
same terms.

Also drops the pointer to a private evaluation repository, which is not being
created. High-fidelity evaluation against real policy text would need one; nobody
has a standing need for it, and an unbuilt plan referenced in shipped docs reads
as a commitment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderbuds

coderbuds Bot commented Aug 28, 2026

Copy link
Copy Markdown

Documentation clarifies the field-test harness’s scope and limitations precisely.

🎯 Quality: 97% Elite · 📦 Size: Small

📈 This month: Your 150th PR — above team average · Averaging Elite

See how your team is trending →

@patchstackdave

Copy link
Copy Markdown
Contributor Author

/review

@patchstackdave
patchstackdave merged commit 019cf97 into main Aug 28, 2026
14 checks passed
@patchstackdave
patchstackdave deleted the docs/scope-the-field-test-claim branch August 28, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants