Say what the field-test harness establishes, and what it does not - #193
Merged
Conversation
The harness is structural regression coverage for the install prompt: a green run means the prompt still survives pressures that once broke it. The personas are synthetic, written from our own analysis rather than from any platform's policy text, so a green run is not evidence that a live platform accepts the prompt and must not be reported as though it were. A hosted product is (prompt x model x runtime x UI) and this reproduces one of the four. That scope was stated in `field-test/README.md` and missing from the two documents that tell a contributor to rely on the gate. It now appears in all three, in the same terms. Also drops the pointer to a private evaluation repository, which is not being created. High-fidelity evaluation against real policy text would need one; nobody has a standing need for it, and an unbuilt plan referenced in shipped docs reads as a commitment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Documentation clarifies the field-test harness’s scope and limitations precisely. 🎯 Quality: 97% Elite · 📦 Size: Small 📈 This month: Your 150th PR — above team average · Averaging Elite |
Contributor
Author
|
/review |
daniloradovic
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only. Three files, no behaviour change.
What changed
field-test/README.mdopens with what the harness is: structural regression coverage for the install prompt. A green run means the prompt still survives pressures that once broke it — and the value in that is real, because every pressure encoded there produced an actual refusal, and every clause in the prompt exists because a run like it failed without one.What a green run is not is evidence that a live platform accepts the prompt. The personas are synthetic, written from our own analysis rather than from any platform's policy text, and a hosted product is (prompt × model × runtime × UI) — this reproduces one of the four. The README said as much already; the two documents that tell a contributor to rely on the gate did not, so
MAINTAINING.mdandCONTRIBUTING.mdnow say it in the same terms.The pointer to a future private evaluation repository is gone. High-fidelity evaluation against real policy text would need one, nobody has a standing need for it, and an unbuilt plan referenced in shipped docs reads as a commitment.
Why bother
The harness is a merge gate for prompt changes, so how strong it is taken to be decides how much weight a green run carries. A gate described as validating the prompt against platforms invites treating a field refusal as surprising, when the correct response is to add coverage — which is what the docs now say to do.