[docs] ENG-3685 - Align the tested install prompt with what setup actually does - #167
Merged
Merged
Conversation
The prompt the field-test harness validates differed from the one README ships, on two points that are not stylistic. It said "dev dependency". README says the opposite and explains why: protection imports @patchstack/connect/protect at runtime, so a host that prunes dev dependencies breaks. Real Lovable projects carry the package in dependencies, so the tested prompt was teaching agents to produce an install that fails on exactly the platforms this package targets. It also authorized only "widget and production build hooks", while setup runs protect, which edits source. Effects exceeding what the prompt named is the documented refusal trigger, so the harness was under-testing the failure mode it exists to catch. Both now match README's wording. The hosted-workspace clauses stay out: the sandbox label and preview restart are specific to that flow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Aligned install prompt to reflect actual setup behavior for disclosure and runtime edits. 🎯 Quality: 100% Elite · 📦 Size: Tiny 📈 This month: Your 64th PR — above team average · Averaging Excellent |
Contributor
Author
|
/review |
daniloradovic
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes ENG-3685
CLAUDE.mdsays the install prompt "appears in three places that must stay identical:README.md,GETTING-STARTED.md, andfield-test/prompt.txt—prompt.txtis the artifact the harness tests." It has drifted, and two of the differences are functional rather than cosmetic.Wrong dependency type
The tested prompt said dev dependency.
README.md:21says the opposite and explains why:README.md:11accordingly says "regular dependency". Real Lovable projects carry it independencies. So the prompt the harness validates was teaching agents to produce an install that breaks on the platforms this package targets.Under-authorized effects
The tested prompt authorized "its widget and production build hooks".
setupalso runsprotect, which edits source — and since #165,scanedits source too.field-test/README.mdrecords that effects exceeding what the prompt named is the refusal trigger, and that the prompt "now authorizes one explicitsetupcommand and names its bounded effects inline" precisely to avoid it. An under-scoped authorization means the harness was under-testing the failure mode it exists to catch.Both points now match
README.md's wording. The hosted-workspace clauses stay out —PATCHSTACK_ENVIRONMENT=sandbox, "restart the preview server", and the sandbox non-persistence instruction are specific to that flow.Still open: are two variants intended?
This closes the functional gap, not the structural one. The remaining differences look deliberate —
README.md:11is a hosted-builder variant, andREADME.md:23documents that sandbox guidance on purpose.If two variants are wanted, the "must stay identical" line in
CLAUDE.mdis stale and should say so, and the harness should exercise both. As it stands the hosted variant — the one facing the refusal-prone audience — is the copy never tested. Worth a decision from someone who owns the testing strategy; I did not want to guess at it inside a correctness fix.Before merging
The field test has not been run.
CLAUDE.mdrequiresnode field-test/run.mjs --persona hostile --rounds 3for any prompt change, and unlike a CLI change this one is fully exercisable by it — the harness mocks the API and only the prompt shape is under test. It is runnable in this environment (claude -pagainstfield-test/mock-api.mjs); I left it for a human to trigger because it spawns nested agents with permissions skipped and costs 30–50k tokens per round.Given the change widens the authorization and corrects an install instruction, I would expect it to score no worse — but that is a prediction, not a result.
npm testunaffected: 1315 passed, 6 skipped. No code changed.🤖 Generated with Claude Code