Skip to content

[docs] ENG-3685 - Align the tested install prompt with what setup actually does - #167

Merged
mariojgt merged 1 commit into
mainfrom
mariot/eng-3685-align-tested-install-prompt
Aug 25, 2026
Merged

[docs] ENG-3685 - Align the tested install prompt with what setup actually does#167
mariojgt merged 1 commit into
mainfrom
mariot/eng-3685-align-tested-install-prompt

Conversation

@mariojgt

Copy link
Copy Markdown
Contributor

Closes ENG-3685

CLAUDE.md says the install prompt "appears in three places that must stay identical: README.md, GETTING-STARTED.md, and field-test/prompt.txtprompt.txt is the artifact the harness tests." It has drifted, and two of the differences are functional rather than cosmetic.

Wrong dependency type

The tested prompt said dev dependency. README.md:21 says the opposite and explains why:

Protection imports @patchstack/connect/protect at runtime, so deployments that prune dev dependencies need the package in dependencies.

README.md:11 accordingly says "regular dependency". Real Lovable projects carry it in dependencies. So the prompt the harness validates was teaching agents to produce an install that breaks on the platforms this package targets.

Under-authorized effects

The tested prompt authorized "its widget and production build hooks". setup also runs protect, which edits source — and since #165, scan edits source too.

field-test/README.md records that effects exceeding what the prompt named is the refusal trigger, and that the prompt "now authorizes one explicit setup command and names its bounded effects inline" precisely to avoid it. An under-scoped authorization means the harness was under-testing the failure mode it exists to catch.

Both points now match README.md's wording. The hosted-workspace clauses stay out — PATCHSTACK_ENVIRONMENT=sandbox, "restart the preview server", and the sandbox non-persistence instruction are specific to that flow.

Still open: are two variants intended?

This closes the functional gap, not the structural one. The remaining differences look deliberate — README.md:11 is a hosted-builder variant, and README.md:23 documents that sandbox guidance on purpose.

If two variants are wanted, the "must stay identical" line in CLAUDE.md is stale and should say so, and the harness should exercise both. As it stands the hosted variant — the one facing the refusal-prone audience — is the copy never tested. Worth a decision from someone who owns the testing strategy; I did not want to guess at it inside a correctness fix.

Before merging

The field test has not been run. CLAUDE.md requires node field-test/run.mjs --persona hostile --rounds 3 for any prompt change, and unlike a CLI change this one is fully exercisable by it — the harness mocks the API and only the prompt shape is under test. It is runnable in this environment (claude -p against field-test/mock-api.mjs); I left it for a human to trigger because it spawns nested agents with permissions skipped and costs 30–50k tokens per round.

Given the change widens the authorization and corrects an install instruction, I would expect it to score no worse — but that is a prediction, not a result.

npm test unaffected: 1315 passed, 6 skipped. No code changed.

🤖 Generated with Claude Code

The prompt the field-test harness validates differed from the one README
ships, on two points that are not stylistic.

It said "dev dependency". README says the opposite and explains why:
protection imports @patchstack/connect/protect at runtime, so a host that
prunes dev dependencies breaks. Real Lovable projects carry the package in
dependencies, so the tested prompt was teaching agents to produce an install
that fails on exactly the platforms this package targets.

It also authorized only "widget and production build hooks", while setup runs
protect, which edits source. Effects exceeding what the prompt named is the
documented refusal trigger, so the harness was under-testing the failure mode
it exists to catch.

Both now match README's wording. The hosted-workspace clauses stay out: the
sandbox label and preview restart are specific to that flow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderbuds

coderbuds Bot commented Aug 24, 2026

Copy link
Copy Markdown

Aligned install prompt to reflect actual setup behavior for disclosure and runtime edits.

🎯 Quality: 100% Elite · 📦 Size: Tiny

📈 This month: Your 64th PR — above team average · Averaging Excellent

See how your team is trending →

@mariojgt

Copy link
Copy Markdown
Contributor Author

/review

@mariojgt
mariojgt merged commit 797d104 into main Aug 25, 2026
6 checks passed
@mariojgt
mariojgt deleted the mariot/eng-3685-align-tested-install-prompt branch August 25, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants