Skip to content

[codex] Refine GUI platform runtime and docs#22

Draft
s-JoL wants to merge 5 commits intomainfrom
codex/gui-runtime-cleanup
Draft

[codex] Refine GUI platform runtime and docs#22
s-JoL wants to merge 5 commits intomainfrom
codex/gui-runtime-cleanup

Conversation

@s-JoL
Copy link
Copy Markdown
Contributor

@s-JoL s-JoL commented Mar 31, 2026

What changed

This PR consolidates the recent GUI/runtime work into a cleaner platform split, restores the real-GUI grounding path, and updates docs to match the current implementation.

Key changes:

  • split the GUI runtime into explicit macOS and Windows platform backends
  • add GUI action session management and a physical resource lock for native GUI actions
  • add a PowerShell-backed Windows MVP for observe/wait/key/type/click/drag/scroll/move
  • wire abort handling through Windows input dispatch
  • harden tool/process/grounding prompt handling and add cross-platform helper scripts
  • update README, Chinese README, and packages/gui docs so platform support reflects the current code

Why

The previous implementation had started to grow platform-specific behavior inside a single runtime path, which made capability reporting, cleanup, and Windows support inconsistent. This change makes the platform boundary explicit, fixes the real GUI grounding regression we found during review, and removes outdated documentation that still said Windows GUI support was only planned.

Impact

  • macOS keeps the full screenshot-grounded GUI path and the real-grounding tests are no longer blocked by the new physical lock in test setup
  • Windows now has a documented GUI MVP instead of partial or misleading capability claims
  • CLI/package scripts use cross-platform helpers instead of shell-specific env assignment and cleanup commands

Validation

  • pnpm lint (passes with 5 pre-existing warnings outside this PR's changed areas)
  • pnpm -C /Users/songliang/workspace/Understudy/understudy --filter @understudy/gui run typecheck
  • pnpm -C /Users/songliang/workspace/Understudy/understudy exec vitest --run packages/gui/src/__tests__/platform.test.ts packages/gui/src/__tests__/runtime.test.ts --testNamePattern="Windows|win32"
  • Earlier in this change series, pnpm test:gui:real:grounding was run successfully after the macOS regression fixes. A final rerun from the current active desktop session did not complete promptly, so it was not used as the final gate for this PR.

Notes

Windows still does not support windowSelector.index or teach-by-demonstration recording. The backend is now an explicit MVP rather than a claim of full parity with macOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant