fix: match the tap hint to the screenshot's orientation

[nongvantinh]

Hi! This is a small fix for #453.

What was going on

When a landscape app is in front on a phone that's normally portrait, the screenshot comes back landscape (for example 1024×461), but the screen size from mobilecli device info is still the phone's natural portrait size (720×1600). describeCoordinateMapping then scales x and y by very different factors (0.703 and 3.471 here), so an agent that follows the hint taps the wrong spot. Sometimes the spot is off the screen entirely.

The fix

Before working out the ratios, describeCoordinateMapping now checks whether the screenshot and the reported screen size have the same orientation. If they don't, it swaps the screen's width and height. The screenshot always shows the display as it's rotated right now, and taps land in that rotation's coordinates, so it's the more trustworthy of the two.

It compares orientation, not exact sizes, so the existing iOS case (a pixel screenshot of a screen measured in points) keeps working.

Tests

I added four cases to test/coordinate-mapping.ts:

• a landscape screenshot with a portrait-reported screen (the Tap hint uses portrait screen size when a landscape app is in front #453 case);
• a full-size landscape screenshot, which now correctly says "coordinates match";
• the iOS points case, rotated;
• a screen that's already reported landscape, which is left untouched.

npm run lint and npm run build are clean, and all the tests that don't need a device pass locally (68, including the new ones).

Checked on a real device

I tried it on a Samsung Galaxy A04 (Android 14) running a landscape-locked Godot game:

• The hint now reads Screen coordinates are 1600x720 … multiply its x by 1.563 and y by 1.562.
• Tapping where it says, (800, 347), hit the game's Play button and opened the next screen.
• With the old factors the same tap would have gone to (360, 771), below the bottom of a 720-pixel-tall screen.

I didn't run the Android device suite in test/android.ts. It exercises the legacy robot rather than this code, and it changes the phone's rotation setting, which I'd rather not do on a personal phone.

What this doesn't touch

The underlying screen size from mobilecli is still the portrait one. So mobile_get_screen_size, mobile_get_orientation (which says "portrait" here), and a swipe() called without coordinates, which centres on the reported height, probably still see the unrotated values. Fixing that properly probably belongs in mobilecli itself. I kept this PR to the hint so it stays small, but I'm happy to look at the rest separately if that would help.

Thanks for taking a look!

Fixes #453

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 8c04707f-fcfa-41af-a782-7562de675a3a

📥 Commits

Reviewing files that changed from the base of the PR and between 18d0e8c and e9d9c4e.

📒 Files selected for processing (2)

• src/coordinate-mapping.ts
• test/coordinate-mapping.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

Walkthrough

describeCoordinateMapping now accepts reported screen dimensions and reorients them to match the screenshot when their orientations differ. Coordinate comparison and scaling use the oriented dimensions. Four tests cover rotated dimensions, coordinate matches, point-to-pixel scaling, and matching landscape orientations.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to e9d9c

The orientation-aware tap hint change has no identified merge-blocking issue; it is ready for normal checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: aligning tap hints with the screenshot orientation.
Description check	✅ Passed	The description explains the orientation mismatch, the implementation, added tests, validation results, and remaining scope. It directly matches the changeset.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}