Abadawi/send has image to router by abadawi591 · Pull Request #308321 · microsoft/vscode

abadawi591 · 2026-04-07T20:19:01Z

Delegate image routing to server-side auto-intent-service

The server-side auto-intent-service now handles has_image via a vision-aware pre-filter instead of returning HTTP 400. Remove the client-side hasImage early return in _tryRouterSelection() and send has_image: true in the router request body when images are present.

Changes

automodeService.ts: remove hasImage bail-out, pass hasImage to router
routerDecisionFetcher.ts: accept hasImage param, include has_image in request body, parse error code from 400 responses
automodeService.ts: handle no_vision_models error as a distinct fallback reason; _applyVisionFallback remains as safety net
automodeService.spec.ts: replace skip-router test with send-has_image test, add no_vision_models error handling test, update transient fallback test

The server-side auto-intent-service now handles has_image via a vision-aware pre-filter instead of returning HTTP 400. Remove the client-side hasImage early return in _tryRouterSelection() and send has_image: true in the router request body when images are present. Changes: - automodeService.ts: remove hasImage bail-out, pass hasImage to router - routerDecisionFetcher.ts: accept hasImage param, include has_image in request body, parse error code from 400 responses - automodeService.ts: handle no_vision_models error as a distinct fallback reason; _applyVisionFallback remains as safety net - automodeService.spec.ts: replace skip-router test with send-has_image test, add no_vision_models error handling test, update transient fallback test

Replace (err as any).errorCode pattern with a typed RouterDecisionError class that carries the errorCode as a proper property.

Add emptyPrompt and emptyCandidateList to the comment so telemetry documentation matches runtime behavior.

Verify the router was actually called and the error was logged with the noVisionModels fallback reason, not just that the model is gpt-4o.

Copilot

Pull request overview

This PR updates the Copilot extension’s auto-mode routing flow so image-bearing chat turns are routed server-side (by sending has_image) instead of being short-circuited client-side, and adds a dedicated fallback classification when the router reports no available vision models.

Changes:

Remove the client-side image-based router bail-out and always send has_image to the router when images are present.
Add RouterDecisionError with parsed error codes from HTTP 400 responses to enable more specific fallback reasons (e.g. noVisionModels).
Update automode routing tests to validate has_image behavior and no_vision_models fallback handling.

Show a summary per file

File	Description
extensions/copilot/src/platform/endpoint/node/automodeService.ts	Sends `hasImage` into router requests and maps `no_vision_models` to a distinct fallback reason.
extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts	Adds `has_image` to the router request body and introduces `RouterDecisionError` with parsed error codes.
extensions/copilot/src/platform/endpoint/node/test/automodeService.spec.ts	Updates/extends tests to cover routing for image turns and the `no_vision_models` fallback case.

Copilot's findings

Files reviewed: 3/3 changed files
Comments generated: 4

- Add this.name to RouterDecisionError - Fall back to raw errorText when error body is not JSON - Replace nested ternary with if/else for fallback reason - Pass server error codes through directly instead of mapping to client-side names, so future codes surface automatically - Add hasImage property to automode.routerFallback telemetry - Update GDPR annotation for new hasImage property

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…tcher.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…vice.spec.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR updates the Copilot extension’s “auto mode” routing flow to let the server-side auto-intent/router handle image-containing requests (via a has_image flag) rather than having the client skip routing when images are present.

Changes:

Remove the client-side early router bail-out for image requests and include has_image: true in router request bodies when references contain images.
Introduce RouterDecisionError to propagate a server-provided router error code and use it as a distinct router fallback reason.
Update automode unit tests to assert router invocation with has_image and to validate fallback behavior for no_vision_models.

Show a summary per file

File	Description
extensions/copilot/src/platform/endpoint/node/automodeService.ts	Sends `has_image` to the router, adds telemetry for `hasImage`, and treats server router error codes as fallback reasons.
extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts	Adds `has_image` request support and parses router error codes from non-OK responses via `RouterDecisionError`.
extensions/copilot/src/platform/endpoint/node/test/automodeService.spec.ts	Adjusts router/vision-related tests to validate `has_image` routing and `no_vision_models` fallback behavior.

Copilot's findings

Files reviewed: 3/3 changed files
Comments generated: 2

Only pass parsed errorCode to RouterDecisionError, not raw errorText. When the router returns a non-JSON body (e.g. HTML gateway timeout), the catch block in automodeService falls through to 'routerError' instead of emitting the raw response body as the telemetry reason. Add tests for non-JSON error body and JSON-without-error-field paths.

Both hasImage (this PR) and routingMethod (from main) modify the getRouterDecision signature and request body. Merged to include both parameters.

abadawi591 added 4 commits April 7, 2026 19:56

Fix no-explicit-any lint warnings

a55ff3a

Replace (err as any).errorCode pattern with a typed RouterDecisionError class that carries the errorCode as a proper property.

List all possible routerFallback reason values in GDPR annotation

9b084c3

Add emptyPrompt and emptyCandidateList to the comment so telemetry documentation matches runtime behavior.

Assert noVisionModels error classification in test

75af0f6

Verify the router was actually called and the error was logged with the noVisionModels fallback reason, not just that the model is gpt-4o.

Copilot AI review requested due to automatic review settings April 7, 2026 20:19

Copilot started reviewing on behalf of abadawi591 April 7, 2026 20:20 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

vs-code-engineering Bot assigned mjbvz Apr 7, 2026

abadawi591 and others added 5 commits April 9, 2026 20:24

Update extensions/copilot/src/platform/endpoint/node/automodeService.ts

4ad5560

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update extensions/copilot/src/platform/endpoint/node/routerDecisionFe…

d9a9133

…tcher.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update extensions/copilot/src/platform/endpoint/node/test/automodeSer…

7bff509

…vice.spec.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update extensions/copilot/src/platform/endpoint/node/test/automodeSer…

895f6a9

…vice.spec.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

bhavyaus requested a review from Copilot April 10, 2026 03:22

Copilot started reviewing on behalf of bhavyaus April 10, 2026 03:23 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Comment thread extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts

Comment thread extensions/copilot/src/platform/endpoint/node/automodeService.ts

bhavyaus reviewed Apr 10, 2026

View reviewed changes

Comment thread extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts

Comment thread extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts

abadawi591 and others added 3 commits April 10, 2026 18:53

Merge origin/main - resolve conflicts with routingMethod feature

b369df4

Both hasImage (this PR) and routingMethod (from main) modify the getRouterDecision signature and request body. Merged to include both parameters.

Merge branch 'main' into abadawi/send-has-image-to-router

6615f66

jinu-jang suggested changes Apr 10, 2026

View reviewed changes

Comment thread extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts

Comment thread extensions/copilot/src/platform/endpoint/node/routerDecisionFetcher.ts

Add JSDoc to RouterDecisionError

b2eee58

bhavyaus approved these changes Apr 13, 2026

View reviewed changes

roblourens approved these changes Apr 13, 2026

View reviewed changes

abadawi591 requested a review from jinu-jang April 15, 2026 00:38

jinu-jang approved these changes Apr 15, 2026

View reviewed changes

aashna merged commit 34754a8 into microsoft:main Apr 15, 2026
25 of 26 checks passed

vs-code-engineering Bot added this to the 1.117.0 milestone Apr 15, 2026

vs-code-engineering Bot locked and limited conversation to collaborators May 31, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Abadawi/send has image to router#308321

Abadawi/send has image to router#308321
aashna merged 13 commits into
microsoft:mainfrom
abadawi591:abadawi/send-has-image-to-router

abadawi591 commented Apr 7, 2026

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

7 participants

Uh oh!

Conversation

abadawi591 commented Apr 7, 2026

Delegate image routing to server-side auto-intent-service

Changes

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Labels

7 participants