Skip to content

Make js_repl image output controllable#13331

Merged
fjord-oai merged 1 commit intomainfrom
fjord/original_image_res---4hxjh7uv9pkj_8
Mar 4, 2026
Merged

Make js_repl image output controllable#13331
fjord-oai merged 1 commit intomainfrom
fjord/original_image_res---4hxjh7uv9pkj_8

Conversation

@fjord-oai
Copy link
Copy Markdown
Contributor

@fjord-oai fjord-oai commented Mar 3, 2026

Summary

Instead of always adding inner function call outputs to the model context, let js code decide which ones to return.

  • Stop auto-hoisting nested tool outputs from codex.tool(...) into the outer js_repl function output.
  • Keep codex.tool(...) return values unchanged as structured JS objects.
  • Add codex.emitImage(...) as the explicit path for attaching an image to the outer js_repl function output.
  • Support emitting from a direct image URL, a single input_image item, an explicit { bytes, mimeType } object, or a raw tool response object containing exactly one image.
  • Preserve existing view_image original-resolution behavior when JS emits the raw view_image tool result.
  • Suppress the special ViewImageToolCall event for js_repl-sourced view_image calls so nested inspection stays side-effect free until JS explicitly emits.
  • Update the js_repl docs and generated project instructions with both recommended patterns:
    • await codex.emitImage(codex.tool("view_image", { path }))
    • await codex.emitImage({ bytes: await page.screenshot({ type: "jpeg", quality: 85 }), mimeType: "image/jpeg" })

git stack

@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 05:34
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from bb627f5 to aa0708e Compare March 3, 2026 05:34
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 05:34
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 05:57
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from aa0708e to 19f2421 Compare March 3, 2026 05:57
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 05:57
@fjord-oai
Copy link
Copy Markdown
Contributor Author

@codex review

Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19f24217d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 16:14
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 19f2421 to 03e2f59 Compare March 3, 2026 16:15
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 16:15
Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03e2f59d28

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 16:29
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 03e2f59 to 3f1753d Compare March 3, 2026 16:30
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 16:30
Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f1753daea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 16:40
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 3f1753d to 879cf67 Compare March 3, 2026 16:40
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 16:40
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 19:52
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 879cf67 to 86ab8ea Compare March 3, 2026 19:52
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 19:52
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 20:34
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 86ab8ea to bb0fe78 Compare March 3, 2026 20:34
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 2f32ae2 to 6c3916e Compare March 3, 2026 21:58
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 21:58
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 22:30
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 6c3916e to 95e0034 Compare March 3, 2026 22:31
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 22:31
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 22:54
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 95e0034 to 2f438d7 Compare March 3, 2026 22:54
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 22:54
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 23:07
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 2f438d7 to 5af28d4 Compare March 3, 2026 23:07
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 23:07
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 23:19
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from 5af28d4 to c64b9e3 Compare March 3, 2026 23:20
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 23:20
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 23:24
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from c64b9e3 to dcd9586 Compare March 3, 2026 23:24
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 23:24
@fjord-oai fjord-oai changed the base branch from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 23:34
@fjord-oai fjord-oai force-pushed the fjord/original_image_res---4hxjh7uv9pkj_8 branch from dcd9586 to c6c0b72 Compare March 3, 2026 23:34
@fjord-oai fjord-oai changed the base branch from main to fjord/original_image_res---4hw7wu3447irye March 3, 2026 23:34
fjord-oai added a commit that referenced this pull request Mar 3, 2026
## Summary

Add original-resolution support for `view_image` behind the
under-development `view_image_original_resolution` feature flag.

When the flag is enabled and the target model is `gpt-5.3-codex` or
newer, `view_image` now preserves original PNG/JPEG/WebP bytes and sends
`detail: "original"` to the Responses API instead of using the legacy
resize/compress path.

## What changed

- Added `view_image_original_resolution` as an under-development feature
flag.
- Added `ImageDetail` to the protocol models and support for serializing
`detail: "original"` on tool-returned images.
- Added `PromptImageMode::Original` to `codex-utils-image`.
  - Preserves original PNG/JPEG/WebP bytes.
  - Keeps legacy behavior for the resize path.
- Updated `view_image` to:
- use the shared `local_image_content_items_with_label_number(...)`
helper in both code paths
  - select original-resolution mode only when:
    - the feature flag is enabled, and
    - the model slug parses as `gpt-5.3-codex` or newer
- Kept local user image attachments on the existing resize path; this
change is specific to `view_image`.
- Updated history/image accounting so only `detail: "original"` images
use the docs-based GPT-5 image cost calculation; legacy images still use
the old fixed estimate.
- Added JS REPL guidance, gated on the same feature flag, to prefer JPEG
at 85% quality unless lossless is required, while still allowing other
formats when explicitly requested.
- Updated tests and helper code that construct
`FunctionCallOutputContentItem::InputImage` to carry the new `detail`
field.

## Behavior

### Feature off
- `view_image` keeps the existing resize/re-encode behavior.
- History estimation keeps the existing fixed-cost heuristic.

### Feature on + `gpt-5.3-codex+`
- `view_image` sends original-resolution images with `detail:
"original"`.
- PNG/JPEG/WebP source bytes are preserved when possible.
- History estimation uses the GPT-5 docs-based image-cost calculation
for those `detail: "original"` images.


#### [git stack](https://github.com/magus/git-stack-cli)
- 👉 `1` #13050
- ⏳ `2` #13331
- ⏳ `3` #13049
Base automatically changed from fjord/original_image_res---4hw7wu3447irye to main March 3, 2026 23:56
git-stack-id: fjord/original_image_res---4hxjh7uv9pkj_8
git-stack-title: Make js_repl image output controllable
@fjord-oai fjord-oai merged commit c4cb594 into main Mar 4, 2026
31 checks passed
@fjord-oai fjord-oai deleted the fjord/original_image_res---4hxjh7uv9pkj_8 branch March 4, 2026 00:26
@github-actions github-actions bot locked and limited conversation to collaborators Mar 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

2 participants