login: add custom CA support for login flows

[joshka-oai]

Stacked PRs

This work is split across three stacked PRs:

• login: add custom CA support for login flows #14178: add custom CA support for browser and device-code login flows, docs, and hermetic subprocess tests
• client: extend custom CA handling across HTTPS and websocket clients #14239: broaden the shared custom CA path from login to other outbound reqwest clients across Codex
• client: apply custom CA handling to websocket TLS #14240: extend that shared custom CA handling to secure websocket TLS so websocket connections honor the same CA env vars

Review order: #14178, then #14239, then #14240.

Supersedes #6864.

Thanks to @3axap4eHko for the original implementation and investigation here. Although this version rearranges the code and history significantly, the majority of the credit for this work belongs to them.

Problem

Login flows need to work in enterprise environments where outbound TLS is intercepted by an internal proxy or gateway. In those setups, system root certificates alone are often insufficient to validate the OAuth and device-code endpoints used during login. The change adds a login-specific custom CA loading path, but the important contracts around env precedence, PEM compatibility, test boundaries, and probe-only workarounds need to be explicit so reviewers can understand what behavior is intentional.

For users and operators, the behavior is simple: if login needs to trust a custom root CA, set CODEX_CA_CERTIFICATE to a PEM file containing one or more certificates. If that variable is unset, login falls back to SSL_CERT_FILE. If neither is set, login uses system roots. Invalid or empty PEM files now fail with an error that points back to those environment variables and explains how to recover.

What This Delivers

Users can now make Codex login work behind enterprise TLS interception by pointing CODEX_CA_CERTIFICATE at a PEM bundle containing the relevant root certificates. If that variable is unset, login falls back to SSL_CERT_FILE, then to system roots.

This PR applies that behavior to both browser-based and device-code login flows. It also makes login tolerant of the PEM shapes operators actually have in hand: multi-certificate bundles, OpenSSL TRUSTED CERTIFICATE labels, and bundles that include well-formed CRLs.

Mental model

codex-login is the place where the login flows construct ad hoc outbound HTTP clients. That makes it the right boundary for a narrow CA policy: look for CODEX_CA_CERTIFICATE, fall back to SSL_CERT_FILE, load every parseable certificate block in that bundle into a reqwest::Client, and fail early with a clear user-facing error if the bundle is unreadable or malformed.

The implementation is intentionally pragmatic about PEM input shape. It accepts ordinary certificate bundles, multi-certificate bundles, OpenSSL TRUSTED CERTIFICATE labels, and bundles that also contain CRLs. It does not validate a certificate chain or prove a handshake; it only constructs the root store used by login.

Non-goals

This change does not introduce a general-purpose transport abstraction for the rest of the product. It does not validate whether the provided bundle forms a real chain, and it does not add handshake-level integration tests against a live TLS server. It also does not change login state management or OAuth semantics beyond ensuring the existing flows share the same CA-loading rules.

Tradeoffs

The main tradeoff is keeping this logic scoped to login-specific client construction rather than lifting it into a broader shared HTTP layer. That keeps the review surface smaller, but it also means future login-adjacent code must continue to use build_login_http_client() or it can silently bypass enterprise CA overrides.

The TRUSTED CERTIFICATE handling is also intentionally a local compatibility shim. The rustls ecosystem does not currently accept that PEM label upstream, so the code normalizes it locally and trims the OpenSSL X509_AUX trailer bytes down to the certificate DER that reqwest can consume.

Architecture

custom_ca.rs is now the single place that owns login CA behavior. It selects the CA file from the environment, reads it, normalizes PEM label shape where needed, iterates mixed PEM sections with rustls-pki-types, ignores CRLs, trims OpenSSL trust metadata when necessary, and returns either a configured reqwest::Client or a typed error.

The browser login server and the device-code flow both call build_login_http_client(), so they share the same trust-store policy. Environment-sensitive tests run through the login_ca_probe helper binary because those tests must control process-wide env vars and cannot reliably build a real reqwest client in-process on macOS seatbelt runs.

Observability

The custom CA path logs which environment variable selected the bundle, which file path was loaded, how many certificates were accepted, when TRUSTED CERTIFICATE labels were normalized, when CRLs were ignored, and where client construction failed. Returned errors remain user-facing and include the relevant path, env var, and remediation hint.

This gives enough signal for three audiences:

• users can see why login failed and which env/file caused it
• sysadmins can confirm which override actually won
• developers can tell whether the failure happened during file read, PEM parsing, certificate registration, or final reqwest client construction

Tests

Pure unit tests stay limited to env precedence and empty-value handling. Real client construction lives in subprocess tests so the suite remains hermetic with respect to process env and macOS sandbox behavior.

The subprocess tests verify:

• CODEX_CA_CERTIFICATE precedence over SSL_CERT_FILE
• fallback to SSL_CERT_FILE
• single-certificate and multi-certificate bundles
• malformed and empty-bundle errors
• OpenSSL TRUSTED CERTIFICATE handling
• CRL tolerance for well-formed CRL sections

The named PEM fixtures under login/tests/fixtures/ are shared by the tests so their purpose stays reviewable.

[joshka-oai]

Comparison to the original PR (#6864), mainly to make review easier for @3axap4eHko:

The main behavioral intent is still the same: login can use a custom CA bundle via CODEX_CA_CERTIFICATE, with fallback to SSL_CERT_FILE, and the login flows share that client-construction path.

The biggest structural change is that most of the CA-specific code now lives in a dedicated custom_ca.rs module instead of adding a few hundred lines to server.rs. That was mainly to keep the login server code focused on the server flow and make the CA-loading logic more additive and easier to review in isolation.

Relative to the original PR, this version also leans harder into robustness and observability:

• the PEM fixtures were moved into named files and made shared across tests
• custom CA parsing now has a typed error surface and clearer user-facing failure messages
• the CA-loading path logs enough context to diagnose which env var/file won and where loading/building failed
• TRUSTED CERTIFICATE handling is documented as a compatibility shim, with rationale and upstream context
• CRLs are handled via rustls-pki-types mixed-section parsing rather than broad text stripping
• env-sensitive tests were isolated into subprocess tests, and the macOS sandbox/proxy panic was contained to the probe path so the tests stay focused on CA behavior

So from my perspective, the main delta versus #6864 is not a change in product intent so much as a reorganization around reviewability, clearer contracts, and more defensive behavior around real-world CA bundles and test environments.

Most of the credit for the actual feature still belongs to @3axap4eHko; this version rearranges and hardens the work substantially, but it is still building on the original implementation and review feedback.

[3axap4eHko]

Thanks for the rework. I re-compared this against #6864, and the main things that still feel lost from the old PR are:

1. The subprocess probe no longer exercises the exact production builder path. login_ca_probe now goes through probe_support::build_login_http_client(), which uses .no_proxy() specifically for tests. I understand why that exists, but it does mean the subprocess coverage is no longer validating the same reqwest builder configuration that production login uses. fix: add custom CA certificate support for login flows #6864's probe was closer to "exercise the real path in a subprocess".

2. TRUSTED CERTIFICATE support now depends on hand-rolled DER boundary parsing (first_der_item / der_item_length) to trim trailing X509_AUX bytes. That is a fair bit more bespoke than fix: add custom CA certificate support for login flows #6864. If this is going to stay, it would be good to justify it with the exact real-world inputs it was validated against, because this is the kind of parsing logic that is easy to carry forward without much confidence.

3. CRL tolerance is weaker than it was in fix: add custom CA certificate support for login flows #6864. The current code ignores well-formed X509 CRL sections, but it still fails the whole bundle if a CRL section is malformed. The old strip_pem_block approach was more forgiving there: malformed CRLs were removed before PEM parsing, so valid certs in the same bundle could still load. For the enterprise-bundle use case, that more permissive behavior still seems desirable.

One note so this doesn’t send anyone on the wrong trail: the env-scrubbing issue from #6864 looks already fixed on the current tip. run_probe() now removes inherited CODEX_CA_CERTIFICATE / SSL_CERT_FILE before applying test-specific values.

[joshka-oai]

Thank you for your review. It's exactly what I needed on this (and very appreciated). I'll take a look at the mentioned concerns tomorrow morning. Are you able to confirm that this works in your environment? It's difficult to anticipate which of the mentioned inconsistencies will lead to failure and which will mostly just work fine as runtime configuration of these proxies is fairly bespoke in the real world.

[joshka-oai]

Thanks, this is helpful. I think the three points break down a bit differently.

On the subprocess probe using .no_proxy()

That was an intentional test-only workaround, not a change to the production login path.

The issue we hit was that the subprocess tests could fail before the CA code under test reported either success or a structured error, because on macOS seatbelt runs reqwest::Client::builder().build() can panic while probing platform proxy settings. That matters in practice for us because these tests are often run from inside Codex, where spawned processes commonly end up under seatbelt.

I tried to document that at the owner level here:

• module docs defining “hermetic” and the seatbelt failure mode:
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/custom_ca.rs#L22-L41
• test-only builder docs:
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/custom_ca.rs#L201-L213
• probe binary pointing back to that explanation:
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/bin/custom_ca_probe.rs#L1-L20

So yes, this is weaker than “exercise the exact production builder in a subprocess,” but that tradeoff was deliberate: without disabling proxy autodetection in the probe, these tests were not reliably testing CA behavior at all.

On the bespoke TRUSTED CERTIFICATE handling

Agreed this is more custom than the earlier version, but it is there to handle a real OpenSSL PEM variant rather than an invented edge case.

I added the rationale and upstream references directly in the code:

• normalization / why this is a local compatibility shim:
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/custom_ca.rs#L543-L582
• why TRUSTED CERTIFICATE may need DER trimming before handing bytes to reqwest:
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/custom_ca.rs#L509-L518
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/custom_ca.rs#L613-L653

This is also covered by a real OpenSSL-generated fixture in the repo rather than a hand-written approximation:

• fixture generated with openssl x509 -addtrust serverAuth -trustout:
  https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/tests/fixtures/test-ca-trusted.pem#L1-L6

In particular, this points at:

• rustls/pemfile declining upstream support for BEGIN TRUSTED CERTIFICATE:
  Certificates in PEM files that starts "BEGIN TRUSTED CERTIFICATE" are ignored rustls/pemfile#52
• OpenSSL docs for x509 -trustout, which emits that PEM form:
  https://docs.openssl.org/master/man1/openssl-x509/

The intent here was to stay narrow: normalize the label, trim off trailing OpenSSL X509_AUX trust metadata, and hand only the certificate DER to reqwest.

On malformed CRLs

I think this is the most valid open point, but I’m comfortable treating it as a follow-up if we find it matters in practice.

Right now the code ignores well-formed CRLs, but a malformed CRL can still fail the bundle before we classify that section as ignorable. I documented that as a known limitation here:

• https://github.com/openai/codex/blob/joshka/login-custom-ca-shared-clients/codex-rs/codex-client/src/custom_ca.rs#L450-L458

My take is that this is a reasonable place to fail closed for now. This is a security-sensitive root-loading path, and in the normal case we should be able to expect CA bundles provided by system administrators to contain well-formed items. If that assumption turns out to be wrong in the field, making CRL handling more permissive is a two-way door and we can relax it later with concrete examples, but I don’t think we need to preemptively broaden acceptance of malformed mixed PEM input now.