Skip to content

Stabilize realtime startup context tests#13876

Merged
aibrahim-oai merged 5 commits intomainfrom
dev/flaky-realtime-startup-context-tests
Mar 9, 2026
Merged

Stabilize realtime startup context tests#13876
aibrahim-oai merged 5 commits intomainfrom
dev/flaky-realtime-startup-context-tests

Conversation

@aibrahim-oai
Copy link
Copy Markdown
Collaborator

@aibrahim-oai aibrahim-oai commented Mar 7, 2026

What changed

  • The realtime startup-context tests no longer assume the interesting websocket payload is always connection 1 / request 0.
  • Instead, they now wait for the first outbound websocket request that actually carries session.instructions, regardless of which websocket connection won the accept-order race on the runner.
  • The env-key fallback test stays serialized because it mutates process environment.

Why this fixes the flake

  • The old test synchronized on the mirrored session.updated client event and then inspected a fixed websocket slot.
  • On CI, the response websocket and the realtime websocket can race each other during startup. When the response websocket wins that race, the fixed slot can contain response.create instead of the startup-context-bearing session.update request the test actually cares about.
  • That made the test fail nondeterministically by inspecting the wrong request, or by timing out waiting on a secondary event even though the real outbound request path was correct.
  • Waiting directly on the first request whose payload includes session.instructions removes both ordering assumptions and makes the assertion line up with the actual contract under test.
  • Separately, serializing the environment-mutating fallback case prevents unrelated tests from seeing partially updated auth state.

Scope

  • Test-only change.
@aibrahim-oai aibrahim-oai force-pushed the dev/flaky-realtime-startup-context-tests branch from 657aca7 to fbfb087 Compare March 8, 2026 04:08
@aibrahim-oai aibrahim-oai requested a review from bolinfest March 8, 2026 06:32
@aibrahim-oai aibrahim-oai merged commit 75e6083 into main Mar 9, 2026
31 checks passed
@aibrahim-oai aibrahim-oai deleted the dev/flaky-realtime-startup-context-tests branch March 9, 2026 17:57
@github-actions github-actions bot locked and limited conversation to collaborators Mar 9, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

1 participant