Skip to content

Give turn dispatch a real cancellation path instead of an abandoning timeout - #483

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-7193-with-timeout-abort
Aug 31, 2026
Merged

Give turn dispatch a real cancellation path instead of an abandoning timeout#483
TheGreatAxios merged 2 commits into
mainfrom
cl-7193-with-timeout-abort

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

Fixes CL-7193. withTimeout raced a timer against a promise with no AbortSignal — it resolved the timeout branch and walked away while the losing promise kept running with nothing able to stop it. This wrapped both waitUntilFree and dispatchTurn. On timeout, the caller posted an "undelivered" notice in the agent's own voice while the original sendMail was still in flight; if it later landed, the turn row was still running, so the real reply could attach to it and land behind the apology. Underneath, waitUntilFree's uncancellable for(;;) loop was built on a waiter pubsub whose backstop timer notify() never cleared — a key that kept timing out accumulated one abandoned waiter per attempt forever.

  • withTimeout now threads an AbortSignal into the work it wraps, firing it the instant its own timer wins.
  • waitUntilFree accepts that signal and throws once aborted, instead of polling until its own stale-turn backstop or silently resolving as though the agent were free.
  • The waiter pubsub (createTurnFreedSignal) gives every waiter full teardown (clear its timer, remove itself, detach its abort listener) so notify(), its own backstop, and an aborted signal can't double-fire or leak — a repeated-timeout test asserts the waiter count never grows.
  • finishTurn is now compare-and-set on status === "running" on both the in-memory and Drizzle stores, so two racing closes (a dispatch deadline's abort-close vs. a late reply's own close) can't clobber each other.
  • dispatchTurn takes the same signal and closes its turn row failed the instant it aborts — not cancelling sendMail itself (no cancellable primitive exists for it), just closing our own bookkeeping early so a later real reply can't reattach to a running row behind the undelivered notice.

Scope boundary

platform-adapter.ts's two other withTimeout call sites (wakeByAddressBounded, sendFoldedMailWithReclaimRetry) wrap wakeByAddress / sendFoldedMail, neither of which has any cancellable primitive — no AbortSignal support anywhere in @corbits/agent-lifecycle or @corbits/folded-runs, and sendFoldedMail does a DB write plus sidecar delivery that shouldn't be half-cancelled. Those two sites only changed call shape to match the new signature; the signal is unused, same abandon-on-timeout behavior as before. Not touched: Interchange, chat-orchestrator.ts (CL-7196), turn-queue.ts, or routes.ts's question handler (CL-7192). agent-turns.ts is also touched by CL-7200 (unstarted, lower priority).

Known consequence — flagging for review

With a timed-out turn now closing immediately, chat-orchestrator.ts's postReply (unchanged in this PR) falls into its existing "no running turn" branch when a late reply arrives after the timeout. A single-workbench-membership agent still posts, untied to any turn. A multi-membership agent with no correlation hint now has its late reply dropped entirely instead of possibly misattaching. This trades a confusing double-post (apology + real reply) for a rarer silent drop, which seems like the right trade but is a product call worth a second look, not something I felt authorized to silently absorb by also touching chat-orchestrator.ts.

Acceptance criteria

  • withTimeout propagates an AbortSignal and the work it wraps actually stops on timeout
  • waitUntilFree accepts a signal and exits its loop when aborted
  • notify clears the backstop timer, and a backstop firing removes its own waiter from the Set
  • A timed-out dispatch cannot later post a reply behind the undelivered notice (closes the turn row immediately; see known consequence above for the one edge this surfaces)
  • A test asserts the waiter Set does not grow across repeated timeouts

Test plan

  • packages/chat/src/with-timeout.test.ts (new) — signal propagation, abort-on-timeout, no-abort-on-success
  • packages/chat/src/agent-turns.test.ts — waiter-Set growth, CAS finishTurn race, waitUntilFree abort-throws
  • packages/chat/test/turn-dispatch-deadline.test.ts — turn row closes failed immediately on dispatch timeout, before sendMail ever settles
  • WORKBENCH_CHECK_SINCE=origin/main bun run typecheck — clean
  • WORKBENCH_CHECK_SINCE=origin/main bun run test — clean (736 pass in @corbits/chat, 0 fail; all other affected packages green)
  • bun run lint — clean
  • bun run check:structural — clean
  • Reviewed by Greybeard (approach) and Critique (implementation, two passes — one verified finding on an unhandled-rejection path in the abort-close handler, fixed and re-verified)

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critique · reviewed the two commits (tests-first, then implementation)

What this branch does: threads an AbortSignal through withTimeout into waitUntilFree and dispatchTurn, fixes the waiter-Set leak in agent-turns.ts's createTurnFreedSignal, and makes finishTurn compare-and-set on status so a dispatch-timeout close and a late reply's own close can't clobber each other.

Findings from two review passes (both already addressed on this branch, not open items):

  • packages/chat/src/workbench-service.ts:1831 — dispatchTurn's closeAsTimedOut originally fired finishTurn with void, no rejection handler. A rejecting finishTurn (e.g. a Drizzle write failure during the abort race) would have surfaced as an unhandled promise rejection, bypassing reportError. Fixed: now chains .catch() into reportError with operation "chat.dispatchTurn.closeAsTimedOut". Re-verified: optional-chaining short-circuit is correct when deps.agentTurns is undefined, and a losing CAS resolves to undefined rather than throwing, so no double-report on the race.
  • packages/chat/src/agent-turns.ts:224-243 — createTurnFreedSignal's waiter teardown (settle() clearing its own backstop timer, removing itself via an id-keyed Map to sidestep a self-referencing-object TDZ/prefer-const issue, detaching its abort listener) verified idempotent under every reachable ordering (notify racing a fired backstop, notify racing abort). notify() snapshots via [...byId.values()] before iterating so in-loop removal doesn't corrupt iteration.
  • finishTurn's CAS change verified safe for existing callers: chat-orchestrator.ts's postReply only calls finishTurn on a turn already read as "running" via findRunningTurn, so the new precondition never rejects a previously-succeeding call; connect-pending.ts never calls finishTurn.

Known, deliberately surfaced (not silently absorbed) consequence: with a timed-out turn now closing immediately, a multi-membership agent's late reply with no correlation hint now gets dropped by chat-orchestrator.ts's existing postReply logic instead of possibly misattaching — worth a second look from whoever owns CL-7196, called out in the PR body.

Scope verified clean: no edits to chat-orchestrator.ts, turn-queue.ts, routes.ts, or any @intx/* package.

@TheGreatAxios
TheGreatAxios force-pushed the cl-7193-with-timeout-abort branch 5 times, most recently from 410a5ba to 5da0ebb Compare August 31, 2026 04:38
withTimeout races a timer against a promise with no way to signal the
loser that it lost. dispatchTurnBatch wraps waitUntilFree and
dispatchTurn in it, so a timed-out call keeps running unbounded — for
waitUntilFree's uncancellable for(;;) loop, and for dispatchTurn's
turn row, which can stay `running` long enough for a later reply to
land behind the undelivered notice already posted for it.

Underneath, waitUntilFree's wait primitive registers a resolve into a
Set and schedules a backstop timer notify() never clears; when the
backstop fires first, the resolve is never removed either, so a key
that keeps timing out accumulates waiters forever.

These tests assert the fix ahead of the implementation: withTimeout
must pass work an AbortSignal it can react to; the waiter Set must
never grow across repeated timeouts; finishTurn must be
compare-and-set on status so a late reply can never clobber a turn a
timeout already closed; and a dispatch deadline must close its turn
row the instant it fires, not whenever the abandoned send eventually
settles.
withTimeout now threads an AbortSignal into the work it wraps, firing
the moment its own timer wins the race, instead of rejecting and
walking away from a promise that keeps running unobserved.

waitUntilFree's for(;;) loop accepts that signal and throws once it
aborts, rather than polling until its own backstop timer -- up to
AGENT_TURN_STALE_MS later -- or silently resolving as though the
agent were free.

The waiter pubsub underneath (createTurnFreedSignal) gives every
waiter its own teardown: notify(), a fired backstop, and an aborted
signal all funnel through one settle() that clears the backstop timer
and removes the waiter, so whichever reaches it first is the only one
that ever runs -- a key that keeps timing out no longer accumulates
one abandoned waiter per attempt.

dispatchTurn takes the same signal and closes its turn row `failed`
the instant it aborts, rather than leaving the row `running` until (or
unless) the abandoned sendMail eventually settles -- sendMail itself
has no cancellable primitive, so this closes the bookkeeping around
the send rather than the send itself. finishTurn is now
compare-and-set on status === "running", so this abort-driven close
and a late reply's own close can race safely: exactly one applies,
and the loser can never clobber a completed turn's replyMessageId.

The two withTimeout call sites in platform-adapter.ts (wakeByAddress,
sendFoldedMail) have no cancellable primitive to hook a signal into,
so they keep today's abandon-on-timeout behavior -- only their call
shape changes to match the new signature.

With a turn closed on timeout, a late reply with no correlation hint
across multiple room memberships now has no running turn to attach
to; chat-orchestrator.ts's existing single-membership fallback still
posts it untied, but a multi-membership agent's late reply in that
situation is dropped rather than landing behind the undelivered
notice -- trading a confusing double-post for a rarer silent drop.
@TheGreatAxios
TheGreatAxios force-pushed the cl-7193-with-timeout-abort branch from 5da0ebb to d9e7059 Compare August 31, 2026 05:21
@TheGreatAxios
TheGreatAxios merged commit bbf7670 into main Aug 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant