Give turn dispatch a real cancellation path instead of an abandoning timeout - #483
Conversation
TheGreatAxios
left a comment
There was a problem hiding this comment.
Critique · reviewed the two commits (tests-first, then implementation)
What this branch does: threads an AbortSignal through withTimeout into waitUntilFree and dispatchTurn, fixes the waiter-Set leak in agent-turns.ts's createTurnFreedSignal, and makes finishTurn compare-and-set on status so a dispatch-timeout close and a late reply's own close can't clobber each other.
Findings from two review passes (both already addressed on this branch, not open items):
- packages/chat/src/workbench-service.ts:1831 — dispatchTurn's closeAsTimedOut originally fired finishTurn with
void, no rejection handler. A rejecting finishTurn (e.g. a Drizzle write failure during the abort race) would have surfaced as an unhandled promise rejection, bypassing reportError. Fixed: now chains.catch()into reportError with operation "chat.dispatchTurn.closeAsTimedOut". Re-verified: optional-chaining short-circuit is correct when deps.agentTurns is undefined, and a losing CAS resolves to undefined rather than throwing, so no double-report on the race. - packages/chat/src/agent-turns.ts:224-243 — createTurnFreedSignal's waiter teardown (settle() clearing its own backstop timer, removing itself via an id-keyed Map to sidestep a self-referencing-object TDZ/prefer-const issue, detaching its abort listener) verified idempotent under every reachable ordering (notify racing a fired backstop, notify racing abort). notify() snapshots via [...byId.values()] before iterating so in-loop removal doesn't corrupt iteration.
- finishTurn's CAS change verified safe for existing callers: chat-orchestrator.ts's postReply only calls finishTurn on a turn already read as "running" via findRunningTurn, so the new precondition never rejects a previously-succeeding call; connect-pending.ts never calls finishTurn.
Known, deliberately surfaced (not silently absorbed) consequence: with a timed-out turn now closing immediately, a multi-membership agent's late reply with no correlation hint now gets dropped by chat-orchestrator.ts's existing postReply logic instead of possibly misattaching — worth a second look from whoever owns CL-7196, called out in the PR body.
Scope verified clean: no edits to chat-orchestrator.ts, turn-queue.ts, routes.ts, or any @intx/* package.
410a5ba to
5da0ebb
Compare
withTimeout races a timer against a promise with no way to signal the loser that it lost. dispatchTurnBatch wraps waitUntilFree and dispatchTurn in it, so a timed-out call keeps running unbounded — for waitUntilFree's uncancellable for(;;) loop, and for dispatchTurn's turn row, which can stay `running` long enough for a later reply to land behind the undelivered notice already posted for it. Underneath, waitUntilFree's wait primitive registers a resolve into a Set and schedules a backstop timer notify() never clears; when the backstop fires first, the resolve is never removed either, so a key that keeps timing out accumulates waiters forever. These tests assert the fix ahead of the implementation: withTimeout must pass work an AbortSignal it can react to; the waiter Set must never grow across repeated timeouts; finishTurn must be compare-and-set on status so a late reply can never clobber a turn a timeout already closed; and a dispatch deadline must close its turn row the instant it fires, not whenever the abandoned send eventually settles.
withTimeout now threads an AbortSignal into the work it wraps, firing the moment its own timer wins the race, instead of rejecting and walking away from a promise that keeps running unobserved. waitUntilFree's for(;;) loop accepts that signal and throws once it aborts, rather than polling until its own backstop timer -- up to AGENT_TURN_STALE_MS later -- or silently resolving as though the agent were free. The waiter pubsub underneath (createTurnFreedSignal) gives every waiter its own teardown: notify(), a fired backstop, and an aborted signal all funnel through one settle() that clears the backstop timer and removes the waiter, so whichever reaches it first is the only one that ever runs -- a key that keeps timing out no longer accumulates one abandoned waiter per attempt. dispatchTurn takes the same signal and closes its turn row `failed` the instant it aborts, rather than leaving the row `running` until (or unless) the abandoned sendMail eventually settles -- sendMail itself has no cancellable primitive, so this closes the bookkeeping around the send rather than the send itself. finishTurn is now compare-and-set on status === "running", so this abort-driven close and a late reply's own close can race safely: exactly one applies, and the loser can never clobber a completed turn's replyMessageId. The two withTimeout call sites in platform-adapter.ts (wakeByAddress, sendFoldedMail) have no cancellable primitive to hook a signal into, so they keep today's abandon-on-timeout behavior -- only their call shape changes to match the new signature. With a turn closed on timeout, a late reply with no correlation hint across multiple room memberships now has no running turn to attach to; chat-orchestrator.ts's existing single-membership fallback still posts it untied, but a multi-membership agent's late reply in that situation is dropped rather than landing behind the undelivered notice -- trading a confusing double-post for a rarer silent drop.
5da0ebb to
d9e7059
Compare
Summary
Fixes CL-7193.
withTimeoutraced a timer against a promise with noAbortSignal— it resolved the timeout branch and walked away while the losing promise kept running with nothing able to stop it. This wrapped bothwaitUntilFreeanddispatchTurn. On timeout, the caller posted an "undelivered" notice in the agent's own voice while the originalsendMailwas still in flight; if it later landed, the turn row was stillrunning, so the real reply could attach to it and land behind the apology. Underneath,waitUntilFree's uncancellablefor(;;)loop was built on a waiter pubsub whose backstop timernotify()never cleared — a key that kept timing out accumulated one abandoned waiter per attempt forever.withTimeoutnow threads anAbortSignalinto the work it wraps, firing it the instant its own timer wins.waitUntilFreeaccepts that signal and throws once aborted, instead of polling until its own stale-turn backstop or silently resolving as though the agent were free.createTurnFreedSignal) gives every waiter full teardown (clear its timer, remove itself, detach its abort listener) sonotify(), its own backstop, and an aborted signal can't double-fire or leak — a repeated-timeout test asserts the waiter count never grows.finishTurnis now compare-and-set onstatus === "running"on both the in-memory and Drizzle stores, so two racing closes (a dispatch deadline's abort-close vs. a late reply's own close) can't clobber each other.dispatchTurntakes the same signal and closes its turn rowfailedthe instant it aborts — not cancellingsendMailitself (no cancellable primitive exists for it), just closing our own bookkeeping early so a later real reply can't reattach to arunningrow behind the undelivered notice.Scope boundary
platform-adapter.ts's two otherwithTimeoutcall sites (wakeByAddressBounded,sendFoldedMailWithReclaimRetry) wrapwakeByAddress/sendFoldedMail, neither of which has any cancellable primitive — noAbortSignalsupport anywhere in@corbits/agent-lifecycleor@corbits/folded-runs, andsendFoldedMaildoes a DB write plus sidecar delivery that shouldn't be half-cancelled. Those two sites only changed call shape to match the new signature; the signal is unused, same abandon-on-timeout behavior as before. Not touched: Interchange,chat-orchestrator.ts(CL-7196),turn-queue.ts, orroutes.ts's question handler (CL-7192).agent-turns.tsis also touched by CL-7200 (unstarted, lower priority).Known consequence — flagging for review
With a timed-out turn now closing immediately,
chat-orchestrator.ts'spostReply(unchanged in this PR) falls into its existing "no running turn" branch when a late reply arrives after the timeout. A single-workbench-membership agent still posts, untied to any turn. A multi-membership agent with no correlation hint now has its late reply dropped entirely instead of possibly misattaching. This trades a confusing double-post (apology + real reply) for a rarer silent drop, which seems like the right trade but is a product call worth a second look, not something I felt authorized to silently absorb by also touchingchat-orchestrator.ts.Acceptance criteria
withTimeoutpropagates anAbortSignaland the work it wraps actually stops on timeoutwaitUntilFreeaccepts a signal and exits its loop when abortednotifyclears the backstop timer, and a backstop firing removes its own waiter from the SetTest plan
packages/chat/src/with-timeout.test.ts(new) — signal propagation, abort-on-timeout, no-abort-on-successpackages/chat/src/agent-turns.test.ts— waiter-Set growth, CASfinishTurnrace,waitUntilFreeabort-throwspackages/chat/test/turn-dispatch-deadline.test.ts— turn row closesfailedimmediately on dispatch timeout, beforesendMailever settlesWORKBENCH_CHECK_SINCE=origin/main bun run typecheck— cleanWORKBENCH_CHECK_SINCE=origin/main bun run test— clean (736 pass in@corbits/chat, 0 fail; all other affected packages green)bun run lint— cleanbun run check:structural— clean