Skip to content

Cut PostHog event volume with generation aggregates and subagent rollups - #694

Merged
TheGreatAxios merged 16 commits into
mainfrom
cl-6816-cut-posthog-event-volume-aggregate-tool-spans-into-the-turn
Aug 26, 2026
Merged

Cut PostHog event volume with generation aggregates and subagent rollups#694
TheGreatAxios merged 16 commits into
mainfrom
cl-6816-cut-posthog-event-volume-aggregate-tool-spans-into-the-turn

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Folds per-tool $ai_spans into $ai_generation aggregates by default (tool_call_count, tool_error_count, subagent_call_count); opt-in spans via CORBITS_TELEMETRY_AI_SPANS
  • Enriches subagent_end with run rollups (model, turns, tokens, tool counts, stop_reason, spawn-time parent_trace_id) and allowlists first-party director ids
  • Stamps $process_person_profile: false on ambient events, dedupes plugin_loaded per plugin per process, and documents a generation sample-rate knob (default 1.0)

Verification

  • bun run typecheck, bun run build, and bun run test pass (bun run check exit 0 locally)
  • Critic re-review clean after parent_trace_id fix

Fixes CL-6816

Unknown project and plugin profile ids still collapse to custom so
employer-named agents never leave the process.
Emit one $ai_generation per turn with tool and subagent counts instead
of an $ai_span per call. CORBITS_TELEMETRY_AI_SPANS restores per-call
spans for debugging; CORBITS_TELEMETRY_GENERATION_SAMPLE_RATE samples
successful generations while errors always ship.
Accumulate turn, token, and tool counts in runSubAgent and send them
on ambient subagent_end so dashboards stay answerable without leaf
AI observability events.
Ambient capture sets $process_person_profile false so PostHog batch
events stay anonymous; intentional survey capture omits the flag.
Emit plugin_loaded once per plugin identity per process.
@linear-code

linear-code Bot commented Aug 26, 2026

Copy link
Copy Markdown

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Primary · Request changes

This aggregates parent-turn telemetry, rolls worker activity into subagent_end, and makes ambient capture anonymous.

Findings

  • src/telemetry/ai-observability.ts:207 — A recoverable inference error always emits a failed generation before the successful retry, so one completed parent turn can emit two generations or look failed-only when success sampling is zero.
  • src/telemetry/ai-observability.ts:47 — Only task is classified as a subagent call, so preferred spawn_agent dispatches inflate tool_call_count and leave subagent_call_count at zero.
  • src/subagent/run.ts:1147 — Worker failures rethrow before accumulated turn, token, tool, and error totals reach subagent_end; clean completion also omits the required stop reason.
  • src/subagent/agent-fleet.ts:575 — Fleet setup failure returns after subagent_start but before the terminal emitter is registered; fused task setup failure reports the default completed status at src/subagent/task-tool.ts:936.
  • src/exec/runner.ts:660 — Exec mode does not install the turn-start observer used to capture the active parent trace, so worker terminal events omit parent_trace_id.
  • src/telemetry/product-events.ts:73 — Plugin dedupe records identity even when capture uses the no-op telemetry handle, permanently suppressing a later real plugin_loaded event in the same process.
  • commit 75df24f6 — The subject includes CL-6816, which the repository commit rules prohibit.
  • CL-6816 — The representative before/after event-count measurement remains unchecked, so the target reduction is not demonstrated.

Notes

  • bun run check exits 0 across lint, typecheck, build, and tests.
  • GitHub prevents a formal self-request-changes action because the PR author and authenticated reviewer are both TheGreatAxios; this comment review records the substantive request-changes verdict.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critic · Request changes

This reshapes AI observability around parent-turn aggregates and per-dispatch worker rollups.

Findings

  • src/session/run-sink.ts:178inference.error immediately emits a failed generation even when the run sink retries. At sample rate zero, an eventually successful turn is recorded only as failed; at rate one, the same trace emits both failed and successful generations.
  • src/telemetry/ai-observability.ts:47 — The subagent tool-name check excludes spawn_agent, so its calls and optional spans are recorded as ordinary tools instead of subagent calls.
  • src/subagent/run.ts:1147 — A worker that fails after prior activity throws away its accumulated rollups before callers can enrich subagent_end; src/subagent/run.ts:1071 also returns clean completion without a stop reason.
  • src/subagent/task-tool.ts:936 — Fused setup failures inherit a default completed status, while src/subagent/agent-fleet.ts:575 returns before fleet dispatch installs its subagent_end emitter. These paths create false-success or unmatched start/end telemetry.
  • src/exec/runner.ts:660 — Exec parent turns never update the current turn trace used by subagent dispatch, so src/telemetry/product-events.ts:44 cannot attach parent_trace_id.
  • src/plugins/loader.ts:143 — Loads without a telemetry handle still pass through global dedupe with NOOP_TELEMETRY; the identity is consumed without capture and later enabled telemetry cannot emit the plugin event.

Notes

  • Permanent coverage should include recovered retries at sample rates zero and one, both dispatch tool names, worker failure after prior activity, clean stop reason, setup failures for both dispatch primitives, exec trace linkage, and no-op-then-enabled plugin loading.
  • Targeted telemetry tests pass with 75 tests and zero failures; the independent full bun run check also exits 0.
Update the event table and AI section for generation aggregates, opt-in
spans, subagent_end rollups, director allowlisting, anonymous ambient
processing, plugin_loaded dedupe, and generation sampling.
Capture parent_trace_id at task/spawn dispatch from the current turn
noted at inference.start, instead of the last completed turn. Treat an
empty generation sample-rate env as unset, and document that sampling
drops opt-in spans with the generation.
@TheGreatAxios
TheGreatAxios force-pushed the cl-6816-cut-posthog-event-volume-aggregate-tool-spans-into-the-turn branch from 3f6a859 to 2ce256f Compare August 26, 2026 20:47
@TheGreatAxios
TheGreatAxios force-pushed the cl-6816-cut-posthog-event-volume-aggregate-tool-spans-into-the-turn branch from 2ce256f to aed3ec0 Compare August 26, 2026 21:16

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Primary · Approve

This reduces ambient telemetry volume with parent-turn aggregates, truthful worker rollups, anonymous processing, and conservative fallback attribution.

No findings.

Notes

  • bun run check exits 0 at ba4c34ad.
  • Required GitHub checks and CLA pass.
  • The deterministic representative fixture records 98 → 18 billable events, an 81.6% reduction.
  • GitHub prevents formal self-approval because the PR author and authenticated reviewer are both TheGreatAxios; this comment review records the approval verdict.

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critic · Approve

This emits one terminal parent generation, enriches one worker terminal event per dispatch, and avoids false provider attribution when runtime source identity is unavailable.

No findings.

Notes

  • Retry, sampling, setup failure, cancellation, interruption, plugin dedupe, exec linkage, and privacy paths have permanent regression coverage.
  • Pre-usage fallback failures retain the attempted model and use the fixed unknown provider bucket instead of claiming a stale selected provider.
  • The final diff contains no Interchange, vendor, dependency, or lockfile changes.
@TheGreatAxios
TheGreatAxios merged commit 0ba8511 into main Aug 26, 2026
5 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-6816-cut-posthog-event-volume-aggregate-tool-spans-into-the-turn branch August 26, 2026 23:26
TheGreatAxios added a commit that referenced this pull request Aug 27, 2026
…ume-aggregate-tool-spans-into-the-turn

Cut PostHog event volume with generation aggregates and subagent rollups
TheGreatAxios added a commit that referenced this pull request Aug 27, 2026
…ume-aggregate-tool-spans-into-the-turn

Cut PostHog event volume with generation aggregates and subagent rollups
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant