Skip to content

Feat/kiro cli executor with tool usage - #330

Open
perezjoseph wants to merge 10 commits into
KeygraphHQ:mainfrom
perezjoseph:feat/kiro-cli-executor-with-tool-usage
Open

Feat/kiro cli executor with tool usage#330
perezjoseph wants to merge 10 commits into
KeygraphHQ:mainfrom
perezjoseph:feat/kiro-cli-executor-with-tool-usage

Conversation

@perezjoseph

@perezjoseph perezjoseph commented May 8, 2026

Copy link
Copy Markdown

feat: Add Kiro CLI executor backend with tool usage logging

Summary

Adds a new executor backend (kiro-cli) that runs Shannon agents via the Kiro CLI in headless mode, as an alternative to the Claude Agent SDK. Includes per-invocation tool usage tracking exposed through a Temporal workflow query.

Changes

Kiro CLI Executor (apps/worker/src/ai/kiro-cli-executor.ts)

  • Full executor implementation: agent JSON generation, subprocess spawning, exit code mapping, error classification
  • Generates .kiro/agents/<name>.json with model, prompt, MCP servers, and hooks
  • Heartbeat signaling to prevent Temporal activity timeouts during long-running agents
  • Credits extraction from kiro-cli stderr output
  • Structured output support via prompt augmentation (queue-writing instructions for vuln agents)
  • Tool usage logging via pre/post hook scripts that write JSONL

Tool Usage Tracking

  • Hook scripts capture every tool invocation (pre + post events) to ~/.kiro/agents/tool-usage.jsonl
  • readAndLogToolUsage() parses JSONL and returns both summary and per-invocation records
  • Workflow query getToolUsage accepts optional detailed boolean arg:
    • false (default): returns Record<string, ToolUsageSummaryMetrics> (counts, failures)
    • true: returns Record<string, DetailedToolUsageMetrics> (includes invocations[] with tool name, timestamp, success)
  • Cumulative tool invocations/failures exposed as Temporal search attributes

CLI Credential Validation

  • validateCredentials() now recognizes SHANNON_EXECUTOR_BACKEND=kiro-cli and only requires KIRO_API_KEY
  • buildEnvFlags() excludes conflicting Bedrock/Vertex env vars when kiro-cli backend is active
  • Worker preflight (preflight.ts) has matching kiro-cli bypass

Dockerfile

  • Installs kiro-cli binary in builder stage via official installer
  • Copies binary to runtime stage

Infrastructure

  • Agent JSON supports MCP server configuration, custom hooks, and model tier resolution
  • Global agents directory (~/.kiro/agents/) used for writable hook scripts inside Docker (repo mounted :ro)
  • Playwright skill installation for browser-based testing agents

Configuration

# .env
SHANNON_EXECUTOR_BACKEND=kiro-cli
KIRO_API_KEY=ksk_your_key_here

Querying Tool Usage (Temporal)

# Summary (default)
temporal workflow query \
  --workflow-id <id> \
  --name getToolUsage \
  --address localhost:7233

# Detailed (per-invocation records)
temporal workflow query \
  --workflow-id <id> \
  --name getToolUsage \
  --input 'true' \
  --address localhost:7233 \
  --output json

Testing

  • Full 5-phase pipeline scan against a live target (10 agents, 2394 tool invocations tracked)
  • Verified getToolUsage query returns correct data in both summary and detailed modes
  • Build passes (pnpm run build, pnpm run check)
jpilier and others added 10 commits April 23, 2026 16:03
@jpilier
feat(worker): Add kiro-cli executor backend and improve infrastructure
a2cd3a5 
- Add Claude SDK and Kiro CLI executor implementations with comprehensive test coverage
- Create executor interface abstraction for pluggable AI backend support
- Implement container and preflight service tests for reliability validation
- Add vitest configuration for worker package testing
- Extend environment variable forwarding for AWS credentials and Kiro API configuration
- Add SHANNON_EXECUTOR_BACKEND mode detection supporting kiro-cli backend
- Create .kiro/agents directory structure for agent definitions mount point
- Add kiro-cli binary installation to Docker build with Go security tool configuration
- Mount kiro-agents workspace overlay for writable agent definitions in containers
- Update .gitignore to exclude .kiro and .vscode directories
- Add licenses.txt documentation file
- Update dependencies in pnpm-lock.yaml
@jpilier
feat(worker): Update agent generation for Docker compatibility and gl…
ad306c8 
…obal agent directory

- Update agent JSON prompt paths to use relative file:// URIs (file://./ prefix)
- Write agents to global ~/.kiro/agents/ directory for reliable kiro-cli resolution
- Add best-effort fallback writes to project-level .kiro/agents/ for host-side visibility
- Change verify-queue-file output extension from .js to .mjs
- Remove unused execSync import from kiro-cli-executor
- Add cpSync import for potential file operations
- Update generateAgentJson to handle both global and project directories with retry logic
- Update generateQueueValidationHooks to write to global directory with project fallback
- Add detailed comments explaining Docker bind-mount strategy and agent resolution
@jpilier
feat(kiro-cli-executor): Improve credits extraction to handle stderr …
5b2f433 
…output

- Update CREDITS_PATTERN regex to match both "Credits used:" and "▸ Credits:" formats
- Rename extractCreditsUsed parameter from stdout to text for clarity
- Check stderr first for credits output, then fall back to stdout
- Enhance JSDoc to document kiro-cli's stderr credit format and accepted patterns
- Handle kiro-cli's new credit output format in stderr with bullet point prefix
@jpilier
feat(executor): Add heartbeat signaling to prevent event loop starvation
e869eca 
- Add optional onHeartbeat callback to ExecutorOptions interface for orchestrator liveness signaling
- Implement heartbeat loop in spawnKiroCli that emits elapsed time and process ID every 30 seconds
- Pass onHeartbeat callback through agent execution pipeline from Temporal activities to CLI executor
- Clean up heartbeat timer on process completion to prevent resource leaks
- Prevents Temporal heartbeat starvation when multiple kiro-cli subprocesses saturate the event loop with I/O callbacks
@jpilier
chore: pull latest shannon from upstream
67cf2a6 
# Conflicts:
#	jpilier/week1/shannon/Dockerfile
#	jpilier/week1/shannon/apps/cli/src/env.ts
#	jpilier/week1/shannon/apps/worker/src/services/preflight.ts
#	jpilier/week1/shannon/apps/worker/src/temporal/activities.ts
#	jpilier/week1/shannon/pnpm-lock.yaml
@jpilier
feat(executor): Add kiro-cli backend support with tool usage logging
c076b38 
- Install kiro-cli binary in Docker image for headless mode execution
- Add SHANNON_EXECUTOR_BACKEND and KIRO_API_KEY environment variables
- Implement credential validation for kiro-cli backend mode
- Exclude conflicting provider variables when using kiro-cli backend
- Add ToolUsageSummary type and tool invocation tracking to ClaudePromptResult
- Implement KiroCliExecutor with tool usage logging via pre/post-tool hooks
- Add mergeHooks utility to combine multiple hook configurations
- Implement readAndLogToolUsage to parse and aggregate tool execution metrics
- Add generateToolUsageLoggerScript to create hook scripts for tool tracking
- Update metrics tracking to record tool usage events with timestamps and duration
- Extend audit logging to capture tool invocation details and performance data
- Add comprehensive test coverage for hook merging and tool usage extraction
@perezjoseph
Merge branch 'main' into feat/kiro-cli-executor-with-tool-usage
39e6290 
# Conflicts:
#	Dockerfile
#	apps/cli/src/commands/start.ts
#	apps/worker/src/services/agent-execution.ts
#	apps/worker/src/services/container.ts
@perezjoseph
Merge remote-tracking branch 'upstream/main' into feat/kiro-cli-execu…
721a2e2 
…tor-with-tool-usage

# Conflicts:
#	apps/worker/src/services/agent-execution.ts
#	apps/worker/src/types/metrics.ts
@perezjoseph
feat(ai): upgrade Opus to 4.8 and add MCP servers support
71126ae 
- Add mcpServers option parameter to Claude SDK executor initialization
- Update Opus model version from 4.6 to 4.8 in model tier mapping
- Fix indentation inconsistencies in kiro-cli-executor for tool usage result interface and heartbeat callback
- Align code formatting in error logging statement for consistency
@perezjoseph
Merge branch 'KeygraphHQ:main' into feat/kiro-cli-executor-with-tool-…
369a59f 
…usage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@perezjoseph @jpilier