Problem
The eval suite drives the provider streaming tool loop directly. It does not
exercise agent-block input wiring, variable resolution, the executor's
retry/fallback policy, or the run/error handling above the loop. A harness change
there can regress without this suite noticing.
Proposal
Add an executor-level harness that builds a serialized workflow with an Agent
block, invokes DAGExecutor.execute, and scores the same checks.
- Location:
apps/sim/evals/agent-tool-use/executor-harness.ts
- Real Start → Agent workflow; only
executeProviderRequest is mocked at the
provider boundary, so the Agent block handler, variable resolution, and the
executor run/error handling are real.
- Reuse
@sim/testing factories (createSerializedWorkflow) and the
@sim/testing mocks already used by
executor/handlers/agent/agent-handler.test.ts.
- Reuse the existing scoring/report so executor and loop results land in one
report.
Acceptance criteria
Follow-up
The provider boundary is mocked, so the executor's retry/fallback policy is not
asserted yet. Add a scenario whose first provider call rejects, with a block
retry config, to cover the retry path.
Problem
The eval suite drives the provider streaming tool loop directly. It does not
exercise agent-block input wiring, variable resolution, the executor's
retry/fallback policy, or the run/error handling above the loop. A harness change
there can regress without this suite noticing.
Proposal
Add an executor-level harness that builds a serialized workflow with an Agent
block, invokes
DAGExecutor.execute, and scores the same checks.apps/sim/evals/agent-tool-use/executor-harness.tsexecuteProviderRequestis mocked at theprovider boundary, so the Agent block handler, variable resolution, and the
executor run/error handling are real.
@sim/testingfactories (createSerializedWorkflow) and the@sim/testingmocks already used byexecutor/handlers/agent/agent-handler.test.ts.report.
Acceptance criteria
DAGExecutorend to endFollow-up
The provider boundary is mocked, so the executor's retry/fallback policy is not
asserted yet. Add a scenario whose first provider call rejects, with a block
retryconfig, to cover the retry path.