Adds integration test scaffolding#176
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
JasonTheAdams
force-pushed
the
add/integration-tests
branch
from
a599825 to
16d9056
Compare
felixarntz
left a comment
felixarntz
left a comment
There was a problem hiding this comment.
@JasonTheAdams Almost LGTM, just two things.
Other than that, for the Anthropic testFunctionCallingWithNoArguments, I know that'll be fixed in the other PR.
Preemptively approving, pending my feedback below.
| 'say_hi_to', | ||
| 'Says hi to a specific person by name', | ||
| [ | ||
| 'type' => 'string', |
There was a problem hiding this comment.
This test fails for me, and I think it should? According to what I could gather from the docs, Anthropic requires object as type. So this shouldn't be expected to pass?
There was a problem hiding this comment.
Good note! I'm going to address and get all function-related tests passing in a subsequent PR. 👍
| 'say_hi_to', | ||
| 'Says hi to a specific person by name', | ||
| [ | ||
| 'type' => 'string', |
There was a problem hiding this comment.
Same here, for OpenAI I don't think this is allowed. So maybe we'll want to expect an exception here? Just to make sure we still cover it for these providers that don't support it.
There was a problem hiding this comment.
I don't think it is. Hahah! But I'm starting with stupid tests and will fix/add/address them in the next PR.
| // Args should be null or empty for a no-argument function | ||
| $args = $functionCall->getArgs(); | ||
| $this->assertTrue( | ||
| $args === null || $args === [] || $args === (object) [], |
There was a problem hiding this comment.
I think we should make this more concrete, i.e. let's see what the API actually returns in this case. Unless it varies between distinct requests, we should expect that concrete result. While this is extremely nit-picky, we'll want to know when something related to this changes in the API, so being as precise as possible helps here, I'd argue.
There was a problem hiding this comment.
Noted! I'll address this subsequently.
| // Args should be null or empty for a no-argument function | ||
| $args = $functionCall->getArgs(); | ||
| $this->assertTrue( | ||
| $args === null || $args === [] || $args === (object) [], |
| // Args should be null or empty for a no-argument function | ||
| $args = $functionCall->getArgs(); | ||
| $this->assertTrue( | ||
| $args === null || $args === [] || $args === (object) [], |
2 participants
This introduces integration tests for running actual prompts against the models. As this has grown it's become more obvious that the only way to really catch Provider issues is by running prompts. While we would do some basic testing locally, having a proper suite of tests is really useful for catching things and avoiding regressions.
These tests can't be run automatically as they require credentials. As such, we'll want to run this manually periodically when developing, and every time before publishing a new release.
API keys can be loaded as part of the environment or put in a
.envfile.NOTE:
There is a purposely failing test for OpenAI function calling. This will be resolved in #175 as a broader effort to correct issues with function calling across the provider APIs.