EDIT 2:
I wanted to point out that I'm going down the rabbit hole here just to explain how to generate separate html reports per spec-file. This is not best practice and you should generally let playwright create one single report and execute all tests with one command. That is the idea behind it. You can create 2 projects with different settings that match different test files if some of your tests have to run on slightly different settings.
In your case I would change the reporter settings to something like this and then share the whole html/ folder:
reporter: [[html', { outputFolder: 'html', open: 'never' }],
['list']],
Yes you can do that but I think there's no setting that allows that easily, gotta do some bash magic! Assuming you have some tests that can run in parallel and some that have to be run serially but you want to run them with one command (npm run test).
If you have a config that goes something like this:
const config: PlaywrightTestConfig = {
// ...
/* Reporter to use. See https://playwright.dev/docs/test-reporters */
reporter: [['junit', { outputFile: `junit/results.xml` }],
['html', { outputFolder: `html/`, open: 'never' }],
['list']],
/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
use: {
/* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
trace: 'on-first-retry',
/* Capture screenshot after each test failure. */
screenshot: 'only-on-failure',
/* Record video only when retrying a test for the first time. */
video: 'retain-on-failure',
},
/* Folder for test artifacts such as screenshots, videos, traces, etc. */
outputDir: `${__dirname}/artifacts`,
};
and a package.json file like this:
{
# ...
"scripts": {
"codegen": "npx playwright codegen",
"test": "npm run test-serial; npm run test-parallel",
"test-parallel": "./run_playwright_tests.sh parallel",
"test-serial": "./run_playwright_tests.sh serial"
},
you can write a script that goes something like this:
#!/bin/bash
set -Eeuo pipefail
mkdir -p junit html artifacts
if [[ $1 == "serial" ]]; then
echo "Running serial tests"
find tests/serial/*.spec.ts | xargs -n 1 -P 20 -I{} sh -c \
'export PLAYWRIGHT_HTML_REPORT="html/$(basename {})"; \
export PLAYWRIGHT_JUNIT_OUTPUT_NAME="junit/results_$(basename {}).xml"; \
npx playwright test {} --reporter=junit,html,list'
elif [[ $1 == "parallel" ]]; then
echo "Running parallel tests"
PLAYWRIGHT_HTML_REPORT="html/parallel_tests" PLAYWRIGHT_JUNIT_OUTPUT_NAME="junit/results_parallel_tests.xml" \
npx playwright test tests/parallel/*.spec.ts --reporter=junit,html,list
else
echo "$1 is not a valid option"
exit 1
fi
The magic lies in the variables PLAYWRIGHT_HTML_REPORT and PLAYWRIGHT_JUNIT_OUTPUT_NAME, check the docs for additional info.
Now it's also easy to add this stuff, if running in the CI, to the job artifacts. Here's my example for Gitlab:
run_your_test:
# ...
script:
- npm run test
- cp -r your/test/dir/results results
artifacts:
name: ${CI_COMMIT_REF_SLUG}-${CI_JOB_NAME}
expire_in: 5 days
paths:
- results/html
reports:
junit:
- results/junit/results_*.xml
when: always
EDIT:
I noticed that the approach with the variables is very limited (other settings from the global config get ignored when using them), so I opted for modifying the playwright.config.ts on every run instead:
#!/bin/bash
mkdir -p junit html artifacts
reset_playwright_settings () {
echo "Settings will be changed back to the defaults"
sed -i "s/junit\/results.*.xml/junit\/results.xml/g" playwright.config.ts
sed -i "s/outputFolder: 'html.*',/outputFolder: 'html',/g" playwright.config.ts
}
modify_playwright_settings () {
echo "Settings will be changed for $1"
sed -i "s/junit\/results.*.xml/junit\/results_$1.xml/g" playwright.config.ts
sed -i "s/outputFolder: 'html.*',/outputFolder: 'html\/$1',/g" playwright.config.ts
}
export -f modify_playwright_settings
if [[ $1 == "serial" ]]; then
echo "Running serial tests"
find tests/serial/*.spec.ts | xargs -I{} bash -c 'modify_playwright_settings $(basename {}); npx playwright test {}'
elif [[ $1 == "parallel" ]]; then
echo "Running parallel tests"
modify_playwright_settings "parallel_tests"
npx playwright test tests/parallel/*.spec.ts
else
echo "$1 is not a valid option"
exit 1
fi
reset_playwright_settings
and I call my script like this (in the package.json):
{
// ...
"scripts": {
"test": "npm run test-serial; npm run test-parallel",
"test-parallel": "./run_playwright_tests.sh parallel",
"test-serial": "./run_playwright_tests.sh serial"
}
}
