Run PHPStan programmatically and get analysis results

[maks-rafalko]

Hello.

My question/request is quite specific because we need to run PHPStan under the hood of @infection - Mutation Testing library - to check if modified files still pass SA.

attention: understanding of Mutation Testing is needed to continue reading, so please read at least this Introduction

Why we need optional Static Analysis for killing Mutants is well explained here: https://github.com/Roave/infection-static-analysis-plugin?tab=readme-ov-file#background - this is a "brutal" implementation of Psalm integration with Infection as a standalone library.

What I want to achieve is basically the same, but with PHPStan, and not as a hacky solution but natively in Infection.

So, keeping Infection's implementation/integration out of this topic, here is what we really need:

• we have project files
• Infection mutates (changes) one single operator on one line of code at a time, e.g. changes $a + b to $a - $b in src/Calculator.php - and creates a tempory file with this change, e.g. /tmp/infection/Calculator.s0m3h4sh.php and runs tests as a separate process (in parallel with other "Mutants")
• we need to run PHPStan programmatically inside Infection execution so that it checks if /tmp/infection/Calculator.s0m3h4sh.php is still valid / passes PHPStan checks.

If new code (/tmp/infection/Calculator.s0m3h4sh.php) does not pass PHPStan check, we need to know result error messages (like Parameter $date of method HelloWorld::sayHello() has invalid typehint type DateTimeImutable.)

This is how it's currently implemented in Roave's infection static analysis plugin for Psalm, where Psalm is executed programmatically from the php code:

https://github.com/Roave/infection-static-analysis-plugin/blob/1.36.x/src/Roave/InfectionStaticAnalysis/Psalm/RunStaticAnalysisAgainstMutant.php#L28-L58

Questions that I have before playing with this:

• is something like that already supported in PHPStan?
• should we run PHPStan against only 1 (changed) file or againts all the original project's files with the only one file replaced? (like src/Calculator.php is replaced with /tmp/infection/Calculator.s0m3h4sh.php? I'm pretty sure that we need to run PHPStan against all the files, because there can be such mutations (changes) that produce error only when modifed Calculator class is used outside in other classes (like changing public to private for methods)

For example, if we do the following mutation:

class Calculator {
-    public function add(int $a, int $b): int 
+    protected function add(int $a, int $b): int 

then PHPStan will not find an issue if we run it against only this file. But it will find an issue if we run PHPStan against all the files in the project.

[ondrejmirtes]

I really like this idea, it'd make mutation testing a lot more useful! As for how to do this, you could certainly run PHPStan programatically inside other PHP application, but you'd lose a lot of benefits. If you simply run PHPStan on the command line as an external program, you get a lot for free and it's certainly easier. You get parallelism out of the box, and also the result cache (which is very important for this).

So in order to achieve your goal, I'd add more CLI options to the main analyse command for your needs. In fact, PHPStan used to contain these options for PHPStan Pro needs (it also modified a single file temporarily and saw what errors it fixed or added). So I could carefully revert and review this commit (phpstan/phpstan-src@f595e35) and this part would be done.

We'd introduce two new options: --tmp-file and --instead-of.

againts all the original project's files with the only one file replaced

Yes, you need to analyse the whole project, otherwise you miss out on errors. Fortunately the result cache can help a lot with the performance.

As for other considerations, I think you need to make sure that PHPStan passes with 0 errors before trying out temporary changes in the mutated file. I think this can be simply a prerequisite for Infection to work, similarly to the test suite being green before you run Infection. The baseline would certainly make that easy.

Also: you need to ask the user how to run PHPStan. So these things would have to be added to Infection config I think:

• Path to PHPStan (could be autodetected from Composer)
• Path to config file (if not specified, phpstan.neon and phpstan.neon.dist are auto-discovered in current working directory)
• Current working directory (usually project root but people do weird things)
• Analysed paths (they can be present in the config but the user can also pass them as arguments when running PHPStan)
• Other CLI options like --level, --autoload-file, --memory-limit

One more important question: Is Infection interested in actual error messages reported by PHPStan, or not? If not, we could also add an option --stop-on-first-error that would speed up the process. If Infection is interested in the actual error messages, we could add a new error formatter (https://phpstan.org/user-guide/output-format) to suit your needs.

Let me know if I'm right and all of this would be sufficient to adopt PHPStan in Infection.

[maks-rafalko]

sounds good, thanks

[Wirone]

Out of curiosity (cleaning up Github notifications): was something implemented in terms of this feature request?

[maks-rafalko]

nope. Waiting for @ondrejmirtes support for --tmp-file and --instead-of

[ondrejmirtes]

Yes, please wait a bit longer, too much stuff going on right now :)

[ondrejmirtes]

I just finished phpstan/phpdoc-parser#264, which was another popular request that other open-source projects were blocked on, and Infection support is now at the top of my todolist, so you should see some movement around this soon.

[romm]

Just commenting to let you know that I'm interested on giving the solution a try, whenever it is ready for early testing. 😊

Thank you for making PHP better everyday! ❤️

[Connum]

Dropping a comment to let you know that I'm also interested in running PHPStan programmatically from a class helping with development in a certain plugin environment.

[ondrejmirtes]

Hey, so I just made the first step and implemented --tmp-file and --instead-of: phpstan/phpstan-src@74b909a

Available in both 1.12.x and 2.1.x (not yet released).

I also forked https://github.com/Roave/infection-static-analysis-plugin as a proof of concept: https://github.com/phpstan/mutant-killer-infection-runner

Please test it on a real project to see if it actually kills some mutants 😊

I expect Infection to add support for PHPStan internally, the linked repo is just a proof of concept.

We can discuss this in Poland on Saturday @maks-rafalko 😊

[ondrejmirtes]

I also expect to add —stop-on-first-error to help with speed.

[ondrejmirtes]

I just published a documentation page about it: https://phpstan.org/user-guide/editor-mode

[maks-rafalko]

For those who is interested: please try dev-master (UPD: released in 0.30.0) of Infection. See a demo of native Infection + PHPStan integration here: Roave/BetterReflection#1510 (comment)

bin/infection --static-analysis-tool=phpstan --threads=4

[...]

A: killed by Static Analysis

.................A.............AAM.A....A.........   (3650 / 4409)
.......A......M.MM.AA..A.......A...A.........A....   (3700 / 4409)
......M.AAA...A.................A.A.M..MA....A....   (3750 / 4409)
.................A.......A.....................M..   (3800 / 4409)
.AA...A..A...............A.AAA......A............E   (3850 / 4409)
E..A..........A...................................   (3900 / 4409)
..........................A.......................   (3950 / 4409)
..............................T...................   (4000 / 4409)
..................................................   (4050 / 4409)
.........T........................................   (4100 / 4409)
..................................................   (4150 / 4409)
T..............E..................AA..............   (4200 / 4409)
.........M................................A.......   (4250 / 4409)
.....................................A..........E.   (4300 / 4409)
E.E.............................................AA   (4350 / 4409)
....................A.............A...............   (4400 / 4409)
......A.A                                            (4409 / 4409)

4409 mutations were generated:
    3660 mutants were killed by Test Framework
+    212 mutants were killed by Static Analysis
       0 mutants were configured to be ignored
      11 mutants were not covered by tests
      47 covered mutants were not detected
       7 errors were encountered
       0 syntax errors were encountered
       9 time outs were encountered
     463 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 98%
         Mutation Code Coverage: 99%
         Covered Code MSI: 98%

It's not yet officially launched, the main problem for now is the speed, this is yet to be addressed. However, if you subscribed to this discussion you are probably interested in this - so please test and report any issues so we can investigate. Let me know if you need any help!

Custom phpstan executable path or custom config path are also supported, just add this to infection.json5 if needed:

{
    "$schema": "./resources/schema.json",
    "phpStan": {
        "configDir": "custom/config/dir",
        "customPath": "custom/phpstan/executable"
    }
}

[Ocramius]

the main problem for now is the speed

Note that speed being an issue has always been true with this approach: MT is slow, and SA on top of it is expectedly slow as well. Optimizations will be forced by introducing the approach in the ecosystem at first, making the pain visible.