Run the regression benchmarks faster by stephenamar-db · Pull Request #543 · databricks/sjsonnet

stephenamar-db · 2025-10-31T23:47:21Z

add ./mill bench.listRegressions
rename runner to ./mill bench.runRegressions and accepts parameters to run select test cases.
tweak numbers of warmup/rounds

TBA:

Add FileTest to verify golden files for the test cases.

Current results from HEAD on my laptop:

[1-113] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
[1-113] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.267          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.064          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       34.823          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       14.813          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       40.949          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.378          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        3.541          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.050          ms/op
[1-113] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.111          ms/op
[1-113] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        0.995          ms/op
[1-113] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        1.688          ms/op
[1-113] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        3.992          ms/op
[1-113] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.437          ms/op
[1-113] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       79.496          ms/op
[1-113] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.864          ms/op
[1-113] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.665          ms/op
[1-113] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt       13.101          ms/op
[1-113] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.217          ms/op
[1-113] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt        7.627          ms/op
[1-113] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       63.315          ms/op
[1-113] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.054          ms/op
[1-113] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt       10.886          ms/op
[1-113] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.658          ms/op
[1-113] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.061          ms/op
[1-113] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.088          ms/op
[1-113] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.068          ms/op
[1-113] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.440          ms/op
[1-113] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.044          ms/op
[1-113] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       16.407          ms/op
[1-113] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.651          ms/op
[1-113] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.652          ms/op
[1-113] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.217          ms/op

stephenamar-db · 2025-11-01T03:59:07Z

For posterity, I also ran the benchmarks before #539

[1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main [1-113] RegressionBenchmark.main bench/resources/bug_suite/assertions.jsonnet avgt 0.256 ms/op
bench/resources/cpp_suite/bench.01.jsonnet avgt 0.068 ms/op
bench/resources/cpp_suite/bench.02.jsonnet avgt 35.366 ms/op
bench/resources/cpp_suite/bench.03.jsonnet avgt 14.803 ms/op
bench/resources/cpp_suite/bench.04.jsonnet avgt 40.943 ms/op
bench/resources/cpp_suite/bench.06.jsonnet avgt 0.381 ms/op
bench/resources/cpp_suite/bench.07.jsonnet avgt 3.584 ms/op
bench/resources/cpp_suite/bench.08.jsonnet avgt 0.051 ms/op
bench/resources/cpp_suite/bench.09.jsonnet avgt 0.112 ms/op
bench/resources/cpp_suite/gen_big_object.jsonnet avgt 1.003 ms/op
bench/resources/cpp_suite/large_string_join.jsonnet avgt 1.707 ms/op
bench/resources/cpp_suite/large_string_template.jsonnet avgt 3.990 ms/op
bench/resources/cpp_suite/realistic1.jsonnet avgt 3.312 ms/op
bench/resources/cpp_suite/realistic2.jsonnet avgt 82.222 ms/op
bench/resources/go_suite/base64.jsonnet avgt 0.850 ms/op
bench/resources/go_suite/base64Decode.jsonnet avgt 0.682 ms/op
bench/resources/go_suite/base64DecodeBytes.jsonnet avgt 13.828 ms/op
bench/resources/go_suite/base64_byte_array.jsonnet avgt 1.266 ms/op
bench/resources/go_suite/comparison.jsonnet avgt 7.756 ms/op
bench/resources/go_suite/comparison2.jsonnet avgt 65.240 ms/op
bench/resources/go_suite/escapeStringJson.jsonnet avgt 0.046 ms/op
bench/resources/go_suite/foldl.jsonnet avgt 11.227 ms/op
bench/resources/go_suite/lstripChars.jsonnet avgt 0.668 ms/op
bench/resources/go_suite/manifestJsonEx.jsonnet avgt 0.063 ms/op
bench/resources/go_suite/manifestTomlEx.jsonnet avgt 0.081 ms/op
bench/resources/go_suite/manifestYamlDoc.jsonnet avgt 0.066 ms/op
bench/resources/go_suite/member.jsonnet avgt 0.577 ms/op
bench/resources/go_suite/parseInt.jsonnet avgt 0.045 ms/op
bench/resources/go_suite/reverse.jsonnet avgt 16.743 ms/op
bench/resources/go_suite/rstripChars.jsonnet avgt 0.659 ms/op
bench/resources/go_suite/stripChars.jsonnet avgt 0.662 ms/op
bench/resources/go_suite/substr.jsonnet avgt 0.222 ms/op

He-Pin · 2025-11-01T05:08:40Z

Is it possible to render it in a table, with some statistics numbers

- add `./mill bench.listRegressions` - rename runner to `./mill bench.runRegressions` and accepts parameters to run select test cases. - tweak numbers of warmup/rounds TBA: - Add FileTest to verify golden files for the test cases. Current results from HEAD on my laptop: ``` [1-113] Benchmark (path) Mode Cnt Score Error Units [1-113] RegressionBenchmark.main bench/resources/bug_suite/assertions.jsonnet avgt 0.267 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.01.jsonnet avgt 0.064 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.02.jsonnet avgt 34.823 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.03.jsonnet avgt 14.813 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.04.jsonnet avgt 40.949 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.06.jsonnet avgt 0.378 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.07.jsonnet avgt 3.541 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.08.jsonnet avgt 0.050 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/bench.09.jsonnet avgt 0.111 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/gen_big_object.jsonnet avgt 0.995 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/large_string_join.jsonnet avgt 1.688 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/large_string_template.jsonnet avgt 3.992 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/realistic1.jsonnet avgt 3.437 ms/op [1-113] RegressionBenchmark.main bench/resources/cpp_suite/realistic2.jsonnet avgt 79.496 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/base64.jsonnet avgt 0.864 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/base64Decode.jsonnet avgt 0.665 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/base64DecodeBytes.jsonnet avgt 13.101 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/base64_byte_array.jsonnet avgt 1.217 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/comparison.jsonnet avgt 7.627 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/comparison2.jsonnet avgt 63.315 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/escapeStringJson.jsonnet avgt 0.054 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/foldl.jsonnet avgt 10.886 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/lstripChars.jsonnet avgt 0.658 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/manifestJsonEx.jsonnet avgt 0.061 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/manifestTomlEx.jsonnet avgt 0.088 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/manifestYamlDoc.jsonnet avgt 0.068 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/member.jsonnet avgt 0.440 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/parseInt.jsonnet avgt 0.044 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/reverse.jsonnet avgt 16.407 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/rstripChars.jsonnet avgt 0.651 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/stripChars.jsonnet avgt 0.652 ms/op [1-113] RegressionBenchmark.main bench/resources/go_suite/substr.jsonnet avgt 0.217 ms/op ```

Run the regression benchmarks faster

e01a529

stephenamar-db merged commit 22e264a into master Nov 1, 2025
6 checks passed

stephenamar-db deleted the bench2 branch November 1, 2025 03:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run the regression benchmarks faster#543

Run the regression benchmarks faster#543
stephenamar-db merged 1 commit intomasterfrom
bench2

stephenamar-db commented Oct 31, 2025 •

edited

Loading

Uh oh!

stephenamar-db commented Nov 1, 2025

He-Pin commented Nov 1, 2025

Labels

2 participants

Conversation

stephenamar-db commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stephenamar-db commented Nov 1, 2025

He-Pin commented Nov 1, 2025

Labels

2 participants

stephenamar-db commented Oct 31, 2025 •

edited

Loading