<!-- This issue template can be used as a great starting point for feature requests. Learn more about the process: https://handbook.gitlab.com/handbook/product/how-to-engage/#customer-feature-requests. The section "Release notes" can be used as a summary of the feature and is also required if you want to have your release post blog MR auto generated using the release post item generator: https://about.gitlab.com/handbook/marketing/blog/release-posts/#release-post-item-generator. The remaining sections are the backbone for every feature in GitLab. The goal of this template is brevity for quick/smaller iterations. For a more thorough list of considerations for larger features or feature sets, you can leverage the detailed [feature proposal](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal%20-%20detailed.md). --> ### Release notes <!-- What is the problem and solution you're proposing? This content sets the overall vision for the feature and serves as the release notes that will populate in various places, including the [release post blog](https://about.gitlab.com/releases/categories/releases/) and [Gitlab project releases](https://gitlab.com/gitlab-org/gitlab/-/releases). " --> ### Problem to solve [MR pipelines](https://docs.gitlab.com/ee/ci/pipelines/merge_request_pipelines.html) are a feature that a number of customers are interested in, however, delivering that feature across all of the Sec templates has proven difficult. As it is a significant change, attempts have been made, and failed during a few different milestones. This issue should be used to coordinate how we might deliver this, including alternate approaches. Some context for the most recent discussions that led to not delivering this in %16.0 can be found [here](https://gitlab.com/gitlab-org/gitlab/-/issues/388988#note_1372629948) and [here](https://gitlab.com/gitlab-org/gitlab/-/issues/388988#note_1372760461). Reminder: [CI/CD Templates are deprecated](https://docs.gitlab.com/ee/development/cicd/templates.html) and that will need to be taken into consideration when delivering this work. ### Additional details MR pipeline support has been added to the `.latest` templates and the remaining blocker to transition the stable template to them is the AutoDevOps support. Indeed, some security features (ex: Container Scanning, DAST, and Fuzzing jobs) have dependencies on the `Build` job, which doesn't run in MR pipelines. Good summary from Lucas: https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2152506266 Some part of the overall problem to solve can be addressed with other approaches (like allowing Scan Execution Policies to use the `.latest` template) but eventually some features really require to support MR pipelines to provide a relevant experience. That's the case for Secret Detection for instance, which uses MR pipelines enablement as a trigger to scan all commits in the MR diff rather than only scanning the head of the branch. <!-- What is the user problem you are trying to solve with this issue? --> ### Proposal <!-- Use this section to explain the feature and how it will work. It can be helpful to add technical details, design proposals, and links to related epics or issues. --> <details> <summary>See original proposals</summary> #### 1. Use CI/CD components Handle MR pipelines support in CI/CD components only. Do not touch existing CI/CD templates. - Pros: - No risk of impacting existing users who don't want to use MR pipelines - Incentivize to migrate over CI/CD components to benefit from newer features - No breaking change. CI templates are not changed and existing CI/CD components that require an update can release a new Major version without impacting users of current version. - Cons: - Customers have to migrate from CI templates to CI/CD components to benefit from MR pipelines support. - CI/CD components are not mature enough for a viable UX in self managed environments. https://gitlab.com/groups/gitlab-org/-/epics/15224#note_2283373852. - Scan Execution Policies do not work with CI/CD components. This requires work not currently defined and planned. https://gitlab.com/gitlab-org/gitlab/-/issues/489904#note_2172295365 - All security features must provide CI/CD components. https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2287739349 - All security features might need to align how their CI/CD components are implemented vs rules and MR pipelines. - Maintains discrepancy between the "latest" and "stable" CI templates. #### 2. Use CI templates, and add dedicated `branch` template for ADO Update the `rules` in the existing "stable" CI templates (similar to what we already have in `latest` templates) to provide the MR pipeline support. Additionally, create new `branch` CI templates to mimic old behavior and use these in ADO to maintain existing support for branch only usage. - Pros: - All existing customers of Sec feature will benefit from the change without any manual action. - Provide MR pipeline support with Scan Execution Policies - Keeps ADO support as it is (do not break features depending on `build` job). - Cons: - Risk of impacting all existing customer, including these who don't use MR pipelines. - Long term maintenance cost for additional "branch only" template - No MR pipeline support in ADO (consistent accross all features) TODO: Would this be considered a breaking change? :thinking: Since ADO is meant for implicit usage, changing an underlying included template should have no impact. => Yes. Some customers might have done some changes in their CI config after enabling MR pipelines to ensure some jobs still run in branch pipeline too, to fulfill the needs for some of our jobs (e.g keeping the build job for CS to work in branch pipeline). If we flip the CS job to support running in MR pipelines, then their custom setup might break. #### 3. Use CI templates, and remove ADO support when necessary Same as proposal 2, but remove ADO support for features no longer compatible with MR pipeline in this context rather than adding dedicated `branch` templates. Compatible features are kept in ADO and provide MR pipeline support. - Pros: - All existing customers of Sec feature will benefit from the change without any manual action - Provides MR pipeline support with Scan Execution Policies - MR pipeline support in ADO for compatible features - Cons: - Risk of impacting all existing customers, including these who don't use MR pipelines. - BREAKING CHANGE: Removes ADO support for some features (introduce discrepancy). #### 4. other alternatives to review: 1. Adjust MR widget, MR comparison logic and policies to consider branch pipelines too as source of scan results? Let's imagine we don't run Security scans at all in CI jobs, but in the backend or an external service once a branch or an MR is created. Their execution would not be tied to the configuration of MR pipeline or branch pieline. The creation of a pipeline or an MR would trigger a scan execution based on the content of that branch. So we might actually don't really care in which type of pipeline the security scan ran? As far as a scan has run for a given ref, should it be relevant to be considered? - While this could alleviate some of the problems, this still won't be enough to cover some usages. For instance, Secret Detection must run in an MR pipeline setup to scan all commits in MRs, rather than just the head of the branch. 2. Make Scan Execution Policies able to work with `.latest` templates. Technically doable, but could tightly couple unrelated concepts and increase the risk of breaking SEPs which are usually leveraged by large enterprise customers who expect reliability. -> this has actually already been implemented with https://gitlab.com/gitlab-org/gitlab/-/issues/415427+ </details> From the original propoals above (expand to see them), and after a synchronous discussion (2025-01-13) between DRIs (@connorgilbert @mbenayoun @gonzoyumo) we've decided the following: - Proposal 1, using CI/CD components is rejected. This would not offer the expected customer results, requires too much work, and represents too much risk due to its current maturity level and it's current usage. - Proposal 2 and 3, using CI templates and either adding `branch` templates and/or (partially) dropping ADO support offers the best path forward. Deciding the details will depend on further investigation and discussion with the impacted groups. - Suggested alternatives are not sufficient to provide the expected customer results. #### UPDATE 2025-01-29 Further testing the proposal, we've realized that the current behavior of the `latest` template is actually enforcing AST CI/CD jobs to run in an MR pipelines. This means users who have their jobs running in a regular pipeline when an MR is created will face the duplicate pipeline issue. See https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2320969911. As a result the proposal is updated to consider an opt-in approach instead of the default enablement we have with the `latest` template. By checking an additional environment variable before enabling MR pipeline support in our CI/CD jobs, the `stable` templates would look like this: ```yaml rules: # Add the job to merge request pipelines if MR Pipelines for AST are enabled and there's an open merge request. - if: $AST_ENABLE_MR_PIPELINES == "true" && $CI_PIPELINE_SOURCE == "merge_request_event" # Don't add it to a *branch* pipeline if MR Pipelines for AST are enabled and there's an open merge request. - if: $AST_ENABLE_MR_PIPELINES == "true" && $CI_OPEN_MERGE_REQUESTS when: never # Add the job to branch pipelines. - if: $CI_COMMIT_BRANCH ``` This approach allows us to modify the `stable` CI/CD templates with: - no impact for existing users of the stable templates, particularly if they don't want to migrate to MR pipelines - no impact for ADO support, as the default behavior is still to run in a branch pipeline - no impact on SEP. If users don't want MR pipelines, they just leave the stable template without any change This means that there is no deprecation announcement to be made in 17.9 as there will be no breaking changes in 18.0. If users of Scan Execution Policies wants to enable MR pipelines, they will be able to use the `stable` template if they configure the additional environment variable or they can continue to use the `latest` templates. Here is an example MR simulating how a user would enable MR pipelines for AST features: https://gitlab.com/gitlab-org/secure/tests/olivier/mr-pipelines-testing/existing-with-mr-pipelines/-/merge_requests/3/diffs TODO: validate the possibility to implement this proposal before 18.0. If there is no breaking change, nothing prevents us from shipping the change today. TODO: consider if/when to align `latest` templates with `stable` templates ~~TODO: we may decide to take the inverse approach for Secret Detection, and have this particular feature enabling the MR pipeline by default. NB: if we want to do this, we can't simply set the shared variable to true at the global level in the SD template, otherwise it will enable it for all other included security features.~~ It has been decided that [SD will follow the same opt-in approach](https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2328738774). TODO: validate Merge **Results** pipelines too. See https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2321629558 ##### Next steps & timeline - [x] By 2025-01-17: Verify using ADO with MR pipelines is actually a thing that makes sense. E.g. if the build job does not support MR pipelines, is ADO at all working then? Is it worth trying to get our features working in ADO with MR pipelines enabled? - [x] By 2025-01-17: Devise an implementation plan and planning breakdown. - [x] By 2025-01-24: Gather feedback and get approval on the implementation plan from impacted group's leader/DRI and Sec leadership? - [x] :boom: 2025-01-29: Feedback from testing lead to an updated proposal - [x] By 2025-31: Test and validate the updated proposal. Get approval from group's leader. - [x] By 2025-02-07 Implement the necessary changes that must ship in 17.9 (announcement, doc update, etc.) ### Implementation plan :warning: The implementation plan has changed after [the proposal update from 2025-01-29](https://gitlab.com/gitlab-org/gitlab/-/issues/410880#update-2025-01-29). <details> <summary>See former proposal implementation plan</summary> - [ ] Each group validate the proposal for its feature(s). Please review [MR pipelines in AutoDevOps](https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2303963301) and [Behavior of each AST feature in the context of AutoDevOps](https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2303963541) to the necessary context - [x] Code Quality (@connorgilbert): make no changes because CodeClimate-based scanning is scheduled for removal in %18.0 (https://gitlab.com/gitlab-org/gitlab/-/issues/471677). - [x] SAST (@connorgilbert): add MR pipeline support to the "stable" CI template in 18.0 and keep it included in AutoDevOps. - [x] IaC (@connorgilbert): add MR pipeline support to the "stable" SAST-IaC CI template in 18.0 and keep it included in AutoDevOps (if it is there in the first place). - [x] Secret Detection (@abellucci): add MR pipeline support to the "stable" CI template in 18.0 and keep it included in AutoDevOps. - [x] Dependency Scanning (@johncrowley): add MR pipelines support to the "stable" CI template in 18.0 and keep it included in AutoDevOps. - [x] Container Scanning (@johncrowley): Pick one option of these options 1. [x] add a specific `Container-Scanning.branch.gitlab-ci.yml` template with no support for MR pipelines and use it in ADO. Then add MR pipelines support to the stable CI template. 2. [ ] Remove Auto Container Scanning feature from AutoDevOps (remove the inclusion of the CS template), and add MR pipelines support to the stable CI template. 3. [ ] Do the necessary work to add MR pipelines support to `Auto Build`, then add MR pipelines support to the stable CI template. 4. [ ] Other suggestion? - [x] DAST (@johncrowley): Pick one option of these options 1. [ ] add a specific `DAST.branch.gitlab-ci.yml` template with no support for MR pipelines and use it in ADO. Then add MR pipelines support to the stable CI template. 2. [x] Remove Auto DAST feature from AutoDevOps (remove the inclusion of the DAST template), and add MR pipelines support to the stable CI template. 3. [ ] Do the necessary work to add MR pipelines support to `Auto Build` and `Auto Deploy`, then add MR pipelines support to the stable CI template. 4. [ ] Other suggestion? </details> | Template | Supports MR pipelnes? | Should migrate | | ---------------------------------------- | --------------------- | -------------------- | | API-Discovery.gitlab-ci.yml | Yes | Yes (in 18.0) | | API-Fuzzing.gitlab-ci.yml | No | Yes | | API-Fuzzing.latest.gitlab-ci.yml | Yes | Yes | | API-Security.gitlab-ci.yml | No | Yes | | API-Security.latest.gitlab-ci.yml | Yes | Yes | | BAS.latest.gitlab-ci.yml | Yes | No (feature removed) | | Container-Scanning.gitlab-ci.yml | No | Yes | | Container-Scanning.latest.gitlab-ci.yml | Yes | Yes | | Coverage-Fuzzing.gitlab-ci.yml | No | No (feature deprecated) | | Coverage-Fuzzing.latest.gitlab-ci.yml | Yes | No (feature deprecated) | | DAST-API.gitlab-ci.yml | No | Yes | | DAST-API.latest.gitlab-ci.yml | Yes | Yes | | DAST-On-Demand-API-Scan.gitlab-ci.yml | No | No (irrelevant) | | DAST-On-Demand-Scan.gitlab-ci.yml | No | No (irrelevant) | | DAST-Runner-Validation.gitlab-ci.yml | No | No (irrelevant) | | DAST.gitlab-ci.yml | No | Yes | | DAST.latest.gitlab-ci.yml | Yes | Yes | | Dependency-Scanning.gitlab-ci.yml | No | Yes | | Dependency-Scanning.latest.gitlab-ci.yml | Yes | Yes | | SAST-IaC.gitlab-ci.yml | No | Yes | | SAST-IaC.latest.gitlab-ci.yml | Yes | Yes | | SAST.gitlab-ci.yml | No | Yes | | SAST.latest.gitlab-ci.yml | Yes | Yes | | Secret-Detection.gitlab-ci.yml | No | Yes | | Secret-Detection.latest.gitlab-ci.yml | Yes | Yes | - [x] In 17.9. announce upcoming breaking change for API discovery template - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/181047+ - [x] Update the CI templates as listed above - [x] For `stable` templates, add the option to enable MR pipeline when setting `AST_ENABLE_MR_PIPELINES` to `"true"`. NB: `API Discovery` will change in 18.0, see https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2340997686 - [x] For `latest` templates, add the same rules, but set this options to `"true"` by default in the template (global variable). See reasoning in https://gitlab.com/gitlab-org/gitlab/-/issues/410880#note_2331261884. - [x] Ensure automated tests are covering the new behavior - [x] Ensure each group validates the change, review, and approve the MR - [x] Update the documentation - https://docs.gitlab.com/ee/user/application_security/detect/roll_out_security_scanning.html#use-security-scanning-tools-with-merge-request-pipelines - notice below the table in https://docs.gitlab.com/ee/user/application_security/policies/scan_execution_policies.html#scan-action-type - notice in https://docs.gitlab.com/user/application_security/policies/merge_request_approval_policies/#merge-request-with-multiple-pipelines - [x] Pending MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/179652 - [ ] add a release post item to announce the new feature - [ ] MR: https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/138176+ - [x] add the `AST_ENABLE_MR_PIPELINES` CI/CD variable to the list of suggested variables in the Scan Execution Policy editor. - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/186739+ - [x] Update API Discovery template in 18.0 to align with this change and default to branch pipeline, assign to DA team and ensure a DRI is assigned. - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/188334+ - [x] advertise internally that we can switch back to using the `stable` template to use MR pipelines (e.g. ping infra team `@andrewn`)