[GHAS CodeQL Series] - Part 2: Implementing Alert-Mode Repository Rulesets

[vishaljsoni]

GitHub Advanced Security CodeQL Series - Part 2: Implementing Alert-Mode Repository Rulesets

Welcome back to our GitHub Advanced Security CodeQL series! In Part 1, we enabled CodeQL scanning across repositories by using security configurations. Once CodeQL is enabled, developers already get pull request scanning, net new alerts, annotations, and Copilot Autofix suggestions for supported alerts during pull requests.

This post is about the next step: using repository rulesets to operationalize that alert-first experience at scale. In other words, rulesets do not create CodeQL alerts in pull requests, but they do help you standardize coverage, require the code scanning check to run, and support compliance goals without blocking merges on new findings yet.

Series Overview

• Part 1: Setting Up Organization-Wide Code Scanning ✅
• Part 2: Implementing Alert-Mode Repository Rulesets (This post)
• Part 3: Blocking Vulnerable Code Merges (Coming Soon)

Why Start with Alert Mode?

Before jumping straight to blocking vulnerable code merges, there is significant value in starting with an alert-first approach.

Developer Benefits

• Shifts security left by catching net new issues before they reach the default branch
• Reduces rework and remediation cost by fixing problems while the change is still fresh
• Brings security into the review flow through pull request checks and annotations
• Provides Autofix assistance for supported alerts while keeping developers in control
• Builds confidence in CodeQL's accuracy and relevance before stricter enforcement

Organizational Benefits

• Establishes baseline metrics for new security findings in pull requests
• Verifies scanning coverage across repositories that should be running CodeQL
• Allows tuning of thresholds and triage practices before blocking merges
• Generates buy-in from development teams through a lower-friction rollout

What Developers Already Get Before a Ruleset

This is the most important framing for Part 2: if CodeQL is already enabled and scanning pull requests, developers can already see code scanning results in their pull requests even when no ruleset exists.

• Code scanning results appear as a pull request check
• Net new security findings are surfaced in the pull request
• Annotations can appear directly in the diff
• Copilot Autofix can be offered for supported alerts

Repository rulesets become valuable when you want to make that experience consistent and governable across repositories or across an organization.

Understanding Repository Rulesets vs. Branch Protection Rules

Repository rulesets represent GitHub's modern approach to repository governance, offering significant advantages over traditional branch protection rules.

Key Advantages of Rulesets

Feature	Branch Protection Rules	Repository Rulesets
Scope	Repository-level only	Repository-level and organization-level management
Flexibility	Limited conditions	Granular control with complex conditions
Code Scanning	Generic required status checks	Native code scanning merge protection with alert thresholds
Future Support	Legacy approach	GitHub's recommended modern approach
Management	Per-repository configuration	Centralized organization rollout or repository-specific rollout

Why Choose Rulesets for Code Scanning

Repository rulesets provide native support for code scanning merge protection, making them the best choice when you want to require the code scanning check itself and standardize thresholds across repositories. That is especially useful for compliance programs and larger engineering organizations that need consistent governance.

Step-by-Step Implementation Guide

Step 1: Navigate to Repository or Organization Rulesets

Rulesets can be managed at either the repository level or the organization level:

• Repository rulesets are useful for piloting or tailoring rollout to a single repository
• Organization rulesets are ideal when you want centralized governance across many repositories

To create an organization ruleset:

1. Go to your organization's main page on GitHub and click Settings
2. In the left sidebar, in the "Code, planning, and automation" section, click Repository, then click Rulesets
3. Click New ruleset

Step 2: Create a New Branch Ruleset

Now create a branch ruleset specifically for non-blocking code scanning governance:

1. Click New branch ruleset
2. Give your ruleset a descriptive name: "CodeQL Alert Mode - Governed Repositories"
3. Add a meaningful description: "Requires CodeQL scanning to run while keeping new security findings non-blocking during rollout."

Step 3: Configure Ruleset Targets

This step determines which repositories the ruleset applies to.

Enforcement Status

• Select "Active" when you are ready to require the code scanning check on the targeted repositories
• Consider "Evaluate" if you want to preview the governance impact of the ruleset before turning it on

Note

Evaluate mode is helpful for understanding how a ruleset would behave, but it does not control whether developers see pull request alerts. Pull request alerts come from CodeQL scanning itself.

Target Repositories

Choose your scope based on your organization's needs:

• All repositories: Applies to all current and future repositories
• Dynamic list by property: Filter repositories by language, visibility, or team ownership
• Selected repositories: Choose specific repositories for a staged rollout

Step 4: Set Branch Targeting

Define which branches the branch ruleset should protect:

Recommended Configuration

• Default branch: Use the "Include default branch" option to target main or whichever branch is your default
• Pattern-based: Use patterns like release/* for release branches, or a specific branch name like main to target it directly

Advanced Targeting

For organizations with complex branching strategies:

• All branches: Comprehensive protection, but potentially too broad for an initial rollout
• Dynamic list by name: Use patterns for flexible targeting

Step 5: Configure Code Scanning Settings (Non-Blocking Findings)

This is the step that keeps new findings non-blocking while still requiring the code scanning check:

1. Under "Branch protections", select Require code scanning results
2. Click Add tool and select CodeQL from the dropdown
3. Configure the thresholds:
   • Alerts: Select "None"
   • Security alerts: Select "None"

Understanding What "None" Actually Means

Setting both thresholds to "None" means:

• New security findings will not block merges
• The code scanning check itself is still required
• Repositories without CodeQL properly configured can still be blocked
• Pull request checks, annotations, and Autofix suggestions still come from CodeQL scanning

This can feel a little strict at first, because you are not blocking on new findings, but you are still making the scan mandatory. That tradeoff is often acceptable when your goal is compliance, coverage verification, or change management across a large set of repositories.

If your goal is only to give developers visibility into CodeQL findings in pull requests, you may not need a ruleset yet. CodeQL already provides that experience when pull request scanning is enabled.

Step 6: Optional Complementary Protection Rules

While keeping security findings non-blocking, you can still layer in other protections that make sense for your software delivery process.

Pull Request Requirements

• Require a pull request before merging: Ensures review happens in the open
• Require approvals: Set the minimum number of reviewers
• Dismiss stale reviews: Keep approvals current with code changes

Status Check Requirements

• Require status checks to pass before merging: Include your CI and test pipeline
• Require branches to be up to date: Ensure code cannot be merged until changes from the target branch have been merged into the source branch

Code Quality

• Require code quality results: Block merging on Errors if GitHub Code Quality is available in your environment

Sensitive File Protections

If you want to restrict changes to specific file paths, treat that as a separate control. File path restrictions are typically handled with a push ruleset, not this branch ruleset.

Step 7: Create and Activate the Ruleset

Final steps to implement your alert-mode ruleset:

1. Review your configuration settings
2. Verify the repository scope matches your rollout plan
3. Click Create to save and activate the ruleset
4. Confirm the targeted repositories are running CodeQL as expected

What This Implementation Achieves

Your implementation now gives you two complementary outcomes.

1. Developer Education from CodeQL Pull Request Scanning

• Net new security findings are surfaced before code reaches the default branch
• Annotations and check results become part of the normal review process
• Autofix assistance can speed up remediation for supported alerts
• Developers stay unblocked while teams learn and build confidence

2. Governance and Compliance from Rulesets

• Coverage verification across targeted repositories
• Consistent thresholds across repositories or across an organization
• A required code scanning check for governed branches
• A cleaner path to future enforcement when you are ready for blocking mode

Monitoring and Measuring Success

Key Metrics to Track

Coverage Metrics

• Repositories with active scanning: Track which repositories are actually running CodeQL
• Repositories covered by rulesets: Confirm governance scope matches policy intent
• Scan completion rates: Monitor for repositories with configuration or workflow issues

Pull Request Alert Metrics

• Net new security findings: Volume of new issues introduced in pull requests
• Alerts fixed with Copilot Autofix: AI-assisted remediation outcomes
• Alerts fixed manually: Findings resolved by developers before merge
• Alerts unresolved and merged: Risk that still reached the default branch
• Alert categories and rule hotspots: The classes of issues being introduced most often
• Dismissal reasons and false positives: Signal for tuning and education

Developer Engagement

• Annotation engagement: Whether teams are acting on in-context pull request feedback
• Time to remediate: How quickly findings are addressed
• Training requests: Signals that teams want deeper support on recurring issues

Monitoring Tools and Approaches

GitHub Security Overview

Access organization-level visibility:

1. Navigate to your organization's Security tab
2. Review Security overview
3. Use the CodeQL pull request alerts report to understand prevented alerts, unresolved and merged alerts, Autofix usage, and remediation trends

Repository-Level Monitoring

For individual repository insights:

1. Open a pull request where CodeQL is running
2. Review the Code scanning results check
3. Review pull request annotations and Autofix suggestions for supported alerts
4. Review the Security tab for alert triage history

API-Based Reporting

For alert inventory reporting, the GitHub CLI can give you better pagination support and more context than a single REST call:

# List open code scanning alerts across an organization with pagination and useful context
gh api --paginate \
  -H "Accept: application/vnd.github+json" \
  "/orgs/$ORG/code-scanning/alerts?state=open&per_page=100" \
  --jq '.[] | {repo: .repository.full_name, number: .number, rule: .rule.id, severity: (.rule.security_severity_level // .rule.severity), state: .state, ref: .most_recent_instance.ref}'

Use API output for inventory and operational reporting, and use Security overview for the richer pull request alert and Autofix metrics.

Best Practices for Alert Mode Success

Communication Strategy

• Explain the default developer experience: CodeQL already surfaces net new findings in pull requests
• Explain what the ruleset adds: governance, required scanning, and standardized thresholds
• Provide training materials on triaging alerts and using Autofix responsibly
• Share success stories of vulnerabilities prevented before merge

Developer Support

• Create documentation for common vulnerability types
• Establish office hours for security questions
• Build remediation guides with code examples
• Integrate findings into existing code review practices

Continuous Improvement

• Review rulesets regularly to adjust scope and thresholds
• Use Evaluate mode when previewing governance changes before enforcement
• Track net new findings and Autofix metrics to measure progress
• Prepare for blocking mode only after teams and tooling are ready

Preparing for Part 3: Blocking Mode

After a period of successful alert-first operation, you'll be ready to transition to blocking controls. Monitor these indicators:

Readiness Signals

• Teams are consistently fixing net new findings before merge
• Autofix and manual remediation rates are healthy
• False positive rates are low and understood
• Scanning coverage is stable across all targeted repositories
• Developers understand why findings are being raised

Pre-Blocking Checklist

• Scanning coverage is verified across targeted repositories
• Alert thresholds are tested and validated
• False positive handling processes are established
• Developer training is complete
• Exception handling procedures are defined

Troubleshooting Common Issues

Ruleset Not Applying

Symptoms: Target repositories are not showing ruleset enforcement

Solutions:

• Verify repository targeting configuration
• Confirm whether you created the ruleset at the repository or organization level
• Check permissions for the administrators managing the ruleset

Missing Code Scanning Results

Symptoms: No code scanning feedback is appearing on pull requests

Solutions:

• Confirm CodeQL is enabled through security configurations or advanced setup
• Verify pull request scanning is configured for the repository
• Check that GitHub Actions is enabled for the repository
• Confirm the repository contains supported languages

Developers Expect the Ruleset to Create Alerts

Symptoms: Teams think the ruleset is what turns pull request alerts on

Solutions:

• Clarify that CodeQL scanning produces the alerts, annotations, and Autofix suggestions
• Explain that the ruleset governs merge protection behavior and scan requirements
• Use screenshots from real pull requests to show the distinction

What's Next?

Excellent work! You now have a clearer operating model: CodeQL gives developers alert-first pull request security feedback by default, and repository rulesets help you standardize and govern that experience across repositories.

In Part 3 of this series, we'll cover the final step: moving from non-blocking findings to blocking merges on the thresholds that match your team's readiness and risk tolerance.

Coming up in Part 3:

• Implementing blocking repository rulesets
• Choosing appropriate alert and severity thresholds
• Managing exceptions and false positives
• Emergency bypass procedures
• Organization-wide rollout strategies

Key Takeaways

• CodeQL pull request scanning already provides alert-first developer feedback through checks, annotations, and Autofix for supported alerts
• Repository rulesets add governance and compliance value by requiring the code scanning check and standardizing thresholds
• Setting thresholds to "None" keeps new findings non-blocking while still making the scan itself mandatory
• Tracking net new findings, Autofix usage, and unresolved merged alerts gives you the right readiness signals for Part 3
• A progressive rollout helps teams adopt security controls with less friction

Additional Resources

Essential Documentation:

• GitHub Docs: About code scanning alerts
• GitHub Docs: Set code scanning merge protection
• GitHub Docs: Available rules for rulesets
• GitHub Docs: CodeQL pull request alert metrics

Changelog and Guidance:

• GitHub Changelog: Prevention and autofix insights for CodeQL pull request alerts
• GitHub Changelog: Enhanced metrics for CodeQL pull request alerts and Copilot autofixes
• GitHub Changelog: Code scanning rulesets to prevent pull requests from being merged
• GitHub Changelog: GitHub Code Quality in public preview
• GitHub Well-Architected

Community Resources:

• GitHub Community Discussions
• CodeQL Documentation
• Security Best Practices Guide

---

This is Part 2 of our GitHub Advanced Security CodeQL series. Continue to Part 3 when you're ready to turn proven, non-blocking visibility into targeted enforcement.

Series Navigation:

• Part 1: Setting Up Organization-Wide Code Scanning
• Part 2: Alert-Mode Repository Rulesets (You are here)
• Part 3: Blocking Vulnerable Code Merges (Coming Soon)