Skip to content

Preserve hand-written entries when regenerating container configs - #4476

Open
pinin4fjords wants to merge 2 commits into
nf-core:devfrom
pinin4fjords:container-config-manual-marker
Open

pinin4fjords wants to merge 2 commits into
nf-core:devfrom
pinin4fjords:container-config-manual-marker

Conversation

@pinin4fjords

Copy link
Copy Markdown
Member

Summary

  • generate_container_configs() (behind nf-core pipelines lint --fix container_configs and bulk modules update/remove/patch) unconditionally deleted and rewrote every conf/containers_*.config file from a fresh meta.yml scan, with no way to keep hand-written entries for modules that don't carry per-platform containers: metadata yet.
  • This is the marker convention the Seqera Containers Part 2 blog post describes (// AUTOGENERATED CONFIG BELOW THIS POINT - DO NOT EDIT), but it was never actually implemented when the generation feature shipped in Command to generate pipeline container config files #3955 - I found this while trying to adopt the scheme in nf-core/rnaseq and hit real data loss risk from it.
  • Content above the marker is now preserved verbatim on every regeneration path (generate_container_configs() full scan and update_module_container_config() single-module update); only the generated section below the marker is replaced. Files with no generated entries but existing manual content are kept rather than deleted. Files without a marker at all (i.e. anything written by a pre-this-PR version of the tool) are treated as fully generated, matching current behaviour, so nothing changes for existing pipelines until they add manual content above a marker themselves.
  • Updated the container_configs lint docstring to document the convention.

Test plan

  • Added tests covering: manual prefix survives a full regen, a removed module's generated entry disappears while manual content stays, a manual-only file (no generated entries) isn't deleted, and the single-module update path also preserves manual content.
  • uv run pytest tests/pipelines/test_container_configs.py tests/pipelines/lint/test_container_configs.py - 14 passed
  • uv run ruff check / uv run ruff format --check on changed files - clean
  • uv run mypy on changed files - no new errors (pre-existing errors elsewhere in the codebase are unrelated to this change)
  • Confirmed against a clean dev checkout that the unrelated tests/modules/ failures in this sandbox (network access to GitHub/GitLab) are pre-existing and not caused by this change

🤖 Generated with Claude Code

pinin4fjords and others added 2 commits September 21, 2026 11:24
generate_container_configs() previously deleted and rewrote each
conf/containers_*.config file purely from a meta.yml scan, silently
discarding any hand-added overrides for modules without per-platform
container metadata. This implements the marker convention the
Seqera Containers Part 2 blog post describes but that never shipped:
content above `// AUTOGENERATED CONFIG BELOW THIS POINT - DO NOT EDIT`
is preserved, only the generated section below it is replaced.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 78.22%. Comparing base (d91668e) to head (1fe2807).

Files with missing lines Patch % Lines
nf_core/pipelines/containers_utils.py 95.45% 1 Missing ⚠️

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant