Skip to content

Switch Montage to the Toolforge buildservice (container images) #563

@lgelauff

Description

@lgelauff

Background

Toolforge's buildservice (docs)
builds container images from a git repository using Cloud Native Buildpacks, replacing the
language-specific toolforge webservice python3.13 setup.

Montage's current install has two sources of pain:

Python webservice: manual venv setup inside a shell pod — pip bootstrap with curl,
--without-pip workaround, shell-pod dance that can't run on the bastion. Breaks across
Python upgrades.

Frontend build: toolforge jobs --image node20 hits a persistent cross-platform esbuild
bug — package-lock.json generated on macOS (npm 10+) is misread by npm 9 in the node20
image, installing the wrong @esbuild/linux-x64 binary. Worked around with --ignore-scripts

  • explicit binary reinstall, but fragile.

The buildservice eliminates both.

Goal

Replace toolforge webservice python3.13 + tools/build_frontend.sh with a single
toolforge build start that builds the Python app and frontend assets together, then:

toolforge webservice buildservice start

No venv creation, no shell-pod dance, no pip workarounds, no esbuild cross-platform bug.

Why the frontend must be folded in (not kept as a separate job)

montage/static/ is gitignored. The buildservice image is built from git, so the image
contains an empty montage/static/. The app serves static files from that path inside the
image — a separately-built job writing to NFS would be invisible to the container.

The clean solution: build the frontend inside the buildpack so assets are baked into the
image. Buildpacks support Node + Python in the same image. Because the build now runs on
Linux, the cross-platform esbuild mismatch disappears entirely.

What needs to change

New files (repo root)

  • Procfile: web: gunicorn --bind=0.0.0.0:5000 --workers=4 --forwarded-allow-ips=* app:app
  • .python-version: 3.13 — the buildpack way to pin Python version

requirements.in / requirements.txt

Add gunicorn — the buildservice does not provide a WSGI server the way python3.13 webservice did (uwsgi).

montage/utils.py — two env detection fixes

get_env_name() uses getpass.getuser() → maps tools.montage-dev to devlabs. That
username doesn't exist inside a container. Fix: check MONTAGE_ENV env var first, fall back
to current behaviour for local dev.

load_env_config() loads config.<env>.yaml from the source tree (PROJ_PATH). Secrets
can't be in the image. Fix: check MONTAGE_CONFIG_DIR env var first; if set, load config
from there (i.e. $TOOL_DATA_DIR on NFS). Fall back to PROJ_PATH for local dev.

Node buildpack — frontend at image-build time

The Node buildpack is triggered by frontend/package.json. It must run npm run build during
toolforge build start so assets land in montage/static/ inside the image. May require
buildpack configuration to point at the frontend/ subdirectory (open question — needs testing).

Deployment scripts

  • tools/build_frontend.sh: removed — frontend built inside the image
  • tools/reinstall_venv.sh: removed — venv managed by buildpack
  • tools/reinstall.sh: remove all venv and frontend job steps; add toolforge build start
    and toolforge webservice buildservice start
  • deployment.md: rewrite install steps

Deployment — env vars (one-time setup per tool account)

toolforge envvars create MONTAGE_ENV         # e.g. devlabs / prod
toolforge envvars create MONTAGE_CONFIG_DIR  # e.g. /data/project/montage-dev
toolforge webservice buildservice restart

Persistent across restarts — set once per tool account.

Multi-environment

Each tool account (montage-dev, montage-beta, montage) builds from its own branch:

toolforge build start https://github.com/hatnote/montage.git --ref <branch>

No structural change to the per-account setup.

Test plan

  • cd frontend && npm run dev still works for local development (Vite dev server, hot reload)
  • python app.py still works locally with config.dev.yaml and SQLite (no env vars needed)
  • npm run build + serving via Python app still works for local production-style testing
  • Node buildpack detects frontend/package.json in subdirectory, or config added to make it work
  • $TOOL_DATA_DIR is set and NFS-mounted in the webservice pod (config and log paths resolve)
  • urllib3.contrib.pyopenssl shim still needed, or safely removed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions