Skip to content

Gateway systemd service fails to auto-restart when browser processes orphaned #1617

@anpicasso

Description

@anpicasso

Problem

When the gateway spawns browser automation (Chrome via remote-debugging-port), those Chrome child processes join the systemd service cgroup. On service stop/restart:

  1. ExecStop triggers graceful Python shutdown
  2. Python's async Telegram disconnect throws errors ('NoneType' object has no attribute 'shutdown', dictionary changed size during iteration)
  3. systemd waits TimeoutStopSec (15s), then tries to kill the cgroup
  4. Chrome orphans prevent clean cgroup teardown: Failed to kill control group: Invalid argument
  5. Service enters Failed with result 'timeout' state
  6. Restart=on-failure doesn't always trigger recovery after this state

The gateway stays down until manually restarted.

Root Causes

  • KillMode=mixed only SIGKILLs the main process, leaving chrome children alive in the cgroup
  • Restart=on-failure doesn't cover all exit scenarios (e.g., SIGKILL after timeout)
  • No cleanup of leaked browser processes after stop
  • No crash loop protection — if something is broken, it could restart infinitely

Fix

Update generate_systemd_unit() in hermes_cli/gateway.py:

  • Restart=always — self-heal on any exit
  • KillMode=control-group — kill entire cgroup including orphan chromes
  • ExecStopPost — force-kill leaked browser processes
  • StartLimitIntervalSec=120 / StartLimitBurst=5 — crash loop protection
  • TimeoutStopSec=20 — slightly more time for graceful disconnect
  • SendSIGKILL=yes — ensure cleanup after timeout
  • RestartSec=15 — prevent rapid restart churn

Users who already installed the service need to run hermes gateway install --force to regenerate the unit file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions