[shim] Pass proxy variables to the container

[un-def]

Fixes: #3906

[un-def]

Test setup

Prepare a VM with HTTP proxy:

Install tinyproxy

sudo apt update
sudo apt install -y tinyproxy
sudo systemctl enable --now tinyproxy

Block egress except for tinyproxy

PROXY_UID=$(id -u tinyproxy)
# let the tinyproxy process talk to the world
sudo iptables -A OUTPUT -m owner --uid-owner $PROXY_UID -j ACCEPT
# let anyone use loopback (so your app can reach the proxy on 127.0.0.1:8888)
sudo iptables -A OUTPUT -o lo -j ACCEPT
# keep already-open connections alive
sudo iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# reject everyone else's outbound HTTP/HTTPS
sudo iptables -A OUTPUT -p tcp -m multiport --dports 80,443 -j REJECT

Configure HTTP proxy for users

It's required since dstack logs in via SSH and calls curl to download the shim binary. Without *_proxy variables, provisioning would fail.

Note, we use the lower case form – both curl and apt ignore HTTP_PROXY due to CGI: https://everything.curl.dev/usingcurl/proxies/env.html#http_proxy-in-lower-case-only

Add these lines to /etc/environment (see pam_env(8)):

http_proxy="http://127.0.0.1:8888"
https_proxy="http://127.0.0.1:8888"
no_proxy="localhost,127.0.0.1,::1"

Configure HTTP proxy for services

At least for Docker and dstack-shim. The easiest way to set proxy variables for all services via systemd.conf drop-in. Create /etc/systemd/system.conf.d/proxy.conf:

[Manager]
DefaultEnvironment="http_proxy=http://127.0.0.1:8888"
DefaultEnvironment="https_proxy=http://127.0.0.1:8888"
DefaultEnvironment="no_proxy=localhost,127.0.0.1,::1"

Reload systemd and services

sudo systemctl daemon-reexec
sudo systemctl restart docker.service

Check the setup

$ ssh debian@192.168.122.58 -i ~/.ssh/vm_debian

$ curl https://api.github.com/zen
Avoid administrative distraction.
$ unset http_proxy https_proxy
$ curl https://api.github.com/zen
curl: (7) Failed to connect to api.github.com port 443 after 0 ms: Could not connect to server

Deploy an SSH fleet

type: fleet
name: vm
ssh_config:
  hosts:
    - hostname: 192.168.122.58
      user: debian
      identity_file: ~/.ssh/vm_debian

Run a job

type: dev-environment
image: ubuntu  # or alpine or fedora, all three package managers should work

Disable env passing and check again

Redeploy the fleet with additional env section

type: fleet

... 

env:
  - DSTACK_DOCKER_PASS_ENV=

Run the same run configuration

Exited (none)
Package openssh-server is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'openssh-server' has no installation candidate