1

I'm trying to build an image from a Dockerfile in a Gitlab pipeline and running into a build issue.

The base image is apache/airflow:slim-2.11.2-python3.10 and within my Dockerfile I have a command:

RUN sudo apt-get update && \
    sudo apt-get install -y <list of packages>

While executing the apt-get update command, the build fails with the following logs:

#11 127.9 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
#11 127.9 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
#11 127.9 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#11 127.9 Get:4 https://packages.microsoft.com/debian/12/prod bookworm InRelease [3618 B]
#11 127.9 Err:1 http://deb.debian.org/debian bookworm InRelease
#11 127.9   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265 NO_PUBKEY F8D2585B8783D481
#11 128.0 Err:2 http://deb.debian.org/debian bookworm-updates InRelease
#11 128.0   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265
#11 128.0 Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
#11 128.0   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#11 128.0 Err:4 https://packages.microsoft.com/debian/12/prod bookworm InRelease
#11 128.0   At least one invalid signature was encountered.
#11 128.1 Get:5 https://download.docker.com/linux/debian bookworm InRelease [46.6 kB]
#11 128.1 Err:5 https://download.docker.com/linux/debian bookworm InRelease
#11 128.1   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
#11 128.1 Get:6 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [180 kB]
#11 128.2 Err:6 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease
#11 128.2   At least one invalid signature was encountered.
#11 157.9 Ign:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 188.9 Ign:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 221.0 Ign:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 255.0 Err:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
#11 255.0   Could not wait for server fd - select (11: Resource temporarily unavailable) [IP: 138.201.152.105 443]
#11 255.0 Reading package lists...
#11 255.0 W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265 NO_PUBKEY F8D2585B8783D481
#11 255.0 E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
#11 255.0 W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY 78DBA3BC47EF2265
#11 255.0 E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
#11 255.0 W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#11 255.0 E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
#11 255.0 W: GPG error: https://packages.microsoft.com/debian/12/prod bookworm InRelease: At least one invalid signature was encountered.
#11 255.0 E: The repository 'https://packages.microsoft.com/debian/12/prod bookworm InRelease' is not signed.
#11 255.0 W: GPG error: https://download.docker.com/linux/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
#11 255.0 E: The repository 'https://download.docker.com/linux/debian bookworm InRelease' is not signed.
#11 255.0 W: GPG error: https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease: At least one invalid signature was encountered.
#11 255.0 E: The repository 'https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease' is not signed.
#11 255.0 E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
#11 255.0 E: Sub-process returned an error code

The build runs successfully on my local machine (running docker v28.1.1), but only fails in Gitlab. The Gitlab CI/CD job is defined with

  image: docker:28.5.2
  services:
    - name: docker:28.5.2-dind

I've also attempted this with other airflow versions and found that any slim base image with v2.7.3 or below will work, while v.2.8.0 and above breaks in the same way.

I've seen many answers that suggest something along the lines of including

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <list of missing keys>

in the Dockerfile, but this fails with "Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8))." (supposedly just a warning but it fails without any other message)

I've also tried

apt-get install -y --no-install-recommends debian-archive-keyring

But that just tells me "debian-archive-keyring is already the newest version (2023.3+deb12u2). debian-archive-keyring set to manually installed." then continues to fail as normal.

Any thoughts on next steps?

Improve this question

1 Answer 1

Reset to default
1

Perhaps there's a caching problem as apt-get update works locally and Docker doesn't pull newer images unless necessary. You can use the digest of your apache/airflow:slim-2.11.2-python3.10 to ensure both computers have the same version. It's surprising that your sudo apt-get update works because I get "sudo: a terminal is required to read the password" on the latest version. You can use the Dockerfile command USER to switch to root and back:

FROM apache/airflow@sha256:32ef1c1927c47e55fd05f65e7da7b60ff7c431d0cca5c27972c7f436fff9cb56
USER root

# Debug 1
RUN sha256sum /etc/apt/sources.list.d/debian.sources /usr/share/keyrings/debian-archive-keyring.gpg
# You should see:
# fba4b66c95952e28af3fda06211991a51dc83d5448c2a4d262ec736b12323edb  /etc/apt/sources.list.d/debian.sources
# 506b815cbb32d9b6066b4a2aa524071e071761e7e7f68c3ac74f3061ba852017  /usr/share/keyrings/debian-archive-keyring.gpg

# Debug 2
RUN curl http://deb.debian.org/debian/dists/bookworm/InRelease
# Then check that the PGP SIGNATURE section printed in the GitLab Runner
# is the same as the one you see in a broswser

RUN apt-get update && \
    apt-get install -y neofetch # example
USER airflow

The other possibility is that your GitLab Runner has the wrong configuration. Let's print the Dockerfile using cat. Your entire .gitlab-ci.yml should look like:

# Replace this first half with no-tls-docker-runner is you're using that
default:
  image: docker:24.0.5-cli
  services:
    - docker:24.0.5-dind
  before_script:
    - docker info
variables:
  DOCKER_TLS_CERTDIR: "/certs"

build:
  stage: build
  tags:
    - tls-docker-runner
  script:
    - cat Dockerfile
    - docker build --no-cache --progress=plain -t my-docker-image .
    - docker run --rm -i my-docker-image bash -c neofetch
# Then verify the printed Dockerfile contents are exactly what you saved
Improve this answer
8
  • Thanks for your help. I used the image digest (and confirmed it still worked locally), used the docker version you suggested, added docker info to the before_script (which confirmed I'm using 24.0.5, not sure if there's anything else there I should note), and confirmed with cat that the Dockerfile Gitlab is using is correct version. The issue still persists. Commented 11 hours ago
  • @MikeS Edited. Compare the cURL inside the container to the GPG signature you see in a browser. If you are using your own GitLab instance and not Gitlab.com, then you might have a firewall blocking it Commented 11 hours ago
  • Hmm. Debug 1 worked as you indicated, but I couldn't even run curl for Debug 2 without an error: "#6 [3/8] RUN curl deb.debian.org/debian/dists/bookworm/InRelease #6 0.496 curl: (6) getaddrinfo() thread failed to start" Commented 10 hours ago
  • Also, yes, I am using my organization's Gitlab instance Commented 10 hours ago
  • @MikeS Does replacing http with https work? Use curl -v to debug. Otherwise, there might be a general networking problem with that GitLab Runner, and you should contact your organization's GitLab admin Commented 10 hours ago

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.