GCP Cloud Functions - How to import local scripts outside of cloud function? How to structure projects that use a global config.py file?

Question

I have one giant repo folder which contains many more sub-repos, and each sub-repo contains multiple cloud functions, i.e.

|-repo_master
    |-repo_a
        |-cloud_function_a
               |-.github/workflows/deploy.yml
               |-main.py
               |-requirements.txt
               |-config.py
        |-cloud_function_b
               |-.github/workflows/deploy.yml
               |-main.py
               |-requirements.txt
               |-config.py
    |-repo_b
        |-cloud_function_a
               |-.github/workflows/deploy.yml
               |-main.py
               |-requirements.txt
               |-config.py
        |-cloud_function_b
               |-.github/workflows/deploy.yml
               |-main.py
               |-requirements.txt
               |-config.py
    |-repo_c
        |-cloud_function_a
               |-.github/workflows/deploy.yml
               |-main.py
               |-requirements.txt
               |-config.py
        |-cloud_function_b
               |-.github/workflows/deploy.yml
               |-main.py
               |-requirements.txt
               |-config.py
    etc...

each cloud function has a config.py file and I would like to move this config.py file into the repo_master directory and have all the cloud functions refer to it.

I am able to achieve this locally by adding the repo_master to sys.path, for instance, my python files (main.py) look like this.

# Import config file
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
from config import CONFIG_VARIABLE

essentially by adding the repo_master dir to the sys.path I can access CONFIG_VARIABLE in all of my main.py files.

The issue is when I deploy this to github's workflow and deploy the cloud function I get the error in GCP

ImportError: cannot import name 'CONFIG_VARIABLE' from 'config' (unknown location)

My cloud function deployment looks like this (with variables defined in deploy.yml env)

gcloud functions deploy ${{ matrix.function.name }} \
            --gen2 \
            --trigger-http \
            --source=cloud_functions/${{ matrix.function.name }} \
            --runtime=${{ env.RUNTIME }} \
            --region=${{ env.REGION }} \
            --memory=${{ env.MEMORY }} \
            --service-account=${{ env.WORKFLOWS_SA }} \
            --timeout=${{ env.TIMEOUT }} \
            --no-allow-unauthenticated \
            --set-secrets="GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT:latest,GOOGLE_CLOUD_PROJECT_NUMBER=GOOGLE_CLOUD_PROJECT_NUMBER:latest"

Potential Solutions

I have thought about having a custom git push bash script which copies the config into all the cloud function directories, however I would prefer to avoid this and I would like to know if there is a better way to access these global variables/scripts or if there is a better way to structure my entire project

Answer 1

For the observed error, try suggestions below and on this SO post:

• Double-checking incorrect module or package name:

  • module exist, not mispelled, case sensitive, present in the specified path

• Installing missing dependencies

• Missing init.py:

  • Creating the file _init_.py is recommended because the existence of an init.py file allows users to import the directory as a regular package, even if (as is the case in this tutorial) init.py is empty. Details here.

As an alternative, create a folder (shared_config) under repo_master containing the config.py file (+ init.py) and installing it as a dependency (via requirements.txt > pip install -r requirements.txt).

|-repo_master
   |-shared_config
      |-_init_.py
      |-config.py
   |-repo_a
      |-cloud_function_a
        |-main.py
        |-requirements.txt
        |-.github/workflows/deploy.yml