0

I have a process in GCP that can be summarized as:

  1. a log sink checks the logs for a specific log (daily GA4 table is added to the dataset)
  2. when sink gets the log, a pub/sub message is sent to trigger python cloud function
  3. python cloud function (below script) runs several scheduled queries.

I would like to have the error resolved so that my queries can run again.

 import time
 from google.protobuf.timestamp_pb2 import Timestamp
 from google.cloud import bigquery_datatransfer_v1

 def runQuery(parent, requested_run_time):
     client = bigquery_datatransfer_v1.DataTransferServiceClient()
     projectid = 'my_project_id'  # Enter your projectID here
     locationid = 'europe'
     transferids = [  #List of transfer IDs
                     'my_config_id1'   # Scheduled Query 1
                     ,'my_config_id2'  # Scheduled Query 2
                     ,'my_config_id3'  # Scheduled Query 3
                     #,'my_config_id4' # Scheduled Query 4 ## not needed anymore
                     ]  # List of transfer IDs

     for transferid in transferids:
         parent = client.location_transfer_config_path(projectid, locationid, transferid)
         start_time = bigquery_datatransfer_v1.types.Timestamp(seconds=int(time.time() + 10))
         response = client.start_manual_transfer_runs(parent, requested_run_time=start_time)

Since 10th of April the function does not work anymore and I get an error (see below): But all I did to my update was that I commented out "my_config_id4". What I do notice is that the function was initially deployed by "Google Cloud Functions" but now it is deployed by "my_email_address".

The requirements.txt only contains "google-cloud-bigquery-datatransfer==1" And Base image: Python 3.12 (Ubuntu 22 Full)

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/execution_id.py", line 106, in wrapper
    return view_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/__init__.py", line 142, in view_func
    return function(request._get_current_object())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: runQuery() missing 1 required positional argument: 'requested_run_time'

I am new to python so I could not do a lot to debug it myself. I copied the script from a "Towards Data Science" article a while ago but that article is removed.

Would really appreciate some help here.

1
  • did you try to run it again with all IDs? What if you remove other IDs? Maybe it is not problem with IDs but some bug in module. Maybe module was updated and now it has bug. maybe it needs to go back to older version - and send this problem (as issue) to author(s) of this module.
    – furas
    Commented Apr 17 at 18:55

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.