Rapidly creating cloud tasks using Cloud Task, Cloud Functions, and GAE

Question

I'm using App Engine to trigger the creation of multiple Cloud Tasks based on items in a list:

function_url="https://us-central1-myproject.cloudfunctions.net/some-cloud-function"

someTasks = [
{'id': 'task-1'},
{'id': 'task-2'},
{'id': 'task-3'},
...
{'id': 'task-1000'},
]

The tasks are currently created using:

Parallel(backend='threading', n_jobs=100)(
        delayed(create_document_task)(function_url=function_uri, data=task) for task in someTasks 
                    )

The above code creates the tasks in parallel and instructs the Task Queue to direct the payload to that specific cloud function.

Is doing this in parallel the correct way to rapidly create tasks?

Answer 1

- I am posting this as an answer, due to the amount of text not fitting in a comment.

It seems that the aforementioned (in the comments) method:

• Queue(“someQueue”).add_async(tasks)

is indeed an old method. This method is implemented within the Task Queue REST API (v1), in order to asynchronously add a task or list of tasks into a task queue.

However as stated here, The App Engine Task Queue REST API (v1), was turned down as of February 20, 2018. The separate product Cloud Tasks provides a REST API that you can use to add tasks from a second generation App Engine standard environment run-time, any of the App Engine flexible environment run-times, or even from entirely outside of App Engine.

This API does not include the “add_async()” functionality. More specifically, here and here is affirmed that the feature of adding task to queues asynchronously, as the users of App Engine SDK have the option to do, is NOT an available feature via Cloud Tasks API.

Nonetheless, when a large number of Cloud Tasks, for example millions or billions, need to be added, a double-injection pattern can be useful.

To implement this scenario, you'll need to create a new injector queue, whose single task would contain information to add multiple(100) tasks of the original queue that you're using. On the receiving end of this injector queue would be a service which does the actual addition of the intended tasks to your original queue. Although the addition of tasks in this service will be synchronous and 1-by-1, it will provide an asynchronous interface to your main application to bulk add tasks. In such a way you can overcome the limits of synchronous, 1-by-1 task addition in your main application.

Note that the 500/50/5 pattern of task addition to queue is a suggested method, in order to avoid any (queue/target) overloads.

As I did not find any examples of this implementation, I will edit the answer as soon as I find one.

I hope this helps.