Python Flask - Asyncio asynchronous Question

Question

I have a web endpoint for users to upload file. When the endpoint receives the request, I want to run a background job to process the file.

Since the job would take time to complete, I wish to return the job_id to the user to track the status of the request while the job is running in background.

I am wondering if asyncio would help in this case.

import asyncio

@asyncio.coroutine
def process_file(job_id, file_obj):
     <process the file and dump results in db>

@app.route('/file-upload', methods=['POST'])
def upload_file():
    job_id = uuid()
    process_file(job_id, requests.files['file']) . # I want this call to be asyc without any await
    return jsonify('message' : 'Request received. Track the status using: " + `job_id`)

With the above code, process_file method is never called. Not able to understand why.

I am not sure if this is the right way to do it though, please help if I am missing something.

Answer 1

Flask doesn't support async calls yet.

To create and execute heavy tasks in background you can use https://flask.palletsprojects.com/en/1.1.x/patterns/celery/ Celery library.

You can use this for reference: Making an asynchronous task in Flask

Official documentation: http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#installing-celery

Even though you wrote @asyncio.coroutine() around a function it is never awaited which tells a function to return result.

Asyncio is not good for such kind of tasks, because they are blocking I/O. It is usually used to make function calls and return results fast.

Answer 2

As @py_dude mentioned, Flask does not support async calls. If you are looking for a library that functions and feels similar to Flask but is asynchronous, I recommend checking out Sanic. Here is some sample code:

from sanic import Sanic
from sanic.response import json

app = Sanic()

@app.route("/")
async def test(request):
    return json({"hello": "world"})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

Updating your database asynchronously shouldn't be an issue; refer to here to find asyncio-supported database drivers. For processing your file, check out aiohttp. You can run your server extremely fast on a single thread without any hickup if you do so asynchronously.