Python async process getting executed in different orders

Question

I have an async function that calls an async subprocess and later on execution, the same function performs a file read. After execution of the subprocess, the file should be generated, but when I run the function for some reason the file read operation gets executed before calling the subprocess, which causes an error saying No file named X.

async def job():
     # creates and calls my CPU intensive command which should take some
     # time to perform
     cmd = "my time consuming command"
     await run_shell_process(cmd)
     
     # opening a file that would be generated as an output
     # to the command, but returning a file not found 
     # I checked and made sure that the file IS getting
     # getting generated, but the order of the process is
     # causing the issue
     with open(my_file, 'r') as file:
         file.read()


async def run_command_shell(command):
    process = await asyncio.create_subprocess_shell(
        command, stderr=asyncio.subprocess.STDPIPE
    )

    # Wait for the subprocess to finish
    stdout, stderr = await process.communicate()

    # Result
    result = stdout.decode().strip()

    # Return stdout
    return result

Is there any way that I can make sure my async subprocess is fully executed before running the following file open operation

Note: Also the job is ran as a background task in Sanic by using app.add_task(job()). I tried to use the multiprocess callback method to invoke a call back after the future got executed, but because it's already running as a child process, it threw an error saying can't generate a child process for already running child process. Also, the command that I am running is a very CPU intensive command, could that lead to re-scheduling the thread and them getting executed in a different order

Answer 1

There are a collection of tools https://docs.python.org/3/library/asyncio-sync.html of which Event looks like a good match.

import asyncio

async def slow_must_finish_first(wev, rev):
    global file
    await wev.wait()
    print('slow writer writing')
    await asyncio.sleep(1.0)
    print('slow writer finished')
    rev.set()

async def fast_must_finish_last(rev):
    await rev.wait()
    print('fast reader reading')

async def main():
    write_event = asyncio.Event()
    read_event = asyncio.Event()
    tasks = [
        asyncio.create_task(fast_must_finish_last(read_event)),
        asyncio.create_task(slow_must_finish_first(write_event, read_event))
    ]
    write_event.set()
    await asyncio.gather(*tasks)

if __name__ == '__main__':
    asyncio.run(main())

Output:

slow writer writing
slow writer finished
fast reader reading