5

I wrote code that seems to do what I want, but I'm not sure if it's a good idea since it mixes threads and event loops to run an infinite loop off the main thread. This is a minimal code snippet that captures the idea of what I'm doing:

import asyncio
import threading

msg = ""

async def infinite_loop():
    global msg
    while True:
        msg += "x"
        await asyncio.sleep(0.3)

def worker():
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    asyncio.get_event_loop().run_until_complete(infinite_loop())

t = threading.Thread(target=worker, daemon=True)
t.start()

The main idea is that I have an infinite loop manipulating a global variable each 0.3 s. I want this infinite loop to run off the main thread so I can still access the shared variable in the main thread. This is especially useful in jupyter, because if I call run_until_complete in the main thread I can't interact with jupyter anymore. I want the main thread available to interactively access and modify msg. Using async might seem unnecessary in my example, but I'm using a library that has async code to run a server, so it's necessary. I'm new to async and threading in python, but I remember reading / hearing somewhere that using threading with asyncio is asking for trouble... is this a bad idea? Are there any potential concurrency issues with my approach?

Improve this question
4
  • You should use call_later() instead of the loop.
    – Klaus D.
    Commented Mar 31, 2018 at 6:14
  • @KlausD. call_later will execute the callback only once. A loop runs in a full-fledged coroutine that may (in a real-world example) maintain state or await other coroutines.
    – user4815162342
    Commented Mar 31, 2018 at 7:11
  • Your callback should call call_later() again. Also global and + on strings are considered bad practice in Python.
    – Klaus D.
    Commented Mar 31, 2018 at 7:15
  • @KlausD. There is no advantage of the repeated scheduling of callbacks with call_later over a simple coroutine, and there is the downside that it is harder to have a shared state. Using a coroutine for periodic execution is fairly idiomatic asyncio.
    – user4815162342
    Commented Mar 31, 2018 at 7:23

1 Answer 1

Reset to default
5

I'm new to async and threading in python, but I remember reading / hearing somewhere that using threading with asyncio is asking for trouble...

Mixing asyncio and threading is discouraged for beginners because it leads to unnecessary complications and often stems from a lack of understanding of how to use asyncio correctly. Programmers new to asyncio often reach for threads by habit, using them for tasks for which coroutines would be more suitable.

But if you have a good reason to spawn a thread that runs the asyncio event loop, by all means do so - there is nothing that requires the asyncio event loop to be run in the main thread. Just be careful to interact with the event loop itself (call methods such as call_soon, create_task, stop, etc.) only from the thread that runs the event loop, i.e. from asyncio coroutines and callbacks. To safely interact with the event loop from the other threads, such as in your case the main thread, use loop.call_soon_threadsafe() or asyncio.run_coroutine_threadsafe().

Note that setting global variables and such doesn't count as "interacting" because asyncio doesn't observe those. Of course, it is up to you to take care of inter-thread synchronization issues, such as protecting access to complex mutable structures with locks.

is this a bad idea?

If unsure whether to mix threads and asyncio, you can ask yourself two questions:

  • Do I even need threads, given that asyncio provides coroutines that run in parallel and run_in_executor to await blocking code?
  • If I have threads providing parallelism, do I actually need asyncio?

Your question provides good answers to both - you need threads so that the main thread can interact with jupyter, and you need asyncio because you depend on a library that uses it.

Are there any potential concurrency issues with my approach?

The GIL ensures that setting a global variable in one thread and reading it in another is free of data races, so what you've shown should be fine.

If you add explicit synchronization, such as a multi-threaded queue or condition variable, you should keep in mind that the synchronization code must not block the event loop. In other words, you cannot just wait on, say, a threading.Event in an asyncio coroutine because that would block all coroutines. Instead, you can await an asyncio.Event, and set it using something like loop.call_soon_threadsafe(event.set) from the other thread.

Improve this answer