All Questions
Tagged with python-multithreading threadpool
79 questions
1
vote
1
answer
63
views
Is the Python ThreadPoolExecutor best used for synchronous database calls?
I need to connect to several different databases and then perform a number of queries on each one. I first tried to use asyncio until I realize that JDBC doesn't have async calls, and wrapping the ...
- 11
0
votes
2
answers
213
views
How to clean up thread-local data after using ThreadPoolExecutor?
I want to use ThreadPoolExecutor to parallelize some (legacy) code with database access. I would like to avoid creating a new database connection for each thread, so I use threading.local() to keep a ...
- 151k
0
votes
1
answer
55
views
KeyboardInterrupt doesn't work as expected
Ask please why press Ctrl + C ( KeyboardInterrupt) does not work as expected. (the process is not interrupted, continues to work)
from concurrent.futures import ThreadPoolExecutor
def sleeper(n):
...
- 15
0
votes
1
answer
180
views
How to dynamically add a thread or pick a thread from the pool of thread in python
I'm looking to schedule tasks to be executed by threads, with the ability to dynamically add or remove tasks from the process. My goal is to pre-create a pool of threads, for example, [th1, th2, th3], ...
- 63
0
votes
0
answers
39
views
Python Thread Deadlock Issue
I have a Python script that reads all XML files in a specified directory, translates them using the Google API, and saves them in another directory.
Since the API is slow, I divide the XML into chunks ...
0
votes
2
answers
178
views
ThreadPoolExecutor too fast for CPU bound task
I'm trying to understand how ThreadPoolExecutor and ProcessPoolExecutors work. My assumption for this test was that CPU-bound tasks such as increasing a counter wouldn't benefit from running on ...
- 1,480
1
vote
1
answer
110
views
Non-blocking thread pool that posts results as soon as they're ready for the main thread to access
import multiprocessing as mp
pool = mp.Pool()
def calc(i):
return i * 2
def done(results):
for result in results:
print(result)
def loop():
pool.map_async(calc, [0, 1, 2, 3], ...
- 1,151
0
votes
1
answer
210
views
QT5, Threading, Sockets : Issue with GUI and QRunnable in Python
I'm looking for help for my project with threading and sockets. I have to make a communication between a server and few clients, and I have to do a GUI.
The issue is that when I use QRunnable, the GUI ...
0
votes
1
answer
112
views
`concurrent.futures.ThreadPoolExecutor` is behaving strangely
So, I am working on a project where I'm provided with some images and some excel sheets containing the information in context to those images. Now the images and excels are day to day data readings. ...
- 1,895
0
votes
0
answers
121
views
Multi-Threading Implementation
I am trying to implement multi-threading for data scraping.
DATA_TABLE is a oracle sql table containing serial_number, Link_URL and Link_Status columns.
Serial_number: To track the number of entries ...
- 381
0
votes
0
answers
90
views
How to get partial result in a concurrency timeout error?
I am doing HTTP GET from some URLs in parallel and I wanna set an overall timeout, my code is pasted below. Sometimes some URLs return okay responses but others would cause the timeout exception to be ...
-2
votes
1
answer
816
views
Python thread pool executor gets slower over time for I/O bound tasks
I use a threading pool executor to run a function over a large number of endpoints.
What I do not understand is that it is getting slower over time - e.g. initially it processes 5-6000 urls in the ...
- 1
0
votes
1
answer
600
views
Joining results from ThreadPoolExecutor as available
I'm using ThreadPoolExecutor to run several simultaneous queries of an API simultaneously. I'd like to return the results as they become available and perform some action on them. I found this post ...
- 139
0
votes
1
answer
93
views
how can i run batches in thread worker parallel
any help please how can i do this process and run batch en parallel with multiprocessing.pool
I have a list of batches (20000 items) splited into multiple batches one batch 2000,
batch_size = 2000
i ...
- 11
2
votes
2
answers
4k
views
How to exit ThreadPoolExecutor with statement immediately when a future is running
Coming from a .Net background I am trying to understand python multithreading using concurrent.futures.ThreadPoolExecutor and submit. I was trying to add a timeout to some code for a test but have ...
- 4,260