All Questions
49 questions
1
vote
1
answer
63
views
Is the Python ThreadPoolExecutor best used for synchronous database calls?
I need to connect to several different databases and then perform a number of queries on each one. I first tried to use asyncio until I realize that JDBC doesn't have async calls, and wrapping the ...
- 11
0
votes
2
answers
213
views
How to clean up thread-local data after using ThreadPoolExecutor?
I want to use ThreadPoolExecutor to parallelize some (legacy) code with database access. I would like to avoid creating a new database connection for each thread, so I use threading.local() to keep a ...
- 151k
0
votes
0
answers
39
views
Python Thread Deadlock Issue
I have a Python script that reads all XML files in a specified directory, translates them using the Google API, and saves them in another directory.
Since the API is slow, I divide the XML into chunks ...
1
vote
1
answer
110
views
Non-blocking thread pool that posts results as soon as they're ready for the main thread to access
import multiprocessing as mp
pool = mp.Pool()
def calc(i):
return i * 2
def done(results):
for result in results:
print(result)
def loop():
pool.map_async(calc, [0, 1, 2, 3], ...
- 1,151
0
votes
1
answer
112
views
`concurrent.futures.ThreadPoolExecutor` is behaving strangely
So, I am working on a project where I'm provided with some images and some excel sheets containing the information in context to those images. Now the images and excels are day to day data readings. ...
- 1,915
0
votes
0
answers
121
views
Multi-Threading Implementation
I am trying to implement multi-threading for data scraping.
DATA_TABLE is a oracle sql table containing serial_number, Link_URL and Link_Status columns.
Serial_number: To track the number of entries ...
- 381
-2
votes
1
answer
816
views
Python thread pool executor gets slower over time for I/O bound tasks
I use a threading pool executor to run a function over a large number of endpoints.
What I do not understand is that it is getting slower over time - e.g. initially it processes 5-6000 urls in the ...
- 1
0
votes
1
answer
600
views
Joining results from ThreadPoolExecutor as available
I'm using ThreadPoolExecutor to run several simultaneous queries of an API simultaneously. I'd like to return the results as they become available and perform some action on them. I found this post ...
- 139
0
votes
1
answer
93
views
how can i run batches in thread worker parallel
any help please how can i do this process and run batch en parallel with multiprocessing.pool
I have a list of batches (20000 items) splited into multiple batches one batch 2000,
batch_size = 2000
i ...
- 11
0
votes
1
answer
134
views
How to get count of threads from user and start Thread Python
I can start function with my own count of threads and it works:
start_time = time.time()
t1 = Thread(target=time.sleep, args=(3, ))
t2 = Thread(target=time.sleep, args=(3, ))
t1.start()
t2.start()
t1....
- 178
0
votes
1
answer
1k
views
How to split a list into multiple batches and assign each batch to a thread worker
I want to split a bigger list (2000 items) into multiple batches, each going to run on a different thread in parallel.
The text of the problem is this:
We need to calculate the sum of squares for a ...
0
votes
1
answer
128
views
Multithread https requests in python
I'm trying to multi-thread web requests in python for web scraping. I want to send multiple requests to the same website using multi-threading, but the time it takes for the script to complete is the ...
- 440
1
vote
0
answers
435
views
In order to parallelize a for loop, how to use concurrent.future for having 1 reused ᴛᴄᴘ connection per thread?
Using concurrent.futures one can do this :
import pandas,concurrent.futures,urllib.request
URLS = ['http://some-made-up-domain.com/sifhzfhihrffhzs',
'http://some-made-up-domain.com/rthrgfgd',
...
- 3,100
1
vote
1
answer
155
views
How to avoid killing a thread until its child threads are finished?
I am developing my school project and it requires to stop killing thread by itself until its child threads are finished. The following code is a sample that simulates my scenario.
Sample is a thread ...
- 31
1
vote
1
answer
224
views
How to start python thread with control?
I am developing my school project and it requires starting threads with control. The following code is a sample that simulates my scenario.
GrandFather has 3 children [Mike, William, John] these ...
- 31