All Questions
14 questions
0
votes
1
answer
146
views
Why threading is much faster than asyncio when processing multiple files in my case
I'm experimenting with asyncio and threading to figure out which mechanism I should choose in processing a large number of files. The experiment is simple that I just want to read the files and add ...
- 407
3
votes
2
answers
748
views
How to execute two different functions concurrently in Python?
I have two different functions (funcA and funcB) that I want to be executed concurrently to cut down on overall execution time.
funcA is an API call that takes somewhere between 5 to 7 seconds.
funcB ...
- 630
0
votes
1
answer
144
views
Create and call a function that "asynchronously" updates a file in a loop until the second function that is started in parallel is done
I'm new to multiprocessing / threading and asyncio in Python and I'd like to parallelise two function calls so that the first function updates a healthcheck text file in an endless loop with 5 min. ...
0
votes
1
answer
3k
views
Keep indefinite connections open on TCP server with asyncio streams?
I'm trying to understand how to use asyncio streams for multiple connections that will keep sending messages until a predefined condition or a socket timeout. Looking at Python docs, they provide the ...
- 168
2
votes
2
answers
1k
views
How to implement Multiprocessing in Azure Databricks - Python
I need to get details of each file from a directory. It is taking longer time. I need to implement Multiprocessing so that it's execution can be completed early.
My code is like this:
from pathlib ...
- 205
3
votes
1
answer
1k
views
Which is the best way to parallelly running multiple tasks in Python
I have a function:
import time
def all_40k():
for _ in range(400000):
print('validate')
print('parsing')
print('inserting')
if __name__ == '__main__':
start_time = ...
- 4,064
3
votes
1
answer
2k
views
Multiprocessing: optimize CPU usage for concurrent HTTP async requests
I need to download a list of sites/URLs (which can vary over time) and I currently use multiprocessing.Manager().Queue() to submit and update said list.
I have to check each URL/task every second: ...
- 133
0
votes
0
answers
596
views
Issue multithreading `terraform init` executable via subprocess module in Python
I am experiencing an issue when multithreading the terraform init command on a set of subdirectories within Python. The output of the terraform init subprocess is showing the expected results, ie. the ...
0
votes
0
answers
137
views
How to increase for loop performance in Python
I have a for loop where there are a series of operations performed but one method in particular I discovered takes about 44% of the time for the entire iteration to execute. In order to increase the ...
1
vote
2
answers
1k
views
await run_in_executor() fails to unblock upon termination of synchronous function
Consider the following simple, self-contained Python3 program:
import asyncio
import multiprocessing
import time
loop = asyncio.get_event_loop()
def sub_process(event, i):
async def ...
- 159
1
vote
0
answers
87
views
Is it good to use asyncio, multiprocessing or multithreading for post requests
I have a story where dynamically filled xml document are sent through a web service. Here is what I'm using in python3.x:
import requests
import pandas as pd
df = pd.DataFrame({'A': ['a', 'b', 'a', '...
- 103
1
vote
0
answers
112
views
discord.py command to connect to a bot
I'm trying to do a command like .addtoken <token> that's connecting to a bot for 15 minutes and disconnect.
The problem is that when I try to connect to the bot, it fails and print:
discord.ext....
- 26
4
votes
0
answers
957
views
How to connect to multiple channels using websocket multiprocessing?
I wrote a script allowing to collect data through different websockets channels but I can't manage to listen to multiple channel at once with this script. I would like to find a solution such as "...
- 41
-1
votes
1
answer
538
views
Parallelizing downloads in Python, what is the optimal number of concurrent downloads, and which method to use?
I am trying parallelize thousands of downloads in python. Each download takes 2-3 seconds. I have looked at multi-threading vs multi-processing and it seems that multi-threading would be better for IO ...
- 449