All Questions
30 questions
3
votes
1
answer
83
views
Multi-Page Web Scraping Code Using Selenium with Multithreading
I have written a web scraping script using Selenium to crawl blog content from multiple URLs. The script processes URLs in batches of 1000 and uses multithreading with the ThreadPoolExecutor to ...
- 31
1
vote
2
answers
157
views
Web scraping spider
I'm currently working on my first web scraping project and I need to scrape a lot of websites. With my current code it takes more than a day but for my project I need to scan the same websites every 5 ...
- 27
1
vote
1
answer
158
views
Training a Reinforcement Learning algorithm
Problem:
I'm trying to reduce the computation time and optimize specific functions of a Reinforcement Learning algorithm in the training phase and observed that one block of code takes too much ...
- 113
2
votes
1
answer
123
views
4
votes
1
answer
705
views
Python multi-connection downloader
This is a simple Python multi-connection downloader primarily using requests, mmap and threads, it downloads a single file using 32 concurrent connections, slices the download using range parameter, ...
- 3,396
2
votes
1
answer
503
views
Brute Force Password Program / Python 3.9
I've been making a brute force program for fun. This program tries to guess a password using brute force. I've tried many things to help increase the efficiency. Here it is: (Sorry for the lack of ...
- 145
7
votes
1
answer
2k
views
Speeding up Python program that converts DOCX to PDF in Windows
This is meant to be a performance-centric question as this type of conversion is obviously very common. I'm wondering about the possibilities for making this process faster.
I have a program that ...
0
votes
1
answer
459
views
Instagram Scraper running to slow
Can someone help me how to speed up this scraping process getting all the usernames from the # 'cats' on instagram. My goal is to make this as fast as possible because currently the process is kinda ...
- 21
2
votes
0
answers
554
views
Multi-threaded image convolution/pooling-python
This multi-threaded code takes an array of 3d images and applies the convolution function with padding, stride, pad values .. as parameters and same applies for creating pooling layers. I need ...
2
votes
1
answer
1k
views
Multithreading to process requests and save results in python
I was presented with a task to come up with a script that generates a CSV with POSTAL codes via bruteforce (I'm in Brazil, so that means CEP to us).
Points to note:
I'm using an external library, but ...
- 387
4
votes
1
answer
195
views
Motor control PLC in Python
Project description
We have different types of devices like sov, motor, analog, digital, control valves, etc. Each type of device has 100 items.
Now our software continuously monitors with the PLC to ...
3
votes
1
answer
586
views
Simple sharding of delimited files to more sophisticated
A while back I came across the issue of having a large delimited file where I wanted to simply parallelize my python code across each line of the file. However, doing so all at once either took up too ...
- 281
2
votes
2
answers
450
views
Python 3 multi-threaded pinger
My goal: I want to ping every single IPv4 address and record whether or not they responded.
The way I have it set up is every IP address corresponds to an index. For example ...
user115383
1
vote
0
answers
1k
views
Using multi-threading efficiently with million GET requests
Disclaimer: The following question is not related to any malicious activity. Only performance testing we are doing.
I am working on querying an internal server with lots of requests (~20 million).
...
- 237
5
votes
0
answers
926
views
Uploading captured video to Google Cloud Storage
My program intends to capture video streams (in mjpeg) with OpenCV and upload the captured frames into Google Cloud Storage for later processing. I am expecting to capture ~15-20 frames per second and ...
- 151