All Questions
15 questions
1
vote
1
answer
887
views
Python requests and cURL never return a response
I am attempting to scrape the NSE website for a particular company, in Python. I am attempting this using the requests library and it's corresponding get() method. This should return a .html file, ...
- 57
0
votes
1
answer
326
views
Why does the following curl works but equivalent python code does not?
The following curl is working:
curl --location 'https://www.lowes.com/rnr/r/get-by-product/50413062/pdp/prod?sortMethod=SubmissionTime&sortDirection=desc&offset=0' \
--header 'authority: www....
- 51
0
votes
1
answer
877
views
Curl works in insomnia but does not work in Postman or Mac Terminal
I am trying to execute the following curl in Postman, Insomnia and Local Mac Terminal. It works with Insomnia but does not work with Postman and my local terminal.
curl --request GET \
--url 'https:/...
- 51
0
votes
1
answer
473
views
Curl Works fine on a site but Getting blocked while using requests
I am trying to make a API call to get reviews from ryviu website.
If I execute the following curl in the terminal, I do get a response from the server.
curl 'https://app.ryviu.io/frontend/client/get-...
- 51
1
vote
0
answers
720
views
Curl works but Python requests does not work for a API call
I am trying to make a API call to get reviews from best buy web site.
If I execute the following curl in the terminal, I do get a response from the server.
curl --location 'https://www.bestbuy.com/ugc/...
- 51
1
vote
1
answer
152
views
Why does my GET query with cURL and Python requests redirects me to an "Internet Explorer not supported" page despite Firefox user-agent?
I am trying to automate some work load with Python requests or cURL (can't use browser automation like Selenium), but I can't seem to get the login page to load properly.
Whether I use a Python ...
- 11
0
votes
1
answer
312
views
Why am I getting status code forbidden with only copying cURL to python?
I'm trying to scrape a website (url = https://sports.betway.be/nl/sports/grp/soccer/belgium/first-division-a) via python requests. I found via the network tab the corresponding name which is ...
2
votes
0
answers
269
views
How to automatically change bearer token in python code
I am scraping a site in an hourly basis through post requests but everyday the authorisation bearer token changes and I have to copy paste new token from the site to my headers dict. How can I ...
1
vote
1
answer
872
views
How to catch redirects with curl or requests
I am new to web scraping and have stumbled upon an unexpected challenge. The goal is to input an incomplete URL string for a website and "catch" the corrected URL output returned by the ...
- 7,536
1
vote
0
answers
673
views
Replicate chrome devtools network XHR in Python
Scraping a "JavaScript heavy" website I see that, on page load and clicks, requests are made to get data via an API. The thing is there's authentication with a token that expires... that ...
- 123
0
votes
1
answer
1k
views
NASDAQ PULL DATA
I have a list of 500 indices. I am trying to just pull the corresponding name of the company for each index. I am using a simply request for each item:
url = 'https://www.nasdaq.com/markey-activity/...
13
votes
3
answers
22k
views
POST request works in Postman, but not in Python Requests (200 response with robot detection)
I have a POST request that works perfectly with both Postman an cURL (it returns a JSON blob of data). However, when I perform the exact same request with Python's Requests library, I get a 200 ...
- 143
1
vote
1
answer
500
views
I try to scrape a website, and I keep getting 404 error
I'm trying to scrape this website:
https://www.nadlan.gov.il
It is a website in Israel, that shows data on historical real-estate deals.
In the search box, you can search in free text to what city/...
- 3,392
0
votes
1
answer
62
views
Requests - post does not returns desired data - ASPX
I'm trying to get an html from one web page which uses ASP.NET. When you want to get the html using browser, you have to get to menu, set what district and city you want and then click on button. I ...
- 18.8k
1
vote
1
answer
341
views
Avoid redirections on my server
When i try to fetch some non existent url on my local machine most of the time i get:
curl: (6) Could not resolve host: some-non-existent-url.co
Sometimes however i get a 301:
<!DOCTYPE HTML ...
- 2,099