All Questions
357 questions
-1
votes
0
answers
70
views
How to scrape the full New York Times article content using Selenium and BeautifulSoup without triggering the "Please enable JavaScript" message?
I'm building a scraper that fetches full article content from the New York Times using both the Article Search API and a hybrid static + Selenium-based HTML scraper. My goal is to extract complete ...
1
vote
2
answers
97
views
Can't find button using playwright (or puppeteer) for web scraping
There are many similar questions (like this: Scraping Websites With Playwright), yet I did not find and solution to this:
I have this url:
https://www.derstandard.at/search?n=&fd=2025-02-17&td=...
- 1,499
0
votes
1
answer
98
views
Why can't I extract listings information
I am trying to extract the EPC rating from each listings. you can only get the EPC rating when you click on the listing. each time i run my script it
it's keeps timing out, what could be the issue ? ...
0
votes
0
answers
84
views
scrape link for jwplayer calculated with JS using python
I'm trying to scrape video link (m3u8) from this website:
https://deaddrive.xyz/embed/fa31e
While inspecting the page, I realized that the link is calculated on the fly using JS in the function:
&...
- 535
-1
votes
2
answers
53
views
unable to click on webelement with selenium ActionsChain or javascript click as it refreshes the page and gives StaleElementReferenceError
I'm trying to close a cookie consent pop-up on a site https://www.meinschiff.com/. I used selenium webdriver for chrome to find the close and accept div element and click on whichever is found first. ...
- 59
0
votes
1
answer
100
views
Not able to download a web page that uses javascript
I have been trying to download a web page that I ultimately intend to scrape. The page uses Javascript, and has in their code a catch to test if javascript is enabled, and I keep getting it is not ...
0
votes
1
answer
52
views
Extracting the text between span tags in a Javascript-rendered page using Selenium in Python
I am trying to scrape all instances of text between tags with a particular class on a web page that dynamically updates. I am using selenium with a chrome WebDriver in Python.
In a normal browser, ...
- 5
0
votes
1
answer
241
views
Using Python with Selenium and BeautifulSoup4 how can i get data after Javascript has loaded all elements on the page?
I'm trying to scrape data from a sandbox website just to practice and start using python to scrape web data.
I have managed to extract a lot of data using the basics however I have found an element ...
-1
votes
1
answer
210
views
How can I scrape a dynamically rendered webpage using Selenium?
This is the page I'm trying to scrape. My goal is to check whether or not the string "Any available Pool will appear here." is on the page.
I've done some research and found out that the ...
- 1
1
vote
1
answer
120
views
How to web scrape a page that has collapsible fields using Selenium?
There is this website - https://www.flashscore.com/ and I'm looking to extract fixture data, including the competition name, team names, and dates. While I've already developed a scraping solution, ...
1
vote
1
answer
81
views
Can't Scrape a webpage whose contents are dynamically generated through JavaScript
I am trying to scrape table data from a webpage but it's not a normal webpage that can be scraped using its html tags and CSS class or ID. The contents of the webpage are dynamically generated using ...
- 11
0
votes
1
answer
78
views
How to obtain and click the URL of a dynamically loaded website?
I would like to scrape the marathon results from the link (call it page A): https://www.marathon.tokyo/2023/result/index.php
Suppose I choose the 'Marathon Men' in the first option and then search, I ...
- 3,058
0
votes
1
answer
97
views
How to interact with a button using selenium
I am trying to scrape some information from this page (Binance announcements)
I am trying to go through all the announcements and scrape the data from the announcements. however, I am having trouble ...
- 47
0
votes
1
answer
76
views
selenium webdriver can't get pass loading of javascript table
I'm building a web-scrapper to extract public data from a website's table
This is my code:
options = webdriver.ChromeOptions()
options.add_argument('--headless')
driver = webdriver.Chrome(options=...
- 119
0
votes
1
answer
64
views
How can I use selenium to iterate over pages on a Website?
I'm trying to write a piece of code in Python to scrape the following website for all the listed urls.
https://yavapai.arizonataxsale.com/index.cfm?folder=previewitems
What I've run into is that the ...
- 205