All Questions
Tagged with web-scraping selenium-webdriver
8,567 questions
0
votes
1
answer
51
views
Can't close cookie pop up on website with selenium webdriver
I am trying to use selenium to click the Accept all or Reject all button on a cookie pop up for the the website autotrader.co.uk, but I cannot get it to make the pop up disappear for some reason.
This ...
- 759
0
votes
0
answers
31
views
Rotating Authenticate Proxy In Selenium Using Chrome Driver
import zipfile
import json
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import time
def ...
-1
votes
0
answers
68
views
How to scrape the full New York Times article content using Selenium and BeautifulSoup without triggering the "Please enable JavaScript" message?
I'm building a scraper that fetches full article content from the New York Times using both the Article Search API and a hybrid static + Selenium-based HTML scraper. My goal is to extract complete ...
-3
votes
1
answer
48
views
How to switch to a popup cookie consent page?
I'm using Python 3.12.3, Selenium 4.31.0, Firefox driver in Ubuntu 24.04.
When I try to open an url, a cookie consent popup, asking to continue without accepting, accept and more options. How can I ...
- 117
-1
votes
2
answers
47
views
How to get attribute value from webpage by css_selector?
I'm using Python 3.12.3, Selenium 4.31.0, Firefox driver.
How do I retrieve the class attribute in this html tag?
<div class="costa-itinerary-tile FS03250425_BCN03A20" data-cc-cruise-id=&...
- 117
0
votes
1
answer
95
views
Python Selenium Scraping problem (combining multiple occurences into N in for loop, for n in x:
I'm trying to do some scraping for educational purposes, I just started and am fairly noob at python.
My problem is, in selenium I am trying to scrape a product page, take the name, price, shipping ...
- 1
0
votes
1
answer
33
views
XHR Endpoint Returning Loading Page Data Only
I want to access the tables of the following website:
https://www.marketbeat.com/ratings/
However, pages can only be changed by setting the "Reporting Date".
I do know that I can change the ...
0
votes
0
answers
29
views
is there any method for managing memory usage in an infinite scrolling scraping?
I'm writing a Python Selenium scraper for a web page that uses infinite scrolling to load content dynamically. Over time, as more posts are loaded, the JavaScript heap memory usage in ChromeDriver ...
- 31
0
votes
1
answer
65
views
selenium does not download chromedriver itself
I am building a google image scraper using python selenium. I have selenium 4.8.0, so I guessed I didn't need webdriver_manager to download chromedrivers. But selenium is giving me some errors.
...
- 1
1
vote
0
answers
44
views
Problems downloading files with Selenium + Scrapy
This code is supposed to download some documents it most locate within series of given links.
While is does seemingly locate the link of the pdf file, its failing to download it. What might be the ...
0
votes
1
answer
41
views
Selenium script for Amazon UK postcode entry triggers CAPTCHA and fails to apply zip code
Automating Amazon UK with Selenium: Handling CAPTCHA, Setting Postcode, and Extracting Product Data
I'm automating Amazon UK (www.amazon.co.uk) using Selenium to:
Decline cookies (if present).
Click ...
- 63
1
vote
1
answer
19
views
Python Selenium clicking a jquery dropdown to expose more dropdowns
I'm new to Selenium and trying to undertake a live example of web-scraping a list using the following URL - https://mcscertified.com/find-an-installer/
However, I'm struggling to click on a drop-drown ...
- 511
0
votes
2
answers
37
views
Selenium Python scrolling dynamic table
page I'm scraping: https://data.nordpoolgroup.com/auction/day-ahead/prices?deliveryDate=2025-01-15¤cy=EUR&aggregation=DailyAggregate&deliveryAreas=AT,FR
I'm trying to scrape the ...
- 15
0
votes
1
answer
47
views
Selenium Web Scraping: Missing 11 Sectional Times from Race Data Grid
I’m new to Python (version 3.13.2) and using Selenium with Firefox to scrape data from a racecard webpage, specifically the "Sectional Times" tab at a URL like https://www.attheraces.com/...
1
vote
1
answer
28
views
How to scroll and click "Load more results" using Selenium in Python booking.com?
I’m new to web scraping with Selenium, and I’m trying to scrape property listings from Booking.com. My code (included below) successfully scrapes 25 results, but I suspect the issue is that more ...
- 47