All Questions
21 questions
1
vote
1
answer
56
views
Why selenium's click method is returning NoSuchElementException?
During scraping a javascript base website(link), I've stuck with a click function which is not producing the desired result. I've pasted the code which should return the data under "Fits the following ...
- 168
1
vote
1
answer
101
views
Unable to get response in css selector or xpath expression
I am trying to get the movie names from https://www.sunnxt.com/movie/inside/ website. When I go to inspect element it shows me elements with movie names, but when I perform css selector or xpath ...
- 655
0
votes
1
answer
501
views
Scrapy Program stops after about 500-600 items
I hired someone off of freelancer to make this spider for ebay. I am just learning python and needed it quick so that was our solution. I have asked him to figure out what is causing the issue and he ...
1
vote
1
answer
158
views
Cannot select element from a javascript webpage using Python scrapy and Selenium
I'm attempting to create an app that gathers some data. I'm using Python 2.7 with Scrapy and Selenium on Windows 10. I've done this with only a few web pages previously, however, I'm not able to ...
- 13
0
votes
0
answers
493
views
parse function not getting called in Scrapy
I'm using scrapy + selenium stack for crawling task. I have worked with them before, the following technique worked fine. But now, my parse function is not getting called. Following is my code snippet
...
- 757
0
votes
1
answer
3k
views
Type Error url is not JSON serializable in Scrapy with Selenium
I am using python scrapy with selenium here i added the code
import scrapy
from examble.items import exambleItem
from selenium import webdriver
class exambleSpider(scrapy.Spider):
name = "examble"...
- 190
2
votes
2
answers
1k
views
Scrapy not calling any other function after "__init__"
OS: Ubuntu 16.04
Stack - Scrapy 1.0.3 + Selenium
I'm pretty new to scrapy and this might sound very basic, But in my spider, only "init" is being getting executed. Any code/function after that is not ...
- 757
3
votes
2
answers
2k
views
Using Selenium after authenticated login session with Scrapy
after looking around, it seems that if you login to a website through Scrapy, the authenticated login session doesn't transfer over if you try to use Selenium within the spider. Is there a way to ...
- 139
0
votes
2
answers
499
views
Selenium inside scrapy does not work
I have a scrapy Crawlspider that parses links and returns html content just fine. For javascript pages however I enlisted Selenium to access the 'hidden' content. The problem is that while Selenium ...
0
votes
1
answer
169
views
scrapy: how to type argument in a blank space
I want to download the data from Yahoo Finance
http://finance.yahoo.com/q/hp?s=^TWII&a=00&b=15&c=2004&d=11&e=4&f=2015&g=m
I want the program to type "2004" in the start ...
- 69
4
votes
2
answers
5k
views
__init__() got an unexpected keyword argument '_job'
I am trying to use scrapyd with scrapy.
When I use this the code below it works fine.
import scrapy
from scrapy.selector import Selector
from scrapy.spiders import CrawlSpider
from shikari.items ...
- 438
2
votes
1
answer
199
views
Scrapy and Selenium StaleElementReferenceException
There is several click-able elements on the page and I'm trying to scrape some pages behind, but I have this error and spider closed after first click:
StaleElementReferenceException: Message: ...
- 6,844
2
votes
2
answers
1k
views
Refresh scrapy response after selenium browser Click
I m trying to scrape a website that uses Ajax to load the different pages.
Although my selenium browser is navigating through all the pages, but scrapy response is still the same and it ends up ...
- 1,801
1
vote
2
answers
101
views
What tools are required to scrape, click, and analyze data from a website
Ok so I just want to know what my best plan of action is here and what tools/frameworks would I need:
1. Log onto webpage
2. Navigate to desired page which would require clicking on buttons and then ...
- 574
0
votes
1
answer
358
views
Using recursion Spider in Scrapy [Python]
Intro
Hello i'm using Scrapy in order to take data in Yahoo Answer.
My achievement is to take all the question and the answer in one precise section.
I try
Using scrapy and selenium first of all i ...
- 1,923