Edit - Stack Overflow

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

That HTTPConnectionPool error in my question is an outlier. The script works perfectly fine until it suddenly stops on page 9 without an error or exception. The only reason I set but didn't use next_button is because I was trying to troubleshoot this, thought that might've had something to do with it, and never reset it. The question/issue here is why does it stop crawling/scraping once it finishes page 9??

oldboy
– oldboy

2018-10-10 15:37:26 +00:00
Commented Oct 10, 2018 at 15:37
oh maybe that was why i never reset it. WebDriverWait(ff, 15).until(EC.text_to_be_present_in_element((By.PARTIAL_LINK_TEXT, 'Next→'), 'Next→')).click() returns a bool object which isn't clickable. you have an error in your code

oldboy
– oldboy

2018-10-10 15:53:10 +00:00
Commented Oct 10, 2018 at 15:53
@Anthony As per your observation I have made some small modifications in my solution. I could have optimized your code and performed the same web scrapping opening the Firefox Browser Client only once and traversing through various products. But to preserve your logic and innovation I have suggested the minimal changes required to get you through. Can you please try the updated solution and let me know the status please?

undetected Selenium
– undetected Selenium

2018-10-11 14:34:58 +00:00
Commented Oct 11, 2018 at 14:34
1

i actually fixed the issue on my own a number of days ago. i had updated my question and just updated it again for the sake of clarity. yeah, there is a specific reason why i'm not traversing the website using a single instance, but instead creating multiple instances. though now that i think about it, creating new instances may not even be necessary with ff.back() functionality, but it's still certainly a lot more straight forward. i will give your answer a read when i get some free time! thanks for trying to solve my issue <3

oldboy
– oldboy

2018-10-12 02:13:40 +00:00
Commented Oct 12, 2018 at 2:13

Add a comment |

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

Collectives™ on Stack Overflow