5

I want to scrape the html generated by javascript , just like what you can see in Firebug.

UPDATE: I've found this article: http://blog.motane.lu/2009/07/07/downloading-a-pages-content-with-python-and-webkit/ which use PyQt to solve the problem and it works well for me.

BUT another problem occur: I have to login the website first, but I don't know how to simulate login in PyQt .... :(

Improve this question
1
  • I'm not recommending to use pyQT for this task. Selenium fits better here and offers a huge community that answered thousands of questions for many challenges. Have a look at my function posted here stackoverflow.com/a/73054340/4293361 Perhaps it helps. For my purposes it worked like a charme...
    – NeuroMorphing
    Commented Jul 20, 2022 at 16:27

3 Answers 3

Reset to default
5

Have a look at this article which describes using Windmill to do scrape a page after Javascript has been executed by the browser.

This article will show how to extract the desired information using the same three steps when the web page is not written directly using HTML, but is auto-generated using JavaScript to update the DOM tree.

They have some examples I am sure you can easily adapt.

Improve this answer
2
  • I've seen this article yesterday.But it seems that the user have to open a web browser to scrape the page, and that's not what I want ... Thanks anyway.
    – wong2
    Commented Apr 25, 2010 at 4:47
  • 1
    It's possible (not easy but possible) to run e.g., Firefox in a headless mode.
    – Ian Bicking
    Commented Apr 25, 2010 at 4:52
0

To be precise with terminology, Javascript does not generate HTML. Javascript generates and manipulates the DOM in your browser. The Firebug is showing you HTML representation of that DOM so that it would be readable. The HTML does not actually exist. :)

I don't think an out-of-the box easy solution exists. You may want to look at this blog post and comments that have some pointers.

Improve this answer
0

You could use python spidermonkey, which is a python wrapper to Firefox's engine: http://code.google.com/p/python-spidermonkey/

But the project seems a bit immature.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.