10

I am scraping a website that has a list of football games generated using JavaScript. I have written the following line that creates a list of all the game elements on the page:

list = browser.find_elements_by_xpath('//*[@data-sportid="1"]')

If I then write

for game in list:
    print game.text

it prints all the text fields contained in each of the games (home team name, away team name, etc.)

However, the loop

for game in list:
    print game.find_element_by_xpath('//*[@class="home-team"]').text

prints the very first home team's name in the page for each iteration of the loop.

It appears that game.find_element_by_xpath is searching the entire page, and not just this game element, and so it keeps returning the first home team name on the page.

How can I search for a child element within each item of the list?

EDIT

Here is the exact page I am working on

2
  • can u give us the site?\
    – user6665922
    Commented Apr 16, 2017 at 22:38
  • 1
    @nooby added it to the question
    – KOB
    Commented Apr 16, 2017 at 23:05

2 Answers 2

25

@Justin Bartz Thank you.

You were using XPath //*[@class="home-team"] but no matter what parent element you are using the // tells XPath to search the whole document not just the children of the parent element. Using the XPath .//*[@class="home-team"] with the period in front of the forward slashes (IE .//) it tells it to search under the current element only.

Hope this expanded explanation helps the understanding.

   driver = webdriver.Chrome()
    driver.get("https://www.betfair.com/exchange/football/coupon?id=2")
    list = driver.find_elements_by_xpath('//*[@data-sportid="1"]')
    for game in list:
        print(game.find_element_by_css_selector('span.home-team').text)

or

    driver = webdriver.Chrome()
    driver.get("https://www.betfair.com/exchange/football/coupon?id=2")
    list = driver.find_elements_by_xpath('//*[@data-sportid="1"]')
    for game in list:
        print(game.find_element_by_xpath('.//span[@class="home-team"]').text)
2
  • 1
    The explanation from Justin's answer is key to learning what happened and to stop making the same mistake again. It should be incorporated into this answer. Commented Apr 6, 2020 at 19:15
  • 5
    I've been searching for the whole internet for how to solve this. And the answer is a single "." at the start of the xpath. Thank you Barney! Commented May 1, 2020 at 2:42
16

This is to add an explanation to Barneys answer.

You were using xpath //*[@class="home-team"] but no matter what parent element you are using the // tells xpath to search the whole document not just the children of the parent element.
Using the xpath .//*[@class="home-team"] with the period in front of the forward slashes (IE .//) it tells it to search under the current element only.

Hope this expanded explanation helps the understanding.

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.