Fetching all href links from the page source using Selenium WebDriver with Java

Question

I am trying to test HTTP RESPONSE of all href links on the page, using WebDriver to fetch all the links from the page and then using http.connect to get the response status.

Code snippet to get links of anchor tag:

List<WebElement> list = driver.findElements(By.cssSelector("a"));
for (WebElement link : list) {
    System.out.println(link.getText());
}

But my page has many more href links which are not having anchor tag <a> and might reside outside body of the page in header section or so. Some examples are as shown below. Above webdriver code wont solve in fetching all types of links. Also need to extract src links in some cases...

<script src="https://www.test.com/js/50/f59ae5bd.js"></script> 
<link rel="stylesheet" href="www.test.com/css/27/5a92c391c7be2e9.css" rel="stylesheet" type="text/css" />
<link sizes="72x72" href="https://www.test.com/css/27/5a92c391c7b/kj32.png" />
<li><a href="https://www.test.com/webapps/mpp/resortcheck">resortcheck</a>

I would appreciate if someone can guide how to go about or has resolved similar issues in getting all href links from page.

Never been through this, but you can try achieve it by using getPageSource function and get each line which contains href and split it with =". — Vivek Singh
– Vivek Singh, Commented Jan 27, 2015 at 5:29

vins · Accepted Answer · 2015-01-27 15:44:21Z

3

You can use Xpath to get all the elements containing the attributes href / src.

List<WebElement> list=driver.findElements(By.xpath("//*[@href or @src]"));

I tried something like this to get all the links to the other resource files. It works fine.

       WebDriver driver = new FirefoxDriver();
       driver.get("http://www.google.com");

       List<WebElement> list=driver.findElements(By.xpath("//*[@href or @src]"));

       for(WebElement e : list){
           String link = e.getAttribute("href");
           if(null==link)
               link=e.getAttribute("src");
           System.out.println(e.getTagName() + "=" + link);
       }

edited Jan 27, 2015 at 15:44

answered Jan 27, 2015 at 6:02

vins

15.5k3 gold badges42 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

newuser Over a year ago

hi, so how to click on each links and get pagesource of each link, in the same for loop.

Uday · Accepted Answer · 2015-01-27 06:56:22Z

1

What do you mean by links exists outside of body?

All links are identifiable by html tag. What other ways to represent links?

Check my below code may help:

public static void main(String[] args)
{
    WebDriver driver = new FirefoxDriver();
    driver.get("http://www.google.com/");
    List<WebElement> links=driver.findElements(By.tagName("a"));
    for(WebElement ele:links)
        System.out.println(ele.getAttribute("href"));
}

answered Jan 27, 2015 at 6:56

Uday

1,50411 gold badges39 silver badges60 bronze badges

2 Comments

KitKarson Over a year ago

The questions clearly says not all the elements have 'a' tag.

QualityThoughts Over a year ago

Sorry for delayed response, the way Vinoth has shown works for me using xpath("//*[@href or @src]"))

Collectives™ on Stack Overflow

Fetching all href links from the page source using Selenium WebDriver with Java

2 Answers 2

1 Comment

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Related