Headless webdriver returns error but non-headless works

Question

I am doing a simple experiment with Amazon and Webdriver. However, using Webdriver Headless cannot find elements and errors out, but non-headless works.

Any suggestions how to get it working headless? There is a comment right above the --headless flag.

from selenium import webdriver
import sys
import os

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def get_inventory(url):

    chrome_options = Options()

    # Making it headless breaks. Commenting
    # this line, making it non-headless works.  
    chrome_options.add_argument("--headless")

    chrome_options.add_experimental_option(
        "prefs", {'profile.managed_default_content_settings.javascript': 2})
    chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'

    driver = webdriver.Chrome(executable_path=os.path.abspath(
        'chromedriver'), chrome_options=chrome_options)

    driver.set_window_size(1200, 1000)

    try:
        driver.get(url)

        add_to_cart_button_xp = '//*[@id="add-to-cart-button"]'
        add_to_cart_button = driver.find_element_by_xpath(add_to_cart_button_xp)
        add_to_cart_button.click()

        driver.get('https://www.amazon.com/gp/cart/view.html/ref=lh_cart')

        qty_field_xp = '//div/input[starts-with(@name, "quantity.") and @type="text"]'
        qty_field = driver.find_element_by_xpath(qty_field_xp)
        qty_field.clear()
        qty_field.send_keys("2")

        update_link_xp = f'//input[@value="Update" and @type="submit"]'
        update_link = driver.find_element_by_xpath(update_link_xp)
        update_link.click()



url = 'https://www.amazon.com/Pexio-Professional-Stainless-Food-Safe-Dishwasher/dp/B07BGBSY9F'
get_inventory(url)

The issue appears to be that when using --headless some elements are not found. Testing to see if it has to do with cookies. — smorhaim
– smorhaim, Commented Nov 6, 2018 at 17:21
You might want to switch to FireFox driver. I had also problems with headless Chrome.which disappeared once I switched to FireFox. — Zvi
– Zvi, Commented Jan 14, 2021 at 21:16

kungphu · Accepted Answer · 2018-11-08 08:16:37Z

I think you just had some selector issues. I checked the elements and updated the quantity setting; everything else should be pretty much the same, aside from the binary locations.

from selenium import webdriver
import sys
import os

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


def get_inventory(url):
    chrome_options = Options()

    chrome_options.add_argument("--headless")

    driver = webdriver.Chrome(
        executable_path='/usr/bin/chromedriver', 
        chrome_options=chrome_options,
    )
    chrome_options.add_experimental_option(
        "prefs", 
        {'profile.managed_default_content_settings.javascript': 2},
    )
    chrome_options.binary_location = '/usr/bin'

    driver.set_window_size(1200, 1000)

    try:
        driver.get(url)
        driver.save_screenshot("/tmp/x1.png")

        driver.find_element_by_xpath('//*[@id="add-to-cart-button"]').click()

        driver.get('https://www.amazon.com/gp/cart/view.html/ref=lh_cart')

        driver.find_element_by_xpath("//span[@data-action='a-dropdown-button']").click()
        driver.find_element_by_xpath("//*[@class='a-dropdown-link'][text()[contains(., '2')]]").click()

        driver.find_element_by_class_name("nav-logo-base").click()

        driver.save_screenshot("/tmp/confirm.png")
        driver.close()
    except Exception as e:
        print(e)


url = 'https://www.amazon.com/Pexio-Professional-Stainless-Food-Safe-Dishwasher/dp/B07BGBSY9F'
get_inventory(url)

I've run this with and without --headless and it's working fine for me. I navigated to the homepage at the end so you can confirm the quantity change worked (hence the screenshot).

user973718 · Accepted Answer · 2018-10-26 20:02:35Z

0

What is the behavior you see?

When I enabled headless, scripts started failing because running headless slows execution down.

I currently run chrome with these options:

'--no-sandbox', '--headless', '--window-size=1920,1080', '--proxy-server="direct://"', '--proxy-bypass-list=*'

The last two options supposedly help with the slowness, but I didn't see any difference.

Hope this helps.

answered Oct 26, 2018 at 20:02

user973718

2452 gold badges5 silver badges17 bronze badges

1 Comment

smorhaim Over a year ago

The elements are not found, it errors out. NoSuchElementException .... so the exact same script, runs fine non-headless, but doesn't in headless mode.

Walter Tross · Accepted Answer · 2018-11-01 21:15:24Z

I verified your claim on my Mac (using /Applications/Google Chrome.app/Contents/MacOS/Google Chrome).

My guess is that, since you are moving from an item page to the cart page of Amazon, the cookies are lost, so that the cart page won't show any item, and therefore won't contain any text input with a name starting with “quantity”, which is what the exception is about.

Googling for headless chrome cookies yields this page, which in turn points to this page, the content of which could also be about your problem. Be it this, or be it a particularly smart behavior of the Amazon website, the fact remains: the cookie that stores the cart (or a key thereof, but the result is the same) is not read by the cart page when in headless mode.

Collectives�� on Stack Overflow

Headless webdriver returns error but non-headless works

3 Answers 3

Comments

1 Comment

Comments

Hot Network Questions

Collectives�� on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Related