Using these options together doesn't work as expected:
options.add_argument('--disable-javascript')
options.add_argument('--headless')
When I disable JavaScript without using headless mode, it looks like this.
But when I try to disable JavaScript with headless mode, it looks like this.
It's now possible to disable JavaScript in headless Chrome if you use the new headless mode:
The new way (2023): --headless=new (Chrome 109 and above)
The previous way: --headless=chrome (Chrome 96 - Chrome 108)
(The old way: --headless)
There's more info on that here: https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c36
For the updated Chrome 109 info, see: https://github.com/chromium/chromium/commit/e9c516118e2e1923757ecb13e6d9fff36775d1f4
You can use the code below as options for completely disabling JavaScript while running headlessly:
prefs = {}
prefs["webkit.webprefs.javascript_enabled"] = False
prefs["profile.content_settings.exceptions.javascript.*.setting"] = 2
prefs["profile.default_content_setting_values.javascript"] = 2
prefs["profile.managed_default_content_settings.javascript"] = 2
options = webdriver.ChromeOptions()
options.add_experimental_option("prefs", prefs)
options.add_argument('--disable-javascript')
options.add_argument('--headless=new')
--headless=new. (They just changed it)