Certificate error with Python requests module

Question

I am trying to code in Python what works for me with CURL:

curl --location -k --user "myuser" -X GET https://some_company_domain_path/cm/open-api/persistency/v1/yaml
Enter host password for user 'myuser':

I enter the password and get the YAML.

This also works:

curl --location -k --user myuser:mypassword -X GET https://some_company_domain_path/cm/open-api/persistency/v1/yaml

The Python code I am trying to use is based on https://requests.readthedocs.io/en/latest/user/authentication/.

import requests
r = requests.get('https://some_company_domain_path/cm/open-api/persistency/v1/yaml', auth=('myuser', 'mypassword'))

and I got this error

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
    ssl_context=context)
  File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 354, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='some_company_domain_path', port=443): Max retries exceeded with url: /cm/open-api/persistency/v1/yaml (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='some_company_domain_path', port=443): Max retries exceeded with url: /cm/open-api/persistency/v1/yaml (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

I don't have a certificate (otherwise I would be using it with CURL), so user/password authentication is the only thing I have for the time being. And CURL is not asking me for certificates or proxies.

I am using Python 3.6.8 on Linux 4.18.0-305.45.1.el8_4.x86_64

I have read similar questions but none gives me any hint of what I am doing wrong.

Another method I have tried, with the same resulting error, is:

from requests.auth import HTTPDigestAuth
url = 'https://some_company_domain_path/cm/open-api/persistency/v1/yaml'
requests.get(url, auth=HTTPDigestAuth('myuser', 'mypassword'))

Thank you in advance.

Answer 1

According to curl man page, the -k option disables SSL verification.

You can add the verify=False option to also disable SSL verification in your Python code:

r = requests.get('https://some_company_domain_path/cm/open-api/persistency/v1/yaml', auth=('myuser', 'mypassword'), verify=False)

From the documentation:

requests can also ignore verifying the SSL certificate if you set verify to False.

>>> requests.get('https://kennethreitz.com', verify=False)`
<Response [200]>

---

It also looks like in your request using curl, you are using BasicAuth. To use BasicAuth in Python with the requests module, you should do the following:

from requests.auth import HTTPBasicAuth

auth = HTTPBasicAuth('myuser', 'mypassword')

You final code should then look like this:

import requests
from requests.auth import HTTPBasicAuth

auth = HTTPBasicAuth("myuser", "mypassword")
r = requests.get(
    "https://some_company_domain_path/cm/open-api/persistency/v1/yaml",
    auth=auth,
    verify=False,
)