CERTIFICATE_VERIFY_FAILED error

Question

Python Django error:

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
unable to get local issuer certificate (_ssl.c:1131)>

Exception Location: /usr/lib/python3.8/urllib/request.py, line 1357, in do_open

Running: Python 3.8.10, Django 4.0.3, Ubuntu 20.04, Apache 2

I'm using Django for a simple contact form app, which currently works and throws no errors. The CERTIFICATE_VERIFY_FAILED error occurs when I use this Django library https://github.com/tiesjan/django-hcaptcha-field to add hCaptcha to the form.

The issue doesn't appear to be with django-hcaptcha-field; it appears to be Ubuntu and SSL certificates when the hCaptcha API is accessed.

I've looked at multiple questions on SO (esp. https://stackoverflow.com/questions/27835619/urllib-and-ssl-certificate-verify-failed-error , even though it's for OS X) and Ask Ubuntu Certificate problems .

I've tried all these "fixes":

pip install pyOpenSSL --upgrade
apt-get install --reinstall python3-certifi
pip install --upgrade certifi --force
apt install --reinstall openssl
apt install ca-certificates
update-ca-certificates --fresh
export SSL_CERT_DIR=/etc/ssl/certs

I "force" updated my Let's Encrypt SSLs.

I tried updated certs:

wget --quiet https://curl.haxx.se/ca/cacert.pem
export SSL_CERT_FILE=$HOME/cacert.pem

Nothing in my Python code requires import ssl

What else can I try?

---

Diagnostic outputs:

dpkg -l | grep cert returns

ica-certificates    20210119~20.04.2    all    Common CA certificates
certbot    0.40.0-1ubuntu0.1    all    automatically configure HTTPS using Let's Encrypt
dirmngr    2.2.19-3ubuntu2.1    amd64    GNU privacy guard - network certificate management service
python-certbot-apache    0.36.0-1    all    transitional dummy package
python3-certbot    0.40.0-1ubuntu0.1    all   main library for certbot
python3-certbot-apache   0.39.0-1    all    Apache plugin for Certbot
ipython3-certifi    2019.11.28-1    all    root certificates for validating SSL certs and verifying TLS hosts (python3)
ssl-cert    1.0.39    all    simple debconf wrapper for OpenSSL

dpkg -l | grep openssl returns

libxmlsec1-openssl:amd64    1.2.28-2    amd64        Openssl engine for the XML security library
openssl    1.1.1f-1ubuntu2.12    amd64        Secure Sockets Layer toolkit - cryptographic utility
perl-openssl-defaults:amd64    4    amd64        version compatibility baseline for Perl OpenSSL packages
python3-openssl    19.0.0-1build1    all          Python 3 wrapper around the OpenSSL library

whereis openssl returns

openssl: /usr/bin/openssl /usr/local/bin/openssl /usr/include/openssl /usr/share/man/man1/openssl.1ssl.gz

which openssl /usr/bin/openssl returns

/usr/local/bin/openssl
/usr/bin/openssl

ldd $(which wget) returns

linux-vdso.so.1 (0x00007ffd9f10f000)
libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007efdb3e3d000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007efdb3e34000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007efdb3e12000)
libssl.so.1.1 => /usr/local/lib/libssl.so.1.1 (0x00007efdb3d7a000)
libcrypto.so.1.1 => /usr/local/lib/libcrypto.so.1.1 (0x00007efdb3a8e000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efdb3a72000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007efdb3a5d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efdb386b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efdb3848000)
/lib64/ld-linux-x86-64.so.2 (0x00007efdb3f6b000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007efdb36c6000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efdb36c0000)

dpkg -l | grep python3-certifi returns

python3-certifi    2019.11.28-1    all  root certificates for validating SSL certs and verifying TLS hosts (python3)

Traceback:

Request Method: POST
Request URL: https://example.com/contact/contact/contact/
Django Version: 4.0.3
Python Version: 3.8.10
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'contactform.apps.ContactformConfig',
 'encrypted_files',
 'hcaptcha_field']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/lib/python3.8/http/client.py", line 1425, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
During handling of the above exception ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)), another exception occurred:
  File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/dist-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, callback_args, callback_kwargs)
  File "/var/www/html/example.com/public_html/contact/contactform/views.py", line 26, in contact
    if form.is_valid():
  File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 205, in is_valid
    return self.is_bound and not self.errors
  File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 200, in errors
    self.full_clean()
  File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 433, in full_clean
    self._clean_fields()
  File "/usr/local/lib/python3.8/dist-packages/django/forms/forms.py", line 445, in _clean_fields
    value = field.clean(value)
  File "/usr/local/lib/python3.8/dist-packages/django/forms/fields.py", line 199, in clean
    self.validate(value)
  File "/usr/local/lib/python3.8/dist-packages/hcaptcha_field/fields.py", line 129, in validate
    response = opener.open(request, timeout=hcaptcha_settings.TIMEOUT)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(args)
  File "/usr/lib/python3.8/urllib/request.py", line 1397, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
Exception Type: URLError at /contact/contact/
Exception Value: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

fields.py of hcaptcha_field of https://github.com/tiesjan/django-hcaptcha-field

import json
import logging
import ssl # added #############
import certifi # added #############
from urllib.error import HTTPError
from urllib.parse import urlencode
from urllib.request import build_opener, Request, ProxyHandler
from django import forms
from django.core.exceptions import ValidationError
from django.utils.translation import gettext_lazy as _
from hcaptcha_field.settings import hcaptcha_settings
from hcaptcha_field.widgets import hCaptchaWidget
LOGGER = logging.getLogger('hcaptcha_field')
DATA_ATTRIBUTE_CONFIG = frozenset([
    'theme',
    'size',
    'tabindex',
    'callback',
    'expired-callback',
    'chalexpired-callback',
    'open-callback',
    'close-callback',
    'error-callback',
])
QUERY_PARAMETER_CONFIG = frozenset([
    'onload',
    'render',
    'hl',
    'recaptchacompat'
])
class hCaptchaField(forms.Field):
    widget = hCaptchaWidget
    default_error_messages = {
        'error_hcaptcha': _(
            # Translators: Error shown when an internal server error occurred.
            'Something went wrong while verifying the hCaptcha. '
            'Please try again.'
        ),
        'invalid_hcaptcha': _(
            # Translators: Error shown when visitor did not pass the hCaptcha check.
            'hCaptcha could not be verified.'
        ),
        'required': _(
            # Translators: Error shown when visitor forgot to fill in the hCaptcha.
            'Please prove you are human.'
        ),
    }
def __init__(self, sitekey=None, **kwargs):
    """
    Initializer for `hCaptchaField` class. It determines data attributes
    for the widget class and constructs a widget if none is given. This
    constructed widget receives the URL of the JavaScript resource for the
    hCaptcha integration and the `sitekey` of the site to protect.
    """
    # Retrieve settings
    DEFAULT_CONFIG = hcaptcha_settings.DEFAULT_CONFIG
    JS_API_URL = hcaptcha_settings.JS_API_URL
    SITEKEY = hcaptcha_settings.SITEKEY

    # Determine widget data attributes
    self.widget_data_attrs = {}
    for setting in DATA_ATTRIBUTE_CONFIG:
        if setting in kwargs:
            self.widget_data_attrs[setting] = kwargs.pop(setting)
        elif setting in DEFAULT_CONFIG:
            self.widget_data_attrs[setting] = DEFAULT_CONFIG[setting]

    # If the `widget` argument is not given, instantiate `self.widget` with
    # the hCaptcha API url and the sitekey
    if 'widget' not in kwargs:
        # Determine hCaptcha API url
        query_params = {}
        for setting in QUERY_PARAMETER_CONFIG:
            if setting in kwargs:
                query_params[setting] = kwargs.pop(setting)
            elif setting in DEFAULT_CONFIG:
                query_params[setting] = DEFAULT_CONFIG[setting]
        if query_params:
            js_api_url = '%s?%s' % (JS_API_URL, urlencode(query_params))
        else:
            js_api_url = JS_API_URL

        # Determine hCaptcha sitekey
        self.sitekey = sitekey or SITEKEY

        # Instantiate widget
        kwargs['widget'] = self.widget(
                js_api_url=js_api_url, sitekey=self.sitekey)

    super().__init__(**kwargs)

def widget_attrs(self, widget):
    """
    Returns the widget attributes, including all the data attributes
    determined in the initializer.
    """
    attrs = super().widget_attrs(widget)
    for key, value in self.widget_data_attrs.items():
        attrs['data-%s' % key] = value
    return attrs

def validate(self, value):
    """
    Validates the field by verifying the value of the hidden field
    `h-captcha-response` with their API endpoint.
    """
    super().validate(value)

    # Build request
    opener = build_opener(ProxyHandler(hcaptcha_settings.PROXIES))
    post_data = urlencode({
        'secret': hcaptcha_settings.SECRET,
        'response': value,
        'sitekey': self.sitekey,
    }).encode('utf-8')
    request = Request(hcaptcha_settings.VERIFY_URL, post_data)

    # Perform request
    try:
        context=ssl.create_default_context(cafile=certifi.where()) # added ############
        response = opener.open(request, timeout=hcaptcha_settings.TIMEOUT)
    except HTTPError:
        LOGGER.exception("Failed to verify response with hCaptcha API.")
        raise ValidationError(
            self.error_messages['error_hcaptcha'],
            code='error_hcaptcha'
        )

    # Check response
    response_data = json.loads(response.read().decode('utf-8'))
    if not response_data.get('success'):
        LOGGER.error("Failed to pass hCaptcha check: %s", response_data)
        raise ValidationError(
            self.error_messages['invalid_hcaptcha'],
            code='invalid_hcaptcha'
        )

Answer 1

It looks like the context needs to be passed into an HTTPSHandler in order to make use of the certifi certificates.

Try patching your local copy of /usr/local/lib/python3.8/dist-packages/hcaptcha_field/fields.py with the following.

To the imports, as you've already done, add:

import certifi
import ssl

In addition, expand the import from urllib.request to include HTTPSHandler:

from urllib.request import build_opener, Request, ProxyHandler, HTTPSHandler

Also as you've already done, create an SSL context that uses the certifi certificates. I would add this right under the import block, though, rather than deep in the module code:

context=ssl.create_default_context(cafile=certifi.where())

Then, immediately below this new context=... line, create a new HTTPSHandler instance using context:

https_handler = HTTPSHandler(context=context)

Then, revise the line where opener is defined using the build_opener(...) call to include the new HTTPSHandler instance:

opener = build_opener(https_handler, ProxyHandler(hcaptcha_settings.PROXIES))

---

If I'm reading the build_opener docs correctly, the build_opener machinery is already using a default instance of HTTPSHandler as it tries to open the URL. Hopefully, replacing that default instance with this new https_handler instance, which will operate with the SSL context that includes the certifi certificates, will allow the urlopen call to work.

If this does work, then an issue/PR to the upstream project might be a good idea, to integrate this logic and either to provide a default behavior that pulls in the certifi certificates automatically if they're available, or to include a configuration option to let the user wire in the certifi certificates.

---

UPDATE 2022-04-14: Since this did fix the problem, I'll add some more detail as to why this fix was necessary and how it worked.

HTTPS requires the website you're accessing to provide a certificate that validates its identity. Otherwise, how do you know your traffic hasn't been messed with, man-in-the-middle attacked, etc.? The certificate that a website provides to you when you browse to it is issued to the operators of the website by a certificate issuing authority. This authority serves as a trusted third party, to "vouch for" the website operators and that, "yes, you can trust this certificate from them".

There are several of these issuing authorities out there. Those authorities issue their own certificates ("certificate authority certificates", or "CA certificates"), which are completely separate from the ones that individual websites provide. Your browser (or URL opener) needs to have access to these CA certificates in order to complete the 'trusted third party' verification of the certificate that the website presents. The key error the OP is facing here is the inability of the URL opener to find a good set of CA certificates to use for this third-party validation:

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

The message unable to get local issuer certificate means that urlopen cannot find the needed issuer certificate (CA certificate) on the local system.

So, the solution is to point urlopen to the CA certificates provisioned by the certifi package:

• In order to tell the opener where to find them, you have to provide their location when you create opener from build_opener

• build_opener works based upon a collection of handler objects, so the CA certificate information must be included within a suitable handler

• The only handler (that I could find in the docs) that includes information about the SSL context is the HTTPSHandler

• The way that HTTPSHandler takes in information about SSL-related configuration is through the context keyword argument when creating a new instance, which takes an 'SSL context' object

• ssl.create_default_context is (as best I understand) a factory function for creating SSL context objects, customizeable in various ways

• One of the ways it's customizeable is with the location of a CA certificate store (via the cafile argument)

• certifi provides the location on disk of its curated CA certificate store through the certifi.where() function

If you scan these bullets from bottom to top, they map pretty well to each of the steps in the answer.