3.15.0a8 fails test_urllib2_localnet with OpenSSL 4.0.0beta1

[hroncok]

Bug report

Bug description:

When building Python 3.15.0a8 with OpenSSL 4.0.0beta1, which has just landed in Fedora Rawhide (not yet on the mirrors, only in the buildroot), we see the following test failures:

======================================================================
ERROR: test_https (test.test_urllib2_localnet.TestUrlopen.test_https)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/test/test_urllib2_localnet.py", line 569, in test_https
    data = self.urlopen("https://localhost:%s/bizarre" % handler.port, context=context)
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/test/test_urllib2_localnet.py", line 473, in urlopen
    l.extend(f.readlines(200))
             ~~~~~~~~~~~^^^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/http/client.py", line 714, in readline
    result = self.fp.readline(limit)
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/socket.py", line 734, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/ssl.py", line 1355, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/ssl.py", line 1154, in read
    return self._sslobj.read(len, buffer)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
ssl.SSLError: A failure in the SSL library occurred (_ssl.c:2982)
======================================================================
ERROR: test_https_sni (test.test_urllib2_localnet.TestUrlopen.test_https_sni)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/test/test_urllib2_localnet.py", line 585, in test_https_sni
    self.urlopen("https://localhost:%s" % handler.port, context=context)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/test/test_urllib2_localnet.py", line 473, in urlopen
    l.extend(f.readlines(200))
             ~~~~~~~~~~~^^^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/http/client.py", line 714, in readline
    result = self.fp.readline(limit)
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/socket.py", line 734, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/ssl.py", line 1355, in recv_into
    return self.read(nbytes, buffer)
           ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/python3.15-3.15.0_a8-build/Python-3.15.0a8/Lib/ssl.py", line 1154, in read
    return self._sslobj.read(len, buffer)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
ssl.SSLError: A failure in the SSL library occurred (_ssl.c:2982)
----------------------------------------------------------------------
Ran 2 tests in 0.142s
FAILED (errors=2)

Considering #146207 is closed, I thought I better let you know.

CPython versions tested on:

3.15

Operating systems tested on:

Linux

[hugovk]

This can be reproduced on our CI with hugovk@4e9bb25:

• 4.0.0-alpha1: https://github.com/hugovk/cpython/actions/runs/24188684863/job/70599883528
• 4.0.0-beta1: https://github.com/hugovk/cpython/actions/runs/24188684863/job/70599883308

#146217 added some initial support for 4.0.0-alpha1, but more is needed.

[vstinner]

I noticed these test failures (test_urllib2_localnet) when testing OpenSSL 4.0.0 alpha 1 and I wrote a patch fixing them: #146217 (comment). I didn't merge this change since I'm not confident that my change makes sense/is correct.

It seems like OpenSSL 4 behaves differently than OpenSSL 3 when reaching EOF.

cc @picnixz @gpshead @dstufft @alex

[vstinner]

I can reproduce the issue on Fedora 43 with commands:

python3 Tools/ssl/multissltests.py --steps=library --base-directory "$PWD/multissl" --openssl '4.0.0-beta1' --system Linux
export LD_LIBRARY_PATH=$PWD/multissl/openssl/4.0.0-beta1/lib64
./configure --with-pydebug --with-openssl=$PWD/multissl/openssl/4.0.0-beta1/
make
./python -m test -v test_urllib2_localnet -u all -v

[picnixz]

I quickly glanced at the different errors we had and it appears we are hitting this path in PySSL_SetError with OpenSSL 4.x:

        case SSL_ERROR_SSL:
        {
            p = PY_SSL_ERROR_SSL;
            if (e == 0) {
                /* possible? */
                errstr = "A failure in the SSL library occurred";
            }

This means that we had some SSL error (SSL_ERROR_SSL) but ERR_peek_last_error did not set an error code. This can be done for various reasons: either they forgot to update something, or another path is being taken. The way we propagate SSL errors is a bit messy but we may have missed one call. Ideally, we should know the exact code path that was taken and to see why ERR_peek_last_error() returned 0 (it's the value of e above).

[vstinner]

I didn't update Modules/_ssl_data*.h for OpenSSL 4 in my first change for alpha 1.

[picnixz]

I don't think the problem would be mnemonics here (though we can try to see what happens if we do an update). What's surprising is ERR_peek_last_error returns 0 (this is an OSSL function). It may be because we are incorrectly calling it at a point where there is no more an error though.