``from future import standard_library`` breaks requests

[AlexHill]

Hello,

Working with Mezzanine's current master (with the Python 3 support) I'm ran into an issue where Requests was mysteriously straight-up broken - every get request raised an AttributeError.

To replicate: make a new virtualenv, install future and requests, then:

Python 2.7.5 (default, Sep  2 2013, 05:24:04) 
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from future import standard_library
>>> import requests
>>> requests.get('http://www.google.com/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/sessions.py", line 361, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/sessions.py", line 464, in send
    r = adapter.send(request, **kwargs)
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/adapters.py", line 369, in send
    r = self.build_response(request, resp)
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/adapters.py", line 179, in build_response
    extract_cookies_to_jar(response.cookies, req, resp)
  File "/Users/al/venvs/future-test/lib/python2.7/site-packages/requests/cookies.py", line 117, in extract_cookies_to_jar
    jar.extract_cookies(res, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cookielib.py", line 1642, in extract_cookies
    for cookie in self.make_cookies(response, request):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cookielib.py", line 1559, in make_cookies
    rfc2965_hdrs = headers.getheaders("Set-Cookie2")
AttributeError: 'HTTPMessage' object has no attribute 'getheaders'

Don't really know enough about future to know exactly what the problem is, but I assume requests is getting a future library internally where it expects the standard Python 2 library. Could it be something to do with the lack of support for http.cookies?

Should the import be behaving like that? Can it be limited to the file it's written in?

[AlexHill]

The exception goes away if you comment out the http entry in RENAMES in standard_library/http/client.py.

Also the HTTPMessage object in question has no getheaders method, but it does have a get_all method, which is what Python 3's cookiejar.py expects instead of getattribute in the equivalent location. Seems like a Python-3-style HTTPMessage is being passed into the Python 2 standard library's libcookie, and they aren't compatible.

[edschofield]

Thanks very much for your detailed bug report, Alex. I'll look into this as a priority ...

[edschofield]

Just a note that I am taking this seriously. I have been experimenting with various implementations in an effort to contain the import hooks to the module that imports future.standard_library. I haven't yet found a good solution, so I will now start reaching out for help from the import-SIG guys.

If it turns out that there's no safe way to do this, a context manager may have to suffice. This is currently in master, although it's not finished yet.

Thanks again for this bug report and your investigations.

[edschofield]

An update (v0.11.3) has just gone out that makes it possible to import and use requests after removing the standard library import hooks, like this:

from future import standard_library
import http.client  # etc
standard_library.remove_hooks()
import requests

requests.get('http://www.google.com')

In v0.12, the import hooks will require an explicit function call to install them to make the usage clearer.

@AlexHill: thanks again for the report.

[merwok]

Is it impossible to fix the root of this bug thanks to Alex’ description of the issue? If it just takes an alias or wrapper to provide a getheaders method, why not try that?

[edschofield]

Hi Eric,

I believe this does solve the root problem. The bug arose because requests already has its own Py2/3 compatibility code using try/except blocks, and this was interacting with the import hooks from future.standard_library - by importing and use Py3 versions of some stdlib modules and incompatible Py2 versions of others.

The v0.11.3 update fixes this in a safe way by removing future.standard_library submodules from the sys.modules import cache upon calling remove_hooks().

I am learning that explicit really is better than implicit! ;)

On 27 Feb 2014, at 16:51, Éric Araujo notifications@github.com wrote:

Is it impossible to fix the root of this bug thanks to Alex’ description of the issue? If it just takes an alias or wrapper to provide a getheaders method, why not try that?

—
Reply to this email directly or view it on GitHub.

[merwok]

I see, thanks for the explanation!