TST/DEPS: new boto breaks tests

[h-vetinari]

In the course of #23731, I wanted the CI to run a higher boto3 (resp. botocore) than before, and this breaks the construction of the s3_resource fixture, which then silently skips all tests, due to a

    except:  # noqa: flake8
        pytest.skip("failure to use s3 resource")

While this is an upstream change/issue, we need to adapt the tests to find the right credentials, and - more pressingly - the travis-36 is the only job that tests boto, AFAICT.

In any case (notice, there's no pandas import, but this is what the constructor of s3_resource tries), this works:

>>> import moto
>>> import boto3
>>> boto3.__version__
'1.7.84'
>>> import botocore
>>> botocore.__version__
'1.10.84'
>>>
>>> s3 = moto.mock_s3()
>>> s3.start()
>>> conn = boto3.resource("s3", region_name="eu-west-3")
>>> conn.create_bucket(Bucket="pandas-test")
s3.Bucket(name='pandas-test')

while with botocore>=1.11 (which I need for testing #23731):

>>> import moto
>>> import boto3
>>> boto3.__version__
'1.9.45'
>>> import botocore
>>> botocore.__version__
'1.12.45'
>>> 
>>> s3 = moto.mock_s3()
>>> s3.start()
>>> conn = boto3.resource("s3", region_name="eu-west-3")
>>> conn.create_bucket(Bucket="pandas-test")
Traceback (most recent call last):
[...]
botocore.exceptions.NoCredentialsError: Unable to locate credentials

Here's a more detailed view using boto3.set_stream_logger('botocore', level='DEBUG'). Seems that the new version has become more restrictive regarding credential discovery. I don't have experience with boto (the debug command was 1min of googling ;-)), but that shouldn't be too hard to set up, hopefully.

Old version:

>>> import moto
>>> import boto3
>>> boto3.__version__
'1.7.84'
>>> import botocore
>>> botocore.__version__
'1.10.84'
>>>
>>> boto3.set_stream_logger('botocore', level='DEBUG')
>>>
>>> s3 = moto.mock_s3()
>>> s3.start()
>>>
>>> conn = boto3.resource("s3", region_name="eu-west-3")
2018-11-17 17:23:37,342 botocore.session [DEBUG] Loading variable profile from defaults.
2018-11-17 17:23:37,343 botocore.session [DEBUG] Loading variable ca_bundle from defaults.
2018-11-17 17:23:37,343 botocore.session [DEBUG] Loading variable profile from defaults.
2018-11-17 17:23:37,343 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x000001B81BB0C6A8>
2018-11-17 17:23:37,344 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x000001B81BAD5378>
2018-11-17 17:23:37,345 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x000001B81E3B5BF8>
2018-11-17 17:23:37,345 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x000001B81BAD5158>
2018-11-17 17:23:37,345 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2018-11-17 17:23:37,346 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2018-11-17 17:23:37,348 botocore.client [DEBUG] Registering retry handlers for service: s3
2018-11-17 17:23:37,349 botocore.client [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
>>>
>>>
>>>
>>> conn.create_bucket(Bucket="pandas-test")
2018-11-17 17:23:39,055 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <function validate_bucket_name at 0x000001B81BB22D90>
2018-11-17 17:23:39,056 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x000001B81F0CCBA8>>
2018-11-17 17:23:39,056 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <function generate_idempotent_uuid at 0x000001B81BB229D8>
2018-11-17 17:23:39,056 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <function add_expect_header at 0x000001B81BB242F0>
2018-11-17 17:23:39,056 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x000001B81F0CCBA8>>
2018-11-17 17:23:39,056 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateBucket) (verify_ssl=True) with params: {'url_path': '/pandas-test', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.7.84 Python/3.6.6 Windows/10 Botocore/1.10.84 Resource'}, 'body': b'', 'url': 'https://s3.eu-west-3.amazonaws.com/pandas-test', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x000001B81F0CC390>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'pandas-test'}}}
2018-11-17 17:23:39,057 botocore.hooks [DEBUG] Event request-created.s3.CreateBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000001B81F0CC358>>
2018-11-17 17:23:39,058 botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <function set_operation_specific_signer at 0x000001B81BB228C8>
2018-11-17 17:23:39,058 botocore.hooks [DEBUG] Event before-sign.s3.CreateBucket: calling handler <function fix_s3_host at 0x000001B81B9C5730>
2018-11-17 17:23:39,058 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.eu-west-3.amazonaws.com/pandas-test
2018-11-17 17:23:39,059 botocore.utils [DEBUG] URI updated to: https://pandas-test.s3.eu-west-3.amazonaws.com/
2018-11-17 17:23:39,059 botocore.auth [DEBUG] Calculating signature using v4 auth.
2018-11-17 17:23:39,059 botocore.auth [DEBUG] CanonicalRequest:
PUT
/

host:pandas-test.s3.eu-west-3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20181117T162339Z
x-amz-security-token:test-session-token

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-11-17 17:23:39,060 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20181117T162339Z
20181117/eu-west-3/s3/aws4_request
97ac24cff3d0f7f251e960da8f54204f43f252b460867434a3151227bf965952
2018-11-17 17:23:39,060 botocore.auth [DEBUG] Signature:
13e52ecf8bb5980cd358811e4cbb5cca5566f88a80e33188b4b3d29837fdef11
2018-11-17 17:23:39,061 botocore.endpoint [DEBUG] Sending http request: <PreparedRequest [PUT]>
2018-11-17 17:23:39,064 botocore.parsers [DEBUG] Response headers: {'Content-Type': 'text/plain'}
2018-11-17 17:23:39,064 botocore.parsers [DEBUG] Response body:
b'<CreateBucketResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01">\n  <CreateBucketResponse>\n    <Bucket>pandas-test</Bucket>\n  </CreateBucketResponse>\n</CreateBucketResponse>'
2018-11-17 17:23:39,071 botocore.hooks [DEBUG] Event needs-retry.s3.CreateBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x000001B81F0CC898>
2018-11-17 17:23:39,074 botocore.retryhandler [DEBUG] No retry needed.
2018-11-17 17:23:39,074 botocore.hooks [DEBUG] Event needs-retry.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x000001B81F0CCBA8>>
2018-11-17 17:23:39,075 botocore.hooks [DEBUG] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x000001B81E3B5E18>
s3.Bucket(name='pandas-test')

In the new version, the problem already starts upon connection, but only shows up when trying to interact with the resource:

>>> import moto
>>> import boto3
>>> boto3.__version__
'1.7.84'
>>> import botocore
>>> botocore.__version__
'1.10.84'
>>>
>>> boto3.set_stream_logger('botocore', level='DEBUG')
>>>
>>> s3 = moto.mock_s3()
>>> s3.start()
>>>
>>> conn = boto3.resource("s3", region_name="eu-west-3")
2018-11-17 17:07:55,712 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2018-11-17 17:07:55,715 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2018-11-17 17:07:55,718 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2018-11-17 17:07:55,718 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2018-11-17 17:07:55,721 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2018-11-17 17:07:55,722 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2018-11-17 17:07:55,723 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2018-11-17 17:07:55,728 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2018-11-17 17:07:55,728 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2018-11-17 17:07:55,728 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2018-11-17 17:07:55,729 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2018-11-17 17:07:55,825 botocore.loaders [DEBUG] Loading JSON file: C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\boto3\data\s3\2006-03-01\resources-1.json
2018-11-17 17:07:56,149 botocore.credentials [DEBUG] Looking for credentials via: env
2018-11-17 17:07:56,149 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2018-11-17 17:07:56,153 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2018-11-17 17:07:56,153 botocore.credentials [DEBUG] Looking for credentials via: custom-process
2018-11-17 17:07:56,154 botocore.credentials [DEBUG] Looking for credentials via: config-file
2018-11-17 17:07:56,154 botocore.credentials [DEBUG] Looking for credentials via: ec2-credentials-file
2018-11-17 17:07:56,155 botocore.credentials [DEBUG] Looking for credentials via: boto-config
2018-11-17 17:07:56,156 botocore.credentials [DEBUG] Looking for credentials via: container-role
2018-11-17 17:07:56,156 botocore.credentials [DEBUG] Looking for credentials via: iam-role
2018-11-17 17:07:57,160 botocore.utils [DEBUG] Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\util\connection.py", line 79, in create_connection
    raise err
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\util\connection.py", line 69, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\httpsession.py", line 258, in send
    decode_content=False,
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
    raise value
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\http\client.py", line 964, in send
    self.connect()
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\connection.py", line 196, in connect
    conn = self._new_conn()
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\urllib3\connection.py", line 176, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x000002C89625F828>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\utils.py", line 292, in _get_request
    response = self._session.send(request.prepare())
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\httpsession.py", line 282, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2018-11-17 17:07:57,197 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2018-11-17 17:07:57,201 botocore.loaders [DEBUG] Loading JSON file: C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\data\endpoints.json
2018-11-17 17:07:57,209 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x000002C8947BF268>
2018-11-17 17:07:57,298 botocore.loaders [DEBUG] Loading JSON file: C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\data\s3\2006-03-01\service-2.json
2018-11-17 17:07:57,307 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x000002C89478B1E0>
2018-11-17 17:07:57,307 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x000002C8961BBB70>
2018-11-17 17:07:57,327 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x000002C894783F28>
2018-11-17 17:07:57,328 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2018-11-17 17:07:57,332 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2018-11-17 17:07:57,334 botocore.loaders [DEBUG] Loading JSON file: C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\data\_retry.json
2018-11-17 17:07:57,334 botocore.client [DEBUG] Registering retry handlers for service: s3
2018-11-17 17:07:57,335 botocore.client [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
>>>
>>>
>>>
>>> conn.create_bucket(Bucket="pandas-test")
2018-11-17 17:09:34,046 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <function validate_bucket_name at 0x000002C8947DFD90>
2018-11-17 17:09:34,047 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x000002C896595C88>>
2018-11-17 17:09:34,047 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <function generate_idempotent_uuid at 0x000002C8947DF9D8>
2018-11-17 17:09:34,049 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <function add_expect_header at 0x000002C8947E12F0>
2018-11-17 17:09:34,049 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x000002C896595C88>>
2018-11-17 17:09:34,050 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/pandas-test', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.9.45 Python/3.6.6 Windows/10 Botocore/1.12.45 Resource'}, 'body': b'', 'url': 'https://s3.eu-west-3.amazonaws.com/pandas-test', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x000002C89653D518>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'pandas-test'}}}
2018-11-17 17:09:34,050 botocore.hooks [DEBUG] Event request-created.s3.CreateBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002C89652DE10>>
2018-11-17 17:09:34,051 botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <function set_operation_specific_signer at 0x000002C8947DF8C8>
2018-11-17 17:09:34,051 botocore.hooks [DEBUG] Event before-sign.s3.CreateBucket: calling handler <function fix_s3_host at 0x000002C894682400>
2018-11-17 17:09:34,051 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.eu-west-3.amazonaws.com/pandas-test
2018-11-17 17:09:34,052 botocore.utils [DEBUG] URI updated to: https://pandas-test.s3.eu-west-3.amazonaws.com/
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\boto3\resources\factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\boto3\resources\action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\client.py", line 610, in _make_api_call
    operation_model, request_dict)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\signers.py", line 157, in sign
    auth.add_auth(request)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\auth.py", line 425, in add_auth
    super(S3SigV4Auth, self).add_auth(request)
  File "C:\ProgramData\Miniconda3\envs\pandas-dev\lib\site-packages\botocore\auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

[h-vetinari]

Here's the output in the most recent travis run of #23731, where I disabled the skips:
https://travis-ci.org/pandas-dev/pandas/jobs/456373794

23 errors, means there are 23 tests using s3 - that should not be such a big burden that it couldn't be added to another job as well, no?

[h-vetinari]

Actually, this might be a moto issue: getmoto/moto#1941