Skip to content

[FR] Allow poolsize and maxpool size settings for http client #648

Open
@sssaang

Description

@sssaang

Is your feature request related to a problem? Please describe.
I am currently facing a performance issue in which the default max pool size of requests library has become a bottleneck.

Describe the solution you'd like
A clear and concise description of what you want to happen.
I was wondering if we could increase the pool size by taking them as parameters for message client constructor
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context


class HttpClient:
    def __init__(
            self, credential=None, session=None, base_url='', headers=None,
            retries=DEFAULT_RETRY_CONFIG, timeout=DEFAULT_TIMEOUT_SECONDS,
            pool_connections=DEFAULT_POOL_CONNECTIONS, pool_maxsize=DEFAULT_MAX_POOL_SIZE
        ):
        if credential:
            self._session = transport.requests.AuthorizedSession(credential)
        elif session:
            self._session = session
        else:
            self._session = requests.Session() # pylint: disable=redefined-variable-type

        if headers:
            self._session.headers.update(headers)
        if retries:
            self._session.mount('http://', requests.adapters.HTTPAdapter(max_retries=retries))
            self._session.mount('https://', requests.adapters.HTTPAdapter(max_retries=retries))
        
        self._session.mount('http://', requests.adapters.HTTPAdapter(pool_connections=pool_connections, pool_maxsize=pool_maxsize))
         self._session.mount('https://', requests.adapters.HTTPAdapter(pool_connections=pool_connections, pool_maxsize=pool_maxsize))
        
        self._base_url = base_url
        self._timeout = timeout

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions