Constant timeouts after multiple calls with async

[Inkorak]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

Constant timeouts after multiple asynchronous calls. It was discovered when using the Llama_Index framework that when calls are made to this library through the openai-python client wrapped with async, constant timeouts begin. If you do this without async or with asynchrony, but on the old version like 0.28, then there are no problems.

To Reproduce

Several calls in a row, for example, to embeddings that are wrapped with asynс.

Code snippets

No response

OS

ubuntu

Python version

Python 3.11.4

Library version

v1.2.0 and newer

[RobertCraigie]

Hi @Inkorak, I can't reproduce the issue you're seeing. Can you share a code snippet?

This snippet passes for me:

import anyio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def main() -> None:
    for _ in range(10):
        await client.embeddings.create(input="Hello world!", model="text-embedding-ada-002")

anyio.run(main)

[ashwinsr]

I can confirm this issue is affecting us as well. We recently upgrade from 0.28 to 1.2.3 and 12 hours later the timeouts began.

[RobertCraigie]

@ashwinsr can you share any more details?

Is this only happening when the client has been in use for a prolonged period of time?

[ashwinsr]

I'm trying really hard to build a minimal failing example, but I haven't gotten one yet. Basically we have a FastAPI server that uses the Async OpenAI client with streaming responses. After a while of running, the vast majority of calls to await client.chat.completions.create will give us timeouts.

We are currently on 1.2.3.

Any suggestions on what we can do to troubleshoot this / help you fix? This is a P0 for us right now.

[RobertCraigie]

Are you seeing connection pool timeouts or is it a request timeout?

[ashwinsr]

We are seeing pool timeouts and some request timeouts. Give me a second and I'll pull some more specific logs for you.

[GCODIN]

It’s just hard to adapt to the correct diagram that is built on each software and since there is so much technology out that is the only way it can be guarded, which is by having barriers in this case cookies, soon it will certificates and badges earned individually so we need to start earning them I believe.

…

On Sat, Nov 11, 2023 at 13:13 Robert Craigie ***@***.***> wrote: @ashwinsr <https://github.com/ashwinsr> can you share any more details? Is this only happening when the client has been in use for a prolonged period of time? — Reply to this email directly, view it on GitHub <#769 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BD4QLGRVSUWVZJVTWJE2Y23YD7TBHAVCNFSM6AAAAAA7F3ZRAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWHEYTQNRTGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[RobertCraigie]

Okay, there was a bug reported recently with streaming responses not being closed correctly. But I did manage to reproduce that and push a fix so I'm surprised you're still seeing connection pool timeouts: #763

Do you have a lot of concurrent requests happening at once?

[ashwinsr]

@RobertCraigie We are seeing

1. Some httpcore.PoolTimeout's
2. Some regular timeouts (although this is possibly FastAPI timing out on waiting on OpenAI, but apologies something just logged the timeout error, we will improve our logging herre)

Thoughts?

[ashwinsr]

Not that many concurrent requests (think <20 at a time)

[ashwinsr]

Here's one traceback:

File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1299, in _request
response = await self._client.send(request, auth=self.custom_auth, stream=stream)
File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/httpx.py", line 137, in send
rv = await real_send(self, request, **kwargs)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1620, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1722, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc

[RobertCraigie]

Okay thanks, do you have debug logging enabled?

If you could share debug logs for openai, httpx & httpcore it would be incredibly helpful.

[ashwinsr]

Also some regular timeout errors:

File "open_ai.py", line 111, in get_function_chat_completion
response = await client.chat.completions.create(
File "/usr/local/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1191, in create
return await self._post(
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1480, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1275, in request
return await self._request(
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1331, in _request
return await self._retry_request(options, cast_to, retries, stream=stream, stream_cls=stream_cls)
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1362, in _retry_request
return await self._request(
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1331, in _request
return await self._retry_request(options, cast_to, retries, stream=stream, stream_cls=stream_cls)
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1362, in _retry_request
return await self._request(
File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1332, in _request
raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.