OSError: [Errno 24] Too many open files during retry

[elisa-abridge]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

Our retry logic was successful before migration to openai v1.0.0. However, now the retry fails as described below.

After a call to openai's API chat.completions.create fails with a Connection Error, our code will catch the exception and retry. However, the retry fails with the following error:

  File "<our code>.py", line xxx, in <our_method1>
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/nest_asyncio.py", line 35, in run
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/nest_asyncio.py", line 90, in run_until_complete
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/asyncio/futures.py", line 201, in result
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/asyncio/tasks.py", line 232, in __step
  File "<our code>.py", line xxx, in <our_method2>
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/openai/_client.py", line 317, in __init__
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/openai/_base_client.py", line 1190, in __init__
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_client.py", line 1397, in __init__
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_client.py", line 1445, in _init_transport
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_transports/default.py", line 272, in __init__
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_config.py", line 51, in create_ssl_context
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_config.py", line 75, in __init__
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_config.py", line 87, in load_ssl_context
  File "/opt/conda/envs/pbap_prompt/lib/python3.10/site-packages/httpx/_config.py", line 145, in load_ssl_context_verify
OSError: [Errno 24] Too many open files

Here is a snippet of our methods:

method1:

        try:
            outputs = asyncio.run(
                                method2
             )

method2:

          AsyncOpenAI(
            api_key=xxx,
            organization=xxx,
            max_retries=0,
        ).chat.completions.create(
                    model=model,
                    messages=input_data,
                    max_tokens=max_new_tokens,
                    temperature=temperature,
                    timeout=timeout,
                )
            )

To Reproduce

1. create openai async client and call chat.completions.create
2. openai returns error such as Connection Error
3. catch error and create new openai async client and call chat.completions.create again
4. receive error OSError: [Errno 24] Too many open files

Code snippets

See above

OS

Unix Ubuntu

Python version

Python v3.10.0

Library version

openai 1.3.5

[RobertCraigie]

You shouldn't be instantiating the client in-line like that, especially without closing it afterwards. You can do so automatically by using the client as a context manager (i.e. with OpenAI()) but I would strongly recommend sharing the client instance between requests.

This is still an odd error though, will look into it.

[elisa-abridge]

I was following the openai Migration guide here which doesn't use the with syntax but instead uses the in-line method.

The guide also explicitly says:

We recommend that you always instantiate a client

If I don't always instantiate a new client, then subsequent requests from the same client will sometimes fail with a connection refused error.

Is there a complete working example where the client is used as a context manager?

Thank you for looking into the retry error.

[RobertCraigie]

Sorry I see how those examples can be confusing.

If I don't always instantiate a new client, then subsequent requests from the same client will sometimes fail with a connection refused error.

I'm surprised this happens, have you verified that it definitely doesn't happen when you instantiate a client for each request and it isn't just a flaky connection?

We did fix a case where we connections weren't properly closed in certain cases in v1.3.7 so I would recommend trying the latest version as well.

Is there a complete working example where the client is used as a context manager?

with OpenAI() as client:
  client.chat.completions.create(...)

[elisa-abridge]

I tried on openai v1.3.7 and faced the same errors of Too many open files. The context manager syntax doesn't work for the async client.

Here is a simple working example showing the context manager syntax doesn't work:

with AsyncOpenAI(api_key=openai_credentials["api_key"],
            organization=openai_credentials["organization"]) as client:
    resp=client.chat.completions.create(
                        model='gpt-3.5-turbo',
                        messages=[{"role": "user", "content": "Say this is a test"}],
                        max_tokens=700,
                        temperature=0,
                    )

which fails with:

AttributeError: __enter__

because the async client doesn't have an __enter__ method.

For the original error of Too many open files, this happens only when sending many requests to openai. A single request is successful, but under heavier load it receives the OSError. Note that this same load was successful with openai v0.28.

[RobertCraigie]

@elisa-abridge ah sorry for the async client you need to use async with instead.

[elisa-abridge]

Is there a complete working example of how to use the async client with a context manager? It doesn't make much sense to me since we want to await the results but this would be outside of the with so the connection is already closed.

If I follow the simple example above and add async with, it only works if both the client create and the await are inside the with block, which kind of kills the point of an async client?

works:

async with AsyncOpenAI(api_key=openai_credentials["api_key"],
            organization=openai_credentials["organization"]) as client:
    resp=client.chat.completions.create(
                        model='gpt-3.5-turbo',
                        messages=[{"role": "user", "content": "Say this is a test"}],
                        max_tokens=700,
                        temperature=0,
                    )
    results = await resp

doesn't work (fails with APIConnectionError: Connection error. RuntimeError: Cannot send a request, as the client has been closed.)

async with AsyncOpenAI(api_key=openai_credentials["api_key"],
            organization=openai_credentials["organization"]) as client:
    resp=client.chat.completions.create(
                        model='gpt-3.5-turbo',
                        messages=[{"role": "user", "content": "Say this is a test"}],
                        max_tokens=700,
                        temperature=0,
                    )
results = await resp

In openai v0.28, we did not use the context manager syntax and didn't face any of the above issues.

[schnerd]

@elisa-abridge Apologies for the issue you're experiencing.

it only works if both the client create and the await are inside the with block, which kind of kills the point of an async client?

I don't totally follow this – when using AsyncOpenAI the client.chat.completions.create call is non-blocking. I'm pretty sure python asyncio won't actually start executing the coroutine until you await it, and it doesn't really make sense to execute API operations when the client library has been closed already (when the context manager exits). Can you share more about why using await resp within the contextmanager block doesn't work for your use case?

In general it seems like creating a new client for each request on a high-throughput system is not desirable because it likely prevents connection pooling in the underlying http library, so each new client creates a new tcp connection, each with an associated file, thus leading to the "too many open files" error.

If I don't always instantiate a new client, then subsequent requests from the same client will sometimes fail with a connection refused error.

Can you share more details about the connection refused error? Specific error message / stack trace would be helpful. Also more information about the environment you're running in (e.g. cloud provider, infrastructure, any proxies, etc.). If you're able to create a script that reliably replicates that issue, that would be extremely helpful in helping us debug!

[elisa-abridge]

Can you share more about why using await resp within the contextmanager block doesn't work for your use case?

I want to await the results in a separate method that is outside of the with block, something like this:

def method1():
  async with AsyncOpenAI(api_key=openai_credentials["api_key"],
              organization=openai_credentials["organization"]) as client:
      resp=client.chat.completions.create(
                          model='gpt-3.5-turbo',
                          messages=[{"role": "user", "content": "Say this is a test"}],
                          max_tokens=700,
                          temperature=0,
                      )
      return resp
def method2():
      results = await method1()

but this fails with APIConnectionError: Connection error. RuntimeError: Cannot send a request, as the client has been closed.

to reproduce the error above, you can run this code:

async with AsyncOpenAI(api_key=openai_credentials["api_key"],
            organization=openai_credentials["organization"]) as client:
    resp=client.chat.completions.create(
                        model='gpt-3.5-turbo',
                        messages=[{"role": "user", "content": "Say this is a test"}],
                        max_tokens=700,
                        temperature=0,
                    )
results = await resp

Can you share more details about the connection refused error?

When using the same client for multiple calls to create() method, the first call succeeds but a subsequent call will sometimes fail with: openai.APIConnectionError and the following stack trace:

"/home/elisa/.cache/pypoetry/virtualenvs/completion-service-B5gMtnZ2-py3.10/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1199, in create
return await self._post(
"/home/elisa/.cache/pypoetry/virtualenvs/completion-service-B5gMtnZ2-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1542, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
File "/home/elisa/.cache/pypoetry/virtualenvs/completion-service-B5gMtnZ2-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1316, in request
return await self._request(
File "/home/elisa/.cache/pypoetry/virtualenvs/completion-service-B5gMtnZ2-py3.10/lib/python3.10/site-packages/openai/_base_client.py", line 1398, in _request
raise APIConnectionError(request=request) from err

[antont]

I want to await the results in a separate method that is outside of the with block, something like this:

There's a couple of bugs in the code there:

def method1():

Misses async, Python would not run this code.

  async with AsyncOpenAI(api_key=openai_credentials["api_key"],
              organization=openai_credentials["organization"]) as client:
      resp=client.chat.completions.create(

There you miss await which is necessary for it to actually do the create call.

      return resp

Therefore this returns the task to do the create, not the result.

async with AsyncOpenAI(api_key=openai_credentials["api_key"],
            organization=openai_credentials["organization"]) as client:
    resp=client.chat.completions.create(

Same problem here, you are missing await.

Corrected code is:

async def method1():
  async with AsyncOpenAI(api_key=openai_credentials["api_key"],
              organization=openai_credentials["organization"]) as client:
      resp = await client.chat.completions.create(
                          model='gpt-3.5-turbo',
                          messages=[{"role": "user", "content": "Say this is a test"}],
                          max_tokens=700,
                          temperature=0,
                      )
      return resp

async def method2():
      results = await method1()

[elisa-abridge]

Thanks for the suggestions. However, the async with syntax isn't working in the following example:

import asyncio
from openai import AsyncOpenAI


async def method1():
    async with AsyncOpenAI(api_key=openai_credentials["api_key"],
                organization=openai_credentials["organization"]) as client:
        tasks = [   
                    client.chat.completions.create(
                            model='gpt-3.5-turbo',
                            messages=input_data,
                            max_tokens=700,
                            temperature=0,
                        )
                    
                    for input_data in [[{"role": "user", "content": "Say this is a test1"}], [{"role": "user", "content": "Say this is a test2"}]]
                    ]
    return await asyncio.gather(*tasks, return_exceptions=True)

async def method2():
      results = await method1()
      return results
results = asyncio.run(method2())
print(results)

The output is as follows:

[APIConnectionError('Connection error.'), APIConnectionError('Connection error.')]

However, I think using the with syntax is a bit of a tangent. I would like to instead focus on the original error reported at the beginning of this ticket (OSError) and the alternative error I get if I don't create new client for every call (APIConnectionError). The original issue occurs under a heavy load so I'm not able to reproduce with a simple example. I've provided stack traces for both errors. Please let me know if there is tracing I can enable in the openai code to further debug the error under heavy load.

[antont]

    async with AsyncOpenAI(api_key=openai_credentials["api_key"],
 (...)
    return await asyncio.gather(*tasks, return_exceptions=True)
(...)
[APIConnectionError('Connection error.'), APIConnectionError('Connection error.')]

Just for the record: that is expected, as your gather is outside the with, so the connection has been explicitly closed before you do the requests. If you put it inside the with block, the requests are done first and the connection is closed afterwards.

However, I think using the with syntax is a bit of a tangent. I would like to instead focus on the original error reported at the beginning of this ticket (OSError) and the alternative error I get if I don't create new client for every call (APIConnectionError). The

Fair enough. There were many reports about that earlier and some fixes etc. but I'm not sure about the current status.

[rattrayalex]

Thank you @antont !

@elisa-abridge , the APIConnectionErrors you are seeing should be resolved by following @antont 's instructions, or by instantiating the client less frequently (eg at the top of the file), which I might also recommend if you can.

The OSError you saw should have been resolved with some recent bugfixes; please update and try again. If you are operating under heavy load, I might recommend increasing your connection pool size from the default (100), as described here.

If you see a recurrence of the OSError issue on the latest library version, please share more details here. To print comprehensive logs, you can set the environment variable OPENAI_LOG=debug